A Curriculum Perspective of Robust Loss Functions
Authors: Zebin Ou
Yue Zhang
Venue: NeurIPS 2022 Submitted (2022)

Abstract: Learning with noisy labels is a fundamental problem in machine learning. A large body of work aims to design loss functions robust against label noise. However, it remain open questions why robust loss functions can underfit and why loss functions deviating from theoretical robustness conditions can appear robust. To tackle these questions, we show that a broad array of loss functions differs only in the implicit sample-weighting curriculums they induce. We then adopt the resulting curriculum perspective to analyze how robust losses interact with various training dynamics, which helps elucidate the above questions. Based on our findings, we propose simple fixes to make robust losses that severely underfit competitive to state-of-the-art losses. Notably, our novel curriculum perspective complements the common theoretical approaches focusing on bounding the risk minimizers.

A Curriculum Perspective of Robust Loss Functions
Anonymous Author(s) Affiliation Address
email
Abstract
Learning with noisy labels is fundamental problem in machine learning. A large body of work aims to design loss functions robust against label noise. However, it remain open questions why robust loss functions underfit and why loss functions deviating from theoretical robustness conditions can appear robust. To tackle these questions, show that broad array of loss functions differs only in the implicit sample-weighting curriculums they induce. We then adopt the resulting curriculum perspective analyze how robust losses interact with various training dynamics, which helps elucidate the above questions. Based on our findings, we propose simple fixes make robust losses that severely underfit competitive state-of-the-ar losses. Notably, our novel curriculum perspective complements the common theoretical approaches focusing on bounding the risk minimizers
2
4
10
11
Introduction
13
Labeling eтrоrs are non-negligible [ in datasets from expert annotation 2 3. crowd-sourcing 14 4 and automatic annotation []. The resulting noisy labels can hamper generalization, as over-
parameterized neural networks can memorize all training samples (7]. To combat the impact of noisy labels, a large body of research aims to design loss functions robust against label noise [8
16
Theoretical results show that loss functions satisfying certain robustness conditions 99 to the same optimum with clean or noisy labels.
will lead
18
with the 20 presence of label noise, which are agnostic to the training dynamics. Though theoretically appealing, they may fail fully characterize the performance of robust losses with noisy labels. particular, it has been shown that (1) robust losses can underfit difficult tasks [ 12 13,, while (2) losses
19
Existing approaches focus on bounding the risk minimizer loss function 1
21
22
23 failing to satisfy theoretical robustness conditions [3 [6] can exhibit robustness. The reasons
24 behind these observations remain open questions. For (1), existing explanations [10 17 can be
25
limited as discussed in 22.3 For (2), to our knowledge, there no work directly addressing
26 To tackle the above questions, we consider training dynamics our analysis, which complements existing theoretical approaches [I] By rewriting loss functions into standard form, we show that many loss function differs the implicitly which connects robust losses to the seemingly distinct [I] curriculum learning approaches [18 22
27
28
curriculums they induce ($3)
29
30 for noise-robust training. The original definition 233 of curriculum learning aims to present training
31 samples with gradually increasing difficulty and diversity to ease learning. We adopt a generalized definition curriculum [24). i.c., a curriculum specifies a sequence of re-weighting of training sample distributions, which can manifest sample weighting (1) sample selection 21 22 25
32
34
The curriculum perspective helps elucidate underfitting and noise robustness from the interaction between the sample-weds weighting curriculums and various training dynamics. We first attribute un- derfitting to the marginal average sample weights with the implicit curriculums ($4.1). We then
35
36
37 show that an increased number of classes can lead marginal initial sample weights with some loss
Our code will be available at gighhb
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

38
underfit perform competitively state-of-ther of-the-art loss functions ($4.2). Finally, we attribute the noise robustness of loss functions to higher average sample weights for clean samples compared to noisy ones ($4.3). We hypothesize that clean samples can receive higher weights with sample-weighting curriculums imm that magnify the learning speed differences and neglect unlearnt samples, which explains functions ($4.2) By adapting their curriculums accordingly, we make robust losses that severely
39
40
41
42
43
our empirical observations ($4.3). Inspired by this hypothesis, we find two unexpected results when viewed from existing theoretical robustness guarantees: by simply changing the learning rate schedule, robust losses can vulnerable to label noise and cross entropy can appear robust ($4.3).
44
45
48
2
Background
47
After formulating classification with label noise, we briefly review typical sufficient conditions and loss functions for noise robustness to ser the context for our novel curriculum perspective. We then summarizing open questions to be addressed in this work.
48
49
50
2.1
Classification with Label Noise and Noise Robustness
The k-ary classification problem with input x E R2 can be solved with classifier arg max; 8, where 8, is the score of the i-th class from the class scoring function $p R2d R..t The class scores (x) can be turned into class probabilities with the softmax function Pt = e / (-j=1 e*),
51
53
54 where pr the probability for class i. Given a loss function [==)(=),)) y) and data (z,y) with ground truth label y E {L....,k) k.. the parameter & of sæ can be estimated with risk minimization 56 arg ming Ex,, sy (s0 (z) y). whose solution are called risk minimizers. For notation simplicity, we omit the dependence
0 and æ if possible.
58 The annotation process may introduce errors, resulting in potentially corrupted label j following
V i
with probability P(ù = y 2/ y) with probability P i y)
59 Label noise is symmetric (or uniform) if P(ū = i a. v) - 7 1), Vi + y, with rj = Pū + y) the noise rate constant. Label noise is asymmetric (or class-conditional if P i|,, y) P = ily) Given data (xx 9) with noisy label J. a loss function L is robust against label noise if
61
arg min R.,,, [ (1)) 5) arg min E, (s00
(1)
0
0
62 Most existing work (9
15 aim to derive bounds for the difference between risk minimizers
63
obtained using noisy and clean data, i.c., ensuring Eq. (I) holds with some conditions. As typical
64 examples, loss functions satisfying the symmetric (9] or asymmetric [IT] conditions are theoretically
65
guaranteed to be robust, loss function I, is called symmetric if
Σ [((so(z).)) i) C, Vx. $0
(2)
88 where C' is a constant. When noise rate 7, < (k-1)/k symmetric loss is robust against symmetric
67 label noise ( Such stringent condition is later relaxed by Zhou et (.]) Suppose loss function
68 can be written as function of softmax probability ps. i.e., L(so(x), i) = Up.)). As an equivalent
69 rephrase of the sufficient condition, L is called asymmetric if
P(ū i|. v) bF P( y/
L(p)) - U(: + Ap) 1(0) ((Ap)
max
-rr= inf OSP; Ap+
(3)
70 where Ap is valid increment of p:. When clean labels dominate the data, i.c., F < 1, an asymmetric
71 loss robust against generic label noise. Notably, both symmetric and asymmetric conditions for
72 noise robustness are agnostic to training dynamics to reach the risk minimizers.
73
2.2 Review of Selected Loss Functions
74 In addition to cross entropy (CE) that is vulnerable to label noise D. we review typical loss functions for later analysis. We ignore differences in constant scaling factors and constant additive bias in the
76 loss functions. They are either equivalent to learning rate scaling SGD or irrelevant in the gradient computation. See Table[[fo for the exact formulas and Appendix A for extended review.

---

Type
Name
Function
Sample Weight w
Constraints
CE
log Py
Py
MAE/RCE
-
Pu(1 Py) YNCB (WCE L =NCEN))
Sym.
NCE
E log
AUL
0--P)) (a-1))
Po(1 Pv)(a pb)
a> 1>1,9>
Asym.
9
AGCE
(a+ (0-1)-(mee (a+py]"
Pu(a + 11,00-(1 PPy)
a> 0,000
9
GCE
P2
p8[1 pp)
0 0<951
Comb.
SCE
q) LCE Q LMAE
- Py) Pa)
0 0<g<1
NCE+MAE
(1 q) LNCE + LMAR
g) WXCE WMAR
0<9<l
Table I: Expressions, constraints of hyperparameters and sample weights of the implicit curriculums 3.1 for loss functions reviewed in $2.2 Note that WNCE. is an approximation discussed in $3.
Symmetric Loss The mean absolute error (MAE) 9 and the subsequent reverse entropy (RCE) [3] are essentially equivalent, both satisfying Eq. (2) Ma et al. [10] normalize generic loss functions satisfying i(s,i) 0, Vi E fl,..., K} into symmetric losses with Lx(s.
79
L(s, v)/ v) T(ss,)) We include normalized cross entropy (NCE) as an example.
81
82
Asymmetric Loss We include two asymmetric losses [T] for our analysis: asymmetric generalized
83
cross entropy (AGCE) and asymmetric unhinged loss (AUL). Notably, AGCE with g 2 1 and AUL with g are both completely asymmetric [1I), i.e., Eq. 3 always holds when
84
Combined Loss Loss functions be combined for both robust and sufficient learning. For example, generalized cross entropy (GCE) [12 can be viewed as a smooth interpolation between
86
87
CE and MAE. Alternatively, symmetric cross entropy (SCE) 13 uses a weighted average of CE and RCE (MAE). Finally, Ma et al. [10] argue that robust and sufficient training requires balanced be rewritten into
88
89
combination of active and passive losses. Suppose loss function
L(8.)) Σ (s,,
(4)
90
where is function of scores and any possible label i. An active loss requires Vi 7 ((8, = 0,
91
which focuses learning the target label. In contrast, a passive one satisfies 3j / y,, ((, + which can improve by unlearning non-target labels. Accordingly. CE and NCE are active while MAE (RCE) passive. We use NCE+MAE an example.
82
93
94
2.3
Open Questions
95
Why do robust losses underfit? Ma al. (10) attribute underfitting to failure in balancing active- passive components. However, different specifications of Eq. can lead to ambiguities in the active-passive dichotomy. For example, with LMAE(S, y) Σn
96
t= [[ = y) Pyl where I[-) is the indicator function, MAE passive; yet the equivalent LMAE (8, (8. x Σ ll(i y) - P.) makes MAE an active loss. Wang et al. [17] instead view VV, y)ll I as weights for sample gradients and attribute underfitting to their low variance, making clean and noisy samples less distinguishable. However,
96
100
show in I4.1 MAE also underfits clean datasets. Why robust losses underfit thus
102
remains an open question.
105
What affects the robustness a loss function? Although combined losses such as GCE and SCE fail to satisfy existing robustness conditions (Eq. (2) and (3) it is unclear why they exhibit robustness against label noise [12 13]. Furthermore, it unclear how training dynamics, which are irrelevant in
105
many theoretical robustness guarantees 19 15), affect the noise robustness of a loss function.

---

107
3
Implicit Curriculums of Robust Loss Functions
108
We derive the standard form of reviewed loss functions and show that each implicitly induces a sample-weighting curriculum, which helps examine how they interact with various training dynamics.
110
3.1
The Implicit
Curriculums
111
Loss functions in Table are generally functions of the target softmax probability Py, i.e. L(s, =
((2y). Note that Py can be rewritten
1
Py Ei=1 e
log Eitt ct c
+1
E - +1
(5)
113 where
Dg = Sy - log Σ e S Sy
max 91 A
(6)
v
114 is the soft margin between sy and the maximum of other scores, smooth approximation of the hard
115 margin Ay. By indicates how well a sample is learnt given classifier arg max, 8t, as Ay 2 0 leads to
116 A 2 O, ensuring successful classification with scores 8. Since V vo((py) = =1' (p(p) ) ply (Aa) VaAy,
117
these loss functions can be rewritten into standard form with equivalent gradients:
L( y) = stop_grad [ur(A [ba)))] Ay
(7)
where stop_ stop_grad(-) avoids backpropagating through wi (A,,,) = (p,) P' (4n)) The equivalence
118
valid only with first-order derivatives. Each loss function in the form of Eq. ( thus implicitly induces a sample-weighting curriculum, where wo (((A) is the sample weight and Ay the implicit loss. By examining how w(( Ag) interacts with different training dynamics, we can elucidate the reasons behind underfitting and noise robustness. Table !
119
120
121
summmaizess w/ (A,) for the reviewed loss functions.
122
123
Wang et al. [7 treat [VaL(s y)llı as weights for sample gradients, which share similar formulas w A in Table Instead of directly weighting sample gradients, derivation identifies the implicit loss Ay. making our sample- weiphe-weighting scheme compatible with the definition of curriculum learning (24). In addition, the extracted Ay and A can serve direct metrics for sample performance
125
127
in curriculums, compared to loss [26] [27] and gradient magnitude (28) that are affected by preference
128
from w( (A,) of loss function. Finally, the A, distribution is essential in analyzing the interaction
129
between loss functions and training dynamics in $4
130
3.2
The Additional Entropy-Reducing Curriculum of NCE
Due normalization, LNCE (s, y) in Table 1 additionally depends on AS where A y. which cannot be be trivially rewritten into Eq. (7). A derivation of the gradient gives
132
1
kLCE(s, V
h
V VgINCE(S.N)
VoLCE(S.y) Σ (CE(s,
LCE(s,))
* Ei=1 i)
INCE |V (8, + ENCE - VaRNCE(s))
133
Thus NCE can be rewritten as
LNCE(S, y) = "NCE LCE(s, y) + NNCE CNCE RNCE($)
(8)
134 In this equivalent form, there is no backpropagation through the computation of INCE and ENCE-
135 The first term results in a similar sample- weightini curriculum as CE, with an additional factor
INCE = 1/ llog p.) ) $ 1/(k log k).. The second term is a regularizer
136
&
RNCE
(cE Σ
137
which reduces the entropy of the softmax output. The regularizer has per-sample weights ENCE
k( - log p,y) (/ log,,). It can thus be interpreted as a regularization curriculum. Notably, the two curriculums work synergically in reducing the entropy of the softmax output.
138
139
The extra regularizer makes NCE incompatible to Eq. (7). However, as shown in Appendix since induces gradients with constant L1 norm, we can approximate the upperbound of WNCE with
140
V LNCE(E v) < ANCE WCCE
k
WNCE
V V.Ayll
ENCE
(10)
See Appendix C for derivations. Note that directions of V VoLNCE(s,, v) and V.A, may be different.

---

CIFAR100
CIFARIO
Underfitting
Loss
Acc.
Acc.
CE
71.33 + 0.23
8.183
92.76
0.30
5.541
GCE
69,95 + 0.40
8.861
92.96
0.13
6.151
No
SCE
71.36 + 0.39
9.541
93.17 0.06
7.018
NCE+MAE
68.89 + 0.23
2.971
92.37 + 0.33
2.414
NCE
43.18 1 1.55
1.769
91.28 1 0.22
1.072
Moderate
AUL
58.75 1 1.07
5.278
92.43 1 0.19
5.171
AGCE
49.27 = 1.03
4.537
92.61
+ 0.18
5.225
MAE
3.69 1 0.59
0,035
91.56 1 0.11
2.492
Severe
AUL
3.13 1 0.43
0.033
91.13 0.06
2.308
AGCE
1.62 1 0.69
0.009
87.14 1 4.96
1.701
Table 2: With clean labels, robust losses can underfit CIFAR 100 but CIFARIO. Hyperparameters of loss functions are tuned on CIFAR 100 and listed in Table We report test accuracy and average effective learning rate oi (scaled by 109) at the final training step with 3 different runs, using learning
rate a 0.1. AUL' and AGCE" with inferior hyperparamters are included as reference. See Appendix D for results with 0.01.
1.0
1.0
1.0
0.8
AUL
0.8
ACCE
0.3
SCE
AGCE
OCE
4
CHARR100
0.6
7 0.6
9
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
-10.0
-10.0
-5.0
0.0
5.0
(a) Severe underfitting
(b) Moderate underfitting
(c) No underfitting
Figure 1: Hample-weighting functions w Ay) for selected loss functions and hyperparameters in
Table 2
We include the initial distributions of Ay on CIFARIO and CIFARI00 for reference.
Understanding Robust Losses with Their Implicit Curriculums
143
144
We empirically investigate the interaction between sample-weighting curriculums and various training dynamics for questions in 22.3 Experiments are conducted on MNIST (29] and CIFAR10/100 [30] with synthetic symmetric and asymmetric label noises following standard settings IT) We also include real human noisy labels provided by Wei et al. [11 CIFAR10/100. We 4-layer CNN for MNIST, an 8-layer CNN for CIFAR 10 and ResNet-34 32 for CIFAR100 By default, models trained with fixed number of epochs using SGD with momentum, weight decay and cosine learning rate annealing. See Appendix for more experimental details. Different from standard settings, we rescale w( (Ay)) to have unit maximum to avoid complications, since hyperparameters of loss functions can change the maximum of w( (Ay), essentially adjusting the learning rate of SGD,
146
148
150
151
152
153
4.1
Underfitting of Robust Losses from WeighttWe Curriculum Perspective
154 Robust losses can underfit. We confirm that on difficult tasks like CIFAR100 [10 12 13), underfiting results from robust losses themselves rather than inferior hyperparameters. We tune hyperparameters of loss functions on CIFAR 100 and report results on CIFAR 100 and CIFARIO without label noise. As shown in Table 2 the performance of NCF, AGCE and AUL lag behind CE by nontrivial margin on CIFAR100. Notably, MAE performs much worse compared to CE, similar AGCE" and AUL. with inferior hyperparameters. In contrast, all loss functions fit CIFAR10 well. See Table 8 Appendix for similar results with a smaller learning rate.
155
156
157
158
159
160
Marginal effective learning rate explains underfitting. We attribute underfitting the diminishing effective learning rate of = aa Wr where w; is the average sample weight of the batch and a, the learning rate at step 1. We use the average effective learning rate up to step I,, az = E of/L, /t, to 184
182
characterize the overall oz- In Table for loss functions that heavily underfit on CIFAR 100, their oj
at the final step is marginal compare to CE.
5

---

AUL-superior
NCB
0.08
AUL-inferian
0.08
0.06
0.06
13
0.04
0.04
0.02
0.02
20k
10k
60k
80k
20k
40k
60%
80k
(a) AUL with inferior/superiio hyperparameters.
(b) NCE with estimated weight upperbound.
Different causes of underfitting: (a) marginal initial sample weights; (b) fast diminishing sample weights. We plot the average effective learning rate az at different training steps & with selected loss functions on CIFAR100.
Figure 2:
186
Marginal effective learning rate due to marginal initial sample weights. In Fig. ! we compare sample-weit weightin functions w( (y of robust losses to the A, distribution of CIFARIO 10 and CIFARI00 initialization. For robust losses that severely underfit (Fig. Tap, the Ay distribution of CIFAR 100 concentrates at regions with marginal sample weights, resulting in small effective learning rate oz It can hard for these samples escape the region with marginal weights before the learning rate attenuates. In contrast, loss functions with non-trivial initial sample weights (Fig. [b and Ic] result in moderate or no underfitting in Table 2 As a corroboration, we plot the average effective learning rate ar of AUL with different hyperparameters in Fig, 2a With superior hyperparameters (AUL in Table D. ag quickly increase to a non-negligible value before annealing. In contrast, aj stays marginal with inferior hyperparameters (AUL" in Table 2).
187
168
170
172
174
Marginal effective learning rate due to fast diminishing sample weights. In Fig. 2b different from other robust losses but similar to CE, the effective learning rate of NCE peaks at initialization. However, it decreases much faster compared to CE, which can be attributed to the synergy between the two implicit curriculums of NCE reducing WXCE- As Ay improves, "NCE, ENCE and wCE all decreases. In addition, the regularizer RNCE (s) further decreases the entropy of softmax output and thus "NCE. Thus WNCE decreases much faster compared to WCE. leading to faster attenuating az.
177
179
181
182
Loss combination mitigates marginal initial sample weights. As WCE and WNCE peak at ini- tialization, they compensate the marginal initial sample weights when combined with other robust losses, helping initial learning and thus avoiding underfitting. In Table the effective learning rate on CIFAR 100 is substantially increased when combining MAE with CE and NCE. Interestingly, CE and NCE are both "active" as their sample weights peak at initialization, while other robust losses are "passive" due to their marginal initial sample weights. Such dichotomy based on sample- weinhiig curriculums complements the active-passive dichotomy [TO] from a distinct perspective.
184
186
186
189
4.2
Addressing Underfitting by Adapting the Sample WeightiWegghting Curriculums
As shown in Table 2 robust losses can underfit on CIFAR 100 but CIFAR1O. Such difference has been vaguely attributed to the increased task difficulty I 12 We further show that with static sample-wedt weighting curriculums, loss functions suffer from marginal initial sample weights due the increased number classes k. By adapting the curriculums accordingly, robust losses that severely underfit become competitive with the ttate-of-the-art, We leave the fix for NCE to future work, and use MAE as a typical example for illustration.
191
193
194
195
196
Intuitively, the larger number of classes, the more subtile differences to distinguished, thus the harder the task is. In addition, the number of classes & determines the A, distribution initialization. Assuming that class scores s; at initialization are i.i.d. variables following the normal distribution, i.c., for most neural networks with standard initializations and normalization layers 33 355 See Appendix E for comparisons between simulations and real settings. The expected Ay can be approximated with
198
N (H, r). In particular, μ 0 and
200
33
E[Ay] ~ - log(k - 1) - 02/2 2(k
(11)

---

0.3-
100
08
-10
-10
(a) Simulated initial distributions
(b) Add [[ A
(c) Shifted/rescaled WMAR (Ay)
Figure 3: (a). Simulated initial Ay distributions with different assuming 8; (N,,, σ). We include the plot of (MAE(Ay) for reference. (b). Adding E[ to Ay 's centers simulated distributions in (a) to the origin. (c). The shifted and rescaled WMAR( (A,) with a = 2.6 and 100. We include
the initial Ay distribution of CIFARIO0 for reference.
Clean
Symmetric
Asymmetric
Human
Loss
1 0
y= 0.4
11 0.8
1 0.4
7= 0.4
CE
71.33 + 0.43
39,92 = 0.10
7.59 0.20
40.17 + 1.31
GCE
63.09 + 1.39
56.11 + 1.35
17.42 +: 0.06
40.91 + 0.57
NCE
29.96 + 0.73
19.54 + 0.52
8.55 + 0.37
20.64 + 0.40
NCE+AUL
68.96 + 0.16
59.25 0.23
23.03 + 0.64
38.59 + 0.48
AGCE
49.27 = 1.03
47.76 = L.75
16.03 0.59
33.40 + 1.57
30.45 + 1.50
AGCE shift
67.50 = 1.48
53.33 1.08
10.47 0.57
38.37 1.55
44.44 : 1.39
AGCE rescale
67.20 + 0.79
56.32 + 0.59
12.75 1.10
40.00 + 0.27
49.08 + 0.74
MAE
3.69 + 0.59
1.29 0.50
1.00 0.00
2.53 1.34
2.09 1 0.55
MAE shift
69.02 + 0.78
44.60 + 0.24
8.08 0.26
40.57 1 0.47
48.31 1 0.31
MAE rescale
69.95 + 1.21
60.70 + 0.30
10.79 1 0.97
39.22 + 1.54
54.65 1 0.73
Table 3: Shifting or rescaling Ay mitigates underfitting on CIFAR 100 with different noise types and
noise rate 7). Human noisy labels are from CIFAR100-N [3I] Test accuracies are reported with different runs. We use a = 4.5 for AGCE and 2.6 for MAF. Results from [1]] are included as context. See Appendix for results on Visvin and CIFAR100 with additional noise rates.
We leave detailed derivations Appendix E With more output classes, the A, distribution will have smaller expectation, corresponding to diminishing initial sample weights with the fixed MAE curriculum, shown in Fig. 3a In Fig. 3b subtracting B(A,) from Ay centers distributions to 0.
203
205
Shifting or rescaling w: Agy) mitigates underfitting from increased number of classes. To assign nontrivial sample weights initialization, the sample-weighting curriculum of robust losses should
206
be adapted according the number of classes kt A simple strategy is make the expected initial sample weights agnostic to Given a sample-weighting function w (Ayy), we can either shift
208
shift (Ag) = w(A, + E[A,] a)
(12)
209
or rescale
w: rescale (a,,cale(A,)_0000)))))) = w'),
(13)
210
it, where is a hyperparameter. The shifted and scaled "MAE(A,) are shown in Fig. 3clas as an illustration. Intuitively, shifting or scaling with B[A,] can cancel the effect of increased i on the expected initial sample weights. With smaller a, samples will get higher weights at initialization.
212
213
In Table we our fixes with different noise types and noise rates on CIFARIO0. See Appendix for more results on the large scale noisy dataset Vision (6 and CIFAR 100 with different synthetic noise rates. Rescaling and shifting alleviate the underfitting issues, making MAE and AGCE
214
215
216
perform comparable to the previous best (NCE+AUL) []I. Notably, the performance of MAE is
217
substantially improved. Interestingly, despite being effective fixes for underfitting, simply scaling or shifting w( A can risk assigning large weights for noisy samples, which have lower A, in general as discuss in $4.3 thus hampering the noise robustness of loss functions. Under symmetric label noise with 0.8, the performance of AGCE decreases after applying the fixes.
219
7

---

Clean
Symmetric
Human
1) = 0.2
=
0.4
0.6
1) = 0.8
= 0.4
Loss
Acc
Aace
Face
Aace
Aace
Aacc
CR
90.49
-15.85
0.39
-32.34
0.58
-51.57
0.77
-71.14
0.95
-28.18
0.53
SCE
91.06
-8.10
0.76
-21.55
1.03
43,86
1.29
-71.10
1.32
-22,96
0.74
GCE
90.85
-2.02
3.25
-5.59
3.16
-14.16
2.95
-50.10
2.29
-12.52
1.14
MAE 90.56
-1.96
3.46
-8.25
3.15
-12.31
2.88
-38.11
2.53
-22.49
1.00
AUL 90.79
-1.90
3.51
-5.06
3,40
-13.43
3.01
-50.99
1.79
-22.36
1.02
AGCE
90.56
-4.28
3.11
-4.47
3.29
-17.76
2.69
-44.87
2.04
-21.62
1.02
Table 4: Robust losses assign larger weights to clean samples. We report and drop in test accuracy with symmetric and human label noise on CIFAR10 at the final step with 3 different runs. We use the "worst" version of CIFAR 10-N [31] human label noise. Standard deviation are omitted due space limitation. Hyperparameters of loss functions are tuned with noise rate 1 = 0.6. See Appendix for detailed hyperparameters.
(1.0)
D (1.00)
D (1.0)
1 (0.7)
1 12.1)
1 (4.4)
(0.8)
(2.2)
49 (2.1)
499 (9.4)
119 (1.8)
119 (2.3)
119 (7.9)
-40.0
-20.0
0.0
20.0
40.0
-40.0
-20.0
0.0
20.0
40.0
-40.0
-20.0
20.0
40.0
By
Ay
(a) CE: 58.05
(b) SCE: 69.62
(c) MAE: 85.86
Figure 4: How Ay distribution of noisy (green, left) and clean (orange, right) samples evolve during training on CIFARIO with 40% symmetric label noise. We include w((y)) curves for reference, and omit vertical axes denoting probability density for brevity. Vertical axes are scaled to the peak of histograms for better readability, with epoch number (axis scaling factor) denoted on the right of each subplot. We also include the final accuracy of the corresponding run for each loss function as reference. See Appendix F for results of more loss functions with human label noise.
221
4.3
Noise Robustness from a Hample-Weightiin Curriculum Perspective
222
Intuitively, loss functions exhibiting noise robustness should weight clean samples more than noisy ones. We provide an explanation based on how (Aay) interacts with two training dynamics.
224
Robust losses assign larger weights to clean samples. The average weight assigned to noisy samples during training, adjusted by learning rate ag, noise = Σr, I(jr.. y4,: ( / I(hr. *
225
226
Vi.)) )oy,, where Wi,, denotes the weight of i-th sample of the batch step Welean for clean samples can be defined similarly. The ratio snr = Welean/ / characterizes their relative contribution
227
228
during training. We report snr and the drop in test accuracy under different label noise on CIFARIO
in Table Loss functions with less performance drop have higher snr in general.
230
To explain what leads to high snr, we first examine how A, distributions of noisy and clean samples evolve during training on CIFARIO with symmetric label noise in Fig. 4 See Appendix for results of more loss functions with human label noise. When trained using loss functions with increased robustness (Fig. 4b and 4c). the noisy and clean distributions of A, gets better separated and more spread. In addition, A, of some noisy samples gets decreased, suggesting that noisy samples can be unlearnt. In contrast, with CE (Fig. 4ah. the noisy and clean distributions of A, are less separated and more compact.
231
232
233
235
8

---

1.0
0.9
Test Accuracy
Soynnn No
0.0. MAE
Test Accuracy
0.0, 1E-01
0.0, CE
0.0
0.6, 1E-01
0.0,
0.6
D.6, CE
0.7
0.6. 1E.02
0.8. MAE
0.0. 1E-04
0.4-
0.8, CE
0.6, 1E-04
0.5-
Am
0.4
100
Epach
Epoch
(a) a = 0.01 with different (r,, loss).
(b) CE with different (7, a).
Figure 5: Learning curves with fixed learning rate and extended training epochs on MNIST, where a is the learning rate and the symmetric label noise rate. Vertical axes are scaled for readability.
237
We now give possible explanation for Fig. 4 with the following two training dynamics: (D1) clean samples are learnt faster than noisy samples: (D2) noisy samples can be unlearnt when trained
238
239
robust training [3 33]. It can result from the dominance of clean samples (F < 1) in the expected gradient. In addition, gradients of clean samples are more correlated than those of noisy samples [4)] Thus performance on clean samples can be improved when training on one another, leading to D1. D2 only become apparent when examining Fig. 4b and 4c which can result from generalization with clean samples. Suppose in MNIST, sample of 0 is erroneously labeled as 9. Then model well-trainee with clean samples of class 9 and 0 can result in a low By for this noisy sample. DI and D2 can act in synergy to separate the clean and noisy distributions of Ay. as shown in Fig.
on clean samples. D1 is identified in 7 37, which later manifests itself in curriculum-basse
240
241
242
243
245
247
We hypothesis that robust losses enhance the synergy of D1 and D2. In Table I w (4y) of loss functions can be decomposed into f(م ((dy).g(A) (A,)) where f(Ay) a monotonically increasing function and g( A,) decreasing one. For example, fCr. A,)) degenerates to constant and (CE(A,) = - 1-Py- while SMAE (A,) = Py and (MA)(4,))
248
249
1 Py. Notably, g(A,) shared by all loss functions converges to 0 as Ay increases, preventing 4, from growing infinitely large. In addition, a non- degenerated f( T(A,)) can enhance the synergy between D1 and D2. Since the initial A, distribution generally lies on the monotonically increasing part of w(A,) determined by f f(A,). faster learning of samples results in their larger weights during training. Thus robust losses magnif'y the difference in learning speed between clean and noisy samples, which can also account for the substantially spread A, distributions in Fig. 4b and 4c As w (Ay)) can assign negligible sample weights with low A, due to the monotonically increasing f(Ay). unlearnt noisy samples
250
252
254
256
neglected with diminishing weights, which can account for the decrease of A, 's for noisy samples in Fig, 44 and 44c In contrast, as wCr WCr(A,)) assign high sample weights for small Ay 'ss it compensates the synergy of DI and D2, thus results in compact A, distribution, larger A, 's for noisy samples, and less separated
258
260
A, distributions in Fig. 4al
With sufficient training, clean samples will eventually have high A, 's with diminishing sample weights thanks to g(A,) Noisy samples will then dominate the expected gradient and can lead to overfitting, leading to two unexpected results when viewed from robustness conditions II:
263
265
Robust losses are vulnerable to label noise with extended training. In Fig. 5a we show the learning curve of CE and MAE using constant learning rate under different symmetric noises on MNIST. Although enjoying theoretically guaranteed noise robustness 9 11, similar to CE, MAE eventually
266
267
268
overfits to noisy samples, becoming vulnerable to label noise as weights of clean samples diminish.
289
Loss functions can become robust by adjusting the learning rate schedule. Interestingly, in Fig. 4a despite the compensation of wCg WCE(Ay), the synergy between DI and D2 still results in partially- separated Ag distributions of noisy and clean samples. We can thus improve the noise robustness of
270
271
272
CE by preventing the weights of clean samples from diminishing due to g (Aay). which can be achieve
273
by slowing down the convergence or early stopping [41] In Fig. 5b we show the learning curve of
CE using fixed learning rates under symmetric noise on MNIST. By simply increasing or decreasing
275
the learning rate, which strengthens the implicit regularization of SGD 42 or directly slows down
the convergence, the noise robustness of CE can be substantially improved.

---

5
Related Work
277
Our work is closely related to robust loss functions 8413 for robust training with noisy labels M. Theoretical results [I] derive sufficient conditions for robustness against label noise without con- sidering the training dynamics. We complement these results by considering the interaction between robust losses and various training dynamies. The underfitting of robust losses has been heuristically mitigated with loss combination [2 13]. We further clucidate the cause of underfitting from a curriculum perspective, based on which we provide an effective solution.
278
280
282
283
prriculuu-based approaches combat label noise with either sample selection [21 22 or sample- weighting [8|20] In particular, sample weights are designed [T6][8]] or predicted by model trained on separated dataset 19 20). In contrast, the sample-weighting curriculums considered in this work are implicitly induced by robust loss functions. Most related to our work, Wang et al. [6] identifies gradient norms as weights for sample gradients of each robust loss. In contrast, as discussed in $3.1 we explicitly extract the implicit loss, which helps draw the connection to standard currieulum learning [24] and facilitates analysis of training dynamics.
285
287
289
Our work is also related to the ongoing debate 124
43
strategies for selecting or weighting 26 or harder first 27 is better. The
232
samples in curriculum learning: whether easier first 23
implicit curriculums of robust losses in this work differ in two important ways. First, the implicit
294
loss identified in $3. more directly measures sample difficulty than loss value [26] 27 and gradient magnitude (28). Second, the implicit sample-weighting curriculums can be viewed as a combination
296
of both weighting strategies by emphasizing moderately difficult samples, as discussed in $4.3
Conclusion
297
298
We identify the implicit
curriculums of selected loss functions. By decoupling the implicit loss as direct sample performance metric and sample weights specifying the implicit
299
300
sample preference, we can analyze how robust loss functions and curriculums interact with different training dynamics. Such a perspective complements existing research theoretical bounds for the risk minimizer, and connects robust loss functions to the seemingly distinct approaches based on curriculum learning. Following the curriculum perspective, we elucidate the reasons behind underfitting and robustness against label noise for existing robust loss functions, and design a simple approach to address the underfitting issue.
302
303
304
References
306
307
[1] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey, ArXiv preprint, abs/2000.088199 2020. URL
308
https://arxis: .ttpsiii.rrg//oooooo org/ abs/ .81999
310
[2] Pete Bridge, Andrew Fielding, Pamela Rowntree, and Andrew Pullar. Intraobserver variability: should we worry? Journal of medical imaging and radiation sciences, 47(3):117-200, 2016.
312
[3] Yoshihide Kato and Shigeki Matsubara. Correcting errors in a treebank based on synchronous tree substitution grammar. In Proceedings of the ACL 2010 Conference Short Papers, pages 74-79, Uppsala, Sweden, 2010. Association for Computational Linguistics. URL https
313
314
315
//aclanthology.oog www.:ooyyy...//000011110000 2014
[4] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. urXiv:1409.0573 [cs], 2015. URL arXiv: 1409.0575.
317
319
http://arxiv.orgge www.oogg//ab.... abs 11111111000077
320
[5]
Kun Liu, Yao Fu, Chuanqi Tan, Mosha Chen, Ningyu Zhang, Songfang Huang, and Sheng Gao. Noisy-labeled NER with confidence estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3437-3445, Online, 2021. Association for Computational Linguistics. doi:
321
322
323
324
10. 11883//11222211 www.la main. 269. URL httpps://acllanboooogg..og////// http:///222...aaaaaaaaa naacl-
325
269
326
[6]
Huda Khayrallah and Philipp Koehn. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation,
10

---

328
pages 74-83, Melbourne, Australia, 2018. Association for Computational Linguisties. doi: 10.18653/v1/W18 2709. battps://accanthiie
329
org/ WW8 2709
[7] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representationss ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
331
333
net/Fooruridd-SygddBx:
[8]
N. Manwani and P. S. Sastry. Noise tolerance under risk minimization. IEEE Transactions on Cybernetics, 13(3):1146-1133) 2013. doi: 10.1 1..1109/mmm22212222246666 URL https
335
336
/ (doi..or//il 10t 11w.922tsmcbbbbo.... 2012. 2223460
337
99]
Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Satinder P. Singh and Shaul Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco. California, USA. pages 1919-1925, AAAI Press, 2017. httpttp: / org/ocs//po ocs/indexx index hhtp//AAA//AAA1//pppp/e//r//////
338
339
340
341
342
[10]
Xingjun Ma, Hanxun Huang. Yisen Wang, Simone Romano, Sarah M. Erfani, and James Bailey, Normalized loss functions for deep learning with noisy labels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Muchine Learning Research, pages 6543-6553. PMLR, 2020. URL. http::/proceediigg .mw.e.gg........................ mw...prrss... press/ htt///1119/mmmmmmm html
343
344
346
[11] Xiong Zhou, Xiamming Liu, Junjun Jiang. Xin Gao, and Xiangyang Ji. Asymmetric loss functions for learning with noisy labels. In Marina Meila and Tong Zhang, editors, Proceedings Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12846-12856. PMLR, 2021. BtRLhttp:// http::://proceeding.e mlr www.pprssssvv..e v133ss/o21////2 zhou2/f....ff.mm html
348
of the 38th International Conference
350
352
[12] Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada. pages 8802-8802, 2018. URL https::///// http:///pceeeen..../me www.ddiinnnnneuiiis..e.ee neurips. ct.pps.cc///eeo wtt.cc/cppppc... cc/paper/2eee 2018/hash/ f2925f977b113ad285222 a55 a50i075551122022 B02fffffeeeeooo. Abstract.htm
353
354
355
356
357
358
359
[13]
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 322-330 IEEE, 2019. doi: 10. 109//C/C.2019.0000000 URL. httph::/tp:/ttt org/ 10. 1109/
361
362
363
[14] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. (arXiv: 1910.03311, Aug 2020. doi: 14855500rxXiv.111000333 URL http:
365
//arxiv. org/ abs// 19100 03231 arXiv:1910.03231 [cs, stat].
367
[15]
Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust label noise? In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, page 2206-2212, Yokohama, Japan, Jul 2020. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-0-9992411-6-55 doi: 10.24963/ ijcai.2020/305. URL httpp:///wwttt
368
389
370
371
www. httpaarrr//rroeeee/////0///////
372
[16] Xinshao Wang, Elyor Kodirov, Yang Hua, and Neil M. Robertson. Derivative manipulation for general example weighting. ArXiv preprint, abs/1905. 11233, 2019. URL https::/tpaat //axxiv org/ attpa/////1...11..
373
374
375
[17] Xinshao Wang, Yang Hua, Elyor Kodirov, and Neil M. Robertson. Imae for noise-robust learning: Mean absolute error does not treat examples equally and gradient magnitude's 's variance matters. ArXiv preprint, abs/1903. 12141, 2019. URL httph:tt////rrtt org/ abs /tab///99..... 12141
376
11

---

378
[18]
Haw-Shiuan Chang, Erik G. Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1002-1012, 2017. URL pttoc:// neur iss. cc/ 2017//sh/ Abbrratt--bttttttta.a html
379
381
382
383
385
[19]
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2309-233 2318. PMLR, 2018. URLhttp! aml.r.iinnn.mm.......m.mm..
386
387
388
389
390
//proceedings.oo mlr. peress/v80// jta//jjangg jang18c. hhtml
[0
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings the the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässaan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4331 -340. PMLR, 2018. URL httt:::/pproceednngss mlr. Prress/v80/eee ren18a........ hhtml
382
393
394
395
396
[21] Tianyi Zhou, Shengjie Wang, and Jeff A. Bilmes. Robust currieulum learning: from clean label detection to noisy label self-correction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpt:://oppereeiieeeee
387
398
399
400
[22] Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019. Long Beach, California, USA. volume 97 of Proceedings of Machine Learning Research, pages 1062-1070. PMLR, 2019. URL UtR./http:/pprccediing mlr. wwwppprsssssee... ppwss/v97 v97/ htte//hhe19.... html
402
404
405
406
[23]
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman, editors, Proceedings of the 26th Annual International Conference
407
Machine Learning. ICML 2009, Montreal, Quebec. Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pages 41 41-48.144 ACM, 2009. doi: 10. 1553380
409
URL https:/::///ot org/
411
10.
412
[24]
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. ArXiv preprint, abs/2010. 1316,, 2020. URL //tttps:///rxxt
org/abs/2010/ 13166
414
[25] Jinchi Huang, Lie Qu, Rongfei Jia, and Bingiang Zhao O2u-net: A simple noisy label detection
415
approach for deep neural networks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3325-3333, 2019. doi: 10. 109/1CCV.20199000344444
416
417
[26] M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In John D. Lafferty, Christopher K. I. Williams, John Shawe-Taylor, Richard S. Zemel, and Aron Culotta, editors, Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pages 1189-1197 Curran Associates, Inc., 2010. URL URL/itpp::/pprcceddis...oo http://eeeddinn..nnnunieeeeeeeee wwt.eeis...cpppper///.....
418
419
420
421
422
423
html
424
[27]
Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. ArXiv preprint, 0/5/1511..6334 2015. httpstt:://arxxi org/ abs/ /1t./.111..11 111/1101111111
426
[28] Siddharth Gopal. Adaptive sampling for SGD by exploiting side information. In Maria-Floring Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICMI. 2016, New York City, NY, USA, June 19-24, 2016, volume 48
427
428
429
of JMLR Workshop and Conference Proceedings, pages 364-372. JML..org. 2016. URI. http://proceedings. mlr, wwwppppwsseeeeeee https///vA/goolll...mmm
12

---

431
[29] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document
432
recognition. Proceedings of the IEEE, 86(11):2278-333) 1998. doi: 10..699/..72671.
433
[30]
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
434
435
[31]
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. Mar 2022. URL
437
https:// http::/ppaaarrreeeee.eeftrorrrrp
438
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10.1 (0.1109/CVP..2016..90
440
BtRLhtts::/ //dotts::///iiooo org/ 10. 1109/111R..... 2016. 990
442
[33] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, page prwess/9//looottiee htmm
443
444
JMLR Workshop and Conference Proceedings, 2010.
445
/Rt.htttpp://prrceeedngg mlr.
448
[34]
Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings. pages 448 456. JMLR.org, 2015. URL http::/prooceciings .mlr press/v37/ioffoee .httflll11..mmm.]].
Sergey Ioffe and Christian Szegedy. Batch normalization:
447
448
449
450
[35] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https: // :
452
www.aaavvv.rr.....aaa org/ hbt//bbb/160/////////// 1000b/0650000500 06450
453
[36]
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data, 2017. URL htttp:://arxiv.ort abs/
455
1708. 02862
456
[37] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-1 1 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 233-242. PMLR, 2017. BtLLhtpp::// http::ttppccceedii/// mlr, prees// v70/arpit17a. html
457
458
460
462
463
[38] Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Kwok. Searching to exploit memorization effect in learning from corrupted labels, 2019. URL /ttpp::///xxii....../ org/ abs/
464
465
1911.0237
466
[39]
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kris- ten Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neu- ral Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada. pages 8536-8546, 2018 URL https://proceeddngg wttprrroweeiinnnnnnennrereeeeeep www..pp..ccppppee////.... pttppppp////////ss... Abstratt..bbbtttt... html
487
468
469
470
471
472
473
[40] Satrajit Chatterjee and Piotr Zielinski. On the generalization mystery in deep learning. ArXiv 474
475
preprint, abs/2203. 10036, 2022. httpss:/t//// http:/iiiwwrrrabb////aaaa/ http///bbs/222.... 10036
[41] Hwanjun Song. Minseok Kim, Dongmin Park, and Jac-Gil Lee. How does early stopping help generalization against label noise?, 2019. URL hattps:://raettt http:://aaaxxv..orabbbabae wttp://///..aa..... 1911110111111099991 08059
477
478
[42] Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttt::/ http:/nnnttttt//eee Qroc11y
480
13

---

482
[43] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, Califorria, USA, volume 97 of Proceedings of Machine Learning Research, pages 2535-2544. PMLR, 2019, http::/tt/////ooeeee/e mml............................. mlr. press/ v97 httph//hchhh......c 19tmmmll.mmmmmmmm
483
485
486
487
[44]
Xuan Zhang, Gaurav Kumar, Huda Khayrallah. Kenton Murray, Jeremy Gwinnup. Mariannal J. Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. ArXiv preprint, abs/1811 1.00399 2018. URL
488
489
490
https://arxe //axivv..vvvv.//.a org/ abs//ass....///////////////////a
1811. 181110071119
491
[45] Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2999-3007. IEEE Computer Society, 2017. doi:
492
493
10. 10.1109/1CCV22011.333 3244 URL https: //doi org/ 10. 1109 /CI0/////.... 2017. 3224
494
[46]
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach, 2020. URL httpps::///xxxv..oo
496
487
0bs/2010.02347
496
[47] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision, 2015. URL https:/t:://are http::::/axxvvv.rpo/ Abs/1512.00567
500
[48] Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smooth- ing mitigate label noise? In Proceedings of the 37th International Conference on Machine Learming, page 6448-6458. PMLR, Nov 2020. URL htttss://proceeddiii: mlr...mm.........m press//
502
504
v119/lukasik20a html
505
[49]
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. (arXiv:2106. 1tt.:200000044999 0414), Jun 2022. doi: 10. 110550arXii.22060444440 URL http://////a wttpoos////////a//oe///// 04149 arXiv:2106.04149 [cs].
506
507
508
509
[50] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust label noise: loss correction approach. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2233-2241 IEEE Computer Society, 2017. doi: 10. 10.1119/CVPR.2017.200 URL. U htt.:httt:///et //db:::://do.... org/ 10.
510
511
512
11000.11110 1009/ /wV.9////R..// 20170001440
514
[51] Yee Whye Teh, David Newman, and Max Welling. A collapsed variational bayesian inference algorithm for latent dirichlet allocation. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann. editors, Advances in Neural Information Processing Sys- tems 19, Proceedings of the Twentieth Annual Conference on Neural Information Process- ing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006, pages 1353- 1360. MIT Press, 2006. URL httpttts:/pppoceeediie wwwpiiiinnnnneiiis... neurips.cc. httpur//p...cpppp/////////////// /atppppa/e//////s//// 53277689700331112288880000222323 Abbrratt--.btttt...ht
515
516
517
518
519
520
521
[52] Barry Cobb, Rafael Rumí, and Antonio Salmerón. Approximating the distribution of a sum of
522
log-normal random variables. 2012.
Checklist
523
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
526
527
(b) Did you describe the limitations of your work? [Yes] In $3.1 we state that the curriculum view is valid when considering the first order derivatives. We also analyze the exception with NCE in (3.2
528
529
(c)
Did you discuss any potential negative societal impacts of your work? [N/A]
14

---

531
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
532
538
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In $42 we explicitly state the assumed distributions of 8; when deriving E[Ay]-
635
538
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix C we include the detailed derivations of [V.A, || and WNCE; in Appendix E we include the detailed derivation of E [[
537
538
539
3. If you ran experiments...
540
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All default settings are in Appendix B specific hyperparameters deviation from default settings
543
stated near each result.
545
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Tables 2 to 8 and TO] improve readability.
548
We omit error bars for figures to
547
548
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Appendix [
550
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
551
(a) If your work uses existing assets, did you cite the creators? [Yes] At the beginning of
552
$4 we cite MNIST, CIFARI0/100.
553
(b) Did you mention the license of the assets? [N/A] MNIST and CIFARIO/100 are classic
benchmarks
(c) Did you include any assets either in the supplemental material or as URL? [N/A]
556
557
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
558
559
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
582
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
583
564
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
565
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
587
A Extended Review of Loss Functions
569
As a general reference, we provide an extended review of loss functions for classification that is relevant the standard form Eq. (7). complementing review in $2.2 Loss functions and their
570
571
sample-weighting functions are summarized in Table We plot how hyperparameters affect their
572
sample-weighting functions in Fig. 6
573
A.1
Loss Functions without Robustness Guarantees
574
Cross Entropy (CE)
(s, log Py
575
the standard loss function for classification.
Focal Loss (FL) 45
LF u) = -1 (1- py) log
577 aims to address the label imbalance in object detection. Note that both CE and FL. are neither symmetric [10] nor asymmetric [I]]
15

---

579
A.2
Symmetric Losses
Mean Absolute Error (MAE) 9
1 k IMAR v)
[T y) -psl = 2py x 1 Py
k 1m1
581 is a classic symmetric loss, where I(i y) is the indicator function.
582
13 Reverse Cross Entropy (RCE)
LRCE($, (s, v)
Σ Pt log 1 = -)= -P (1 Py [MMAE(s
iØy
equivalent to MAE in implementation, where log is truncated to a negative constant A to avoid 583
numerical overflow.
Ma et al. [00 argued that any generic loss functions with L(s,t) > 0, D,Vie E (1,...,k)) k) symmetric by simply normalizing them. As an example,
can become
586
587
Normalized Cross Entropy (NCE)
INCE(S, v) = LCE(S,
-logPs
Σ (cE(
-
log
is symmetric loss [10. However, NCE does not follow the standard form of Eq. 7 It involves
589
additional regularizer as discussed in $3.2 and Appendix C thus being more relevant to discussions
590
in Appendix A.4
591
A.3
Asymmetric Losses
Zhou et al. [T] derived the asymmetric condition for noise robustness, and propose an array of 592
598
asymmetric losses:
Asymmetric Generalized Cross Entropy (AGCE)
LAGCE(s, y) (a + 1) (a + py))
9
where a > and > 0. It is asymmetric when I(q $ 1)( ( - + U(q 1) $ 1/7.
595
596 Asymmetric Unhinged Loss (AUL)
(sU (a - Pyy) - (a- 1)9
9
where a > and > O. It is asymmetric when T(q S 1) (1) + T(q S 1) $ 1/F
597
598
Asymmetric Exponential Loss (AEL) LAEL v) e PP/9
where > 0. It is assymetric when 1/9 < 1/F.
599
A.3.1 Combined Losses
601 Loss functions can be combined to enjoy better learning.
Generalized Cross Entropy (GCE)
LGcE (8, v)
q
can be viewed as smooth interpolation between CE and MAE, where 0 < S 1. CE or MAE can 604 be recovered by setting - or = 1.
605 Symmetric Cross Entropy (SCE) [K]
LSCE(s, y) LCE(S, y) b LRCE(S. v) (1 (1-9) (L-9)-(-logpp)) + (1 (p))
16

---

Name
Function
Sample Weight
Constraints
CE
loggy Py
Py
FL
(1 Puy log
(1 P1) Py Py log
q
MAE/RCE
1 Pv
Po(1 - -Pu)
AUT
(a+1) (a+tr))
Po Pu)(a -Py)
a> 1,9 0
AGCE
(n-Pu) pul 9- 1a-119
Pa + Pw)
(1 Pa)
a> 0,g
AEL
c Pa/4
}rv(L - -Pu)e -Du)e-Puee
PPu/a
g>
GCE
(1 - p8)/22
pb[1 PPy)
0 q S
SCE
-11 9) log + 9(1 pb)
(1 - pp) Pv)
0< <<1
TCE
(1- -pp--pil //
Ps Ei - pb)
9
Table 5: Expressions, constraints of hyperparameters and sample-weighting functions of loss functions in Appendix A that follows the standard form Eq. (7)
606 a weighted average of CE and RCE (MAE), where a > 0, > 0. and 0 < 1.
607
Taylor Cross Entropy (TCE) 15
Pb)
LICE(s,9) Σ 1=1
is originally derived from Taylor series of the log function. TCE reduces to MAE when 1. Interestingly, the summand of TCE (1- Py)'/i with > 2 is proportional to AUL with a = and q i. Thus TCE be viewed as combination of symmetric and asymmetric losses.
609
Ma al. [10 propose to additively combine active and passive loss functions. We review NCE+MAE
612 as an example:
-logP
(NCE+MAF((. v) = a LNCCE + b. IMAE(8. x (1 - 9) Σ/=1 - llog p
+ 9- (1 Py)
613
where a > 0,b >>>>0 0, and < < 1.
A.4 Loss Functions with Additional Regularizers
We additionally review loss functions that implicitly involve a regularizer and a primary loss function 516 that fits the standard form Eq. 77. See 6 summary, We leave investigation on how these additional regularizers affect noise robustness for future work.
617
Mean Square Error (MSE) 9
LMSE(8, v) =
Σ
(I[ii = 1) - pp)2 1- 2p,
1-2p, +
p
1=1
1=1
1 Pv
1
Σ
1 IMAR(S,
RXSE(S)
2
i=1
is argued [9] to be more robust than CE, where = 1/2 and the regularizer
RMSE(s) Σ
P
(14)
==1
620 reduces the entropy of the softmax output. We can generalize a a hyperparamter, making MSE combination of MAE and entropy regularizer RMSE-

---

1.0
0-0.5
1.0
0-20 g-0.1
1.0
08-
2-2
get.
0.8-
2-2
0.2.0
900.5
0.8-
100-
0.6
0.05
9-7.0
0-2.0
9-3.0
0.
0.4
0.4
02-
0.2-
0.2-
00
0.0
-15.0
-10.0
-15.0
-10.0
-10.0
(a) FL
(b) AUL., 0 = 2.0
(c) AUL, q = 0.1
1.0-
a=1.0 11100.mmm
x-0.1. qm0.5
1.0
g-0.1
08-
x=100
qm0.9
0.0
3.0.0
--100 gm0.5
0.8
0.6
a=1.0. 001.0.CH40
0.6
8-7.0 Q
0.6
Q-2.0
a-1.0 100 9-
0.4
0.4-
0.4
02-
0.2-
0.2-
0.0
0.0
0.0
-15.0
-10.0
-5.0
0.0
5.0
-15.0
-10.0
-5.0
0.0
5.0
-15.0
-10.0
-5.0
0.0
5.0
4,
(d) AGCE, a 1.0
(c) AGCE, @ = 0.5
(f) AEL
1.0-
q-0.2
0-0.2
1.0
Q=1.0
0.8-
0.8-
ge0.i
gm3a
9-4.0
066
Q-0.8
0.8
q=0.9
0.8
Q-6.0
1=10.0
0.4-
0.4
0.4
0.2-
0.2-
0.2-
0.0
0.0
0.0
-15.0
-10.0
-5.0
0.0
5.0
-1500
-10.0
-5.0
0.0
5.0
-15.0
-10.0
-5.0
0.0
5.0
4
(g) GCE
(h) SCE
(i) TCE
Figure 6: How hyperparameters affect the sample-weighting functions of loss functions in Table 5 The initial A, distribution of CIFAR 100 are included as reference.
Given generic loss function L(s, v), Peer Loss (PL) 11
LeI v) = L(s, 9) L(Bn))(8iee Un2)
623
can make it robust against label noise, where 8ny and Yon denote scores (of input æx,)) and labels randomly sampled from the noisy data. PL is inspired by the peer prediction mechanism truthfully elicit information when there is no ground truth verification. Its noise robustness is theoretically established for binary classification and extended to multi-class setting [14]. Cheng al. [46] later
624
625
628
627
show that PL in its expectation is equivalent to the original loss plus Confidence Regularizer (CR):
RCR(s) = - EEILLLL (L 3)
628
Substituting L with the standard LEE, RCR( (s) becomes
(c)(s) -Egl- lagiloog2 log Pyl
Σ
(15)
629
Minimizing (s)(8) thus makes the softmax output distribution p:'s deviate from the prior label distribution of the noisy dataset P(ÿ ii'ss, reducing the entropy of the softmax output.
630
Label smoothing [47 has been shown to mitigate overfitting with label noise [48]. With the standard cross entropy, Generalized Label Smoothing (GLS) [49]
632
LCLS+CE(8,U)
[[i -1 a) log
h
k
= - (1 a) log Py - a
log Ps
k
i=1
k
a
1
x - log
log LcE(s. v) + a' RoLs(s)
1
k
im1
18

---

Name
Original
4 Primary Loss
Regularizer
MSE
2py + Ei= P
Py
Li=1
PL
log Ps + log Purng log.ung lttns
Po
> P(ù = 1) log
GLS
Σ
[I(i = 1) a) logps
-logPy
Σ *
i= log
NCE
Py
stop hop_grad
logps
D log
>>
lo
Σ
log
Table 6: Original expressions, primary losses following the standard form Eq. 7 and regularizers for loss functions reviewed in Appendix A..4 We view PI. in its expectation derive its regularizer. is the softmax output with a random input Iin and random label Ynz sampled from the
Pang 22n noisy data.
Loss
CIFARIO
CIFAR100
SCE
9 0.7
9 0.95
GCE
4 0.3
4 0.9 4 ==
NCE+MAE
4 = 0.3
AUL
a 1.1,9 =
a 7.0, = 0.5
AGCE
0 = 0.1, 0.1
a 3.0, = 1.2
AUL
3.0, q 0.7
AGCE
a = 1.6, g 2.0
AEL
4 1.5
TCE
9 = 6
Table 7: Hyperparameters of each loss function different datasets. AUL and AGCE are with inferior hyperparameters.
639 where = aa/(1 a)), has regularizer RGLS
RGLs(s) = --- 1=1
log
(16)
634 With a' > O, RGLS corresponds to the original label smoothing 47). which increases the entropy of softmax outputs. contrast, a' corresponding to negative label smoothing [49] which decreases the output entropy similar to RCR.
635
637 Finally, with equivalent derivatives, NCE discussed in $.2 and Appendix C can be decomposed into
k loggy
(NCCE(.9)
Σk= - log log Pv +
log Ps
Σ logp:
K 1=1
hupp_gad(<NNc)) [LCE(S. v) + stop (oo__grad(eNce) RNCB(s)
638
where
RNCE(s)= logp:
(17)
1=1
639
is the same regularizer as RGLS with negative weight -ENCE.
B Detailed Experimental Settings
641
Our settings follow [T0] [1], with differences explicitly stated in the main text. All models on
CIFAR10/100 and MNIST are trained on NVIDIA 2080ti gpus with FP32. For models on the large scale dataset Web Vision 36., we use FP16 to accelerate training.
643
644
Synthetic noise generation The noisy labels are generated following 10 50 For symmetric
label noise, the training labels randomly flipped to different class with with probabilities
19

---

648
7 E (0.2,0.4.0.6.00. 0.8). Asymmetric label noise are generated by a class- desssdeednddntt flipping pattern. On CIFAR-100, 100, the 100 classes are grouped into 20 super-classes each having sub-classes. Each class are flipped within the same super-class into the next in circular fashion. The flip probabilities
647
649
are 7 E (0.1,0.2, 0.3.0...,.44
650
Models and Training We use a 4-layer CNN for MNIST, an 8-layer CNN for CIFARIO,
ResNet-34 [32) for CIFARI00 and ResNet-50 [32] for WebVision, all with batch normalization
652
34 Data augmentation including random width/height shift and horizontal flip are applied to
653
CIFAR10/100, On Web WisVis, we additionally include random cropping and color jittering. Without
654
further specifications, all models are trained using SGD with momentum 0.9 and batch size 128
655
for 50, 120, 200 and 250 epochs on MNIST, CIFARIO, CIFARIOG 100 and Web WebVision, respectively.
656
Learning rates with cosine annealing are 0.01 on MNIST and CIFARIO, 0.1 on CIFARI00, and 0.2 II,ARIO, 10-3 on CIFARIO 100 and 3 x 10-5 on WebVision Notably, all loss functions are normalized to have unit maximum sample
on WebVision Weight decays are 10-3 on MNIST, 10-4
658
weights, which is different from [10]. See Table 7 for hyperparameters of loss functions on different datasets.
860
C Deriving the Upperbound of Sample Weights of NCE
662
We provide detailed derivations for results in $32
663
Constant Norm of V V. 1.Ayl11' | Since
0
1,
i =
Üs;
Eisy
I-py"
p
i i
664
then
2A
IV = Σ
0s:
1
Σ
Σ
=
=1+1=2 = =2
koy
685
Approximating upperbound of WNCE in Eq.
(10
V VeLNNC(SS,V)l L
WNCE
IV,A,11
$ 2 INCE.
(IVvLLCC(8) w)ll + ENCE :
1
[ ENCE 5 V VoLCE((.)))l
$
2 INCE
k- k
= INC
WCE
+
ENCE
The derivation is based the inequality |x yl < |x|| + 1y). following the intuition 16 17 that |/ LNCR(88 y)]l can be regarded as sample weights.
667
D Underfitting of Robust Losses: Additional Results
689
In Table S we report similar results as Table 2 in 4 with smaller learning rates. Although settings that severe underfits slightly improve, the performance gap compared to CE is still substantial. Such results further confirms that underfitting results from robust losses themselves.
670
E Fixing Underfitting: Derivations and Additional Results
672
We include detailed derivations and additional results for
Simulated A 's well approximate real settings. We compare the simulated A, distributions to that of real datasets initialization in Fig. 7 Although less accurate with the variance, the simulated expectations mostly follow real settings, which supports the analysis in $42
675
20

---

CIFAR 100
CIFARIO
Underfitting
Loss
Acc.
Acc.
No
CE
68.76 + 0.21
0.962
90.24 + 0.14
0.624
GCE
69.00 + 0.24
0.956
90.83 + 0.20
0.644
No
SCE
68.89 + 0.05
1.165
91.07 + 0.09
0.726
NCE+MAE
68.21 + 0.51
0.520
90.14 0.09
0.344
NCE
57.95 + 0.26
0.330
85.96 0.21
0.206
Moderate
AUL
47.98 = 3.48
0.485
88.94 + 0.29
0.604
AGCE
43.51 = 2.58
0,406
90.71 + 0.19
0.549
MAE
9.11 + 0.83
0.025
90.65 1 0.10
0.355
Severe
AUL
10.04 1 2.33
0.023
90.77 T 0.04
0.337
AGCE
5.34 1 0.67
0.008
81.59 1 8.55
0.243
Table 8: Similar results as Table hyperparameters, AUL and AGCE losses can underfit regardless of hyperparameters of training.
except with learning rate a = 0.01. See Table 7 for detailed
with inferior hyperparamters are included reference. Robust
0.44
0.8
0.3-
0.2-
0.2
0.4
0.1
0.2-
0.0-
-5.0
A,
(a) CIFARIO
(b) CIFARI00
(c) VesVisio
1.0-
0.6
0.-
0.8
0.6-
0.44
02-
0.2
0.2-
0.0-
(d) Visiino
(c) WebVision200
(f) WebVision400
Figure 7: Comparing simulated and real A, distributions at initialization. We simulate with class scores following standard normal distribution, i.e., 8; ~ NO, 1). Histograms are real distributions while the curves are from simulations, with the vertical axis denoting probability density f f(Ay).
Deriving B(Ag) in Eq.
EE E(),,) E(s, log
E
11 Ellog
Σ e
jFy
jFy V jzy ed5]
R31 1 log E
e*1 +
21E
jæy ed
jf9
(k: 1)V(c*>
=2 14 log((( - 1) +
2( 1)B(ee=)]
(k - 1) 1)
=3 1 log((k: 1)elto0/21
2 12
-log(kk - 1) - 02/2 + 80? 1
2(k 1)
678 where = follows E[log X] ~ E[]]
v
51 =2 utilize properties of sum of log- normnolmal
variables [52). and =3 substitutes the expression of E Ele"" and Vie"v] for log- lormalllal distributions.
21

---

Clean
Symmetric Noise (Noise Rate 7?)
Loss
0
7) = 0.2
4= 0.4
? 0.6
7 = 0.8
CE
71.33 + 0.43
56.51 = 0.39
39.92 0.10
21.39 + 1.17
7.39 + 0.20
GCE
63.09 + 1.39
61.57 1.06
56.11
1.35
45.28
0.61
17.42 + 0.06
NCE
29.96 + 0.73
25.27 + 0.32
19.54 + 0.52
13.51 + 0.65
8.55 + 0.37
NCE+AUL 11
68.96 + 0.16
65.36 + 0.20
59.25 + 0.23
46.34 + 0.21
23.03 + 0.64
AGCE
49.27 +: 1.03
49.17 + 2.15
47.76
1.75
38.17 + 1.43
16.03 1 0.59
AGCE shift
67.50 +: 1.48
61.95 + 1.48
53.33 +: 1.08
33.26 1 0.37
10.47 1 0.57
AGCE rescale
67.20 = 0.79
64.28 = 1.27
56.32 0.59
38.52 + 1.67
12.75 + 1.10
MAE
3.69 + 0.59
2.92 + 0.46
1.29 1: 0.50
2.27 + 1.24
1.00 1 0.00
MAE shift
69.02 + 0.78
59.75 0.84
44.60
0.24
24.27 1 0.26
8.08 + 0.26
MAE rescale
69.95 + 1.21
66.42 +
60.70 0.30
45.17 2.37
10.79 + 0.97
Table Shifting or rescaling Ay mitigates underfitting on CIFAR100 with symmetric label noise.
We use 2.6 for MAE and AGCE and = 4.5 for AGCE. Test accuracies are reported with different runs. We also include results from as context.
Clean 0
Asymmetric Noise (Noise Rate 7))
Loss
1) 0.1
0.2
7) 0.3
1) 0.4
CE [
71.33 + 0.43
64.85 + 0.37
58.11 + 0.32
50.68 + 0.55
40.17 + 1.31
GCE
63.09 + 1.39
63.01 1.01
59.35 1.10
53.83 0.64
40.91 + 0.57
NCE
29.96 + 0.73
27.59 + 0.54
25.75 0.50
24.28 + 0.80
20.64 + 0.40
NCE+AUL
I
68.96 + 0.16
66.62 + 0.09
63.86 + 0.18
50.38 + 0.32
38.59 + 0.48
AGCE
49.27 + 1.03
47.53 + 0.73
46.77 2.37
39.82 + 2.70
33.40 1 1.57
AGCE-shift
67.50 + 1,48
64.07 = 0.90
56.16 + 1.44
46.73 + 1.39
38.37 + 1.55
AGCE-rescale
67.20 = 0.79
65.69 = 0.24
60.80 = 0.77
48.72 + 1.39
40.00 + 0.27
MAE
3.69 1: 0.59
3.59 1 0.56
3,19 1 0.98
2.11 + 1.93
2.53 1: 1.34
MAE-shift
69.02 1 0.78
63.82 1 0.84
56.38 0.45
48.93
0.53
40.57 1 0.47
MAE-rescale
69.95 = 1.21
68.01 1.08
65.71
= 0.47
57.40 0.35
39.22 + 1.54
Table 10: Shifting or rescaling Ay mitigates underfitting on CIFAR100 with asymmetric label noise. We use 2.6 for MAE and AGCE and = 4.5 for AGCE. Test accuracies are reported with
different runs. We also include results from [T context.
680
Additional results shifted and rescaled fix robust losses. We report results with symmetric
(Table 9 and asymmetric (Table I0) label noise with diverse noise rates For real world noisy datasets, we subsample Web Vision following standard settings [00 11 with different number of
682
classes, and report results with MAE and ResNet50 in Table [I] See Appendix B for detailed experimental settings. Notably, Web Vision50 corresponds to the mini setting adopted in previous
684
work 10 I1]. Shift and rescale A, mitigate underfitting of MAE and AGCE in general, resulting in performance similar to the state-of-the-arts.
686
Understanding Robustness: Additional Results
687
688
As a more extended exploration to Fig. in in Fig. we plot how distribution of A. evolve with more loss functions and more number of epochs human label noise of CIFARIO-N 3I. They all follow similar trends as in Fig.
690
22

---

k: 10
k: = 50
k = 1.8
200
k = 400
2.2
a 2.0
1.6
CE
62.40
66.40
70.26
/
MAE
10.0
3.68
0.50
/
MAE-shift MAE-rescale 48.40
58.40
60.76
59.31
66.72
71.92
Table 11: Shifting or rescaling Ag mitigates underfitting on real noisy dataset WesVis [36 with
different number of classes. Due to the scale of the dataset, we only report test accuracy with single run.
(1.00)
(1.00
(1.0)
(1.2)
(0.9)
(5.5)
(1.3)
(0.9)
(9.9))
25 (1.2)
25 (1.9)
25 (11.7)
48 (1.6)
48 (1.0)
49 (12.6)
79
(1.2)
79 (11.2)
(2.3)
119
(10.3)
-40.0
-20.0
0.0
20.0
40.0
-40.0
-20.0
0.0
20.0
40.0
-40.0
-20.0
0.0
20.0
40.0
Ay
By
(a) CE: 61.96
(b) FL: 61.02
(c) MAE: 68.34
(1.00)
D
(1.00)
1 (5.6)
(2.8)
(2.2)
B (1L.1)
9 (2.9)
9 (2.6))
25 (14.4)
25 (2.7)
25 (3.0)
(14.5)
49
(13.2)
(2.7)
(12.2)
139 (2.7)
(3.8)
-00.0
20.0
40.0
-40.0
-20.0
0.0
20.0
-20.0
20.0
40.0
A,
Ay
(d) AUL: 68.51
(e) SCE: 68.61
(l) TCE: 72.42
0.1100
0
01.000)
1. (5.5)
(5.1)
1 (4.7)
B (10.4)
9 (7.0)
9 (6.5)
(12.3)
(7.2)
25 (7.7)
(13.7)
(7.3)
(7.3)
79 (10.4)
(6.9)
79 (6.9)
139 (9.6)
119 (6.4)
119 (6.2)
40.0
-20.0
20.0
40.0
400.0
-20.0
-40.0
-20.0
20.0
40.0
4,
A
4,
AGCE: 73.19
(b) AEL: 78.13
(i) GCE: 78.78
Figure& 8: Additional results as Fig. 4 for more loss functions in Table 5 on CIFAR 10-N 31 with "worst" noisy labels (1) = 0.4). Note that CE and FL do not enjoy robustness guarantees.
23


Reviews:
[{"review_id": "a4H7gY2Hb_C", "reviewer": "Reviewer_a8RC", "summary": "When learning with noisy labels under certain robust loss designs, the authors focus on addressing two open questions: (1) Why do robust losses underfit? and (2) What affects the robustness of a loss? The two questions are explained through the view of sample-weighting curriculum perspective for certain robust losses. To overcome the underfitting issue, the authors propose to adapt the sample-weighting curriculums. To make loss more robust, the authors further show that adjusting the learning rate schedule helps.", "questions": "My concerns/questions/suggestions are listed below:\n\n1. In Line 62, a pointer/reference to the argument \"With $\\eta<(k-1)<k, L$ is robust against symmetric label noise\" would be appreciated.\n\n2. In Line 64, $\\ell$ is never defined, also for $\\Delta p$ in Eqn. (2).\n\n3. Typo: Line 108 \"a sampled is learnt\".\n\n4. In Table 3, although it seems that MAE has pretty bad performances on the mentioned tasks, would fine-tuning influence much on the performance of MAE? Since 1.00 accuracy on CIFAR-100 is merely random guessing or predicting all samples to be the same class.", "limitations": "See Weakness 2.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\textsf{Strengths:}$\n1. This work connects several popular robust loss designs to sample-weighting curriculum.\n\n2. Empirically, the authors explain the two open questions in the literature of learning-with-noisy-labels. The introduce of marginal effective learning rate looks interesting and helps with explaining the underfitting issue. And the shifting of soft-margin mitigates the underfitting, especially when the number of classes is large.\n\n$\\textsf{Weaknesses:}$\n\n1. Presentation could be further improved, i.e., notation definitions, and argument statements  (please refer to Section \"Questions\" for more details.)\n\n2. A potential weakness of this work is that all the analysis focuses on the class-dependent label noise, although it enables easiness of theoretical analysis, it was shown that real-world label noise tends to be instance-dependent [1]. And the majority section of this work falls into empirical explorations. It would be interesting to see whether the conclusions still hold for real-world noisy data, either small-scaled ones (CIFAR-10N, CIFAR-100N real-world noisy CIFAR [2]) or the large-scale Clothing 1M dataset.\n\n$\\textsf{References:}$\n\n1. Part-dependent Label Noise: Towards Instance-dependent Label Noise. [NeurIPS 2020]\n\n2. Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. [ICLR 2022]\n\n3. Learning From Massive Noisy Labeled Data for Image Classification. [CVPR 2015]", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JRaJqFxCw9a", "reviewer": "Reviewer_xcSK", "summary": "This paper offers a curriculum perspective in analyzing robust loss functions, which argues that each loss function implicitly induces a sample weighting curriculum. This analysis reveals that underfitting of robust losses can be explained by marginal effective learning rate and marginal initial sample weight. The paper proposes to address underfitting by adapting the sample-weighting curriculums.", "questions": "Please refer to my weakness section.", "limitations": "No", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengthes:\n\n1. The paper is well-motivated from the existing problems of robust loss functions.\n2. The analysis on the sample weighting curriculum offers an insightful perspective on the underfitting effect of robust loss functions.\n3. Motivated by the sample weighting analysis, the paper proposes sample-weighting curriculums that can address the underfitting issue.\n\nWeaknesses:\nMy main concern is that it is unclear how general the findings would hold. The experiments are conducted in a posthoc manner where it is first known that CIFAR100 suffer from the underfitting issue. Given that CIFAR100 is the only dataset with underfitting, it is unclear whether the findings can generalize to other datasets with similar issues. How to ensure that the sample weight curriculum is not a spurious correlation imposed by the CIFAR100 dataset?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0Vc03aZvPyR", "reviewer": "Reviewer_CAua", "summary": "In learning with noisy labels, designing certain loss functions is a promising direction to achieve noise robustness. This paper aims to use a new perspective (curriculum) to understand the key issues of robust loss functions. Typically, there are two issues in existing loss functions. 1). noise robustness, which is usually analyzed by symmetric condition; 2). underfitting problem, which is an open question to be explored.\n\nOverall, this paper has following contributions: \n1. This paper claims that the robust loss functions are implicitly weighted. Based on this finding, the authors explain the reason of underfitting problem (i.e. either due to small initialized weight or weight diminishing during training) and noise robustness (i.e., weights are different for clean and noisy samples).\n2. This paper addresses underfitting problem by shifting or scaling the initialized weights.\n", "questions": "1. In Table 3, settting $a=2.6$ achieves optimal results for MAE. How to find this optimal hyperparameter? Does it depend on the complexity of dataset?\n\n2. Does shitting or rescaling also work for other robust loss functions? such as NCE or AGCE?\n\n3. Existing works explain the underfitting problem by gradient analysis (gradient coefficients are smoothed) or regularization theory (fitting ability is limited). How will you compare this work with other works?\n", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Using sample-weighting curriculum to understand underfitting problem is interesting.\n2. The overall paper is well-written and makes sense.\n\nWeaknesses or concerns:\n1. Since this paper attempts to explain some open questions, no SOTA method is proposed. The overall novelty is limited.\n2. The performance improvement of the proposed techniques (shifting and scaling) is marginal, especially at high noise rate. Compared to the existing methods (NCE+AUL or NCE+RCE), the performance of MAE+rescale does not outperform them much. \n3. It seems that the proposed approaches are not practical and dependent on empirical observations. For example, it has to estimate the expectation of $\\Delta_{y}$ in Eq. (9) based on several assumptions.  In Eq. (10), it seems that the hyperparameter $a$ also depends on complexity of data, which can be hard to find in practice. I am curious about the performance of MAR+rescale on real-world noisy datasets, such as Clothing1M and Webvision.\n4. In Table 2, it shows that both AUL and AGCE suffer from severe underfitting problem. The paper only provides the results on MAE. Do shift or rescale work for AUL or AGCE? More experiment results should be conducted on these complex loss.\n5. One of the proposed claim that \"cross entropy can appear robust by adjusting learning rate\" is not new idea. [1] has observed the similar results and adjusts the learning rate cyclically to achieve noise robustness.\n6. In Eq. (5), the loss can be represented as a form of $w(\\Delta_{y})$ and $\\Delta_{y}$. This paper only considers the impact of $w(\\Delta_{y})$ but does not further investigate the effect of $\\Delta_{y}$ distribution (only provide the $\\Delta_{y}$ distribution at at initialization in Figure 1). Will the $\\Delta_{y}$ distribution change in training?\n\nIn summary, this paper proposes an interesting perspective to understanding key problems of noise robust loss functions. However, some claims are \"hand-waving\" to me and the theoretical analysis is incomplete. The experimental results are not fully supporting the advantage of the proposed methods when compared with recent noisy-label learning methods. I hope that the authors can clarify these points in the rebuttal.\n\n[1] Huang, Jinchi, et al. \"O2u-net: A simple noisy label detection approach for deep neural networks.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0Au6FUxizY", "reviewer": "Reviewer_racf", "summary": "The paper provides a new understanding for two open questions in the learning with noisy labels literature: why robust loss functions can underfit the data distribution and why robust functions can be robust. The analyses are delivered by decoupling the equivalent sample weight from each loss function, which is further viewed as an implicit sample-weighting curriculum. Based on the proposed analytical framework, the authors experimentally studied the relationship between the underfitting of robust loss functions and the average effective learning rate (due to equivalent sample weights), and proposed solutions to mitigate underfitting. Besides, by putting the equivalent sample weights and training dynamics together, the authors showed the robust loss functions assign lower weights to wrong labels compared with CE. ", "questions": "There are some minor questions:\n1. The considered loss functions are relatively limited due to the fast-moving literature. There are several recent robust loss functions, such as peer loss and its extensions [R1-R2], Taylor cross-entropy [R3], label smoothing (positive or negative) [R4-R5]. It would be interesting to provide a library for the equivalent weights for these loss functions.\n2. In addition to the loss functions that do not require estimating the noise transition matrix, is it possible to also find the equivalent weights for the loss correction/reweighting based methods, such as [40], [R6]?\n3. It would also make the paper more convincing if Figure 4 could also be plotted for the above loss functions.\n4. Current experiments only consider the synthetic label noise with simple noise generation regime. It is interesting to see whether the findings hold on real-world human annotations, e.g., CIFAR-N [R7].\n\n[R1] Liu Y, Guo H. Peer loss functions: Learning from noisy labels without knowing noise rates. ICML 2020.\n\n[R2] Cheng H, Zhu Z, Li X, Gong Y, Sun X, Liu Y. Learning with instance-dependent label noise: A sample sieve approach. ICLR 2021.\n\n[R3] Feng L, Shu S, Lin Z, Lv F, Li L, An B. Can cross entropy loss be robust to label noise? IJCAI 2021.\n\n[R4] Lukasik M, Bhojanapalli S, Menon A, Kumar S. Does label smoothing mitigate label noise? ICML 2020.\n\n[R5] Wei J, Liu H, Liu T, Niu G, Sugiyama M, Liu Y. To Smooth or Not? When Label Smoothing Meets Noisy Labels. ICML 2022.\n\n[R6] Liu T, Tao D. Classification with noisy labels by importance reweighting. IEEE TPAMI 2015.\n\n[R7] Wei J, Zhu Z, Cheng H, Liu T, Niu G, Liu Y. Learning with noisy labels revisited: A study using real-world human annotations. ICLR 2022.", "limitations": "They discussed the limitations in Sec. 3.1 Potential negative societal impacts are not applicable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**\n\nThe idea and analytical framework are original and novel.\n\n**Quality, clarity**\n\nBoth the theoretical part and the experiments are clear and of high-quality.\n\n**Significance**\n\nThe reviewer believes this new finding could contribute sufficiently to the noisy learning literature, and guide the design of robust loss functions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject