A Curriculum Perspective to Robust Loss Functions
Authors: Zebin Ou
Yue Zhang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Learning with noisy labels is a fundamental problem in machine learning. Much work has been done in designing loss functions that are theoretically robust against label noise. However, it remains unclear why robust loss functions can underfit and why loss functions deviating from theoretical robustness conditions can appear robust. To elucidate these questions, we show that most robust loss functions differ only in the sample-weighting curriculums they implicitly define. The curriculum perspective enables straightforward analysis of the training dynamics with each loss function, which has not been considered in existing theoretical approaches. We show that underfitting can be attributed to marginal sample weights during training, and noise robustness can be attributed to larger weights for clean samples than noisy samples. With a simple fix to the curriculums, robust loss functions that severely underfit can become competitive with the state-of-the-art.

Under review as a conference paper at ICLR 2023
A CURRICULUM PERSPECTIVE TO ROBUST Loss FUNCTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning with noisy labels a fundamental problem in machine learning. Much work has been done in designing loss functions that are theoretically robust against label noise. However, it remains unclear why robust loss functions can underfit and why loss functions deviating from theoretical robustness conditions pear robust. To elucidate these questions, we show that most robust loss func- tions differ only in the sample-weighting curriculums they implicitly define. The curriculum perspective enables straightforward analysis the training dynamics with each loss function, which has not been considered in existing theoretical ap- proaches. We show that underfitting can be attributed to marginal sample weights during training, and noise robustness can be attributed to larger weights for clean samples than noisy samples. With a simple fix to the curriculums, robust loss functions that severely underfit can become competitive with the state-of-the-art.
ap-
INTRODUCTION
Labeling errors are non-negligible from automatic annotation (Liu et al., 2021; Khayrallah & Koehn, 2018), crowd-sourcing (Russakovsky al., 2015) and expert annotation (Kato & Matsubara, 2010; Bridge et al., 2016). The resulting noisy labels may hamper generalization since prramearamttei neural networks of noisy labels in classification tasks, a large body of research (Song et al., 2020) aims to design loss functions robust against label noise. Most existing approaches derive sufficient conditions (Ghosh et al., 2017: Zhou et al., 2021b) for noise robustness. Despite the theoretical appeal being agnostic to models and training dynamics", they may fail to comprehensively characterize the performance of robust loss functions. Specifically, it has been shown that (1) robust loss functions can underfit difficult tasks (Zhang& Sabuncu, 2018; Wang et al., 2019c; Ma et al., 2020), while (2) loss functions violating existing robustness conditions (Zhang& & Sabuncu, 2018: Wang et al., 2019c:b) can exhibit robustness. For (1), existing explanations (Ma et al., 2020; Wang et al., 2019a) can be limited as discussed in $2.2. For (2), to our knowledge, there has been no work directly addressing it.
memorize the training set (Zhang et al., 2017). To combat the adverse impact
We analyze training dynamics with various loss functions elucidate the above observations, which complements existing theoretical approaches. Specifically, we rewrite a broad array of loss functions into a standard form with the same implicit loss function and varied sample-weighting functions ($3), each implicitly defining a sample-weigg weighting curriculum. The interaction between the sample- weighting function and the distribution of implicit losses of samples thus reveals aspects of the train- ing dynamics with each loss function. Here a curriculum by definition (Wang et al., 2020) specifies a sequence of re- weweghtting for the distribution of training samples, e.g., sample weighting (Chang et al., 2017) sample selection (Zhou et al., 2021a), based on a metric for sample difficulty. No- tably, our novel curriculum perspective first connects robust loss functions to the seemingly distinct curriculum learning approaches (Song al., 2020) for noise-robust training.
With our curriculum perspective, we first attribute the underfitting issue of robust loss functions to marginal sample weights during training (84.1). In particular, for classification tasks with numerous classes, the initial sample weights under the curriculum of some robust loss functions can become marginal. When modifying the curriculums accordingly, robust loss functions that severely underfit can become competitive with the ttate-of-the-ae We then attribute noise robustness of loss functions to larger sample weights for clean samples than for noisy ones during training ($4.2). By examining
'Our code will be available at g thub.
Changes of model states during training except for trivial metrics like evaluation metries and loss functions.

---

Under review as a conference paper at ICLR 2023
the changes of implicit losses during training, we find that dynamics of SGD suppress the learning of noisy samples. Curriculums of robust loss functions further suppress the learning of noisy samples by magnifying the difference in learning pace between clean and noisy samples while neglecting unlearned noisy samples. Based on our analysis, we present two unexpected phenomenons when viewed from existing theoretical results. By simply changing the learning rate schedule, (1) robust loss functions can become vulnerable to label noise, while (2) cross entropy can appear robust.
2 BACKGROUND
Classification k-ary classification with input a E Rd can be solved by classifier arg max; 8; where is the score for the i-th class in the class scoring function s R + R* parameterized by 9. Class scores (<;; 0) can be turned into class probabilities with the softmax function p; e where p; is the probability of class i. Given a loss function ((s(<;; 0), v) and data (xz, y) with y }I..k} the ground truth label, 0 can be estimated by risk minimization arg ming Ez,y ((s(2; 0), y), whose solutions are called risk minimizers. We use s in place of s(x; 0) for notation simplicity.
--1= <"((E= e*x),
Noise robustness Mistakes in the labeling process can corrupt the clean label into a noisy label
y i
with probability P( = y x y) i ||z y/
ÿ
with probability P
Label noise is symmetric (or uniform) if P(ū - il a| v) n/ (/ (k- 1). Vi + y, with 1/ = P(ū + y) the noise rate constant. Label noise is asymmetric (or class conditonn if P(ù = i|æ, y) = P ily). Given data (xx 3) with noisy label J. a loss function L is robust against label noise if
arg ming Ex, . (sx;;; 0), 5) arg ming En, (s< V)
(1)
Conditions for noise robustness Most existing approaches on robust loss function (Ghosh al., 2017: Ma et al., 2020; Liu & Guo, 2020; Feng et al., 2020; Zhou et al., 2021b) focus on bounding the difference between risk minimizers obtained with noisy and clean data, i.e., ensuring that Eq. (1) approximately holds. These bounds only depend on the loss functions and mild assumptions about the dataset. To contrast our curriculum perspective with these approaches, we review two typical sufficient conditions for noise robustmess. Loss function L is symmetric (Ghosh et al., 2017) if
Σ # ((s C, Vs R
where C' is constant. It is robust against symmetrrc label noise with 17 (A- 1)/k. This stringent condition was later relaxed to the asymmetric condition. To rephrase, a loss function as a function of the softmax probability pt, i.e., [(s,i) i) = I(pc), is asymmetric (Zhou al., 2021b) if
P = i (0,,)) P ,)
1(p)) - 1(p; + Ap)
max (
inf DEp.AppI P-tApS1
7(0) 1(Ap)
where is a valid increment of pi. When clean labels dominate the data, i.e., F < 1, an asymmetric loss is robust against generic label noise.
The active-passive dichotomy Ma et al. (2020) draw a distinction between active and passive loss functions. By rewriting loss function I. into a sum of basic functions, I(ss, y) = - ((s, i). active loss functions can be defined with vi / y, [(s, i) = O, which emphasizes learning the target label. In contrast, passive loss functions defined with di # ((s, * 0 can be improved by unlcarning the non-targer labels. However, since there is no canonical guideline to specify 1(s, i)), different specifications can lead to ambiguities in the active-passive dichotomy as discussed in $2.2.
In summary, the above research fails to address open questions in $2.2. Since many loss functions degrade less in performance than cross entropy under label noise, exhibiting various degrees of noise robustness, as a slight abuse of terminology, we refer to them as robust loss functions hereafter.
2.1 TYPICAL ROBUST Loss FUNCTIONS
We review typical robust loss functions for our analysis besides cross entropy (CE) that is vulnerable to label noise (Ghosh et al., 2017). Differences in constant scaling factors and additive biases are ignored, as they are either equivalent to learning rate scaling in SGD or irrelevant in the gradient computation. See Table for the formulas and Appendix A for extended review of loss functions.
2

---

Under review as a conference paper at ICLR 2023
Symmetric The mean absolute enror (MAE; Ghosh al., 2017) and the equivalent reverse cross entropy (RCE; Wang et al., 2019c) are both symmetric as they satisfy Eq. (2). Ma et al. (2020) make loss functions satisfying ((s,t) 0, vi {1..k} symmetric by normalizing them with LN (s. y) L(s, w)/(Zi ii-1 i)). We include normalized cross entropy (NCE; Ma et al. 2020) as an example. Asymmetric We include asymmetric generalized cross entropy (AGCE) and asymmetrie unhinged loss (AUL) proposed by Zhou al. (2021b) as typical asymmetric loss functions. Notably, AGCE with q 2 1 and AUL with S 1 are completely asymmetric, i.c., Eq. (3) always holds when < 1. Combined Loss functions can be combined for both robustness and sufficient learning. For exam- ple, generalized cross entropy (GCE; Zhang & Sabuncu, 2018) is a smooth interpolation between CE and MAF. A Alternatively, symmetric cross entropy (SCE; Wang et al., 2019c) is weighted average of CE and RCE (MAE). Ma et al. (2020) argue that robust and sufficient training requires a balanced combination of active and passive loss functions. Accordingly to their active-passive dichotomy, CE and NCE are active while MAE (RCE) is passive. We include NCE+MAE as an example.
2.2 OPEN QUESTIONS
Why do robust loss functions underfit? Ma et al. (2020) attribute underfitting to failure in balanc- ing active- active-passive components. However, the active- passive dichotomy can be ambiguous. Given
IMAE v) & T(i = psl x Σ I(i v) - ps)
where II() is the indicator function, MAE is passive with ((s, [[ y) pal but active with [(s, i) = T(ii = () - p:). Wang et al. (2019a) view LV,, y) | as weights for sample gradients and attribute underfitting to their low variance, making clean and noisy samples less distinguishable. However, as we show in $4.1, MAE also underfits data with clean labels. In summary, neither Ma al. (2020) nor Wang al. (2019a) fully explain the underfitting issue.
What affects the robustness of a loss function? Although combined loss functions such as GCE and SCE fail to satisfy Eq. (2) and (3), it is unclear why they exhibit robustness against label noise (Zhang & Sabuncu, 2018; Wang et al., 2019c). Furthermore, it is unclear how training dynamics with loss functions, which are irrelevant in theoretical robustness guarantees (Ghosh et al., 2017; Zhou et al., 2021b; Ma et al., 2020; Liu & Guo, 2020; Feng et al., 2020), affect their noise robustness.
3 IMPLICIT CURRICULUMS OF LOSS FUNCTIONS
Loss functions in Table except NCE and NCE+MAE can be written as a function of the target softmax probability Py, i.e. L(s, 3) ((,,). A close examination of Py gives
1 c A, +1 sigmoid (A,),
Pg
(4)
Σ E
log Eiou
where
A, = log Σ it t S 8y
max 8;
(5)
indicates how well sample is learned, as A, 2 0 ensures successful classification y arg max, 8;- Loss functions with the form L(8,8) ((p,) can thus be rewritten into a standard form with equivalent gradients, i.e.,
L(8.1) v) = ((vb) = / Val(py)ds ('(r)))(AA))) (
-- Ay
(6)
is scalar weight wrapped with the stop-gradient operator, and A, is an implicit loss function embedded in L(s,y). Loss functions following Eq. (6) thus implicitly define different sample-weighting curriculums, with w(A,) the sample-weighting function and A, the metric for sample difficulty. Notably, A, factors out the preference from w (A,)), making it more direct metric for sample difficulty than those based on losses (Kumar et al., 2010) or gradient magnitudes (Gopal, 2016). In addition, the interaction between w (Ay) and Ag distributions reveals aspects of the training dynamics with each loss function. See Table for a summary of wo( (A,) for the reviewed loss functions, and Appendix A for how hyperparameters affect w/ w(A,).
where w( v((A,) = 1 ('(p))''
3

---

W. where W; is the average sample weight of the Under review as a conference paper at ICLR 2023
Type
Name
Function
Sample Weight a
Constraints
/
CE
Po
Pv
MAE/RCE
- Py
Pu (1 PP)
Sym.
NCE
p (Ei=1 -log
/
AUL
[[a Pb)
{a 1/9]/4
Pb py)(a py)
1.90
Asym.
AGCE
[(a 1) (a 1y)
Po(a + p,) (1)00-(111 Pb)
a> 0,g20
GCE
--p$8//q
P8(1 -Py)
0 0<gs1
Comb.
SCE
9) LCe + LMAE
9 Pb) pa)
0<g<l <1
NCE+MAE
(1 - q) LNCE + 9 LMAR
0<9<1
Table
Expressions, constraints and sample-weightin functions ($3) for loss functions in $2.1.
3.1
THE ADDITIONAL REGULARIZER OF NCE
NCE does not follow Eq. (6) as it additionally depends i + Yet it can be rewritten into
LNCE y) = 'NCE LCE(s, y) "NCE ENCE RNCE(S),
where "NCE 1/( -= log Pt) and ENCE = lo p/ - po are scalar weights wrapped with the stop nop-gradient operator. In Eq. (7), the first term a primary loss function defining sample-weighting curriculum similar to CE. The second is regularizer
(NCE(8) Σx &
logp:
reducing the entropy of the softmax output. Although the training dynamics of NCE are complicated by the additional regularizer, we can use the upperbound for the L1 norm of gradient V V.LNCE( v),
UNCE = 27NCE WCE (1 + ENCE) Z V y
(8)
the weight of each sample in parameter updates. Examining how ÚNCE changes during training helps understand why NCE underfits in $4.1. We leave derivations of Eq. (7) and (8) and discussions on similar loss functions with additional regularizer Appendix A.5.
ROBUST LOSS FUNCTIONS FROM THE CURRICULUM PERSPECTIVE
We examine the interaction between wi (A,) and A, distributions to address questions in $2.2. Re- sults are reported on MNIST (Lecun et al., 1998) and CIFAR10/100 (Krizhevsky, 2009) with syn- thetic symmetric and asymmetric label noise following Ma al. (2020); Zhou et al. (2021b). For real-world scenarios, we include CIFAR 10/100 with human label noise (Wei et al., 2022) and the large-scale noisy dataset WebVision (Li et al., 2017), which exhibit more complex noise patterns than symmetric and asymmetric label noise. Unlike standard settings, we scale (Ay) to unit maxi- mum to avoid complications, since hyperparameters of loss functions can change the scale of w(A,), essentially adjusting the learning rate of SGD. See Appendix B for more experimental details.
4.1 UNDERSTANDING UNDERFITTING OF ROBUST LOss FUNCTIONS
Robust loss functions underfit We confirm that difficult tasks like CIFAR100 (Song al., 2020), underfitting can result from robust loss functions themselves rather than inferior hyperpa- rameters. As shown in Table 2, CE outperforms NCE, AGCE and AUL by nontrivial margin CIFAR100. In particular, MAE performs much worse than CE, similar to AGCE and AUL with inferior hyperparameters. See Table 9 for similar results with learning rate 0.01.
Marginal sample weights explains underfitting The effective scale of parameter update at step t during SGD can be approximated by
Very small small be important.
4

---

Under review as a conference paper at ICLR 2023
CIFARIO0
CIFARIO
Underfitting
Loss
Acc.
Acc.
CE
71.33 + 0.23
8.183
92.76
0.30
5.541
SCE
71.36 + 0.39
9.541
93.17 + 0.06
7.018
No
GCE
69,95 + 0.40
8.861
92.96 + 0.13
6.151
NCE+MAE
68.89 1 0.23
92.37 1 0.33
AUL
58.75 + 1.07
5.278
92.43 0.19
5.171
Moderate
AGCE
49.27 = 1.03
4.537 /
92.61 + 0.18
5.225
NCE
43.18 + 1.55
91.28 4 0.22
MAE
3.69 = 0.59
0.035
91.56 + 0.11
2.492
Severe
AUL
3.13 = 0.43
0.033
91.13 + 0.06
2.308
AGCE!
1.62 = 0.69
0.009
87.14 4.96
1.701
Table 2: Without label noise, robust loss functions can underfit CIFARIO0 but CIFARIO. Hyperpa- rameters of loss functions are tuned on CIFAR100 and listed in Table 8. We report test accuracy and or (scaled by 10²) at the final training step from different runs with learning rate a = 0.1. Loss functions with inferior hyperparameters (denoted with f) are included as references. See Table 9 for similar results with learning rate a 0.01.
1.0
MAE AUL!
AUL ACCE
CE
0.8
SCE
OCE
3
CIRAR100
0.6
CHAARIO
0.4
0.4-
0.2-
0.2
0.0
11.00
-5.0
0.0
5.0
-10.0
-5.0
Ag 5.0
0.0
-10.0
-5.9
0.0
5.0
A,
A,
(a) Severe underfitting
(b) Moderate underfitting
(c) No underfitting
Figure 1: Sample-weighting functions w/ (Ay) of loss functions in Table 2 with hyperparameters in Table 8. We include the initial Ay distributions of CIFAR1O and CIFAR100 for reference, which are obtained by computing Ay with a randomly initialized model for all training samples.
batch and 0g the learning rate. The overall o, up to step t can be a; = Ei= az /t. In Table 2, for loss functions that heavily underfit on CIFARI00, az the final step is marginal compared CE, suggesting a marginal overall sample weight during training given the same learning rate schedule. Underfitting from fast diminishing sample weights Similar to CE, in Fig. 2a, ar of NCE based on ONCE peaks at initialization. However, it decreases much faster than CE since both "NCE and WCE decrease with improved Ay- In addition, the regularizer RNCE(s) further reduces the entropy of softmax output and thus 'NCE The resulting fast decreasing ÚNNCE hampers the learning of training samples, which can lead to underfitting.
Underfitting from marginal initial sample weights In Fig. la, unlike NCE, loss functions that severely underfit in Table 2 assign marginal weights to samples in CIFAR at initialization, which leads to marginal initial at. Dy of these samples can barely improve before the learning rate van- ishes, thus leading to underfitting. In contrast, loss functions with non-trivial initial sample weights (Fig. 1b and Ic) result in moderate or no underfitting. As further corroboration, we plot ā; of AUL with superior and inferior hyperparameters (AUL and AUL" in Table 2) in Fig. 2b. az stays marginal with AUL", but quickly increases to a non-negligible value before gradually decreasing with AUL.
Loss combination can mitigate underfitting. As ÚNCE peaks at initialization but quickly dimin- ishes while WMAE is marginal at initialization but peaks later during training, combining NCE with MAE mitigate the underfitting issue of each other. In Table 2, combining NCE and MAE suffers less from underfitting compared to both individuals.
Increased number of classes leads to marginal initial sample weights. Unlike CIFAR100, all loss functions in Table 2 perform equally well on CIFARIO. Such a difference has been vaguely attributed to the increased task difficulty of CIFAR 100 (Zhang & Sabuncu, 2018; Song et al., 2020). Intuitively, the more classes, the more subtle differences to be distinguished. In addition, the number of classes l determines the initial distribution of Ag. Assume that class scores s; initialization ii.d. normal variables s; ~ ((p )). In particular, μ = 0 and a = 1 for most neural networks with standard initializations (Glorot & Bengio, 2010; He et al., 2015) and normalization layers (loffe &
5

---

Under review as a conference paper at ICLR 2023
1.00
0.10
CE NCE
CE
0.80
0.08
AUL
AUL
0.60
0.40
0.04
0.20
0.02
0.00
20%
40k
60k 80k
20k 40k 60k a0k
(a) NCE with estimated weight upperbound.
(b) AUL with inferior/superior hyperparameters.
Figure 2: Different explanations for underfitting: (a) fast diminishing sample weights; (b) marginal initial sample weights. We plot the variation of a with training step & on CIFAR 100 without label noise for each loss function. ag NCE is estimated with wNCr- Since ÚICE is not comparable to WCE. we normalize ag with its maximum in (a) to emphasize its variation during training.
Clean
Symmetric
Asymmetric == 0.4
Human
Loss
"1 =
17 = 0.4
1 == 0.8
7 0.4
CE
71.33 1 0.43
39.92 1 0.10
7.59 0.20
40.17 1.31
GCE
63.09 + 1.39
56.11 + 1.35
17.42 = 0.06
40.91 0.57
/
NCB
29.96 + 0.73
19.54 + 0.52
8.55 0.37
20.64 + 0.40
NCE+AUL
68.96 0.16
59.25 + 0.23
23.03 0.64
38.59 0.48
/
AGCE
49.27 1.03
47.76 + 1.75
16.03 0.59
33.40 1.57
30.45 1.50
AGCE shift
69.39 0.84
48.21 + 1.06
14,49 = 0.17
40.76 0.74
48.71 + 0.45
AGCE scale
7
56.69 = 0.33
14.64 = 0.79
39.71 + 0.17
50.85 + 0.11
MAE
3.69 1 0.59
1.29 1 0.50
1.00 = 0.00
2.53 1 1.34
2.09 1 0.55
MAE shift
68.57 + 0.54
49.95 = 0.16
13.10 = 0.41
39.83 + 0.18
47.91 + 0.36
MAE scale
70.97 = 0.41
60.57 = 1.04
24.44 = 0.73
44.48 1.05
54.70 + 0.48
Table 3: Shifting or scaling w(A,) mitigates underfitting on CIFARI00 100 under different label noise.
We report test accuracies with 3 different runs. Results from Zhou et al. (2021b) are included as context (denoted with $). See Table 10 for hyperparameter r of each setting, and Tables 11 and 12 for results with more noise rates.
Szegedy, 2015; Ba al., 2016). The expected Ay can be approximated with
e0²
E[A,] - log(k - 1) - 0² /
2(k - 1)
We leave derivations and comparisons between our assumptions and real settings to Appendix C.I. large k results in small initial Ay; with sample-weigee weighting functions in Fig. Ia it further leads to marginal initial sample weights, which results in underfitting on CIFAR100 as discussed previously.
4.1.1 ADDRESSING UNDERFITTING FROM MARGINAL INITIAL SAMPLE WEIGHTS
Our analysis suggests that the fixed sample-weightinn function wi( (Ay) is to blame for underfitting. To make the initial sample weights agnostic to the number of classes, we can simply scale
w (Ay) w/ A = wr( / [ .T)
shift
w (A,) = w A = w(A, + |AA -7)
the sample-weighting functions, where т is a hyperparameter. Intuitively, |E[ in w wAyy) and t (A,) cancels the effect of k the weight of the expected initial 4,- A small thus leads to high initial sample weights regardless of k. In Appendix C. we visualize (MAe)(A)) and wHA (Ay) in Fig. and discuss the robustness of the loss functions they induce.
Results CIFAR 100 with different label noise are reported in Table 3. See Tables 11 and 12 in Appendix 1. for results with additional noise rates. We also report results on the large-scale We- bVision dataset with different numbers of classes in Table 4. In summary, shifting and scaling allevi- ate underfitting, making MAE and AGCE comparable to the previous state-of-ther -ftth--rt (NCE+AUL; Zhou et al. 2021b). Notably, w" (A,)) leads to dramatic improvements for MAE under all settings.

---

Under review as a conference paper at ICLR 2023
A 50
k 200
k = 400
Settings
T = 2.0
T = 1.8
T 1.6
CF
66.40
70.26
70.16
MAE
3.68
0.50
0.25
MAE shift
60.76
59.31
47.32
MAE scale
66.72
71.92
71.87
Table 4: Shifting or scaling wr (Ayy) mitigates underfitting on WebVision subsampled with different
numbers of classes. k = 50 is the standard "mini" setting in previous work (Ma et al., 2020; Zhou al., 2021b). We report test accuracy with single run due to a limited computation budget.
Clean
Asymmetric
Symmetric
Human
0.2 diff
1
0.2
1J
0.4
1J
0.8
1/
0.4
Loss
Acc
snr
diff
snr
diff
snr
diff
snr
diff
snr
CE
90.64
-7.06
0.32
-15.47
0.39
-31.95
0.57
-50.87
0.77
-28.51
0.53
SCE
89.87
-5.39
0.51
-3.84
0.99
-00.47
1.27
22..2
1.51
-15.84
0.86
GCE
90.44
-7.42
0.36
-6.80
0.96
-23.23
0.89
-45.32
1.04
-21.94
0.72
AUL 89.90
-2.51
0.81
-2.07
3.10
-5,87
2.96
-13.90
2.83
122.8
1.11
MAE
89.29
-2.21
1.00
-1.92
3.56
-4.36
3.33
-11.53
3.22
-10.35
1.32
AGCE 82.62
-9.42
0.92
-1.55
3.02
-19.90
2.16
-41.11
1.83
-21.73
1.28
Table 5: Robust loss functions assign larger weights to clean samples. We report snr and diff from the best of 5 runs on CIFARIO under each noise setting, as inferior initialization can heavily degrade the performance. Hyperparameters listed in Table 13 are selected cover more variants of sample- weighting functions (plotted in Fig. 8), which are not necessarily optimal.
Although w² (Ay) and w+ (Ag)) are agnostic to the number of classes at initialization, their perfor- mances differ significantly. Intuitively, w+ (A,) diminishes much faster than w² (A,,) with increased Ay, which can lead to insufficient training of clean samples and thus inferior performance.
4.2 UNDERSTANDING NOISE ROBUSTNESS OF LOSS FUNCTIONS
We show that robust loss functions following Eq. (6) implicitly assign larger weights clean sam- ples. The underlying reasons are explored by examining how A, distributions change during train- ing. Notably, similar sample-weighting rules are explicitly adopted by curriculums for noise robust training (Ren et al., 2018). We leave NCE to future work as it involves an additional regularizer.
Robust loss functions assign larger weights to clean samples. We use the ratio between the aver- age weights of clean (Wetean) and noisy (1niise) samples, smr = Welean/ Wnoise, to characterize their relative contribution during training. See Appendix C.2 for the exact formulas. Noise robustness is characterized by differences in test accuracy compared to results with clean labels (diff). We report diff and snr under different label noise on CIFARIO in Table 5. Loss functions with higher snr have less performance drop with label noise in general, thus being more robust.
To explain what leads to large snr, plot changes of A,, distributions during training on CIFARIO with symmetric label noise in Fig. 3. See Fig. 9 and 10 for similar results with additional types of label noise and loss functions. When trained with loss functions that are more robust against label noise (Fig. 3b and 3c), A,, distributions of noisy and clean samples spread wider and get better separated. In addition, the consistent decrease of Ay for noisy samples suggests that they can be unlearned. In contrast, training with CE (Fig. 3a) results in more compact and less separated A,, distributions. Furthermore, Ay of noisy samples consistently increases.
Dynamics of SGD suppress learning of noisy samples. As shown in Fig. 3a, noisy samples are learned slower than clean samples as measured by improvements of Ay, which can be explained by more coherent gradients among clean samples (Chatterjee & Zielinski, 2022). Similar results have been reported (Zhang al., 2017; Arpit et al., 2017) and utilized in curriculum-basee robust training (Y (Yao et al., 2019; Han et al., 2018). In addition, noisy samples can be unlearned as shown in Fig. 3b and 3c, which can stem from generalization with clean samples. Both dynamics suppress the learning of noisy samples but clean ones, thus leading to robustness against label noise.
Robust w[ (An) synergizes with SGD dynamics for noise robustness. In Fig. 1, the bell-shaped w(A,,) of robust loss functions only assigns large weights to samples with moderate Ay Since A, distributions initially concentrate at the monotonically increasing interval of w/ (Ay), (1) samples
7

---

Under review as a conference paper at ICLR 2023
11.00
(1.00
[0.6)
(1.)7)
(2.66)
9 (0.6)
(1.8)
(4.4)
25 (0.7)
25 (1.8)
25 (5.8)
49 (1.1)
49 (1.9)
49 (0.2)
79 (1.5)
(1.7)
79 (5.8)
119.(1.7)
119 (3.6
(5.2)
-20.0
0.0
20.0
40.0
-40.0
-20.0
-40.0
-20.0
Ag
Ay
Ay
(a) CE: 57.74
(b) SCE: 79.56
(c) MAE: 83.88
Figure 3: How By distributions of noisy (green, left) and clean (orange, right) samples change on CIFARIO during training with symmetrie label noise and 0.4. Vertical axes denoting probability density are scaled to the peak histograms for readability, with epoch number (axis scaling factor) denoted on the right of each subplot. We plot wo( (y)))) and report the test accuracy of each setting for reference. See Appendix C.2 for results with additional types of label noise and loss functions.
1.0-
0.9
1est Accuracy
0.8
0.0,
1ost Accuracy
0.0, CE
0.8
0.6, 1E-01
0.6,
0.0, 1E-02
0.6, CE
0.6, 1E-02
0.8, MAE
0.0, 1E-04
0.1
0.5
0.4
Epoch
Epoch
(a) = 0.01 with different (1,, loss).
(b) CE with different (n, a).
Figure 4: Learning curves with fixed learning rate and extended training epochs on MNIST under symmetric label noise, where is the learning rate and r) the noise rate.
with faster improving Ay. due to either larger initial weights faster learning as clean samples, are weighted more during early training and learned faster. The magnified learning pace difference explains the widely spread distributions in Fig. 3b and 3c. In addition, (2) the unlearned samples with small A, receive diminishing weights from w(A,, which hampers their pace of lcarning. Noisy samples in Fig. 3b and 3c are consistently unlearned and ignored with marginal sample weights, leading to consistent decrease in Ag. In addition to the SGD dynamics, (1) and (2) further suppress the learning of noisy samples and enhance that of clean samples, thus leading to increased robustness against label noise. In contrast, the monotonically decreasing WCF( (Ay) emphasizes samples with smaller Ay, essentially acting against the SGD dynamics for noise robustness. Thus training with CE results in increased vulnerability to label noise shown in Table 5.
4.2.1 TRAINING SCHEDULES AFFECT NOISE ROBUSTNESS
Although the learning pace of noisy samples gets initially suppressed, the expected gradient will eventually be dominated by noisy samples, since well- clean samples receive marginal sam- ple weights thanks to the monotonically decreasing interval of w/ (Ay) Models with extended train- ing thus risk overfitting noisy samples during the late training stage. Adjusting the training sched- ules to enable avoid such overfitting therefore affect the noise robustness of models. Based on this intuition, we present two phenomenons on noise robustness as examples that follow our curriculum perspective but deviate from existing theoretical results:
Extended training can make robust loss functions vulnerable to label noise. The learning curves of CE and MAE with constant learning rates MNIST are shown in Fig. 4a. Despite the theoreti-
EEoough training steps without early stopping diminishing learning rates for small training loss.
8

---

Under review as a conference paper at ICLR 2023
cally guaranteed noise robustness (Ghosh et al., 2017), similar to CE, with extended training, MAE eventually overfits noisy samples, resulting in vulnerability to label noise.
CE can become robust by adjusting the learning rate schedule. To avoid overfitting noisy sam- ples, we can avoid learning when noisy samples dominate the expected gradient. It can be achieved with either early stopping (Song et al., 2019), or a constrained learning pace that prevents sufficient learning of clean samples, which avoids diminishing weights for them. We show the learning curve of CE using fixed learning rates under symmetric label noise on MNIST in Fig. 4b. By simply in- creasing or decreasing the learning rate, which strengthens the implicit regularization of SGD (Smith et al., 2021) or directly slows down the learning pace, CE can become robust against label noise.
5 RELATED WORK
Our work closely relates to robust loss functions against label noise (Song et al., 2020). Most ex- isting studies (Ghosh al., 2017: Zhang & Sabuncu, 2018: Wang et al., 2019c; Ma et al., 2020; Liu & Guo, 2020; Cheng et al., 2021; Feng et al., 2020; Zhou et al., 2021b) focus on bounding the difference between risk minimizers obtained with noisy and clean data, which are agnostie to train- ing dynamics. In contrast, with our novel curriculum perspective, we analyze the training dynamics with robust loss functions for reasons behind their underfitting issue and noise robustness. The un- derfitting problem has been heuristically mitigated with loss combination (Zhang & Sabuncu, 2018; Wang et al., 2019c; Ma et al., 2020). We identify the cause and provide effective solutions.
Curriculum-based approaches combat label noise with either sample selection (Chen et al., 2019; Zhou et al., 2021a) or sample weighting (Chang et al., 2017: Jiang et al., 2018; Ren et al., 2018)- In particular, sample weights are explicitly designed (Wang et al., 2019a;b; Chang et al., 2017) or predicted by a model trained on a different dataset (Jiang et al., 2018; Ren et al., 2018). In contrast, the sample weights in this work are implicitly defined by robust loss functions. Notably, the implicit loss function we identified is a more direct metric for sample difficulty compared to common metrics based on loss functions (Kumar et al., 2010; Loshchilov & Hutter, 2015) and gradient magnitudes (Gopal, 2016), which are implicitly affected by the preference from the sample-weighting functions of loss functions. Our work is also related the ongoing debate (Hacohen & Weinshall, 2019; Wang et al., 2020) on strategies for selecting or weighting samples in curriculum learning: either casier first (Bengio et al., 2009; Kumar et al., 2010) or harder first (Loshchilov & Hutter, 2015; Zhang et al., 2018). The implicit curriculums of robust loss functions can be viewed as a combination of both strategies, emphasizing samples with moderate difficulty.
Most related to our work, Wang et al. (2019h) identify gradient norms as weights for sample gra- dients and propose heuristic designs of weighting functions for noise-robust training. In contrast, we explicitly identify the implicit loss function, which connects robust loss functions to curriculum learning, facilitates our analysis of the training dynamics and helps elucidate the robustness of loss functions from a curriculum perspective.
Altering noise robustness by adjusting the learning rate is reminiscent of (Huang et al., 2019). They use a cyclic learning rate to make models change back and forth between overfitting and underfitting to collect statistics for noisy label detection. To achieve noise robustness, they discard samples with detected noisy labels and retrain the model from scratch. In contrast, our results show that simply changing the learning rate can achieve noise robustness.
6 CONCLUSION AND DISCUSSION
We identified the implicit sample-weighting curriculums of broad array of loss functions. Our novel curriculum perspective enables examining the training dynamics with loss functions through the interaction between the sample-weighting function and distributions of implicit losses. It con- nects robust loss functions to the seemingly distinct curriculum learning. Notably, the implicit loss function we identified is direct metric for sample difficulty in curriculum learning as it factors out the preference of sample-weighting functions. We elucidate the reasons behind underfitting and robustness against label noise and propose simple approach to address the underfitting issue.
As with previous work on robust loss functions, our empirical results are based on image classifica- tion using convolutional neural networks with and without residual connections. We have extended our experiments to cover larger-scale classification tasks, human label noise and a broader array of robust loss functions. Although our derivation does not depend on the models and task specifica- tions, additional experiments should be performed in future work to extend our conclusions to more models and tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 233-242, PMLR, 2017. URL arpit17an metitt7a.hm....
mlr mtt.sssss/.p./////////
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL aat//bbs///660 06450.
https: / //traaxivv./../////////////aaavm
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman (eds.), Proceedings of the 26th Anmual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pp. 41-48. ACM, 2009. doi: 10.1145/1553374.111 1553380. URL https:tttthet...t..ee .tt.//..1//0../..1 1553374. 1553380.
Pete Bridge, Andrew Fielding, Pamela Rowntree, and Andrew Pullar. Intraobserver variability: should we worry? Journal of medical imaging and radiation sciences, 47(3):217-220) 2016.
Haw-Shiuan Chang, Erik G. Learned-Miller and Andrew McCallum. Active bias:
Training more accurate neural networks by emphasizing high variance samples. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1002-1012, 2017. URL ptttps://proceeddnngg hwwp:eeeeddiinn..........ee.eeee neurips iis. cc httpp/ppprr///1//2 20////hh/hhh///// 2E37d10131f2a40 htr.://...22222eee00oooaaaae--e-
Satrajit Chatterjee and Piotr Zielinski. On the generalization mystery in deep learning. ArXiv
preprint, abs/2203.10036, 2022. URL httts::/arxiv..ore wtw.or////////////////..//eeaeee 2203.2233..2 10036.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1062-1070. PMLR, 2019. URL ht.p:: ppwceediiggga...ooo mlr wa....press/v.. chen 9g. html.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: sample sieve approach. In 9th International Conference on Learning Representationn, ICIR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URI. https://cpeezeviae
net/ forum? iddiddd20 2VdyyyyyyyII
Barry Cobb, Rafael Rumí, and Antonio Salmerón. Approximating the distribution of sum of log-normal random variables. 2012.
Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 10. 249663/jcai22220/3300 URL
ijcai.org, 2020. doi: http::///ooo.oo//////./o /11444993aaai httpcaa..2200/330 2020/305.
Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Satinder P. Singh and Shaul Markovitch (eds.), Proceedingy of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 1919-1925, AAAI Press, 2017. URL http:/ aaai. org//ca. www.xxxwpppp.eaaea /AAI// httaaper/viwww/ee 14759.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010. URL https: / //proceedings... ml 1 prr.ssresss...... / oooo Ca html.
10

---

Under review as a conference paper at ICLR 2023
Siddharth Gopal. Adaptive sampling for SGD by exploiting side information. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 364-372. JMLR.org, 2016. URL http: / //proceedings mlr prrss..resss.... / gopal 6. html.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2535-2544, PMLR, 2019. URL
mll. wwwps/ v97 hww.hhhhhhhhncceeeccc 9a. html.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa Desa-Bianchi, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 31: Anmual Con- ference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8536-8546, 2018. URL https: pww.eedocceeeeeeooooooooooooo.cc ngs nwurips. Abstract .tml..
paper/2018/ haa.//haah/aae www.1194222550 701re2268754450 754 4001 400000118600111 4474444477777477771111
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In 2015 IEEE International Confer- ence Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1034-1034. IEEE Computer Society, 2015, doi: 10.1 N0.1109/CCC..201..222 URL https: /doii /0. 1109/ /CCV.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp- 770-778. IEEE Computer Society, 2016. doi: 90.
10.1109/CVPR.20166 URL https: http//oo.oorrggg 10.
Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. O2u-net: A simple noisy label detection approach for deep neural networks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 3325-3333. IEEE, 2019. doi: 10.1 20,100//CCVV2201900000 URL https: /tte:://dooo /tw.//ooooioo/oooo org.oi..rg/// 10 1109 1w111199IICCCC II1.///19CVV.... 00342.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448 456. JMLR.org, 2015. URI http:://pp /ttp:://ppcceedli// ngs mlm press / v37 /ioffefff5555 html.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 Proceedings of Machine Learning Research, pp. 2309-2318. PMLR, 2018. URL http:///pooeeeen...../ mlr. www.ppesssvvv. /01/80/jiangla html.
Yoshihide Kato and Shigeki Matsubara. Correcting errors in treebank based on synchronous tree substitution grammar. In Proceedings of the ACL 2010 Conference Short Papers, pp- 74-79, Uppsala, Sweden, 2010. Association for Computational Linguistics. URL https: //aclante micclanhhooogg www.gy..rrg/P... www./ogg/////// 20144.
Huda Khayrallah and Philipp Koehn. On the impact of various types of noise neural machine translation. In Proceedings of the 2nd Workshop on Newral Machine Translation and Generation, pp. 74-83, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10. 1 URL https: /ates///cclll wowalannoooooyyoooooo -022982779.19
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
11

---

Under review as a conference paper at ICLR 2023
M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In John D. Lafferty, Christopher K. I. Williams, John Shawe- Richard S. Zemel, and Aron Culotta (eds.), Advances in Neural Information Processing Systems 23: 24th An- nual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pp. 1189-1197. Curran As- sociates, Inc., 2010. URL http::://ppccceeinnnnge/o////eet iewpeeeedinns..euiis.... httpp/sccccpppp//////eeeeee c57c6b956a 1ee.1122222886ce8e/e2
Abstract html.
Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recog- nition. Proceedings of the IEEE, 86(1 86(11):228--2324 1998. doi: 10. 109/5.726791.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data, 2017. URL 02862.
Tsung-Y Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 9999-3007. IEEE Computer Society, 2017. doi: 10.1109/ICCV. 2017.324. URL https:://///. / /oii org/ 10. 1109/ Itt///CC//0// 2017.322
Kun Liu, Yao Fu, Chuanqi Tan, Mosha Chen, Ningyu Zhang, Songfang Huang, and Sheng Gao. Noisy-labeled NER with confidence estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 3437-3445, Online, 2021. Association for Computational Linguis- tics. doi: 10.1 1011186661v// http////222..aaaa mmacc-mmin.22. 269. URL https: http:///aaaaaaota///.a/r.. org/2021. naacl-mate iiniinninn. n.n.n...... 269.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13- 18 July 2020, Virtual Event, volume 1 119 of Proceedings of Machine Learning Research, pp. 6226- 6236. PMLR, 2020. URL http: / propeediocceggsseeeeeeeeepops.se mlr press / /v11vv11111 / 1iu20e. html.
Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks ArXiv preprint, 1ls11511,,6643,,, 2015. URL https:://aaatt
wwwp/org////ooooooooot...ot..sot abs/1511. .0644..
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In Proceedings of the 37th International Conference on Ma- chine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 6448 6458. PMLR, 2020. htthttp: htt:://pocceedneee mlr. press press/vils vv11vv11//9 lukaslkkii..... k20a. html.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah M. Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In Proceedingy of the 37th In- ternational Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, vol- ume 119 of Proceedings of Machine Learning Research, pp. 6543 6553. PMLR, 2020. URL http::/ www.eeeiiggg...oooooo..pp.... ml wtw..mmrrr/mm//mmmmmmmmmmmmmmm / vv11 9/ma m0.. ht.ml
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2233-2241. IEEE Computer Society, 2017. doi: 10. (09/CVPR.2017.244 URL https: ///doi.or//// /t.rg1000. 10/1119P/PPP// 2017 240.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018, Stockholmsmmsscan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4331-434) PMLR, 2018. ht.t:: ttattt::/prooeeee www.ceedinng..... ml wwwppress/vve v80/ ren .htmm...
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. arXiv:1409.0573 [cs], 2015. URL http://an /tt////rxivv/// www.wv..rg//////.. /09 0575. arXiv: 1409.0575.
12

---

Under review as a conference paper at ICLR 2023
Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpss: /tps:///peervvii ew net forum? i drriiirrrrrrreeeee Hyo.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jac-Gil Lee. How does early stopping help generalization against label noise?, 2019. URL aattps://arxiinn org abs/ 1911 11991.000 08059. 1.05.99999.00.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey, ArXiv preprint, abs/2007.08 08190,19999900 2020. URL https:
//arxiv. org/ abs 2ww.00000..0000
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens. and Zbigniew Wojna, Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Com- puter Vision and Patlern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818-2826. IEEE Computer Society, 2016. doi: 10.1 2.1109/CVPR2201.33000 URL https: //doi. 10. 1.00.1100
308.
Yee Whye Teh, David Newman, and Max Welling. A collapsed variational bayesian inference algorithm for latent dirichlet allocation. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann (eds.), Advances in Neural Information Processing Systems 19, Pruceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006, pp. 1353-1360. MIT Press, 2006. URL https: http:::::////ppeeeeeeeeee hwwp:eeweeiiin....ei....eeeeeee. www.nnnurii.cmm http:/accaaaa.aaaaooooo.oe/////p //ash/// 75227Ch0770333112222988800023383 www.:rattt..a.............aaa... html.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey curriculum learning. ArXiv preprint, abs/2010.13166. 2020. URL pttpss//aariin wwt.:/ogg/aab/oo/oo/o
Xinshao Wang, Yang Hua, Elyor Kodirov, and Neil M. Robertson. Imae for noise-robuse learning: Mean absolute error does not treat examples equally and gradient magnitude's 's variance matters. ArXiv preprint, abs/1903. 121411 2019a. URL https::///// //tp://aaxvvvv wwwrxiii....r..eoo abs/ 1991. 12141.
Xinshao Wang, Elyor Kodirov, Yang Hua, and Neil M. Robertson. Derivative manipulation for general example weighting, ArXiv preprint, abs/1905. 11219955.113,,,,. 2019b. URL https://arxiv: org/abs/ 1905 11233.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 322- 330. IEEE, 2019c, doi: 10. N0.1109/CCV.2019.0000.. URL https: http://cc...r.... ott.oo..///./////// ICCV 2019.00041
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. ArXiv preprint, abs/2106.04149, 2021. URL https: //arxiv...ii/aa/aa org/ abs 2210..11222. 04149.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Confer- ence Learning Representations, 2022. URL https: /ttp///pppeevveeeeeeeeeeeeeeeeee net forum?id TBWA6PLJZCC.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Kwok. Searching to exploit memoriza- tion effect in learning from corrupted labels, 2019. URL IRLhhtts://aaxiii orw. abs/ 02377.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:ttpp.........tt. httptttttt/////peeeeeeeeeeee http///epeeeeeeeeeeeeeeee. net. forum? id=
13

---

Under review as a conference paper at ICLR 2023
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J. Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of cur- riculum learning for neural machine translation. ArXiv preprint, abs/1811.00739, 2018. URL https: /ttp///axvvvvv/// wtg/abs/////ooooooogoooooos...cr 007399.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neu- ral networks with noisy labels. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8792-8802, 2018. URL https: http://ooceeiin...neecee Abbrtr-bbstttatcc html.
Tianyi Zhou, Shengjie Wang, and Jeff A. Bilmes. Robust curriculum learning: from clean label detection to noisy label self-correction. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL https:://openreviers net. (aermmmdd=1TTWnmmmmmmm
Xiong Zhou, Xianming Liu, Junjun Jiang. Xin Gao, and Xiangyang Ji. Asymmetric loss functions for learning with noisy labels. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 18846-18566 PMLR, 2021b. URL http::/prrceeeingg . mlr. httppp/eeddnnn.......ssss/////// https///11////21111/1 html.
A EXTENDED REVIEW OF LOsS FUNCTIONS
Due to limited space, we only briefly describe typical robust loss functions in $2.1. As a general reference, here we provide comprehensive review of loss functions related to the standard form Eq. (6). Similar $2.1, we ignore the differences in constant scaling factors and additive bias. Loss functions and their gample-weighting functions are summarized in Table 6. We examine how hyperparameters affect their sample-weightiee functions in Fig.
A.1 Loss FUNCTIONS WITHOUT ROBUSTNESS GUARANTEES Cross Entropy (CE)
LCE(s, log Py
is the standard loss function for classification. Focal Loss (FL; Lin et al. 2017)
LF. v) = -
Pyy) P log
aims to address label imbalance when training object detection models. Both CE and FL are neither symmetric (Ma al., 2020) nor asymmetric (Zhou et al., 2021b).
A.2 SYMMETRIC Loss FUNCTIONS
Mean Absolute Error (MAE; Ghosh et al. 2017)
IMAR(S, v)
Σ T(i y) p:l = 2pp Py
is a classic symmetric loss function, where Il(i = y) is the indicator function.
Reverse Cross Entropy (RCE; Wang al. 2019c)
LRCE(S. v) =
Σ pi log 1(i = 3) = " (1- -1- Py = LMAE(S.
if
is equivalentt to MAE in implementation, where log 0 is truncated to a negative constant A to avoid numerical overflow.
Ma et al. (2020) argued that any generic loss functions with L(s, > 0, Vi e {1..k} can become symmetric by simply normalizing them. As an example,
14

---

Under review as a conference paper at ICLR 2023
Name
Function
Sample Weight
Constraints
CE
llggy
1 Py
FL
(1 P2) log
(1 p1) Py 9py log Py)
g
MAE/RCE
1 Py
Po(1-19) - Pp))
AUT
(a+1)-(atpy))
Po pu)(a -py)
a> 1,9>0 0
AGCE
(a- Pe- (a=1)
Pa + Po)0-1(1 (1-pa)
<> 0,g
AEL
e Pb/4
P - -Pu)e
g>>
GCE
(1 (1-p8)/9
P5[1 Pp))
0 0<951
SCE
(11 2) log + q(1 Pb)
(1 - q
q Pw](1 - pv)
0<q<1
TCE
11
Pv P (1 - pv)
9
Table 6: Expressions, constraints of hyperparameters and cumple-weightee functions of loss func- tions reviewed in Appendix A that follow the standard form Eq. (6).
Normalized Cross Entropy (NCE; Ma et al. 2020)
(NCE(s,y) y) LCE(S.
-loogg Py
Σ LCE(s,
Σ log
symmetric loss function. However, NCE does not follow the standard form of Eq. (6) as additionally depends p;. i * y. It involves additional regularizer, thus being more relevant to discussions in Appendix A.5.
A.3 ASYMMETRIC LOSS FUNCTIONS
Zhou al. (2021b) derived the asymmetric condition for noise robustness and propose numerous asymmetric loss functions:
Asymmetric Generalized Cross Entropy (AGCE)
LAGCE(s, y) (a+ 1) (a + Py)
q
where a > 0 and q 0. It is asymmetric when 1lg S (1(==1))100)) 1(g 1) s 1/8.
Asymmetric Unhinged Loss (AUL)
LAUL(S, v) = (a - (p)p (a - 1}9
q
where a > 1 and > 0. It is asymmetric when 1l(g S 1)( (-) 8-1 + 1ig S 1) 1/F.
Asymmetric Exponential Loss (AEL)
LAEL(S, v) PP//
where q 0. is asymmetric when 1/4 1/F.
A.4 COMBINED Loss FUNCTIONS
Generalized Cross Entropy (GCE; Zhang & Sabuncu 2018)
LGce(s,V)
q
can be viewed as a smooth interpolation between CE and MAE, where 0 < q S 1, CE or MAE can be recovered by setting g or =
15

---

Under review as a conference paper at ICLR 2023
Q-0.5
0-0.1
1.0
0.8
0-0.5
0.8-
q-4.0
4-3.0
0.8
Q-7.0
Q-9.0
0.7.0
0.-
0.4
0.4
02-
0.2-
0.2-
0.-
0.0
0.0
-15.0
-10.0
-5.0
0.0
5.0
-15.0
-10.0
-5.0
0.0
5.0
-15.0
-10.0
-5.0
0.0
5.0
(a) FL
(b) AUL, a = 2.0
(c) AUL, = 0.1
2-0.1
2+0.1
q=0.1
0.8
2-1.0
q-0.
000
0.0
q=2L0
2-9.
0-0.0
Q=1.0
04
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
15.0
10.0
5.0
0.0
5.0
15.0
10.0
0.0
5.0
15.0
10.0
-5.0
0.0
5.0
4,
4,
(d) AGCE, a = 1.0
(c) AGCF, 9 = 0.5
(f) AEL
1.0-
gmo.2
Q-0.6
q-0.0
Q-1.0
066-
0.0.3
0.6
9-9.9
9-0.0
q0.0.
04-
0.1-
0.1-
02
0.2
0.2
0.0
0.0
0.0
-15.0
-10.0
-5.0
0.0
5.11
-15.2
-10.0
-5.0
0.0
5.0
1550
-10.0
-5.0
5.0 A,
0.0
4,
(g) GCE
(h) SCE
(i) TCE
Figure 5: How hyperparameters affect the sample-weighting functions in Table 6. The initial A,
distributions of CIFARI00 extracted with randomly initialized model are included reference.
Symmetric Cross Entropy (SCE; Wang et al. 2019c)
LSCE(s, u) [CE(S, b. INCE(s,
x (1 - 4) log + (1 -1-10 pı)
is weighted average of CE and RCE (MAE), where 0, b 0. and 0 9 1. Taylor Cross Entropy (TCE; Feng et al. 2020)
Po)
LTCE(S,
derived from Taylor series of the log function. reduces to MAE when g = Interestingly, the summand of TCE - p////i with >> 2 is proportional to AUL with a = 1 and q = i. Thus TCE (1-P,)''/
can be viewed as a combination of symmetric and asymmetric loss functions.
Active-Passive Loss (APL; Ma et al. 2020)
Ma et al. (2020) propose weighted combinations of active and passive loss functions. We include NCE+MAE as an example:
LNCE+MAE(S.U) (8,)) v) = a LNCEES, v) + bb LMAE(S, y)
-logyy
x (1 - 9)
+ q (1 - po)
llogy
1=
where a > 0, b, 0, and 0 < < 1.
A.5 LOSS FUNCTIONS WITH ADDITIONAL REGULARIZERS
We additionally review loss functions that implicitly involve a regularizer and a primary loss function following the standard form Eq. (6). See Table 7 for a summary. In addition to the -eephe-weighting curriculums implicitly defined by the primary loss function, the additional regularizer complicates the analysis of the training dynamics. We leave investigations on how these regularizers affect noise robustness for future work.
16

---

Under review as a conference paper at ICLR 2023
Name
Original
Primary Loss
Regularizer
MSE
2py + Ei=ip
Py
Σ Pi
PL(CR)
log Py - -log Pyn 100m
log Py
Σ P = 1) log
CE+GLS
Σ E [L(i
y) - (3) log
log Py
= Σ log
NCB
- -logpa/(E. - -logr))
INCE
logpt
Σ 1 logp:
Table 7: Original expressions, primary loss functions in the standard form Eq. (6) and regularizers for loss functions reviewed in Appendix A.S. We view PL in its expectation to derive its regularizer.
Py am.. is the softmax probability of a random label yn with a random input zm sampled from the
noisy data. 7NCE
1/(E/= - p;) is scalar wrapped with the stop-gradient operator.
Mean Square Error (MSE; Ghosh al. 2017)
k
L (s. v)
(I(i v) -p,)²= = = -2p
Pi
2
k
x 1 Py +
Σ
po LMAE(S. y) + a. RMSE(S)
1=1
more robust than CE (Ghosh al., 2017), where a 0.5 and the regularizer
k
(MSE(S)
Σ
P
(10)
increases the entropy of the softmax output. We generalize hyperparamter, making MSE a combination of MAE and an entropy regularizer RMSE.
Peer Loss (PL: Liu & Guo 2020)
LPL(s, L(s, v) - L(s,, Vm)
makes generic loss function L(s, y) robust against label noise, where sn denotes the score of an input E,, and ym a label, both randomly sampled from the noisy data. Its noise robustness is theoretically established for binary classification and extended multi-class setting (Liu & Guo, 2020).
Confidence Regularizer (CR; Cheng et al. 2021)
Rcn(s) = EE [(s 9)1
shown (Cheng al., 2021) be the regularizer induced by PL in expectation. Substituting L with cross entropy leads to
k
(s) -EyI-logpe =
Σ P( = log
(1I)
1=1
Minimizing (cR(s) thus makes the softmax output distribution deviate from the prior label dis- tribution of the noisy dataset P i), reducing the entropy of the softmax output.
Generalized Label Smoothing (GLS; Wei et al. 2021)
Lukasik et al. (2020) show that label smoothing (LS; Szegedy et al. 2016) can mitigate overfitting with label noise, which is later extended to GSL. Cross entropy with GLS is
LCE+GIS(8.V) = Σ
[[i y) a) log
k
k
= -(1 a:) log - a
log Ps
To
i=1
1
x - logg,
log,; LCE(s, v) + +a' RELS(8)
k
17

---

Under review as a conference paper at ICLR 2023
where a' aa(11 a), has regularizer RGLS
RGLS(S)
log Pi
(12)
With a' > O, RGLS corresponds to the original label smoothing, which increases the entropy of softmax outputs. contrast, a' < corresponding negative label smoothing (Wei et al., 2021), which decreases the output entropy similar to RCR-
A.5.1 DERIVATIONS FOR NCE
Deriving Eq. (7) With equivalent derivatives, since
V=ICE(s.U) Σ (CE(s, V. E LcE(s, LCE(s
VaLNCE(S.N)
k Li=1 Lon(s,
VOLCE(S.y) + kLCE(s,8) Σ Et LCE(s,i)
Va
Σ k
LCE(),))
E (c
'NCE [V (VLC(()) + ENCE RxC((()))
NCE can be rewritten
LNCE(S, v) = "NCE (CE(s. v) + INCE CNCE (NCE($)
where "NCE 1/(E/= i=1 - log pı) and ENCE = k-- (w / log pi) are scalar weights wrapped with the stop-gradient operator discussed in $3.1. The regularizer
(NCE(S)
log
has a form similar to RGLS.
Deriving UNCE of Eq. (8) Here we derive the upperbound of || V LNCE(S v) discussed in $3.1:
L v)ll INCE (IV ENCE V (NCC())))))
S INCE
[V&LCE(B,)))
ENCE
| VVLCE((,)))l
INCE
WCE | + ENCE
WCB V.
2YNCE WCE (1 + ENCE)
-UNCE
The derivation is based on the inequality Jor + yl < |x|| + lvl and the fact that |/,,;;|1 = 2. The latter can be proved by straightforward calculations. Given
a
1,
0sj
Etyee ex
I-p,"
j i
then have
Σ
JA:
1+
Pj
=111=2 Pi
0sj
B DETAILED EXPERIMENTAL SETTINGS
Label noise The synthetic noisy labels are generated following (Ma et al., 2020: Zhou et al., 2021b; Patrini et al., 2017). For symmetric label noise, the training labels randomly flipped to a different class with probabilities 17 E {0.2, 0.4, 0.6, 0.8). Asymmetric label noise is generated from a class- dependent flipping pattern. On CIFAR 100, the 100 classes grouped into 20 super-classes, each
18

---

Under review as a conference paper at ICLR 2023
SCE
GCE
NCE+MAE
AUL
AGCE
AUL"
AGCE
/
1.1
0.1
3.0
1.6
0.7
0.3
0.3
5.0
0.1
0.7
2.0
Table 8: Hyperparameters of different loss functions for results in $4.1 and Appendix C.I. They tuned on CIFARI00 without label noise. Settings with inferior hyperparameters are denoted with f.
CIFARIO0
CIFARIO
Underfiting
Loss
Acc.
Ace.
CE
68.76 + 0.21
0.962
90.24 + 0.14
0.624
SCE
68.89 + 0.05
1.165
91.07 + 0.09
0.726
No
GCE
69.00 = 0.24
0.956
90.83 + 0.20
0.644
NCE+MAE
68.21 = 0.51
90.14 + 0.09
AUL
47.98 = 3,48
0,485
88.94 + 0.29
0.604
Moderate
AGCE
43.51 = 2.58
0,406
90.71 + 0.19
0.549
NCE
57.95 = 0.26
85.96 0.21
MAE
9.11 = 0.83
0.025
90.65 + 0.10
0.355
Severe
AUL
10.04 + 2.33
0.023
90.77 0.04
0.337
AGCE
5.34 = 0.67
0.008
81.59 + 8.55
0.243
Table 9: Similar results as Table 2 with learning rate = 0.01. Hyperparameters for loss functions are listed in Table 8.
having 5 sub-classes Each class is flipped within the same super-class into the next in circular fashion. The flip probabilities are 17 E {0.1, 0.2, 0.3, 0.4]. Human label noise for CIFAR10/100 are adopted from Wei et al. (2022). We use the "worst" labels of CIFARIO-N and the "fine" labels of CIFARION 100-N, both leading to = 0.4.
Models and hyperparameters We use 4-layer CNN for MNIST, an 8-layer CNN for CIFAR1O, ResNet-34 (He et al., 2016) for CIFARI00, and a ResNet-50 (He et al., 2016) for WebVision, all with batch normalization (loffe & Szegedy, 2015). Data augmentation on CIFAR 10/100 include random width/height shift and horizontal flip. On Web we additionally include random cropping and color jittering. Without further specifications, all models trained using SGD with momentum 0.9 and batch size 128 for 50, 120, 200 and 250 epochs on MNIST, CIFAR1O, CIFAR100 and Visiin, respectively. Learning rates with cosine annealing are 0.01 on MNIST and CIFARIO, 10, 0.1 on CIFAR 100, and 0.2 on WebVision. Weight decays are 10-3 on MNIST, 10 10-4 on CIFAR 10, 10-3 on CIFARI 100 and 3 x 10-3 WebVision. All loss functions are normalized to have unit maximum in sample weights, which is different from (Ma et al., 2020). Hyperparameters of loss functions are listed in Tables 8 and 13 for different experiments.
C ADDITIONAL RESULTS TO UNDERSTAND ROBUST Loss FUNCTIONS
We complement $4 in the main text with detailed derivations and additional results. Appendix C. extends discussions in $4.1 while Appendix C.2 extends $4.2
C.I UNDERSTANDING UNDERFITTING OF ROBUST LOss FUNCTIONS
Hyperparameters We list the hyperparameters tuned on CIFAR100 without label noise in Table 8 for experiments in $4.1 and Appendix C.I.
Robust loss functions can underfit. In Table 9 we report results similar to Table with learning rate a 0.01. Although settings that severe underfit slightly improve, they still perform much worse than CE, which further confirms that underfitting results from robust loss functions themselves.
Simulated Ay approximate real settings. We compare the simulated D, distributions based on our assumptions in $4.1 to distributions of real datasets at initialization in Fig. 6. The expectations of simulated A, follow real settings, which supports the analysis in $4.1.
19

---

Under review as a conference paper at ICLR 2023
0.4
0.4
03
0.3
0.3
02-
0.2-
0.1-
0.1
0.0
0.0
0.0
-10.0
-5.0
0.0
-10.0
-5.0
0.0
-10.0
-5.0
0.0
4,
4,
(a) MNIST
(b) CIFARIO
(c) CIFARI00
1.0-
0.6
0.8
0.8-
0.6
066-
0.4-
0.4-
0.-
0.0
0.0
10.0
0.0
10.0
5.0
0.0
10.0
5.00
0.0
4,
A,
(d) WebVision50
(e) Web
(f) WchVision400
Figure 6: Comparisons between simulated and real 4, distributions initialization. The simu- lations are based on the assumption that class scores follow normal distribution s; N 1) at initialization and plotted as curves. Real distributions are extracted with randomly initialized mod- els and plotted histograms. The vertical axis denotes probability density f(Ay).
Clean
Symmetric 0.4
Asymmetric 0.2
Human 0.4
Settings
0.0
0.2
0.6
0.8
0.1
0.3
0.4
MAE shift
2.0
2.6
3.0
4.0
4.0
2.6
2.6
3,0
3.0
2.6
MAE scale
2.0
2.6
2.6
3.0
4.0
2.6
2.6
3.0
3.0
2.6
AGCE shift
4.0
4.0
4.0
5.0
5.0
4.0
4.0
4.0
4.0
4.0
AGCE scale
2.6
3.0
4.0
5.0
5.0
3.0
3.0
3.0
3.0
4.0
Table 10: Hyperparameter of w (A,,) (A,,) and wit((,,)) (A,) for different noise settings on CIFAR 100. For
reference, [B[A,] = 5.097 on CIFAR 100 and B[A,] = 2.717 on CIFARIO. Better performance can be achieved with more thorough hyperparameter search.
Derivation of B(A,)) at initialization in Eq. (9):
E E(A,) = E(sy - log
e = 1 - E Blogg Σ Ed
jy ear]
13 log E
+
2E
igy e0i12
ity
(k 1) [}*])
=2 1 log (k [[{
2 1)TE(e=>))
(k: 1) e) 1 2222u+02
3 / logl(k: 1)
2[ 1)
60²
= - log(k - 1) - 02/2+ +
2(k - 1)
where , follows the approximation with Taylor expansion E[log X| ~ log E[g] -I (Teh et al., 2006), =2 utilizes properties of sum of log-normal variables (Cobb et al., 2012), and =3 substitutes E[e*>] and V[e"y] with expressions for log-normal distributions.
C.I.1 ADDRESSING UNDERFITTING FROM MARGINAL INITIAL SAMPLE WEIGHTS
Hyperparamter for different settings The hyperparameter controlling the shape of modified waight-we functions w (A,,) and w* (A,) can affect the noise robustness. Thus we tune 7 for the best performance under different noise types and noise rates, which are listed in Table 10.
20

---

Under review as a conference paper at ICLR 2023
Clean
Symmetric Noise (Noise Rate 7))
Loss
" = 0
1) = 0.2
") 0.4
") ==
1= 0.8
CE
71.33 + 0.43
56.51 + 0.39
39.92 0.10
21.39 1.17
7.59 0.20
GCE
63.09 + 1.39
61.57 = 1.06
56.11 = 1.35
45.28 0.61
17.42 + 0.06
NCE
29.96 0.73
25.27 = 0.32
19.54 0.52
13.51 0.65
8.55 + 0.37
NCE+AUL'
68.96 0.16
65.36 = 0.20
59.25 = 0.23
46.34 + 0.21
23.03 + 0.64
AGCE
49.27 1.03
49.17 = 2.15
47.76 1.75
38.17 = 1.43
16.03 0.59
AGCE shift
69.39 0.84
62.81 = 0.42
48.21 = 1.06
36.70 + 2,89
14.49 + 0.17
AGCE scale
70.57 = 0.62
64.73 = 0.98
56.69 = 0.33
39.02 + 1.20
14.64 + 0.79
MAE
3.69 + 0.59
2.92 + 0.46
1.29 = 0.50
2.27 + 1.24
1.00 + 0.00
MAE shift
69.02 + 0.78
59.75 + 0.84
44,60 0.24
24.27 + 0.26
8.08 + 0.26
MAE scale
70.97 + 0.41
66.83 + 0.84
60.57 1.04
49.23 1.22
24.44 + 0.73
Table 11: Addition results to Table with more symmetric label noise rates on CIFARI00.
Clean ") = 0
Asymmetric Noise (Noise Rate x))
Loss
= 0.1
1== 0.2
y=
y= 0.4
CE
71.33 + 0.43
64.85 + 0.37
58.11 = 0.32
50.68 0.55
40.17 + 1.31
GCE*
63.09 1 1.39
63.01 1: 1.01
59.35 = 1.10
53.83 0.64
40.91 + 0.57
NCE*
29.96 + 0.73
27.59 1 0.54
25.75
0.50
24.28
0.80
20.64 1 0.40
NCE+AUL
68.96 + 0.16
66.62 + 0.09
63.86 = 0.18
0.32 50.38
38.59 + 0.48
AGCE
49.27 +: 1.03
47.53 1 0.73
46.77
2.37
39.82
2.70
33.40 1.57
AGCE shift
69.39 1 0.84
63.03 1 0.42
55.84 - 0.78
49.05 0.81
40.76 1 0.74
AGCE scale
70.57 + 0.62
67.13 1 0.60
59.71 = 0.10
48.23 1 0.29
39.71 1 0.17
MAE
3.69 + 0.59
3.59 1 0.56
3.19 = 0.98
2.11 + 1.93
2.53 + 1.34
MAE shift
68.57 + 0.54
63.44 + 0.32
56.47 0.48
48.79 + 1.22
39.83 + 0.18
MAE scale
70.97 = 0.41
69.50 = 0.24
64.80 0.49
59.04 1.52
44.48 + 1.05
Table 12: Addition results to Table 3 with more asymmetric label noise rates on CIFAR100
Additional results with w (A,) and w (Ay). We report additional results under symmetric and asymmetric label noise with diverse noise rates in Table 11 and Table 12, respectively. Perfor- mance of MAE and AGCE gets substantially improved with w' (*,)) and w (Ay).
Visualization of w" (An) and u- (Ay). In Fig. we visualize the shifted and scaled sample- weighting functions of MAE on CIFAR100. Although both achieve the same initial sample weights | of CIFAR 100, w (A,,) diminishes much faster as A, increases, leading to insufficient learning of training samples, which can explain its inferior performance in Tables 3, 4, 11 and 12. Robustness of loss functions from w* (Ay) and t (Ay)) Our proposed w" (A,) and w+ (A,) aim to address the underfitting issue of robust loss functions with marginal initial sample weights. They modify Py into
Pyy e (AAA +1 e-(Aya)) +1
and
P e (A,
e (A,
where a r/|E[A,]1 and 8 |B| ||| 7, which induces new loss functions L L' y) I(p)) and (s, = ((tt). respectively. Commonly a < 1 and | since a small r leads to large initial sample weights and underfitting results from small E A Notably, 7 can determine the robustness of the induced loss functions. As shown in Table 10, a larger noise rate requires a larger 7 for better performance, which assigns less weights to samples with small Ay in general. However, our prelim- inary exploration find straightforward derivation from L(s, y) being symmetric/asymmettii to L L'(s.) y) and L ((s y) being symmetric/asymmetric. We leave the theoretical discussions to future work.
21

---

Under review as a conference paper at ICLR 2023
MAF-shift
MAE-scale
CIPAR100
CIFARIO
0.0
-15
10
-5
10
D
Figure 7: Shifted, scaled and the vanilla sample-weighting functions of MAE CIFARIOO. equals [][A,]] on CIFAR10. We include the initial Ay distributions of CIFAR 10/100 extracted with randomly initialized model reference.
AUI.
AGCE
GCE
SCE
2.0
3.0
2.0
0.4
0.95
Table 13: Hyperparameters different loss functions for results in $4.2 and Appendix C.2. They are selected for broad coverage of shapes, scales and horizontal locations of sample- sample-weightin functions instead of optimal performance on CIFARIO.
C.2 NOISE ROBUSTNESS OF Loss FUNCTIONS
Computation of Welean and Wnoise for snr in Table 5 The average weight for clean samples, adjusted by the learning rate at each step azz can be
Eie a 1(Jr, y1,1)
Welean Ee, I(Ji. Vi.e)
where W denotes the weight of i-th sample of the batch at step t, Ur. is the potentially corrupted noisy label and yi the uncorrupted label. Similarly, for noisy samples,
E.e a 1(Vi # wi,
Wnoise
Ei. 04 T(Ti, + Vi.e)
Hyperparameters We list the hyperparameters for different loss functions in Table 13 for results in $4.2 and Appendix C.2. In Fig. 8, we plot the sample-weighting functions of different loss functions. Changes of By distributions with different label noise and loss functions Complementing Fig. 3, in Fig. 9 we plot how distributions of Ag change during training on CIFARIO with additional types of label noise using hyperparameters in Table 13. They follow similar trends in Fig. 3, thus supporting analysis in $4.2. As MAE is not robust against asymmetric label noise with high n (Ghosh et al., 2017), results in inferior performance. We also include results with additional loss functions in Fig. 10. Since optimal hyperparameters will result in similar weeihiin functions, we choose hyperparameters for broad coverage of w (A,,) to better understand how they affect robustness.
22

---

Under review as a conference paper at ICLR 2023
1.0
MAE
AUL
0.8
AGCE
0.6
GCE
SCE
0.4
0.2
0.0
-15.0
-10.0
-5.0
0.0
5.0
Ay
Figure 8: Plots of sample-weighting functions of loss functions used in Table with hyperparameters Table 8.
23

---

Under review as a conference paper at ICLR 2023
(11.0)
(1.00
0 (1.00
(1.00
(2.2)
1 (2.8)
9 (0.9)
(2.9)
9 (9.2)
25 (L.1)
25 (2.5)
25 (6.5)
(2.4)
49 (6.7)
(2.1)
-000
-20.0
By
Av
(a) CE, Sym., 0.2: 74.49
(b) SCE, Sym., 0.2: 85.31
(c) MAE, Sym., 0.2: 86.71
(1.0)
(1.00
(1.0)
(1.00
(2.4)
(3.4)
25
(2.5)
25 (6.6)
(2.2)
(G.0)
(1.8)
(2.2)
-40.9
-20.0
0.0
20.0
00.0
-00.0
-20.0
0.0
20.0
00.0
-00.0
-20.0
0.0
20.0
00.0
4,
(d) CE, Human, 0.4: 61.12
(c) SCF, Human, 0.4: 72.56
MAF, Human, 0.4: 78.99
0.1.0)
011.0)
01.0.0)
(1.00
(0.7)
49 (1.2)
(3.0)
(10.0)
78 (1.7)
79 (3.1)
79 (9.7)
(3.8)
119 (3.0)
119 (9.4)
-40.0
-20.0
0.0
20.0
40.0
-00.
-20.0
0.0
20.0
40.0
-00.0
-20.0
0.0
20.0
40.0
(g) CE, Asym., 0.4: 73.75
(h) SCE, Asym., 0.4: 72.44
(i) MAE, Asym., 0.4: 61.11
Figure 9: Additional results Fig. with different label noise: (a-c) symmetric label noise with
0.2; (d-f) human label noise with n 0.4; (g-i) asymmetric label noise with 77 = 0.4. Noisy samples are colored green (on the left) and clean samples are orange (on the right). Test accuracies are included in the caption for reference.
24

---

Under review as a conference paper at ICLR 2023
(1.0))
(1.00
(1.0)
1 (7.00
(1.3)
1 (3.0)
(4.0)
25 (21.0)
(1.7)
25 (5.1)
(1.7)
(24.2)
(1.9)
(24.9)
(1.8)
-40.9
-20.0
0.0
20.0
00.0
-00.0
-20.0
0.0
20.0
00.0
-40.0
-20.0
0.0
20.0
00.0
4,
Av
by
(a) AGCE, Sym., 0.4: 44.52
(b) GCF, Sym., 0.4: 66.60
(c) AUT., Sym., 0.4: 84.12
0.1.0)
0110.))
(1.0.
(10.6)
(1.7)
(21.7)
(6.4)
(29.4)
(2.3)
49
79 (29.4)
79 (2.3)
79 (4.6)
119 (30.5)
119 (2.3)
119 (4.3)
40.0
-20.0
0.0
20.0
40.0
-00.0
-20.0
0.0
20.0
40.0
-00.0
-20.0
0.0
20.0
40.0
(d) AGCE, Human, 0.4: 35.73
(c) GCE, Human, 0.4: 68.22
(f) AUL, Human, 0.4: 77.22
011000
(1.00)
(10.5)
(1.7)
(3.4)
(21.7)
(1.9)
(c.4))
25 126.7)
(1.9)
25 (6.0)
49 (29.4)
49 (2.3)
49 (6.9)
79 (29.4)
79 (2.3)
79 (4.6)
119 (30.5)
119 (2.3)
119 (4.3)
400.0
-20.0
0.0
20.0
40.0
-00.0
-20.0
0.0
20.0
-40.0
-20.0
0.0
20.0
40.0
4,
A
(g) AGCE, Asym., 0.4: 46.75
(h) GCE, Asym., 0.4: 73.23
(i) AUL, Asym., 0.4: 67.72
Figure 10: Additional results Fig. with more robust loss functions under different label noise: (a-c) symmetric label noise with 0.4; (d-f) human label noise with 7 0.4; (g-i) asymmetric label noise with 7 = 0.4. Test accuracies are included in the caption for reference. Noisy samples colored green (on the left) and clean samples orange (on the right). Hyperparameters of these loss functions are selected for broad coverage rather than optimal performance.
25


Reviews:
[{"review_id": "yPrEeSwAVVS", "reviewer": "Reviewer_Xqtt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents work on learning with noisy labels. Interestingly, the paper provides a new curriculum perspective based on robust loss functions. The perspective aims to explain (1) why robust loss functions can underfit; (2) why loss functions deviating from theoretical robustness conditions can appear robust. Theoretical analyses and empirical evaluations are provided to justify the paper's claims.", "summary_of_the_review": "This paper provides a new perspective on learning with noisy labels. However, as mentioned, there are still many unclear explanations in the present version. Therefore, before rebuttal, the reviewer is negative about this paper. ", "strength_and_weaknesses": "**Strengths** \n\n- The research problem is realistic and important. It is popular to design robust loss functions to handle noisy labels. A new perspective on the robust loss function could provide insights into the research community.\n- Experimental results are detailed with extensive ablation studies. \n\n**Weaknesses**\n\n- The contributions of this paper are overclaimed. \n- The writing and organization of this paper should be improved. It is hard to follow the implementation and claims. \n- Theoretical results and discussions do not correspond to intuition and are often contradictory. \n\n**Specification of weaknesses**\n\n1. The paper claims that one of its main contributions is to understand why robust loss functions can underfit, and argue that prior works fail to comprehensively characterize the performance of robust loss functions. However, existing works, such as GCE, have analyzed the model robustness in training from the perspectives of gradients. It is not agnostic to training dynamics. Besides, it seems that the theoretical results of this paper are not related to a specific deep model. \n\n2. How to understand \"the weights are marginal\"? Although I carefully check this paper, experimental parts, in particular, there is still no clear understanding of \"marginal weights\". \n\n3. The definition of the sample weights should be given before Table 1 since $w$ appears in Table 1. \n\n4. From Section 2.2, the paper argues that the MAE loss is passive with the first specification but active with the second. The conclusion is completely contradictory to the definitions of active and passive losses in [1]. \n\n5. The paper claims that most loss functions in Table 1 can be written as a function of the target softmax probability, which is a bit rough. From this claim, some loss functions cannot be written as this function. Therefore, theoretical results cannot be applied to them?\n\n6. The derivation of Eq. (6) is important. More details should be provided. \n\n7. How do Eq. (8) and the following discussions help analyze the underfitting issue? It is not clear. Besides, the theoretical results of peer loss functions are different from GCE and SCE. There is no clear explanation for relating two types of theoretical results.\n\n8. The experimental results in Table 2 are confusing. Specifically, there are moderate underfitting for NCE and severe underfitting for MAE. Then, there is no underfitting for the combination of NCE and MAE. This is very strange. Can the underfitting issues of two loss functions be addressed by each other?\n\n9. Eq. (9) gives the explanation that $\\Delta_{y}$ is related to the class number $k$. However, it is still unclear how $\\Delta_{y}$ causes underfitting. \n\n10. Both \"scale\" and \"shift\" methods make the initial sample weights agnostic to the number of classes. From Table 4, there is a large gap between the performance of the two methods. Could the paper provide more discussions about the results? Besides, it seems that MAE totally fails in this case. \n\n11. Does extended learning mean training with a fixed learning rate?\n\n12. There is no clear understanding of why slowing down the learning pace can improve the robustness of a loss function. Intuitively, a slow learning pace still makes the model overfitting to mislabeled data. \n\n----\n[1] Xingjun Ma et al. Normalized loss functions for deep learning with noisy labels. ICML 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1WdusAEQuZV", "reviewer": "Reviewer_JUqW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Learning with noisy labels is an important task in weakly supervised learning, which has received extensive attention in recent years. This paper aims to answer two questions: why robust loss functions can underfit and why loss functions deviating from theoretical robustness conditions can appear robust? Specifically, the authors rewrite the loss function into a standard form with equivalent gradients and show that most robust loss functions differ only in the sample-weighting curriculums they implicitly define. Thus, they propose to scale or shift the sample-weighting functions to address underfitting. Additionally, they also show that training schedules can affect noise robustness. \n", "summary_of_the_review": "Overall, I think this paper is a relatively good work on learning with noisy labels because it provides a new perspective to explain the robustness of loss function against label noise. However, there are some typos in this paper and some details of the proposed methods are missing. So, I think the authors should carefully check the manuscript repeatedly and provide the necessary details and discussions to further polish this paper.", "strength_and_weaknesses": "Strengths:\n1. This paper explains the robustness of the loss function against label noise from a new perspective and shows most robust loss functions differ only in the sample weighting. Therefore, the robustness of some loss would be improved by adjusting the implicit sample-weighting function.\n2. This paper not only analyzes why some robust losses underfit but also gives two effective solutions, scaling or shifting the sample-weighting functions.\n3. Comprehensive experimental results in different cases clearly demonstrate the effectiveness of the proposed methods.\n4. This paper is well-written and well-organized.\n\nWeaknesses:\n1. There is a question, that is, does robust loss (e.g. MAE) with scaled or shifted sample-weighting function still satisfy the conditions for noise robustness?\n2. In nature, the methods proposed in this paper aim to adjust the sample weights. However, the authors neither compare the proposed methods with existing sample reweighting methods nor provide details and discussions for the differences between the proposed methods and existing sample reweighting methods.\n3. The authors claim that robust loss functions are vulnerable to label noise with extended training and provide the result of MAE on MNIST as an example. However, such observation does not mean that all robust losses would be vulnerable to label noise with extended training. They should provide more results of other robust losses to support this statement.\n4. I found some typos in this paper, so I would suggest that the authors could carefully check the manuscript repeatedly. The following are some typos I found:\n- \"Although the training dynamics of NCE is complicated by ...\" might be \"Although the training dynamics of NCE are complicated by ...\" in section 3.1.\n- The word \"distribution\" in the title of Figure 3 and Figure 5 may be in the plural.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0Uk9DFw6Ium", "reviewer": "Reviewer_BHcb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies why robust loss in the literature of learning with noisy labels may suffer from underfitting. Even though some early papers have already studied this question, authors provide a different perspective by showing that most robust loss functions differ only in the sample-weighting curriculums they implicitly define. Inspired this observation, authors explain the under-fitting phenomenon from the weighting term and propose an adjustment of weighting funciton to improve the model performance. Experiments are conducted on CIFAR-10, CIFAR-100 and Webvision datasets.", "summary_of_the_review": "- The paper has merits on explaining the underfitting phenomenon of certain robust losses in the literature of learning with noisy labels. Even though\nthe performance is somehow limited, I think the claims and experiments may inspire further research in this area. Thus, I initially give the score of 6.", "strength_and_weaknesses": "**Strength:**\n\n- By writting the losses into the form of Equation (6) is intersting, which enables us to examine the effectiveness of each loss by observing the dynamics of  $w(\\delta y)$ and $\\delta y$. The explanation is supported by the extensive experiments.\n\n- The proposed shifted and scaled MAE significantly improve the performance comapred to vanilla MAE.\n\n**Weakness:**\n\n- Even though the explanation and claims are interesting. The performance of adjusted loss is not competetive to SOTA. For example, the performance in this paper is even not stronger than [1] which also improves the robust losses. \n\n- [2] also observes that CE with adjusted learning rate can improve performance. Authors should cite and discuss the difference with [2].\n\n\n[1] Normalized loss functions for deep learning with noisy labels\n\n[2] O2U-Net: A Simple Noisy Label Detection Approach for Deep Neural Networks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-h9fZcWbng", "reviewer": "Reviewer_N4nc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a novel view to examine robustness against label noise and its under-fitting issue. It first points out previous arguments could not sufficiently address causes of the under-fitting issue. In contrast with the previous approaches, it factorizes a loss function with the sample-weighting function and the distribution of implicit losses, which provides a novel view to see it as a type of curriculum learning. It provides several experimental results showing training dynamics along with several measures based on the factorized terms, and makes arguments about the under-fitting issue and robustness.", "summary_of_the_review": "This paper provides a novel, and potentially impactful view about under-fitting problems of robust loss functions. They made several sub-claims based on their observations, which is intuitively understandable. Experiments are well designed, and the claims are sound in some extent, but they should clearly state that to what extent they want to generalize the claims. Also there are some minor issues including writing clarity and reproducibility. I am willing to raise the score upon the authors response.", "strength_and_weaknesses": "## Strength\nThis paper provides an interesting view for under-fitting issues in loss functions known to be robust against noisy labels. Various loss functions with various parameter settings are considered to be examine and observations are well aligned with our intuition. \n\n## Weakness\n- The experiments are mainly focusing on image classification, and on convolutional layer based models. The authors should discuss in the paper if they want to claim their findings can be generalized to any other tasks, and model architectures. Or it should clearly mention to what extent was this experiment covers in the main text. It is important because the paper's main claims are mainly supported my experimental results. \n- The main manuscript is not sufficiently self-inclusive. Readers should check the supplementary material to check important experimental settings, and term definitions. I understand this is mostly because this paper have to introduce many terms, measures, and acronyms. But such a paper should be more cautious in clarity. See comments in Clarity for actionable feedbacks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject