Active Negative Loss Functions for Learning with Noisy Labels
Authors: Xichen Ye
Xiaoqiang Li
Songmin Dai
Tong Liu
Yan Sun
Weiqin Tong
Venue: NeurIPS 2023 poster (2023)
Keywords: noisy label learning
robust loss function
multiclass classification
computer vision

Abstract: Robust loss functions are essential for training deep neural networks in the presence of noisy labels. Some robust loss functions use Mean Absolute Error (MAE) as its necessary component. For example, the recently proposed Active Passive Loss (APL) uses MAE as its passive loss function. However, MAE treats every sample equally, slows down the convergence and can make training difficult. In this work, we propose a new class of theoretically robust passive loss functions different from MAE, namely *Normalized Negative Loss Functions* (NNLFs), which focus more on memorized clean samples. By replacing the MAE in APL with our proposed NNLFs, we improve APL and propose a new framework called *Active Negative Loss* (ANL). Experimental results on benchmark and real-world datasets demonstrate that the new set of loss functions created by our ANL framework can outperform state-of-the-art methods. The code is available at
https://github.com/Virusdoll/Active-Negative-Loss.

Active Negative Loss Functions for Learning with Noisy Labels
Xichen Ye Shanghai University Shanghai, China yexi cex wwwp:/oossuueuuu.... edu. cn
Xiaoqiang Li Shanghai University Shanghai, China xql i Beli@@shueee edu.
Songmin Dai
Shanghai University Shanghai, China laodar@shu. edd.ar cn
Tong Liu Shanghai University Shanghai, China Long_liu@shu, edu. cn
Yan Sun Shanghai University Shanghai, China yansun@shu. edu.
Weiqin Tong
Shanghai University
Shanghai, China wqtongōshu. edu. cn edu.
Abstract
Robust loss functions are essential for training deep neural networks in the pres- ence of noisy labels. Some robust loss functions Mean Absolute Error (MAE) as its necessary component. For example, the recently proposed Active Passive Loss (APL) uses MAE its passive loss function. However, MAE treats every sample equally, slows down the convergence and can make training difficult. this work, propose new class of theoretically robust passive loss functions different from MAE, namely Normalized Negative Loss Functions (NNLFs), which focus
memorized clean samples. By replacing the MAE in APL with our proposed NNLFs, we improve APL and propose new framework called Ac- tive Negative Loss (ANL). Experimental results on benchmark and real-world datasets demonstrate that the new set of loss functions created by our ANL framework can outperform state-of-the-art methods. The code is available https::/githuub.com com/ https//ggthhb.omm/iimnddl////// Vitssool///ctveeeeaateeeloo/
Introduction
Relying large-scale datasets with high quality annotations, such ImageNet [1], deep neural networks (DNNs) achieve good performance in various supervised classification tasks. However, in practice, the process of labeling large-scale datasets costly and inevitably introduces noisy (mislabeled) samples. Moreover, empirical studies show that over-parameterized DNNs can easily fit randomly labeled dataset
(2). which implies that DNNs may have poor evaluation performance when trained on a noisy dataset. As result, noisy label learning has received lot of attention.
Different approaches have been proposed to solve the noisy label learning problem, and popular research line is to design noise-robust loss functions, which is also the main focus of this paper. Ghosh et al. have theoretically proved that, symmetric loss functions such Mean Absolute Error (MAE), are robust to noise, while others like commonly used Cross Entropy (CE) are not. However, MAE treats every sample equally, leading significantly longer training time before convergence and even making learning difficult, which suggests that MAE not suitable for training DNNs with challenging datasets [4. Motivated by this, several works proposed partially robust loss functions, including Generalized Cross Entropy (GCE) 4. generalized mixture of CE and MAE, and Symmetric Cross Entropy (SCE) (5). combination of CE and scaled MAE, Reverse Cross
Conresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Entropy (RCE). Recently, Active Passive Loss (APL) 66 framework has been proposed to create fully robust loss functions.
APL is one of the state-of-the-at methods, and shows that any loss function can be made robust to noisy labels by simple normalization operation. Moreover, to address the underfitting problem of normalized loss functions, APL first characterizes existing loss functions into two types and then combines them. The Iwo types are 1) "Active" loss, which only explicitly maximizes the probability of being in the labeled class, and 2) "Passive" loss, which also explicitly minimizes the probabilities of being in other classes. However, by investigating several robust loss functions created by the APL framework, we find that their passive loss functions are always scaled versions of MAE. As we mentioned before, MAE is not conducive to training. As result, to address the underfitting problem. APL combines active loss functions with MAE, which again may lead to difficulty in training and further limits its performance.
The fact that APL still struggles with MAE motivates us to investigate new robust passive loss functions. In this paper, we propose new class of passive loss functions different from MAE, called Negative Loss Functions (NLFs). We show that, by combining 1) complementary label learning 77] g and 2) a simple "vertical flipping" operation, any active loss function can be made into a passive loss function. Moreover, to make it theoretically robust to noisy labels, we further apply the normalization operation on NLFs to obtain Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with NNLF, we propose a novel framework called Active Negative Loss (ANL). ANL combines normalized active loss function and a NNLF to build a new set of noise-robust loss functions, which can be seen as an improvement of API. We show that under our proposed ANL framework, several commonly-used loss funetions can be made robust to noisy labels while ensuring sufficient fitting ability to achieve state-of-the-an performance for training DNNs with noisy datasets. Our key contributions are highlighted as follows:
We provide a method to build a new class of robust passive loss functions called Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with our proposed NNLFs, we propose a novel framework, Active Negative Loss (ANL), to construct new set of robust loss functions.
We demonstrate the thcoretical robustness of our proposed NNLFs and ANL to noisy labels, and discuss how replacing the MAE in APL with our NNLFs enhances performance in noisy label learning.
Our empirical results show that the new set of loss functions, created using our proposed ANL framework, outperform existing state-of-the -het--------tttttt methods.
2 Preliminaries
2.1 Risk Minimization and Label Noise Model
Consider a typical K-class classification problem. Let X C Rd be the d-dimensional feature space from which the samples are drawn, and y = [k] = {1,... .K} be the label space. Given a clean training dataset, S {w b where each (æ,. yn. is drawn i.d.d from an unknown distribution, D, over X' x y. We denote the distribution over different labels for sample æ by q(k(z)) and Σx (k)())= = 1. Since there is only one corresponding label y for a æ, we have q(v/>) =
=== + y/
and q
A classifier, h(x)= arg max; fæ);, where f: -C 0, 1] Vc C. c= = is function that maps feature space to label space. In this work, we consider f as a DNN ending with a softmax output layer. For each sample x. fxx) computes its probability p(k|æ=) each label & (1..... ,K), and Ek= p(k]aa)) = Throughout this paper, as a notation, we call f itself the classifier. Training a classifier is to find an optimal classifier f" that minimize the empirical risk defined by loss function: Σ [(f(xx,) yn), where L C R+ is loss function, and [(f(w), k) is the loss of f(x) with respect to label k.
When label noise is present, our model can only access corrupted dataset S, {(ar.: Un.) I==1 where each sample drawn i.i.d. from an unknown distribution, D,, this paper, we consider a popular approach for modeling label noise, which simply assumes that, given the true label y. the
2

---

corruption process is conditionally independent of input features æ [9 So we can formulate noisy label y as:
with probability (1 - (-my))
ú J.jEl, with probability Tly
(I)
where Tly denotes the probability that true label y is corrupted into label j. and Fyy = E Nud denotes the noise rate of label y. Under our assumption of label noise model, label noise can be either symmetric or asymmetric.. The noise is called symmetric, if Mi = - Vjj # and 7); vi E [k]. where 1) is a constant. And for asymmetric noise, Ti is conditioned on both the true label i and corrupted label j.
2.2 Active Passive Loss Functions
Ghosh et al. [3] have shown, under some mild assumptions, a loss function C is noise tolerant if it is symmetric: > [(f()), = C, Vo E X, where C' is some constant. Based on this, Ma et al. [6] proposed the normalized loss functions, which normalize loss function by:
((((x),)))
Loorn
EEEr ((f(x),k)
This simple normalization operation can make any loss function robust to noisy labels, since we always have Σ mmorm = 1. For example, the Normalized Cross Entropy (NCE) is:
Σk ΣK = j|a)) log (kka))
plgg)()))))))
NCE
Similarly, we can normalize FL, MAE, and RCE to obtain Normalized Focal Loss (NFL), Normalized Mean Absolute Error (NMAE), and Normalized Reverse Cross Entropy (NRCE), respectively.
But normalized loss function alone suffers from the underfitting problem. To address this, Ma et al. 66] characterize existing loss functions into two types: Active and Passive. Denote the function of loss
[(f(x),)) y) by ((f(w). k), that is ((f()), y) Σx (x) k) (e.g., let C be CE, then ((f(x), v) = = ((<(a=) log p(k ()), and {(f(x), = (A{[=)( log p(k]=)))) we have the following
definitions:
Definition (Active loss function). Lacivv is an active loss function if V(æ.v) E D, +
y, {ff()), k) O.
Definition 2 (Passive loss function). Prsssive is passive loss function if V(x, v) D, D, {(f()) k) + O.
+
Active loss functions only explicitly maximize p(ylar), the classifier's output probability at the class position specified by the label y. In contrast, passive loss functions also explicitly minimize p(k * f the probability at least one other class positions. Accordingly, the active loss functions include CE, FL, NCE, and NFL, while the passive loss functions include MAE, RCE, NMAE, and NRCE. These
types of loss functions can mutually boost each other to mitigate underfitting, and we refer the reader to [6] for more detailed discussions. By combining them, Ma et al. proposed the Active Passive Loss (APL):
CAPL = a LActvve + B Lasssive
(4)
where a, 3 > 0 are parameters. As an example, by combining NCE and RCE, Ma et al get NCE+RCE, of the state-of-the-ar methods.
2.3 APL struggles with MAE
As shown in the previous subsection, there are four passive loss functions available to APL, including MAE, RCE, NMAE, and NRCE. However, we can show that all these passive loss functions are scaled versions of MAE. Specifically, NMAE = 2) MAE, RCE -A MAE, and
N NRCE- शę M AE (detailed derivations can be found in appendix A. Thus, we can rewrite API, follows:
CAPL LActive + --MMAR MAE.
(5)
3

---

loss
1/4
4)
p(Al))
Figure 1: "Vertical flipping" operation. (()())), k) an active loss function, e.g., CE. Loss function A - [(f(x), k) is obtained by flipping ((f(2), k) vertically with axis loss 24.
This indicates that MAE is a necessary component the current APL. However, as we mentioned before, MAE requires longer training time and even makes learning difficult. Thus, on the one hand, API needs passive loss functions to mitigate active loss functions underfitting, yet on the other hand, MAE is not training friendly, which may limit the performance of APL. This motivates to investigate new robust passive loss functions.
Active Negative Loss Functions
3.1 Method
Normalized Negative Loss Functions. Our goal find method that creates robust passive loss functions from existing active loss functions. This method must consist of three components that: 1) let the loss function optimize the classifier's output probability for at least one other class position that is not specified by the label y. 2) let the loss function minimize the classifier's output probability instead of maximizing it, and 3) let the loss function robust to noisy labels. Inspired by NLNL [TO]
and APL (6), we use 1) complementary label learning, 2) "vertical flipping" and 3) normalization operation as these three components respectively.
Complementary label learning 7 g is an indirect learning method for training CNNs, which randomly selects complementary labels and trains the CNN recognize that "input does not belong to this complementary label". Unlike the usual training approach, complementary label learning focuses on the loss of classifier's predictions with complementary labels, which naturally fits with the passive loss function. Here, we only use its basic idea of letting the loss function focus on all classes (1,.... KK} except the labeled class y.
"Vertical flipping" is a simple operation that can convert the loss function from "maximizing" to "minimizing". As shown in the fig. [ given an active loss function ((f(x), k), the new loss function A - (x)(x, k) is obtained by flipping C(f(x), a.).K k) vertically with axis loss = = A.A. It should be noted
that, A - [(((w), k)) the opposite of L( (x)) k). and it focuses optimizing p(k(æ)) to O.
Based on these two components, given an active loss function L, we propose Negative Loss Functions (NLFs) as follows:
Lueg (f()) E
[())))
(6)
A= C( P.iin P.m......11 I".))
(7)
Here, [... Puiin .'- ". 1' is some probability distribution that may be output by the classifier f. where ply x)) = Pmin the minimum value of p(k| (c.g., 0). Therefore is some constant, the maximum loss value of L. In practice, setting Pmin 0 could cause some computational problems, for example, L is CE and Pmii = O. then A= log = +oo. So in this paper, unless otherwise specified, we define Pmin 1 x 10 10-7 This technique is similar to the clipping operation implemented by most deep learning frameworks.
Our proposed NI_F can transform any active loss function into passive loss function, where 1) (1 (k(aa))) ensures that the loss function focuses on classes (1..... .......K, K) (v), and 2) (A C(f()),k) ensures that the loss function aims to minimize the output probability p(k(æ))

---

Next, to make our proposed passive loss functions robust to noisy labels, we perform normalization operation on NLFs. Given an active loss function L, we propose Normalized Negative Loss Functions (NNLFs) as follows:
A [((()))))
L (f)()
Σx=1 A (())(,)))
where has the same definition as eq. 77) The detailed derivation of NNLFs can be found in appendix A.2 Additionally, NNLFs have the property that Lm E [0, IJ. Accordingly, we can create NNLFs from active loss functions as follows.
The Normalized Negative Cross Entropy (NNCE) is:
A log p(yla))
NNCE- =
(9)
Σ A (- logp(( (k()))
where A - llog Pmin Pmin'
The Normalized Negative Focal Loss (NNFL) is:
A (-(1-p(ylap)) log p()l)))
NNFL=1 Σk= (-(1 (-(1-p(k)=))) log logp()))z))
(10)
where A = -11 - Pmin)
log Pmin'
ANL Framework. We create robust loss functions by replacing the MAE in APL with our proposed NNLF. Given an active loss function L, we propose the Active Negative Loss (ANL) functions follows:
CANL Lnom + 3 Lun-
(11)
Here, and 3 are parameters greater than 0, Lnor denotes the normalized L and Con denotes the Normalized Negative Loss Function corresponding to C. Accordingly, we can create ANL from the two mentioned active loss functions as follows.
For Cross Entropy (CE), have ANL-CE:
LANL.CE LNCE LNNCE.
(12)
For Focal Loss (FL), we have ANL-FL:
LANL-FL α. LNFL 3 LNNFL.
(13)
3.2 Robustness to noisy labels
NNLFs are symmetric. We first prove that our proposed Normalized Negative Loss Functions (NNLFs) are symmetric. Detailed proofs can be found in appendix B
Theorem 1. Normalized negative loss function Can is symmetric.
3 it has been proven that symmetric loss functions, under some mild assumptions, exhibit noise tolerant in the f'ae of both symmetric and asymmetric noise. Given that our NNLFs fall under the category symmetric loss functions, they inherently possess the attribute of noise tolerant.
NNLFss are robust noise. reference to Theorem and Theorem 3 from
ANL is robust to noise. In light Lemma 3 from 6 it is understood that the combination two noise tolerant loss functions retains the noise tolerant attribute. It is noteworthy that both Caoen and C.n within our ANL are noise tolerant, which makes ANL whole noise tolerant.
3.3 NNLFs focus more well-learned samples
As shown the fig. 2 by replacing MAE with our proposed NNCE, NCE+NNCE and ANL-CE show better fitting ability. This raises the question: why does NNLF perform better than MAE? In the following, taking NNCE as an example, we analysis the gradients of MAE and NNCE to provide preliminary answer this question. Detailed derivations and proofs can be found in appendix C
The gradient of the MAE with respect to the classifier's output probability can be derived as:
aLMAE
1, j + y
(14)
Op(ijxx)
-1 j y-
5

---

(a) CE
(b) MAE
(c) NCE+RCE
(d) ANL-CE Lw/L2) (c) ANL-CE (w/ LI)
(1) CE
(g) MAE
(h) NCE+RCE
(1) ANL-CE (w/ L2) ( ANL-CE (w/ LI)
Figure 2: Training and test accuracies of different loss functions. (a) (c): CIFAR-10 under 0.8 symmetric noise. (f) (): CIFAR- 100 under 0.6 symmetrie noise. The accuracies of noisy samples in training set should as low as possible, since they are mislabeled.
The gradient of the NNCE with respect to the classifier's output probability can be derived as:
A- log p(ale)
dLNNEE 0p(j(z)
púlm)
x
A+ Atggp(k))) A+logp(Nlz)
(15)
Plylæ)
EE A+logp(k|as)
For the purpose of analysis, we consider how the gradients of NNCE would differ from MAE in the following scenarios: 1) given the classifier's output probability of sample æ, analyze the difference in gradient for each class, 2) given the classifier's 's output probabilities of sample æ and 22, analyze the difference in gradient between these two samples.
Theorem 2. Given the classifier's 's output probability p(-(2) for sample and normalized negative cross entropy LNNCE. p(jilz) p(j2(z), * j2 + y. then aEwer
0Csyr
UpU: [2)
"p(12 ø)
Theorem 3. Given the classifier's output probabilities p( (1-)aa)) and p()()))) of sample æı and
where ply 2x)) 2 p(k ax1), p(y 2 p(k))2)), Vk E (1..... K}. p(j 21) p(j]z)) j # y, and normalized negative cross entropy LNNCE- p(y(() p x2)) and ((k(z))) (kk)22)) Ak {1,
K) Mi. v). then op())m))
&.ggCc BENNE
Opb|=s)
theorem 2 and theorem demonstrate that, for the gradient of the non-labeled classes, our NNCE focuses more on the classes and samples that have been well learned compared to MAE, which treats every class and sample equally This property may enhance the model's performance noisy label learning. Some studies [1] have shown that during the training process, DNNs would first memorize clean samples and then noisy samples. According to the property revealed by theorem P and theorem 3 apart from robustness, our NNLF may potentially help the model to continuously learn the clean samples that the model has memorized the previous stages of training and ignore the unmemorized noisy samples.
4 Experiments
In this section, we empirically investigate our proposed ANL functions on benchmark datasets, including MNIST 12], CIFAR-1OCCIFAR-100 13 and a real-world noisy dataset Visiin [14].
4.1 Empirical Understandings
In this subsection, explore some properties of our proposed loss functions. Unless otherwise specified, all detailed experimental settings are the same as those section 4.2 More experiments
and discussions about gradient, parameter analysis, and Pmin can be found in the appendix D.]

---

(a) w/o regularizar
(b) L2
(c) Jacobian
(d) L1
Figure 3: Test accuracies of ANL-CE on CIFAR-IO under 0.8 symmetric noise with different
regularization methods and different parameters. 8 is the weight of regularization term of ANL-CE.
Table 1: Test accuracies (%) of different methods on CIFAR-10 10 datasets with clean and symmetric (17 E {0.2, 0.4, 0.6, 0.8)) label noise. The top-1 best results are in bold.
Methods
Clean (17 = 0.D)
= 0.2
= 0.4
1 = 0.6
1 = 0.8
NCE
88,68
81.65
74.80
63.14
37.52
NNCE
91.51
90.09
86,91
82.16
57.06
ANL-CE
91.66:+0.04
90.02=1..33
87.28-10002
81.12:10.30
61.27::0.55
Overfitting problem. In practice, we find that ANL can lead to overfitting in some experimental settings. To motivate this problem, as an example, we train networks using different loss functions on CIFAR-10 under 0.8 symmetric noise and CIFAR-100 under 0.6 symmetric noise, and the experimental results are shown in fig. 2 As can be observed, in the setting of CIFAR-10 under 0.8 symmetric noise, the training set accuracy of ANL-CCE (w/ I.2) keeps increasing while the test set accuracy keeps decreasing. We identify this problem as an overfitting problem.
It is worth noting that although overfitting occurs, unlike CE, the gap between the clean sample accuracies and the noisy sample accuracies of the training set does not shrink, which indicates that our ANL has some robustness to noisy labels even in the case of overfitting, Moreover, we conjecture that the overfitting caused by the property of NNLF focusing more on well-learned samples. When the noise rate is high, one might assume that the model has been trained on only a fairly small number of clean samples, with a very large gradient, which can lead overfitting.
The choice of regularization method. We also find that the commonly used 1.2 regularization may struggle mitigate the overfitting problem. To address this, decide try using other regularization methods. We consider 2 other regularization methods: L.1 and Jacobian [15] 16). To compare the performance of these methods, we apply them to ANL-CE for training on CIFAR-I0 under 0.8 symmetric noise. For simplicity and fair comparison, we use ó as the coefficient of the regularization term and consider in section 4.2 tune 5 for all three methods by following the parameter tuning setting in appendix D.2 We also train networks without using any regularization method. The results reported in fig. 3 As can be observed, among the three regularization methods, only LI can somewhat better mitigate the overfitting problem. If not otherwise specified, all ANLs in this paper use regularization.
an additional parameter of ANL. We keep a and / the same as
Robustness and fitting ability. We conduct a set of experiments on CIFAR- 10/-100 to verify the robustness and fitting ability of our proposed loss functions. We set the noise type to be symmetric and the noise rate to 0.8 for CIFAR-10 and 0.6 for CIFAR-100. In each setting, we train the network using different loss functions, including: 1) CE, 2) MAE, 3) NCE+RCE, 4) ANL-CE (w/ L2), and 5) ANL-CE (w/ LI). For ANL-CE (w/ L2), we set its parameters a and / to be the same as ANL-CE (w/ L1) and set its weight decay be the same as NCE+RCE.
As can be observed in fig. 2 1) CE is not robust to noise, the accuracies of clean and noisy samples in the training set continuuusly close to each other, 2) MAE is robust to noise, the accuracies of clean and noisy samples in the training set keep moving away from each other, but its fitting ability is insufficient, especially when the dataset becomes complex, NCE+RCE is robust to noise and has better fitting ability compared to MAE, 4) ANL-CE (w/ L2) is robust to noise and has stronger fitting ability, but suffers from over-fitting and 5) ANL-CE is robust to noise and mitigates the impact of overfitting to achieve the best performance. To summarize, our proposed loss functions are robust to noise, NNLF shows better fitting ability than MAE, and LI regularization addresses the overfitting problem of NNLF.
7

---

Table 2: Test accuracies (%) of different methods benchmark datasets with clean, symmetric (1 E {0.4, 0.6, 0.8)) or asymmetric (1) E {0.2, 0.3, 0.4)) label noise. The results std) are reported over 3 random runs under different random seeds 21, The top-2 best results are in bold.
Datasets
Mothods
Clean (rp-D.d))
0.4
Symmeriic Naise Rate (r))
0,6
0.8
0.2
Auy mentrric Noise Rote (?)
0.3
99 20.20+00002
74,40=0..25
0.05
88.00+00.077
MAR
99. 0.220003
98.
0..0
97.
97.69=0.20
70. 31
00.3
98.424-0.02
87.40:24011
GUE
99.18:40.01
96.81
80.86=0.31
53.55 53.59:200488
96 96.59-40007
88, 88.99+00.27
81.91 0155
SCE
99.304-0.07
97,48-0.15
58.35-02 0.77
48.28 0...8110.811
97.95 1-0.3
96,00-10.41
44114-0.11
MNIST
NLNL
0161-40..1
07.17=009
05,42=0.90
96.34-1.45 .4
98. 3000
97.51+0.155
95.84
RCE 1
99.3-+00022
18.55=00.09
25.61=0.12
74.04-11.85
98.79-20.10
95. 164:0.00
91.36:10.22
NCE-AGCE
17
99.10:4-.03
98,91 0.04
98.50 0.07
96.93-1-.13
99.0441-000 1-002
98.94:-0003
98.41 0.04
ANL-CE
00:00-2000
96.6220112
004
V8.01
ANL-FL
99.13440.05
98.90-0.05
95.73--0222
98.93+00002
98.18-00.01
CE
90.38:00111
58,19 0.221
18.75:0.19
09:09
8300:10333
78.1510.37
0..09
MAE
19.15:1:027
81.76133.1
76.82-0.84
46.42:1.66 66
67,35-1341
$7.36
OCE
00.661:0000
82.44
68 6.36
25. 51
19.30:10.52
0.
SCE
91.38:1:0.12
19,96:0.22
62.16-0.33
27.98 9.9.
26.22.00.44
80.20:1:.000
74.01 0.52
CIFAR-10
NLNE 1T0
00:73:1:0200
0.44
30.68-0.477
20.53 50.53:111 1.55
84 81.74:1:008
81 81.2661:::33
76. 06.97.10.35 07.10..52
NCEARCE 1
90:94:1:001
86 0.133
79.8
55 25.57722.74
88. 36::06000
84.84 84.10
77,75 0.37
NCE-AGCE
91.08
55.62-1-4778
1:009
84.79100.15
ANL-CE
91 06.0600004
87.28
0.02
81.12 0.30
61.27:10.55
89.3110.11
07.65220331
ANL-FL
91.79+-0.19
87.25 - 0.11
81.67 0.19
61.22 0.88
89.09-10.11
05.81+-0225
27.23+0.31
71.14:0.38
40.72=0.72
22.9K-0.07
0:
1.00
30.3093.119
41.55=0.34
1.31
3.33-0.35
1.35
6.194-0.2
5,82-0096
0.96-00.5
OCF
61 0.44
56 0..55
46.27_1.300
19:51 00:11:088
59 00
$3.88220.66
41.51 0.552
SCE
70 50.50=20..57
39.8 09.8
21.97=0:92
7.87-20.48
57 57.78:4 0333
50.15-0.112
41.35=0086
CIFAR-100
68.73:1:060
16.60=0.00
50 19:19:055
42.811:1:13
0.:10
NCEARCE
68:22:1:0.88
07397=0.30
46.26-1.07
62.77::0.33
$5.62-0.56
42.46=0042
NCE-AGGEE 1R
08.11:00012
59.74-0068
47.96-0.44
24. 134-0-0007
64.05:02225
56, (6.699
44.90-00622
ANL-CE ANI-FI
61.80-0500
5152 0.53
25007
66.77:0019
59,7610.444
45,41
10
61.78-0.88
0.34
27.97:10.58
66.26-10.44
59.68:00.66
0..65
Table 3: Top-1 I validation accuracies(%) on clean ILSVRC12 and WebVision validation set of
ResNet-50 models trained on WisVision using different methods. The top-2 best results are in bold.
Methods
CE
GCE [4]
SCE [S]
NCE+RCE [6
NCE+AGCE 17
ANL-CE ANL-FL
ILSVRCI2 Val
58.64
56.56
62.60
62.40
60.76
65.00
65.56
WebVision Val
61.20
59.44
68.00
64.92
63.92
67.44
68.32
Active and passive parts separately. In table I we show the results of the active and passive parts separately. We separately train NCE and NNCE on CIFAR-10 with different symmetric noise rates while maintaining the same parameters as ANL-CE. Specifically, for a NCE, we set a to 5.0 and for 3 NNCE, we set 3 to 5.0, while is set to 5 x 10 for both. As indicated in the results, the test set accuracies of NNCE are very close to those of ANL-CE. except in the case with 0.8 noise rate. This suggests that NNCE performs well on its own at low noise rates. However, at very high noise rates, a combination of active losses is needed to achieve better performance.
4.2 Evaluation on Benchmark Datasets
Baselines. We consider several state-of of-the-art methods: 1) Generalized Cross Entropy (GCE) 4: 2) Symmetric Cross Entropy (SCE) [5): 3) Negative Learning for Noisy Labels (NLNL) I0: 4) Active Passive Loss (APL) 66, including NCE+MAE, NCE+RCE, and NFL+RCE; 5) Asymmetric Loss Functions (AFLs) 177. including NCE+ NCE+AGCE, and NCE+AUL. For our proposed ANL, we consider two loss functions: 1) ANL-CE and 2) ANL-FL. Additionally, we train networks using Cross Entropy (CE), Focal Loss (FL) 18] and Mean Absolute Error (MAE).
Experimental Details. The full experimental results and the detailed settings of noise generation, networks, training and parameters can be found in the appendix D.2
Results. The main experimental results under symmetric and asymmetric label noise are reported in table 2 For more experimental results, please see appendix D.2 As can be observed, our ANL- CE and ANL-FL show significant improvement for most label noise settings of CIFAR-100-100 especially when the data become more complex and the noise rate becomes larger. For example, on CIFAR-10 under 0.8 symmetric noise, our ANL-CE outperform the state-of-the-art method (55.62% of NCE+AGCE) by more than 5.0%. Overall, the experimental results demonstrate that our ANL can show outstanding performance on different datasets, noise types, and noise rates, which validates the effectiveness of our proposed NNLFs and ANL.
8

---

4.3 Evaluation on Real-world Noisy Labels
Here, we evaluate our proposed ANL methods on large-scale wolld noisy dataset Web Vision [14], which contains more than 2.4 million web images crawled from the internet by using queries generated from the 1,000 class labels of the ILSVRC 2012 [ benchmark. Here, we follow the "Mini" setting in [9] and only take the first 50 classes of the Google resized image subset. We evaluate the trained networks on the same 50 classes of both the ILSVRC 2012 validation set and the WebVision validation set, and these can be considered as clean validation sets. We compare our ANL-CE and ANL.-LL with GCE, SCE, NCE+RCE, and NCE+AGCE. The experimental details can be found in the appendix D.3 The results are reported in table As can be observed, our proposed methods outperform the existing loss functions. This verifies that our proposed ANL framework can help the trained model against real-world label noise.
Moreover, addition to Web Vision, further validate the effectiveness of our method on real-world noisy datasets, we also conduct set of experiments on CIFAR- 1-N/-10ONN [20], Animal-10N 22] and Clothing-IM [22]. The experimental details and results can found in the appendix DD55 which
demonstrate the effectiveness of our method on different real-worke noisy datasets.
5 Limitations
We believe that the main limitation of our approach lies in the choice of regularization method. Although we have experimentally verified that LI is the most efficient among the three regularization methods (L1, L.2, and Jacobian), we lack further theoretical analysis of it. Furthermore, although only consider relatively simple regularization methods for the sake of fair comparison, other regularization methods, such as dropout [23] or regmixup (24), might be more effective in mitigating the overfitting problem caused by NNLF And believe that better solution to overfitting can further improve the performance of our method.
Related Work
In recent years, some robust loss-based methods have been proposed for robust learning with noisy labels. Here, we briefly review the relevant approaches. Ghosh et al. [3] theoretically proved that symmetric loss functions, such as MAE, are robust to label noise. Zhang and Sabuncu 4 proposed Generalized Cross Entropy (GCE), a generalization of CE and MAE. Wang et al. 5 suggested combination of CE and scaled MAE, and proposed Symmetric Cross Entropy (SCE). Menon et al. 25 proposed composite loss-based gradient clipping and applied it to CE to obtain PHuber-CE. Ma et al. 6 proposed Active Passive Loss (APL) to create fully robust loss functions. Feng et al. 266 applied the Taylor series to derive alternative representation of CE and proposed Taylor- CE accordingly. Zhou et al. [17] proposed Asymmetric Loss Functions (ALFs) to overcome the symmetric condition. Inspired by complementary label learning, NLNL [10] and JNPL [77 use complementary labels to reduce the risk of providing the wrong information to the model.
7 Conclusion
In this paper, propose new class of theoretically robust passive loss functions different from MAE, which we refer to as Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with NNLF, we propose Active Negative Loss (ANL), robust loss function framework with stronger fitting ability. We theoretically demonstrate that our NNLFs and ANLs are robust to noisy labels and also highlight the property that NNLFs focus more on well-learned samples. We found in our experiments that NNLFs have a potential overfitting problem, and we suggest using LI regularization to mitigate it. Experimental results verified that our ANL can outperform the state-of-the of-the-art methods on benchmark datasets.

---

Acknowledgements
This work is supported in part by Shanghai Science and Technology Committee under grant No. 21511100608 and No. 22511 106005.
References
[1] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 248 255, 2009.
[2] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. Proceedings of the Sth International Conference Learning Representations, 2017.
[3] Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust Loss Functions under Label Noise for Deep Neural Networks, Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 1919-1925, 2017.
[4] Zhilu Zhang and Mert R. Sabuncu, Generalized Cross Entropy Loss Training Deep Neural Networks with Noisy Labels. Advances in Neural Information Processing Systems 31, pages 8792-8802, 2018.
[5] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric Cross Entropy for Robust Learning with Noisy Labels. Proceedings of the 2019 IEEE International Conference on Computer Vision, pages 322-330, 2019.
[6] Xingjun Ma, Hanxun Huang. Yisen Wang, Simone Romano, Sarah M. Erfani, and James Bailey. Nor- malized Loss Functions for Deep Learning with Noisy Labels. Proceedings of the 37th International Conference on Machine Learning, pages 6543-6553, 2020.
[7] Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from Complementary Labels. Advances in Neural Information Processing Systems 30, pages 5639-5649, 2017.
[8] Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with Biased Complementary Labels. 15th Proceedings of the European Conference Computer Vision, pages 69-85, 2018
[9] Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with Noisy Labels. Advances in Neural Information Processing Systems 26, pages 1196-1204, 2013.
[10] Youngdong Kim, Junho Yim, Juscung Yun, and Junmo Kim. NLNL: Negative Learning for Noisy Labels. Proceedings of the 2019 IEEE International Conference on Computer Vision, pages 101-110, 2019.
[11] Mingehen Li, Mahdi Soltamolkotahi, and Samet Oymak. Gradient descent with early stopping is prov- ably robust to label noise for overparameterized neural networks. International conference on artificial intelligence and statistics, pages 4313-4324, 2020.
[12] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) (6(1)):277-2233 1998.
[13] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
[14] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision Database: Visual Learning and Understanding from Web Data. CoRR, abs/1 008.728626323 2017, URL http:tttt//iiv.oooooot org/ abs/ 1708 02862
[15] Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust Large Margin Deep Neural Networks. IEFE Trans. Signal Process., 65(1 42:455442880 2017.
[16] Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust Learning with Jacobian Regularization. CoRR, Abs/1908.02729 2019. htthtttttp///rae org/ abs// /aabb////9.... 1ww..998.000 .07299
[17] Xiong Zhou, Xianming Liu, Junjun Jiang, Xin Gao, and Xiangyang Ji. Asymmetric Loss Functions for Learning with Noisy Labels. Proceedings of the 38th International Conference on Machine Learning. pages 12846-12856, 2021.
[18] Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. Proceedings of the 2017 IEEE International Conference on Compuier Vision, pages 2999-30090 2017.
10

---

[19] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks Corrupted Labels. Proceedings of the 35th International Conference on Machine Learning, pages 2309-2318, 2018.
[20] Jiaheng Wei, Zhaowei Zhu, Hao Cheng. Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. Proceesdings of the 10th International Conference on Learning Representations. 2022.
[21] Hwanjun Song, Minseok Kim, and Jae-Gil Gil Lee. SELFIE: Refurbishing Unclean Samples for Robust Deep Learning. Proceedings of the 36th International Conference on Machine Learning, 2019.
[22] Tong Xiao, Tian Xia, Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, pages 2691-2699 2015.
[23] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56); 1929-1958, 2014.
[24] Francesco Pinto, Harry Yang. Ser Nam Lim, Philip H. S. Torr, and Puneet K. Dokania. Using mixup as regularizer surprisingly improve accuracy & cut-of-distributinn robustness. Advances in Neural Information Processing Systems 35, 2022.
[25] Aditya Krishna Menon, Ankit Singh Rawat, Sanjiv Kumar, and Sashank Reddi. Can gradient clipping mitigate label noise? Proceedings of the 8th International Conference on Learning Representations, 2020.
[26] Lei Feng. Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can entropy loss be robust label noise? 29th International Joint Conference on Artificial Intelligence. 2020.
[27] Youngdong Kim, Juscung Yun, Hyounguk Shon, and Jummo Kim. Joint Negative and Positive Learning Noisy Labels. Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, pages 9442 9451, 2021.
[28] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making Deep Neural Networks Robust to Label Noise; A Loss Correction Approach. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 2233-2241, 2017.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEF Conference on Computer Vision and Pattern Recognition, pages 770-778,
2016.
[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference Learning Representations, 2015.
[31] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. Proceedings of the 2018 IEEE Conference Computer Vision and Pattern Recognition, pages 5552-5560, 2018.

---

A Loss functions
A.1 NMAE, RCE, and NRCE are scaled versions of MAE
Here, inspired by S 6 we show how to reformulate NMAF, RCE, and NRCE as scaled versions of MAE. First, for convenience, we reformulate the Mean Absolute Error (MAB) follows:
K
MAE
Σ
k=1
(1 -(ulke)) + Σ (kk(a)
(16)
2(1 p(yl=)))
where the last equality holds due to Σx (+ = 1. For Normalized Mean Absolute Error (NMAE), we have:
Σ (p( (k(22)
NMAE = ΣØ= Ek=1 [p(k((=)
q(v = j|
2(1 (()
2Ø=1 j= 2(1 p x))
(17)
K (1 p(y)x))
1
2(K 1) MAF.
This shows that the NMAE a scaled version of the MAE with a factor of -(K-1)
For Reverse Cross Entropy (RCE), we have:
K
RCE Σ [- (k())))
p(yl log 1) Σ pkks)z)(-
(18)
pp(k(z))
key
-AA(1 p(y(xx))
A
MAE,
recall that in RCE we set log A, where A < 0 is some constant (e.g., A = -4). This shows that the RCE is a scaled version of the MAE with a factor of -
For Normalized Reverse Cross Entropy (NRCE), we have:
ΣK km
(kk(z))
NRCE
K ンラ=
Σk K
k- p(k|=)( log q(v j|x))
A(1 p(y)x=)
K
(-A(1 p(i(x)))
(19)
K p(p(l))
2(K 11 .MAE.
This shows that NRCE is scaled version of the MAE with factor of -パ-1J
12

---

A.2 Normalized Negative Loss Functions
Here, we show how to derive Normalized Negative Loss Functions (NNLFs) into its proper form.
(1 - (1
NNLF-
Σ Σ - -
Ekyy A - k()
2j=1 Ekzj A (z)())
K
Enty A (z)
(20)
(K 1) Σ A - (z)(2)) k)
K - (1
A A-f(f(x).u) v)
A A
A
A ((f(x)),1))
Σ A - (z)
B
Noise tolerant
Theorem 1. Normalized negative loss function Lun is symmetric.
Proof. For all æ E X and all f, we have:
K
A -(f(x).)) R))
Lml(f()), =
Σ
Σ)=1 A ((f(z),))
k=1
Σ K
k=
A ((f(x)),
(21)
K
Σ A ((f(x))))
K
K
where K 1 is a constant and L. satisfies the definition of the symmetric loss function.
C
Gradient analysis
C.1 Gradient of MAE
The complete derivation of the Mean Absolute Error (MAB) with respect to the classifier's output probabilities as follows:
In the case of j +
dLMAE Op(ilx)
a
K
pkkk2z) g(k(z)
=
(p(jlx)
k=1
1
(22)
((1 p(y)=))) E((k()))
Op(jiz)
kty
=1 1.
In the case of j = y::
aLMAE
2 Øp(j(x) 1
1p(k/2=) q(k] x))
Øp(j(x)
k=1
(23)
((1 p(u)=)) Σ p(k])))
Op(j)x)
=11-
13

---

C.2 Gradient of NNCE
The complete derivation of our proposed Normalized Negative Cross Entropy (NNCE) with respect
to the classifier's s output probabilities is as follows:
In the case of j * y:
NN ap(j)a))
A (- log plgl(y))))
0p(j)z)
Σ k=
log log (k)))))))
A log ((y)
Öp(ilz)
Σk=1 A log (Aklaa))
K
(24)
0 (A log p(y) ar) p(jaa
(E A log p 22))
A + log p xx)
p(j a:)
(E K
A + log (())))
In the case of j
ÔLNNCE
0 Op(j)z) (1
A - (- log ((yl(())))))
0p(j(z)
=
Σk=
K
(- log log p(k))z))
θ
A log p(yl yx)
Öp(j(z)
2k=1 k=1 A + log p (k()z)))
K
(25)
A + log p k: x)) (A + log p(y(z)) p(
p(y
< + lkg
Enzy A + pk(z))
p(y
(EK A+ log p(k))))))))
C.3 Properties of NNCE
Theorem 2. Given the classifier's output probability p(-[x) for sample x and normalized negutive cross entropy LNNCE. If p(jl()) < p(jal=), j1 # j2 # y, then BLacr
0p (0) 0p(jz|m)
Proof. As per the assumption, we have:
p(i|=) p()2(z))
1
p(1 (x)
p(12(z)
A + p(yl
1
+ log p(y|z) x))
(26)
p(j1 x)
A + log pkkl))))
p(12(z)
|Ek> A + log logp(k)z)))
1=
ÔCNNCE
dL
Op(iila) (z)
Ap(āzla)
This completes the proof.
Theorem 3. Given the classifier's output probabilities p(- (11) and p(- x22) of sample æi and
I2. where p(y|æ1) 2 p(k[21)) p(y(222) 2 p(kk (2)))2)) Vk E f1..... K). p(j(x,) = p(j] and normalized negative cross entropy LNNCE- If p(y p(y) + y. and p(k)z,) S p(k(x)),
yk E (1, K) Mi.v), then dLanx 0C
op(j(æ1) @p((=2]
Proof. We first consider two functions f1 and fz. The function fi is defined as follows:
(1)
Σ log Pk.
(27)
14

---

where the input p is discrete probability distribution, Σx= Ps = 1 and 0 S Ps 1 Ak E {1, K.. Given discrete probability distribution p and real number D as input, the function fz is defined as follows:
f2(p, D) = log(p, + D) logp; +
Σ kデjデy
log(Ps - -dk),
(28)
where Py 2 Pk. Wk E (1,... K). 0 S Py. And {dk), k E (1, KK \ (i.y) is a set of random variables which satisfy following conditions: 0 - Pk= d2 I Pk: and Σk dr D.
Next, we consider whether (1()) - f2(p, D) > O. Given p and D, we have,
fi(p) - D) = Σ
log pg) - (Log(p, + D) + log +
Σ
log(Px - dz))
#=1
b
log Py log(p, D) + Σ
logp:
log(P; dx)
Py + Py + D
log Pt:
= log
Pk
d2
kdity
2 log
Py +
Σ
log
Py
Py + D
Py da
= logPy Py - log Py + D) +
Σ
logp, log(p, - -dx)
logPy py
Σ logpy)
(oog(p, + +D) Σ
log(p, - dx))
log Py +
logpy)
(log(p, + D) +
log(p, - dzz)
sup (
(29)
The first inequality holds because for a fraction f 2 1, we always have 8 2 6 where e 2 0.
To get the maximum value of log(p, D) E EnajAy log(p, de), we must solve the following minimization problem subject to constraints:
(LOB(P, + DD+ min (d. Aoriv
Σ
log(p, - de)).
(30)
kボjデy
s.t.
Σ Kデjテy
dk D.
(31)
We can define the Lagrange function as follows:
Ldd, dg, A) - [oo((, + D)
Σ kFjfy
log(p, + +x)) +x.
+x.E dx dx D).
Σ
(32)
Now we can calculate the gradients:
aL Ddl
+
Pv tue
(33)
BL
Σ Ektsto dx D.
0X
Let 0L_ ad,
and 0L 0x
0, we have:
py td +
(34)
Nzjty de -
and therefore:
dj D
k kナ 2
(35)
15

---

So, the minimization value is:
-(1og(P, +D+
Σ
log(p,
D K 2
(36)
Now, back to the eq. (29 we have:
fi(p) - 12(p, D) 2 log +
Σ
log Py
sup (de),kestjty
[log(p, + D) +
Σ
log(P, - dx)
D
Po
+
log Ps (log(py + D) + Σ
log(Py
K 5-22 2
logp, - log(p, ,, + D) + (K -2) (108P,, log(P,,
2 -22)
D
K
K -----) 222) (oog(p, + D) - log logPy)
D
(K 2) log log(p,
log logpy log(P, D
log Py + D)
log Py
K-2
D
K-2
(37)
Recall the nature of the difference, we have
log P. log(p, R
d log di
where
R-
x=
Pg
D 0 S a S K-2 and log(p, + +DD)-log logg,
log
where 0 0S3S S D. And therefore:
D
P tB
log Py log(p, x²22
1
1
log(p, + +D) - log - Pv
>
(38)
K-2
Py
Py +
So we have:
logp, log(p,
log(p, + D) loo py
K-2
D
(39)
K-2
Si(p) > f2(p, D).
Now, let Pa = Ak (LE K}. D = p(y)(21) - p(y)x22, and du = p(k)22) x1).
A * j # Following eq. 33
we have:
Si(p) > f2(p, D)
K
log log(p, D) + logp, +
Σ kズ
log(Ps da)
K
log Sogp(klan
log p(k()r)))
A=
(40)
>> + log logp(k)z)))
Dk= A+ + log p(ki (x2))
1
+ log x
A ++ log p y (x22)
p(j(z))
(Ek=] + log p xx))
p(j(0z))
(EK, A+ log p(k))))))))
Ô.
OLNNCE
0p(jl x1)
Ap(j(zz)
The eq. (40) holds because we have x()) p(j(az)), and log log p(ylær)) logl(y)(x2). where
p x1) p))) This completes the proof.
16

---

10a
Gradient
10a
Clean samples
107
Noisy so mamlee
10a
10a
100
110
120
Epochs
(a) CIFAR-10 1- under 0.6 symmetric noise
10a
10a
10a
Gradlent
101
samples
10a
Noley sa samples
10:
10a
10
20
30
40
50
60
70
80
90
100
110
120
Epochs
(b) CIFAR-10 under symmetric noise
Figure 4: The gradient maxj.ivv ppl as)
C for clean and noisy samples. Blue boxes indicate clean
samples, red boxes indicate noisy samples, blue triangles indicate the means of clean samples, and red triangles indicate the means of noisy samples. The outliers ignored.
D Experiments
D.1 Empirical Understandings
Gradients of NNCE. We conduct experiments with ANL-CE on CIFAR-I0 under 0.6 and 0.8 symmetric noise. For each sample, we calculate the gradient our NNCE with respect the predicted probability of each non-labeled class and take their maximum value, which is maxj,jvy (pp() a+)
(0CNNr
The results are shown in fig. in the form of box plots. can observed, as the number training epochs increases, the gradients of the clean samples are concentrated at larger values and the gradients of the noisy samples are concentrated smaller values. The means of the gradients for noisy samples are higher than the medians and smaller than the means for clean samples' gradients. This indicates that although the model still incorrectly learning some noisy samples, these learned noisy samples constitute only small fraction of all noisy samples in the training set. In general, this result verifies that our NNLF's have the ability to focus more on clean samples and ignore noisy samples during training process, and also verifies discussion in section 3.3
Parameter Analysis. We apply different values of a and 3 to NCE+NNCE for training on CIFAR- under 0.8 symmetric noise. We test the combinations between a E {0.1, 0.5. 1.0, 1.0,5.0. 10.0) and E (0.1, 0.5, 1.0, 5.0, 10.0), where a the weight of NCE and the weight of NNCE. The weight decay of NCE+NNCE is set to be the same as NCE+RCE. As can be observed in fig 5 regardless of how a (the weight of NCE) varies, when 3 (the weight NNCE) increases, both the robustness and the fitting ability of the model improve, although overfitting occurs. This verifies that our NNLFs are robust to noisy labels and have good fitting ability. Although in ANL we use LI regularization instead of L.2 regularization (weight decay), the effect of and / in the loss functions created by ANL should be similar to that in NCE+NNCE.
17

---

shece Nante
0.1, B 0.1 (b)
0.1,
(c)
0.1,8 1.0 (d)
0.1,8 5.0
0.1,
10.0
0.5,
0.5,
(h)
0.5,
0.5,
1.0, 8 0.1
1.0, 8 0.5
(m) a
1.0,
1.0
(n)
a
1.0,
5.0
(0)
0 1.0, 10.0
(p)
5.0, 0 0.1 (4)
5.0,8 0.5
(1)
5.0,2 9 1.0
(8) 01 5.0,
5.0
(1)
5.0, 3 10.0
(1)
10.0, (V) 10.0, 0.5 (w) 10.0, 1.0 (x) 10.0, 5.0 (y) 10.0, 10.0
Figure Training and test accuracies NCE+NNCE on CIFAR-10 under 0.8 symmetric noise with different parameters. is the weight of NCE and / is the weight of NNCE. The accuracies of noisy samples in training set (red dashed line) should as low as possible, since they are mislabeled.
2.5
0.0
3.5
3.5
ANI-GE
ANI-EE
8-18-7
5=50-7
0-30-7
5.2
02-
0.2
5.7
20
40
60
80
100
120
60
50
100
120
20
40
60
80
100
120
Epoc1s
Epocht
Spocns
(a) L2
(b)
(c) Jacobian
Figure 6: Test accuracies of NCE+RCE 10 under 0.8 symmetric noise with different regularization methods and different parameters. is the weight regularization term of NCE+RCE. As comparison, the blue line shows the test accuracy of our ANL-CE with the same data set and noise setting.
18

---

0.9
0.6
0.7
Ance
Pa
0.5
Por
0.2
Rgu 1
Pore le
Pape le-
0.4
Por 1e-8
0.1
1+-B
le
120
300
200
Epocts
Esocts
(a) CIFAR-10
(b) CIFAR-100
Figure 7: Test accuracies of different Pmin applied to ANL-CE on CIFAR- 100 under 0.6 symmetric noise.
Can APL outperform our ANL by changing the regularization method? We apply different regularization methods to NCE+RCE for training on CIFAR-IO under 0.8 symmetric noise and compare the results with our ANL-CE. Specifically, we train networks using loss functions in the form of a NCC + 3- RCE + 8---RRe Reg, where a, 3. 6 > are parameters and Reg the regularization term. Similarly, we consider regularization methods: L2, 2) LI, and 3) Jacobian
[1]]6]] We
keep the a and of NCE+RCE the same as section 4.2 and tune E { x 10 x 10-7 1 x 10-6 5 10-6 x 10-5 5 x 10-5 1 x 10-13 for all three methods. As can be observed in fig. 6 the performance of NCE+RCE can be improved by adapting the regularization method. However, the improvement from changing the regularization method is limited, and the test accuracies of NCE+RCE are consistently lower than our ANL-CE, no matter how the regularization method is changed and how the parameter is varied. This verifies that APL cannot outperform our ANL by changing the regularization method and also verifies the performance of our proposed NNLFs.
Does the Pmin affect the performance of ANL? As shown in the definitions of NLF and NNLF, Pmin controls the lower bound of the classifier's output probability, and further controls the constant A. In fig. 7 we show different Pmin E x 10-5, 1 10-6, x 10-7, 1 10- x 10-9) applied ANI -CE for training on CIFAR- 100 under 0.6 symmetric noise. As can be observed, for both datasets, Pmin does not significant affect the performance of ANI.
D.2 Evaluation on Benchmark Datasets
Noise generation. The noisy labels are generated following standard approaches in previous works
28 6 17]. For symmetric noise, we flip the labels each class randomly to incorrect labels of other classes. For asymmetric noise, we flip the labels within specific set of classes. For MNIST, flipping 7 - 1, 2 + 7,5 ++ 6, 3 8. For CIFAR-10, 10, flipping TRUCK - AUTOMOBILE, BIRD + AIRPLANE, DEER HORSE, CAT < DOG. For CIFAR- 100, the 100 classes grouped into 20 super-classes with each has sub-classes, and each class are flipped within the same super-class into the next in a circular fashion. We vary the noise rate {0.2, 0.4, 0.6, 0.8) for symmetric noise and E 10 0.2, 0.3, 0.4} for asymmetric noise.
Networks and training. We follow the experimental setting in previous works 66] I7 A 4-layer CNN is used for MNIST, an 8-layer CNN is used for CIFAR- 10, and a ResNet-34 (29] is used for CIFAR-100. For MNIST, CIF and CIFAR-100 the networks are trained for 50, 120, and 200 epochs, respectively. For all the training. we use SGD optimizer with momentum 0.9 and cosine learning rate annealing. Weight decay is set to 1 x 10-3 1 x 10-1, and 1 x 10-5 for MNIST, CIFAR-10, and CIFAR-100, respectively. Particularly, for our proposed ANL methods, weight decay is set to 0 for all datasets. The initial learning rate is set to 0.01 for MNIST/CIFAR-10 and 0.1 for CIFAR- 100. Batch size set to 128. For all settings, we clip the gradient norm to 5.0. Typical data augmentations including random width/height shift and horizontal flip are applied.
Parameter tuning. For each dataset, we tune the parameters of ANL-CE under 0.8 symmetric noise and then use them directly for all other noise settings and ANL functions. Specifically, we use 10% of the original training set as the validation set, and generate 0.8 symmetric noise on the remaining 90% of the original training set as the training set by the standard noise generation approach. We tune the parameters a E (0.1, 0.5, 1.0, 5.0, 10.0), ). B E 10.1, 0.5, 1.0, 5.0, 10.0), and ă E { x 10-7,
5 x 10-7, 1 x 10 5 10 1 x 10-5 5 10 1 10
19

---

Table 4: Parameters settings for different methods.
Method
MNIST
CTFAR-10
CIFAR-100
WelVvvion
CLC)
()
[-]
HL(N)
(0.5)
(0:5)
(0.5)
MAB (-)
(-)
(-)
(-)
GCE(g) (g)
(0.7)
(0.7)
(0.7)
(0.7)
SCE
(10.0,
(100 10.0)
1.0)
(10.0,
NCE+RCE (ax, B)
(1.0, 10.0)
(L.0, 1.0)
(10.0, 0.1)
(50.0,00)
NFL+RCE (a.
(1.0, 1900 0.5)
(1.0, 0.5)
(10.0, o. 0.5)
NCE+AEL (..
(oc. 3.5)
(1.0,
(10.0. 1.5)
NCEAAGCE (,,
(1.0. 6,0
(50.0, 0.1, 3.0)
NCE+AUL (a, 9.0,p)
(u0, L0, 3.0, 0.1)
(1.0. 4.0, 6.3, 1.5)
110.0, 0.015, 6.0. 3.)
ANL-CE
(1.0, 10-9)
10-*)
(20.0 10-9)
ANI-FT. (ox, 9,
(1.0, LO, x 10 0.5)
(5.0.50,.5 5 10 0.5)
(10.0, 10,5 10 10-7,05 0.5
(20.0, 10.6 10 .05.)
Table 5: Test accuracies (%) of different methods benchmark datasets with symmetric (1) (0.2, 0.4, 0.6, 0.8)) or asymmetric (1) E {0.1, 0.2, 0.3, 0.4}) label noise. The results (mean of our methods are reported over random runs under different random seeds (1, 2, The reported results of PHuber-CE and Taylor-CE are directly taken from the original paper of Taylor-CE. The top-2 best results in bold.
Dutneets
Melhods
Symmetric Noise Kate (r)
0.4
Asymmetnc Neise Kate (T)
0.5
PHube-CC
85.81:00.11
20:5:::22
67.7120.19
52.97:10.33
07.91=0013
84-87:1:026
79.01=0.88
TaylarCCF 20
85.9620.09
8151
66.36220.32
33.48-2:044
87.34:0.122
85:02:0:111
79.37-0.12
7265420.1
CIFRR-10
ANL-CE (ound
87.43:00.24
83.811554
73.00::0.24
45.18:11.27
88.29
86.54100.7
83.12-0.56
75.0:10443
86.75-0.70
82.971146
45.52:12.24
0..42:0.16
86.36100.44
7.99511.21
PHuhesCC
58.11=0.11
5039:0013
35.8520.29
13
00:07:0:09
53 10
44.39-0.14
35.36:00113
CIFAR-100 100
ANL
TaylorCC 26
19.11=0.11
59.20
5059-0.099
53442
38.31-0.12
44.42.00099
26.27:00.11
15.96-00311
60.96:0.21
60.58-0.688
55 +-.12
58.11 1..61-1-.60
45.81
0.42
35.45-0025
3855710.72
(ows))
59.23-00117
53.9910.50
2639:0:622
61.51-0.33
58.65:0.111
50.27-0.54
39.14
Parameter settings. For all baseline methods, the parameters are set to match their original papers. Detailed parameter settings can be found in table
Results. The experimental results of symmetric and asymmetric label noise are reported table and table 7 respectively.
D.3 Evaluation Real-world Noisy Labels
We train a ResNet-50 (29 using SGD for 250 epochs with initial learning rate 0.4, nesterov momentum 0.9 and weight decay 3 x 10 (for our proposed ANL methods, the weight decay is set to 0) and batch size 512. The learning rate multiplied by 0.97 after every epoch of training. For all settings, clip the gradient norm 5.0. All the images resized to 224 224. Typical data augmentations including random width/height shift, color jittering and random horizontal flip are applied. Detailed parameter settings can be found in table
D.4 More comparisons with other baseline methods
We also compare our proposed ANL with two other baseline methods, PHuber-CL CE 25 and Taylor- CE [26]. For fair comparison, we followed the experimental setup in the Taylor- CE paper and compared it directly with its reported experimental results. We use Adam [30 optimizer with the number of epochs set to 200 and the batch size set to 256. We ResNet-34 for CIFAR- 10/- 100. We tune our parameters and select the learning rate Ir from {10 2, 10 10
10-4 10 53. For set the parameters (o, 3, Ir) (5.0. 5.0, 10-7 x 10-1) and (10.0, 1.0, 1 x 10-7, x 10-1), respectively. The results in table demonsttett that our proposed ANL can outperform both PHuber-CE and Taylor-CE on the CIFAR-10/-100 dataset with symmetric or asymmetric label noise.
CIFAR-10 and CIFAR-I00,
20

---

Table 6: Test accuracies (%) of different methods on benchmark datasets with clean or symmetric label noise (1) {0.2, 0.4, 0.6, 0.8)). The results (mean are reported random runs under different random seeds (1, 2, 3) and the top-2 best results are boldfaced.
Symmetric Noise Rate (r))
Datascts
Methods
Clean (1==0.0)
0.2
0.4
CE
99.20=#0.02
91,40-E0.28
74,46+0.228
49. 19.19+0.05
22.51=0.23
FL
99.16+0.008
91.66++0.18
75.42++0.25
50.58+0.38
22.93++0.11
MAE
99.16.10.03
99.03.10.01
98.80-10.02
97.69110.20
1.13
GCE
99.18.10.01
98.84110.03
96.81110113
80.86.10.31
33.59110.48
SCE
99.30-10.07
98.91110.04
97.48.10.16
88.35110.77
48.28.10.81
NLNL
98.61
98.02=#0.14
97.17+0.009
95,42:40.30
86.34+1.43
MNIST
NCE+MAE
99.22=00006
98.96=20.04
98.15=0.114
92.94=1..66
59.541:0.76
NCE+RCE
9..43+1002
99.20110.05
98.53+00.09
95.61#0.12
7..04
NFL+RCE
99.33-+0.04
99.17-10.03
98.62440.03
95.54+00.26
1..05
NCE+AEL
99.06+00.06
99.03++0.08
98.87+00.05
98.39+4.11
96.71+0.10
NCE+AGCE
99.10++0.00
99.00+0.05
98.91+00.04
98.50++0.07
96.93++0.13
NCE+AUL
99.1911.01
99.0411.005
99.00-10.03
98.52-10.14
96.9710.11
ANL-CE
99.08110055
98.9710.002
98.84
98.42.10.08
96.62:10.12
ANL-FL
99.1311005
98.94.10.07
98.90.1005
98.4610.12
95.73110.22
CE
00.3
70.05
18.0910.21
38.75+0.19
19.09
FL
89.84.10228
74.52.10.10
57.54.10.75
38.8310.49
19.3
MAE
89.1510.27
87.1910.11
81.76.1.117
76.82.10.84
46.42.13.66
GCE
89.66.10.20
87.1710.01
82.4410.26
68.62.10.35
25.45.1.551
SCE
91.38+0012
87.86+0.112
79.96+
62.16+0.33
07.98++0.8
NLNL
90.73+00.0
73.70
63.90
50.68+00447
1.55
CIFAR-10
NCE+MAE
88.94:10.13
87.37
83.70+0.21
76.35+00.08
44.68+1.12
NCE+RCE
90.94-10.01
89.19-1.118
86.03+-0.13
19.89+00111
55.52+2.74
NFLARCE
90.86+1.551
89.04
86.08+0..3
79.79-L0.16
54.18+42.06
NCE+AEL
88.51+00.6
86.59+0.24
83.07+0.46
75.06++0.26
41.79+1.40
NCE+AGCE
91.08100066
89.11=0.007
86.16110.10
80.14:10.27
55.6244.78
NCE+AUL
91.26110.1
89.0811.114
86.1111.227
79.39110441
54.4912.77
ANL-CE
91.66110.04
90.02.10.23
87.2810.02
81.1210.30
61.27:0.555
ANL-FI
91.79-+0.19
89.95-+.200
87.25+0.11
81.67+0.19
61.22+00.5
CE
71.1410.38
55.9711.111
40.72.10.74
22.98:10.07
7.55
71.02-10.36
55.9411.53
39,5511.24
23.21110.49
7.80.1.27
MAE
1.39
7.914
3.61+0.21
3.63+0.35
2.83+1.35
GCE
61.62+0..3
11.50
56.46+0.95
46.27+1..00
19.51+0.86
SCE
70.80
55.04
39.84+0.11
21.97+00.2
7.87+:0.4
NLNL
68.72-10.60
46.99
30.29.1..4
16.60
11.01
CIFAR-100
NCE+MAE
67.52-10.21
52.68-10.22
35.71110.47
19.44-0..15
7.08
NCE+RCE
68.22-10.28
64.20-10.47
57.9740.300
46.26-11.07
25.65
NFL+RCE
68.04
64.33-1.223
58.48+0.50
47.20110.58
26.26::0.19
NCE+AEL
64.98
48.13:1..2
32.11110.89
20.75+2.00
1.97
NCE+AGCE
68.6110.11
65.30110.21
59.7410.68
47.96-10.44
24.131.0.07
NCF+AUL.
69.91+4.188
65.26-H0.17
56.67+-0.1
39.98+0.42
13.30+H0.08
ANL--CE
70.68
66.79-H0.34
60.50
51.52+0.53
28.0740.28
ANL-FI
70.40++0.15
66.54-10.29
61.73-1-0.48
51.32++0.34
27.97+-0.58
21

---

Table 7: Test accuracies (%) of different methods on benchmark datasets with asymmetric label noise (n E (0.1, 0.2, 0.3, 0.4)). The results (mean-sstd) are reported over 3 random runs under different random seeds 21 and the top-2 best results boldfaced.
ASYMMETRIC NOISE RATE (7)
DATASETS
METHODS
0.1
0.2
0.3
0.4
CE
97.70=1.02
94.0210.18
88.90:10.07
81.79-10.34
FL
97.62=0.05
94.41+0.11
81.99+0.61 0..8
МАE
99.05=0.06
99.11=0.03
98.42=0.09
87.40=4.001
GCE
98.97=0.02
96.59:0.07
88.9910.27
81.91=0.58
SCE
99.05=0.02
97.95=0.23
94.00=t0.41
84.54+0.14
NLNL
98.63+0.06
98.35+0.01
97.51+00.15
95.8440.26
MNIST
NCE+MAE
99.12+0.09
98.33+0.14
95.16+0.21
85.79+0.47
NCE+RCE
99.31+0.06
98.79+0.10
95.16+00.08
91.36+0.22
NFL+RCE
99.27=0.02 98.7920.15
96.97=0.09
91.36110.0
NCE+AEL
99.06=0.05
99.0510.07
98.88+1.02
98.29:10227
NCE+AGCE
99.11=0.04 99.0410.02
98.9410.03
98.41=0.04
NCE+AUL
09.11=0.05
99.09=0.03
98.98:0.06
98.70=10.01
ANL-CE
99.10=0.05
99.0410.04
98.91:0.07
98.01=0.10
ANL-FL
99.00=0.05
99.05=0.09
98.9340.02
98.1840.01
CE
86.85=0.15
83.0010.33
78.15:0.17
73.6910.20
FL
86.3220.28
83.03=0.10
78.53:0.16
73.78:0011
MAE
88.22=0.05
79.6320.74
67.3543.41
57.3612.37
GCE
88.1920.21
85.5510.24
79.32+0.52
72.8340.17
SCE
89.57=0.11
86.22+0.44
80.20+0.20
74.01=0.52
NLNL
88.5410.25
84.74-0.08
81.26+0.43
76.97=0.52
CIFAR-10
NCE+MAE
88.22=0.25
86.1620.18
82.98-0.15
75.23+0.24
NCE+RCE
89.98-0.11
88.36110.13
84.84.0.16
77.75
NFL+RCE
90.11:0.06 88.2610.27
84.72-10.18
77.29+0..3
NCE+AEL
87.64=0.16 85.6410.07
81.95+0.38
74.55+10.42
NCE+AGCE
90.21=0.16 88.4810.09
84.79-0.15
78.60=10.41
NCE+AUL
90.1220.06 88.2910.15
84.8410.06
76.99=0.26
ANL-CE
90.90-0.13
89.1310.11
83.52.0.24
77.63
ANL-FL
90.81=0.20
89.09=0.31
85.81+0.23
77.73+0031
CE
65.1720.45
58.2511.00
50.3010.19
41.5310.34
FL
64.55-0.39
58.0011.38
50.7740.41
41.88
MAE
6.63+1.60
6.1940.42
5.8240.96
3.96=0.35
GCE
64.29=0.90
59.0620.46
53.88+0.96
41.51
SCE
64.6420.57
57.78-0.83
50.15+0.12
41.3310.86
NLNL
59.55#1.22
50.1920.56
42.81+1.13
35.1010.20
CIFAR-100
NCE+MAE
60.52+0.36
52.92+0.50
44.41+0.22
36.71+0.16
NCE+RCE
66.18=00.23
62.77+0.53
55.62+0.56
42.46+00.22
NFL+RCE
66.16:2.44 63.43+0.71
55.63+0.37
42.54+0.52
NCE+AEL
57.4710.47 50.4910.12
42.46.10.51
35.0440.29
NCE+AGCE
66.86-0.23 64.05.0025
56.36.10.99
44.90-10.62
NCE+AUL
66.2340.21 57.7910.40
47.64.1.24
38.65+0.30
ANL-CE
68.78-0.11 66.27
59.76:10.4
45.41
ANL-FL
68.36-0.15
66.26-0.44
59.68-0.886
46.65-0.04
22

---

D.5 Comparisons on more real-world datasets.
CIFAR-1ON--10ON 20 are CIFAR-10/-100 equipped with suman-annotateer real-world noisy labels.
CIFAR- -ON contains five noisy label sets with noise rates of 9.03% (Aggregate). 17.23% (Random 1). 18.12% (Random 2). 17.64% (Random 3) and 40.21% (Worst). CIFAR-10ON contains one noisy label set with noise rate of 40.20% (Noisy).
We the same experimental settings and parameters CIFAR-I0 and CIFAR-100 for CIFAR-10N and CIFAR- 1000, since the only difference is the noise label distribution. The results are reported in Table and
and Table
Animal-10N PI is real-world noisy data set of human-labeled online images for 10 confusing animals, with 50, 000 training and 5, 000 testing images, and its noise rate was estimated 8%.
We follow the experimental setting in previous works [21]. We use VGG-19 with batch normalization. The SGD optimizer is employed. We train the network for 100 epochs and use an imitial learning rate of 1, which is divided by 5 at 50% and 75% of the total number of epochs. Batch size is set 128. Typical data augmentations including random horizontal flip are applied.
We compare our ANL-CE with CE and GCE. We use 1.2 regularization (weight decay) for GCE, and LI regularization for ANL-CE. We denote the regularization coefficient by 6. We tune the parameters {8), {g, 8). {a, 3, 6) for CE, GCE and ANL-CE respectively. We the best parameters {1 x 10-3), ). {0.5. x 10 10-1], {0.5, 1.0, x 10-b) } for each method in our experiments. The results
are reported in Table IO
Moreover, we experiment with NCE+RCE on this dataset and tune the parameters {a, 8, 6] . but we find that the performance is very poor for some unknown reason. The best test accuracy achieve is 28.28% with {10.0, 0.1,5 x 10-63. Since this result is too low and inconsistent with its performance on other datasets, do not include it in the table for comparison.
Clothing- 1M [22 is a large-scale clothing dataset contains 14 categories and I million training samples with nearly 40% mislabeled samples.
We follow the experimental setting in previous works 31 We the 14k and 10k clean data for validation and test, respectively, and we do not use the 50% clean training data. We use ResNet-50 pre-trained on ImageNet. For preprocessing, we resized the images to 256 256, performed mean subtraction, and cropped the middle 224 x 224. We use SGD with a momentum of 0.9, a weight decay of 10-3, and batch size of 32. We train the network for 10 epochs with learning rate 1 x 10-3 and x 10-4 for 5 epochs each. Typical data augmentations including random horizontal flip are applied.
We compare our ANL-CE with CE, GCE and NCE+RCE. In this experiment, we use L2 regularization (weight decay) for ANL-CE and set the coefficient the same those for CE, GCE and NCE+RCE. We tune the parameters {g). 3 and {a, 3} for GCE, NCE+RCE and ANL-CE respectively. We use the best parameters {0.6). {10.0, 1.0) and (5.0, 0.1} for each method in our experiments. The results are reported in Table
23

---

Table 8: Test accuracies (%) of different methods on CIFAR-ION dataset. The results (mean=std) reported over 3 random runs under different random seeds 2,) and the top-2 best results boldfaced.
Methods CE GCE
Clean
Aggregate
Random
Random 2
Random
Worst
90.38
85.09:0.330
79.09=1.22
78.59=1..2
78.39=0.50
61.43:0.52
89.66=0.20 87.38=0.07
85.87=0.27
85,43=0.13
85.51=0.15 75.19=0.23
SCE
91.38+0.12
88.48+0026
85.65+0.30
85.71+0.19
85.87=0.13
73.65+00229
NCE+RCE
90.94+00001 89.17+0.28
87.62+0.34
8.166
87.70-0.18
79.74+0009
NCE+AGCE
91.08+00066 89.27=+0288
87.92+0.02
87.61=0.20
87.62=0.116
79.91+0.37
ANL-CE (ours)
91.66:10.04
89.66:1.112
88.6810.13
88.1910.08
88.24
80.23:10.28
Table Test accuracies (%) of different methods on CIFAR-IOON dataset. The results (mean=std) are reported over 3 random runs under different random seeds (1, 3) and the top-2 best results are boldfaced.
Methods CE
Clean
Noisy
71.14:0.38
48.63310.53
GCE
61.62:10.43
50.97:10.60
SCE
70.80:10.37
48.52:10.11
NCE+RCE
68.22:10.28
54.27=#0.09
NCE+AGCE
68.6110.12
55.96:10.20
ANL-CE (ours)
70.68+0.23
56.37:10.22
Table 10: Test accuracies (%) of different methods on Animal-10N dataset. The results (mean=sst)
are reported over 3 random runs under different random seeds (1, 2) and the top-1 best results are boldfaced.
Methods
CE
GCE
ANL-CE (ours)
Test Acc. (%)
80.39 0..7
80.72
Table 11: Test accuracies (%) of different methods on Clothing- 1M dataset. The top-1 1 best results are boldfaced.
Methods
CE
GCE
NCE+RCE
ANL-CE (ours)
Test Acc. (%)
68.07
68,94
69.07
69.93
24


Reviews:
[{"review_id": "016Gnqv2qs", "reviewer": "Reviewer_j1Cc", "summary": "This paper introduces a novel type of loss function called Active Negative Loss (ANL), which builds upon the Active Passive loss function (APL) framework. The authors identify a limitation in APL, where the passive loss function, being a scaled version of Mean Absolute Error (MAE), can lead to slower convergence and underfitting issues. To address this, the authors propose replacing the passive loss in APL with a normalized negative loss, drawing inspiration from negative learning and vertical flipping techniques. The paper also provides theoretical justifications for the superiority of ANL over APL. The proposed ANL framework is evaluated through experiments conducted on CIFAR10 and CIFAR100 datasets with synthetic label noise, as well as ILSVRC12 and WebVision datasets with real-world label noise.", "questions": "See **weaknesses**", "limitations": "I think this work does not bring negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clearly articulated and accompanied by well-explained reasoning.\n\n2. The proposed ANL framework can be considered an enhanced version of the APL framework, offering improved performance with theoretical backing. Specifically, it demonstrates robustness against both symmetric and asymmetric label noise under certain conditions.\n\n3. Code is provided to facilitate reproducibility.\n", "weaknesses": "1. Although the proposed loss is theoretically sound, many of the techniques or analyses employed are not very new. The ANL framework appears to be a simple combination of the NL loss [R1] and APL loss [R2]. While I acknowledge the validity of this combination, I do not believe it constitutes a significant contribution to the LNL community.\n\n2. The authors mention that regularization, such as L1 regularization, is applied to the ANL framework. I am curious to know if other approaches in your experiments also utilize this regularization. Additionally, the absence of ablation studies investigating the impact of regularization on the ANL framework is worth considering.\n\n3. I believe conducting further experiments to verify the effectiveness of the ANL framework is necessary. For instance, experiments involving CIFAR with controlled instance-dependent label noise, CIFAR-10N, CIFAR-100N, and the Clothing1m dataset would provide valuable insights.\n\n\n[R1] NLNL: Negative Learning for Noisy Labels\n\n[R2] Normalized Loss Functions for Deep Learning with Noisy Labels", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the suggestion from the reviewer and will incorporate it into the updated version of our paper.\n\n1. A simple combination of the NL and APL.\n\n   First of all, although our ANL may seem like a simple combination of existing techniques, we have a strong motivation for doing so: finding a passive loss function that performs better than MAE to enhance APL.\n\n   Secondly, our NLF and NL are still fundamentally different. Take NLF(CE) as an example:\n   $$\n   L_\\text{NLF}=\\sum_{k=1}^K (1-q(k|x))\\big(A-(-\\log (p(k|x))\\big).\n   $$\n   And the NL loss is:\n   $$\n   L_\\text{NL}=\\sum_{k=1}^K (1-q(k|x)) \\big( - \\log (1-p(k|x)) \\big).\n   $$\n   Although the $q(k|x)$ in NL is actually obtained by randomly selecting complementary labels rather than directly from the given labels as we did, we use the same form for comparison purposes.\n\n   Next, let's take a closer look at the gradient of these two losses on $p(j|x), j \\ne y$. For NLF, we have:\n   $$\n   \\frac{\\partial}{\\partial p(j|x)} L_\\text{NLF} = \\frac{1}{p(j|x)}.\n   $$\n   And for NL loss, we have:\n   $$\n   \\frac{\\partial}{\\partial p(j|x)} L_\\text{NL} = \\frac{1}{1-p(j|x)}.\n   $$\n   We can see that for NL loss, its gradient decreases as $p(j|x)$ decreases. However, for our NLF, the gradient increases. This means that our NLF can keep focusing on well-learned samples. This property holds even after normalization (as shown in Theorem 5 and 6). In fact, if we were to use NL instead of our NLF, Theorem 5 and 6 would not hold.\n\n2. Regularization.\n\n   For the experiments of other methods, we followed their original papers and used L2 regularization. However, to ensure fairness and investigate whether other regularizations would have positive effects on other methods, we also conducted a set of experiments using L1 regularization for APL. The details and results of these experiments are presented in Appendix on L530. We found that changing the regularization methods did not significantly affect the performance of APL. Moreover, regarding the ablation study of the impact of regularization on the ANL framework, we have already done the relevant experiments and discussions on L247 of the paper.\n\n3. More experiments.\n\n   Please see the global response."}, {"review_id": "OZh4RROHda", "reviewer": "Reviewer_u2rQ", "summary": "This paper proposes robust loss function to improve training of DNNs using noisy labels. Previously proposed work of Active Passive Loss functions is improved.", "questions": "The same as above.", "limitations": "More theoretical insights are required for the proposed loss functions.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Paper is well written and contains theoretical foundation of the proposed work. There are a lot of derivations which give a broad understanding of loss functions. Table 2 shows improved performance of the proposed loss functions compared to [6] and [18] in some cases. Though the improvement is not consistent. ", "weaknesses": "1. The  terminology used in this paper is mainly based on previous work of reference [6]. The terms like active and passive loss functions do not clearly separate the loss functions. As shown in the appendix, MAE=1(1-p(y|x)), shows that MAE also aims to maximise p(y|x) therefore it is also an active loss function. In [6], MAE was considered as a passive loss function which is not correct Mathematically in cases when we are dealing with a prediction as \\sum_k{p(k|x)}=1. This paper also inherits the same definitions from [6]. \n\n2. Changing the sign of a loss function also changes the max to min. If we are maximising L, we should be minimising -L. Therefore change of sign may not be considered making any fundamental difference. The term `Vertical Flip' is. derived from the graph. Mathematically, there is no thing like a vertical flip. \n\n3. Definition of A in Eq (7) is not clear. How a loss can be computed between a vector [ . .........p_min, .........] and y ? In the appendix, A is defined as a constant value. \n\n4. As shown in the appendix, the proposed normalisation  by [6] is  scaling and the same is inherited in the current work. If all loss values are scaled by the same number then how it will be robust to noise? The loss generated by the noisy and the clean samples will be scaled equally and noisy labels are not known at the time of training. \n\n5. In Eq 9, denominator can be simplified as: \\sum_k{A+ log p(k|x)}=KA+1, reducing Eq 9 to: NNFL=1-A/(KA+1)-logp(y|x)/(1+KA) . It is again a scaled version of the original function added a constant value. What fundamental change makes it robust to noise? \n\n6. The range of asymmetric noise is kept quite small {.2,.3.4} compared to symmetric noise {.4,.6,.8} in Table 2. A larger range would have revealed more insights.\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the suggestion from the reviewer and will incorporate it into the updated version of our paper.\n\n1. Active and passive loss functions.\n\n   We agree that there is no clear distinction between active and passive loss functions in [1]. Since our work follows [1], we directly use these definitions. Regarding why to distinguish between active and passive loss functions, we suggest referring directly to [1], which has a detailed discussion on this.\n\n2. Vertical flip.\n\n   Firstly, the name “vertical flip” does come directly from the graph because it is very intuitive. Secondly, taking CE as an example, $A-(-\\log p(i|x))$ obtained by “vertical flip” is not the same as $-(-\\log p(i|x))$ obtained by simply changing the sign because we do not directly minimize it but also apply normalization to it. Specifically, after normalizing $\\sum_{i \\ne y}^K-(-\\log p(i|x))$, we get $\\frac{1}{K-1}\\cdot(1-\\frac{-\\log p(y|x)}{\\sum_k - \\log p(k|x)})$, which shows that when we minimize it, we are actually minimizing $p(y|x)$ and maximizing $p(i\\ne y|x)$, which contradicts our optimization goal. In contrast, after normalizing $\\sum_{i \\ne y}^K A - (- \\log p(i|x))$, we get $\\frac{1}{K-1}\\cdot (1-\\frac{A-(-\\log p(y|x))}{\\sum_k A - (-\\log p(k|x))})$, which shows that when we minimize it, we are actually maximizing $p(y|x)$ and minimizing $p(i\\ne y|x)$, which is consistent with our optimization goal.\n\n3. Definition of $A$.\n\n   Thanks to the reviewer for pointing out that the definition of $A$ is unclear. $A$ is indeed a constant, and we will revise its definition to make it more clear and precise.\n\n4. Normalization.\n\n   The normalization proposed by [1] does not mean that all the loss values are scaled by the same number. For example, for $NCE=\\frac{1}{\\sum_k - \\log p(k|x)} \\cdot \\big(- \\log p(y|x)\\big)$, the factor $\\frac{1}{\\sum_k - \\log p(k|x)}$ varies for different samples $x$ with different prediction probabilities $p(i|x), i \\in [1, \\cdots, K]$. Moreover, the noise robustness of the normalized loss function is determined by its symmetry: $\\sum_{y=1}^K \\Big( \\frac{1}{\\sum_k - \\log p(k|x)} \\cdot \\big(- \\log p(y|x)\\big)\\Big) = 1$, and [2] has proved that the symmetric loss functions are robust to noise.\n\n5. The denominator of Eq (9).\n\n   The denominator of Eq (9) can be expressed as: $\\sum_k (A + \\log p(k|x) ) = K \\cdot A + \\sum_k \\log p(k|x)$. We know that $\\sum_k p(k|x)=1$ and $p(k|x) \\in [0, 1]$. Therefore, we can conclude that $\\sum_k \\log p(k|x) \\in [-\\infty, K\\cdot \\log \\frac{1}{K}]$ and $\\sum_k ( A + \\log p(k|x) ) \\ne K \\cdot A + 1$.\n\n6. The range of asymmetric noise.\n\n   For synthetic asymmetric noise, we often mislabel one class as a specific other class. For instance, on CIFAR-10 with synthetic asymmetric noise, we randomly label bird images as planes with probability $\\eta$ (noise rate). If $\\eta$ is larger than 0.5, it means that most of the birds are labeled as planes, which is very rare in the real world. To the best of our knowledge, no previous work has experimented with synthetic asymmetric noise at noise rates $\\eta$ greater than 0.5.\n\n[1] Normalized Loss Functions for Deep Learning with Noisy Labels, ICML, 2020\n\n[2] Robust Loss Functions under Label Noise for Deep Neural Networks, AAAI, 2017"}, {"review_id": "O1QXa6Tvvd", "reviewer": "Reviewer_btFc", "summary": "This paper proposes a robust loss function for learning from label noise. The authors first find that negative losses proposed in APL are scaled versions of MAE, which is not training-friendly. To solve this problem, the authors propose the Active Negative Loss (ANL) framework which improves the APL loss by replacing the passive losses with a newly designed set of negative losses. Theoretically, the ANL losses are proved to be noise tolerant and the designed negative loss is shown to focus more on the well-learned classes and samples compared with MAE. Experimentally, the ANL framework outperforms the state-of-the-art robust losses on multiple datasets.", "questions": "- What motivates the three components of NNLF?\n- In Eq. (7), is $A$ a constant over all $p(k|x)$? If not, the derivation of gradients in Eq. (15) should consider the dependency of $A$ on $p(k|x)$. \n- The proposed ANL framework performs less well on the MNIST dataset (Tables 2, 6, 7). Are there any explanations? ", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy to read.\n- This paper provides new insights into the negative losses proposed in APL.\n- The proposed loss framework is provably robust and its gradient is theoretically understood.\n- The experiments are extensive and sound.", "weaknesses": "- In Section 3.1, the motivation for incorporating the three components in NNLF is lacking. Why do we need the three components in NNLF? And why do the three components make NNLF a good robust loss function?\n- The theoretical analysis is less original. Most discussions of the relationship between symmetric loss function and noise tolerance can be found in previous works, e.g. APL. Accordingly, the proofs of Theorems 2, 3, and 4 could be simplified.\n- Typos. Lines 196, 201, 220. \"theorem\" should start with a capital T.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the suggestion from the reviewer and will incorporate it into the updated version of our paper.\n\n1. The three components of NNLF.\n\n   Our goal is to create a new loss function to replace the MAE in APL. This means that: 1. The loss function must conform to the definition of passive loss functions in APL. 2. The loss function is different from MAE and does not treat all samples equally, which leads to training difficulties. 3. The loss must be robust to noisy labels. The three components of NNLF: complementary label learning, “vertical flipping”, and normalization operation correspond to these three motivations respectively. We believe that NNLF is better than MAE because it can make the model pay more attention to clean samples that have already been memorized, as described in our discussion of Theorem 5 and Theorem 6. Theorem 5 and Theorem 6 are due to the “vertical flipping” operation, which means that the gradient increases as the prediction probability of non-labeled classes decreases.\n\n2. The theoretical analysis of noise tolerance is less original.\n\n   We appreciate your comment on the originality of our discussion on noise tolerance. We kept this part for the sake of completeness of our proofs, but we agree that it can be simplified. We will revise our paper accordingly.\n\n3. Is $A$ a constant ?\n\n   Yes, $A$ is a constant.\n\n4. ANL framework performs less well on the MNIST.\n\n   We appreciate the reviewer’s insightful question. We conduct experiments on the MNIST dataset with 0.8 symmetric noise rate using ANL-CE and NCE+AGCE, and calculate the entropy $H=-\\sum_i p(i|x) \\log p(i|x)$ of the model’s prediction probability on clean samples at each step. The lower the entropy, the closer the prediction probability is to a one-hot vector, indicating that the model is more confident in its prediction results. We find that in the first 5 epochs, the entropy values of both methods are very close and have similar trends. After 5 epochs, the entropy of NCE+AGCE stabilizes at around 0.09 and keeps oscillating, while that of ANL-CE shows a continuous decreasing trend and approaches 0. This indicates that our model is too confident in its prediction results.\n\n   We believe that this phenomenon is caused by our Theorem 5 and Theorem 6, which force the loss function to continuously make the model learn already learned samples, causing overfitting of the model to clean samples in the training set and loss of generalization performance. Although we use L1 regularization to alleviate this problem, considering that MNIST is a very simple dataset, the effectiveness of L1 regularization may be affected. As described in the Limitations section, we believe that regularization methods are the main limitation of our method.\n\n5. Typos\n\n   Thank you for pointing out the typos in our paper. We will carefully check our paper and correct all the spelling and grammar errors. We apologize for the inconvenience caused by this."}, {"review_id": "becO7W9NmF", "reviewer": "Reviewer_abGG", "summary": "In this work, the authors propose a new class of theoretically robust passive loss functions named Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with the proposed NNLFs, this paper improves APL and proposes a new framework called Active Negative Loss (ANL).", "questions": "1. Some mathematical notations should be clearly defined, such as N in Section 2.1.\n2. The authors should carefully proofread their paper. Some typos should be corrected, such as \"at at least one other class position...\". In supplementary material, the equation index is missing in \"So, from Eq () we have...\" above Eq. 25.\n3. This paper only investigates the proposed method on one real-world dataset, namely WebVision, which I think is not adequate. In fact, there are many benchmark datasets with real-world noise, such as Animal-10N, Clothing-1M, etc.", "limitations": "See my comments above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clearly stated. The writing of this paper is also clear.\n2. The theoretical study in this paper is solid.", "weaknesses": "1. More real-world datasets are needed for algorithm verification. \n2. This paper contains many typos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the suggestion from the reviewer and will incorporate it into the updated version of our paper.\n\n1. Typos and mathematical notations.\n\n   We thank the reviewer for pointing out the typos and the unclear mathematical notations. We will check our paper carefully and correct the relevant errors. We apologize for the inconvenience caused by this.\n\n2. More experiments.\n\n   Please see the global response."}]

Author rebuttal:
Following the suggestions of the reviewers and in order to further validate the effectiveness of our method, we conducted a set of experiments on CIFAR-10N [1], CIFAR-100N [1], Animal-10N [2], and Clothing-1M [3]. For some experiments, we can only compare a few methods due to time constraint. The table of the experimental results can be found in the pdf file. Overall, our method outperforms all the other compared methods on all four datasets.

**CIFAR-10N and CIFAR-100N**

We use the same experimental settings and parameters as CIFAR-10 and CIFAR-100 for CIFAR-10N and CIFAR-100N, since the only difference is the noise label distribution. The results are reported in Table 1 and Table 2.

**Animal-10N**

We follow the experimental setting in previous works [2]. We use VGG-19 with batch normalization. The SGD optimizer is employed. We train the network for 100 epochs and use an initial learning rate of 0.1, which is divided by 5 at 50% and 75% of the total number of epochs. Batch size is set to 128. Typical data augmentations including random horizontal flip are applied.

We compare our ANL-CE with CE and GCE. We use L2 regularization (weight decay) for GCE, and L1 regularization for ANL-CE. We denote the regularization coefficient by $\delta$. We tune the parameters $\\{\delta\\}, \\{q, \delta\\}$ and $\\{\alpha, \beta, \delta\\}$ for CE, GCE and ANL-CE respectively. We use the best parameters $\\{10^{-3}\\}, \\{0.5, 10^{-4}\\}, \\{0.5, 0.1, 10^{-6}\\}$ for each method in our experiments.  The results are reported in Table 3.

Moreover, we experiment with NCE+RCE on this dataset and tune the parameters $\\{\alpha,\beta,\delta\\}$, but we find that the performance is very poor for some unknown reason. The best test accuracy we achieve is 28.28 with $\\{10.0, 0.1, 5 \times 10^{-6}\\}$. Since this result is too low and inconsistent with its performance on other datasets, we do not include it in the table for comparison.

**Clothing-1M**

We follow the experimental setting in previous works [4]. We use the $14k$ and $10k$ clean data for validation and test, respectively, and we do not use the $50k$ clean training data. We use ResNet-50 pre-trained on ImageNet. For preprocessing, we resized the images to $256 \times 256$, performed mean subtraction, and cropped the middle $224 \times 224$. We used SGD with a momentum of 0.9, a weight decay of $10^{-3}$, and batch size of 32. We train the network for 10 epochs with learning rate $10^{-3}$ and $10^{-4}$ for 5 epochs each. Typical data augmentations including random horizontal flip are applied.

We compare our ANL-CE with CE, GCE and NCE+RCE. In this experiment, we use L2 regularization (weight decay) for ANL-CE and set the coefficient the same as those for CE, GCE and NCE+RCE. We tune the parameters $\\{q\\}, \\{\alpha, \beta\\}$ and $\\{\alpha, \beta\\}$ for GCE, NCE+RCE and ANL-CE respectively. We use the best parameters $\\{0.6\\}, \\{10.0, 1.0\\}$ and $\\{5.0, 0.1\\}$ for each method in our experiments. The results are reported in Table 4.


[1] Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations, ICLR, 2022

[2] SELFIE: Refurbishing Unclean Samples for Robust Deep Learning, ICML, 2019

[3] Learning From Massive Noisy Labeled Data for Image Classification, CVPR, 2015

[4] Joint Optimization Framework for Learning with Noisy Labels, CVPR, 2018



Meta-review: This paper presents the Active Negative Loss (ANL) framework, which aims to improve the robustness of deep neural networks against noisy labels by introducing a new class of passive loss functions called Normalized Negative Loss Functions (NNLFs). The paper has several strengths, such as a clear motivation, having theoretical foundations, and code availability. The theoretical foundations seem solid and the empirical results are promising. However, there also remain some notable weaknesses. All the reviewers unanimously agree that the paper bears a strong technical contribution. Thus, the AC decides to accept this paper.
Decision: Accept (poster)