Title: Active Negative Loss Functions for Learning with Noisy Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: noisy label learning

robust loss function

multiclass classification

computer vision
Abstract: Robust loss functions are essential for training deep neural networks in the presence of noisy labels. Some robust loss functions use Mean Absolute Error (MAE) as its necessary component. For example, the recently proposed Active Passive Loss (APL) uses MAE as its passive loss function. However, MAE treats every sample equally, slows down the convergence and can make training difficult. In this work, we propose a new class of theoretically robust passive loss functions different from MAE, namely *Normalized Negative Loss Functions* (NNLFs), which focus more on memorized clean samples. By replacing the MAE in APL with our proposed NNLFs, we improve APL and propose a new framework called *Active Negative Loss* (ANL). Experimental results on benchmark and real-world datasets demonstrate that the new set of loss functions created by our ANL framework can outperform state-of-the-art methods. The code is available at
https://github.com/Virusdoll/Active-Negative-Loss.

Active Negative Loss Functions for Learning with Noisy Labels

Abstract

Robust loss functions are essential for training deep neural networks in the presence of noisy labels. Some robust loss functions use Mean Absolute Error (MAE) as its necessary component. For example, the recently proposed Active Passive Loss (APL) uses MAE as its passive loss function. However, MAE treats every sample equally, slows down the convergence and can make training difficult. In this work, we propose a new class of theoretically robust passive loss functions different from MAE, namely Normalized Negative Loss Functions (NNLFs), which focus more on memorized clean samples. By replacing the MAE in APL with our proposed NNLFs, we improve APL and propose a new framework called Active Negative Loss (ANL). Experimental results on benchmark and real-world datasets demonstrate that the new set of loss functions created by our ANL framework can outperform state-of-the-art methods. The code is available at https://github.com/Virusdoll/Active-Negative-Loss.

1 Introduction

Relying on large-scale datasets with high quality annotations, such as ImageNet [1], deep neural networks (DNNs) achieve good performance in various supervised classification tasks. However, in practice, the process of labeling large-scale datasets is costly and inevitably introduces noisy (mislabeled) samples. Moreover, empirical studies show that over-parameterized DNNs can easily fit a randomly labeled dataset [2], which implies that DNNs may have a poor evaluation performance when trained on a noisy dataset. As a result, noisy label learning has received a lot of attention.

Different approaches have been proposed to solve the noisy label learning problem, and one popular research line is to design noise-robust loss functions, which is also the main focus of this paper. Ghosh et al. [3] have theoretically proved that, symmetric loss functions such as Mean Absolute Error (MAE), are robust to noise, while others like commonly used Cross Entropy (CE) are not. However, MAE treats every sample equally, leading to significantly longer training time before convergence and even making learning difficult, which suggests that MAE is not suitable for training DNNs with challenging datasets [4]. Motivated by this, several works proposed partially robust loss functions, including Generalized Cross Entropy (GCE) [4], a generalized mixture of CE and MAE, and Symmetric Cross Entropy (SCE) [5], a combination of CE and a scaled MAE, Reverse Cross

Songmin Dai

Weiqin Tong

Entropy (RCE). Recently, Active Passive Loss (APL) [6] framework has been proposed to create fully robust loss functions.

APL is one of the state-of-the-art methods, and shows that any loss function can be made robust to noisy labels by a simple normalization operation. Moreover, to address the underfitting problem of normalized loss functions, APL first characterizes existing loss functions into two types and then combines them. The two types are 1) 'Active' loss, which only explicitly maximizes the probability of being in the labeled class, and 2) 'Passive' loss, which also explicitly minimizes the probabilities of being in other classes. However, by investigating several robust loss functions created by the APL framework, we find that their passive loss functions are always scaled versions of MAE. As we mentioned before, MAE is not conducive to training. As a result, to address the underfitting problem, APL combines active loss functions with MAE, which again may lead to difficulty in training and further limits its performance.

The fact that APL still struggles with MAE motivates us to investigate new robust passive loss functions. In this paper, we propose a new class of passive loss functions different from MAE, called Negative Loss Functions (NLFs). We show that, by combining 1) complementary label learning [7, 8] and 2) a simple 'vertical flipping' operation, any active loss function can be made into a passive loss function. Moreover, to make it theoretically robust to noisy labels, we further apply the normalization operation on NLFs to obtain Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with NNLF, we propose a novel framework called Active Negative Loss (ANL). ANL combines a normalized active loss function and a NNLF to build a new set of noise-robust loss functions, which can be seen as an improvement of APL. We show that under our proposed ANL framework, several commonly-used loss functions can be made robust to noisy labels while ensuring sufficient fitting ability to achieve state-of-the-art performance for training DNNs with noisy datasets. Our key contributions are highlighted as follows:

We provide a method to build a new class of robust passive loss functions called Normalized Negative Loss Function s (NNLFs). By replacing the MAE in APL with our proposed NNLFs, we propose a novel framework, Active Negative Loss (ANL), to construct a new set of robust loss functions.

We demonstrate the theoretical robustness of our proposed NNLFs and ANL to noisy labels, and discuss how replacing the MAE in APL with our NNLFs enhances performance in noisy label learning.

Our empirical results show that the new set of loss functions, created using our proposed ANL framework, outperform existing state-of-the-art methods.

2 Preliminaries

2.1 Risk Minimization and Label Noise Model

̸

Consider a typical K-class classification problem. Let X ⊂ R d be the d -dimensional feature space from which the samples are drawn, and Y = [ k ] = { 1 , · · · , K } be the label space. Given a clean training dataset, S = { ( x n , y n ) } N n =1 , where each ( x n , y n ) is drawn i.i.d. from an unknown distribution, D , over X × Y . We denote the distribution over different labels for sample x by q ( k | x ) , and ∑ K k =1 q ( k | x ) = 1 . Since there is only one corresponding label y for a x , we have q ( y | x ) = 1 and q ( k = y | x ) = 0 .

Aclassifier, h ( x ) = arg max i f ( x ) i , where f : X → C , C ⊆ [0 , 1] K , ∀ c ∈ C , 1 T c = 1 , is a function that maps feature space to label space. In this work, we consider f as a DNN ending with a softmax output layer. For each sample x , f ( x ) computes its probability p ( k | x ) of each label k ∈ { 1 , · · · , K } , and ∑ K k =1 p ( k | x ) = 1 . Throughout this paper, as a notation, we call f itself the classifier. Training a classifier f is to find an optimal classifier f ∗ that minimize the empirical risk defined by a loss function: ∑ N n =1 L ( f ( x n ) , y n ) , where L : C × Y → R + is a loss function, and L ( f ( x ) , k ) is the loss of f ( x ) with respect to label k .

When label noise is present, our model can only access a corrupted dataset S η = { ( x n , ˆ y n ) } N n =1 , where each sample is drawn i.i.d. from an unknown distribution, D η . In this paper, we consider a popular approach for modeling label noise, which simply assumes that, given the true label y , the

corruption process is conditionally independent of input features x [9]. So we can formulate noisy label ˆ y as:

̸

̸

where η yj denotes the probability that true label y is corrupted into label j , and η y = ∑ j = y η yj denotes the noise rate of label y . Under our assumption of label noise model, label noise can be either symmetric or asymmetric . The noise is called symmetric , if η ij = η i K -1 , ∀ j = y and η i = η, ∀ i ∈ [ k ] , where η is a constant. And for asymmetric noise, η ij is conditioned on both the true label i and corrupted label j .

2.2 Active Passive Loss Functions

Ghosh et al. [3] have shown, under some mild assumptions, a loss function L is noise tolerant if it is symmetric: ∑ K k =1 L ( f ( x ) , k ) = C, ∀ x ∈ X , where C is some constant. Based on this, Ma et al. [6] proposed the normalized loss functions, which normalize a loss function L by:

This simple normalization operation can make any loss function robust to noisy labels, since we always have ∑ K k L norm = 1 . For example, the Normalized Cross Entropy (NCE) is:

Similarly, we can normalize FL, MAE, and RCE to obtain Normalized Focal Loss (NFL), Normalized Mean Absolute Error (NMAE), and Normalized Reverse Cross Entropy (NRCE), respectively.

But a normalized loss function alone suffers from the underfitting problem. To address this, Ma et al. [6] characterize existing loss functions into two types: Active and Passive . Denote the function of loss L ( f ( x ) , y ) by ℓ ( f ( x ) , k ) , that is L ( f ( x ) , y ) = ∑ K k =1 ℓ ( f ( x ) , k ) (e.g., let L be CE, then L ( f ( x ) , y ) = ∑ K k =1 q ( k | x )( -log p ( k | x )) , and ℓ ( f ( x ) , k ) = q ( k | x )( -log p ( k | x )) ), we have the following definitions:

̸

Definition 1 (Active loss function) . L Active is an active loss function if ∀ ( x , y ) ∈ D , ∀ k = y, ℓ ( f ( x ) , k ) = 0 .

̸

Definition 2 (Passive loss function) . L Passive is a passive loss function if ∀ ( x , y ) ∈ D , ∃ k = y, ℓ ( f ( x ) , k ) = 0 .

̸

̸

Active loss functions only explicitly maximize p ( y | x ) , the classifier's output probability at the class position specified by the label y . In contrast, passive loss functions also explicitly minimize p ( k = y | x ) , the probability at least one other class positions. Accordingly, the active loss functions include CE, FL, NCE, and NFL, while the passive loss functions include MAE, RCE, NMAE, and NRCE. These two types of loss functions can mutually boost each other to mitigate underfitting, and we refer the reader to [6] for more detailed discussions. By combining them, Ma et al. proposed the Active Passive Loss (APL):

where α, β > 0 are parameters. As an example, by combining NCE and RCE, Ma et al get NCE+RCE, one of the state-of-the-art methods.

2.3 APL struggles with MAE

As shown in the previous subsection, there are four passive loss functions available to APL, including MAE, RCE, NMAE, and NRCE. However, we can show that all these passive loss functions are scaled versions of MAE. Specifically, NMAE = 1 2( K -1) · MAE , RCE = -A 2 · MAE , and

NRCE = 2( K -1) · MAE (detailed derivations can be found in appendix A.1). Thus, we can rewrite

1 APL as follows:

̸

0.0

0.2

0.4

0.6

0.8

1.0

p

(

k

|

x

)

0

1

2

A

A

loss

(

f

(

x

),

k

)

A

(

f

(

x

),

k

)

This indicates that MAE is a necessary component of the current APL. However, as we mentioned before, MAE requires longer training time and even makes learning difficult. Thus, on the one hand, APL needs passive loss functions to mitigate active loss functions underfitting, yet on the other hand, MAE is not training friendly, which may limit the performance of APL. This motivates us to investigate new robust passive loss functions.

3 Active Negative Loss Functions

3.1 Method

Normalized Negative Loss Functions. Our goal is to find a method that creates robust passive loss functions from existing active loss functions. This method must consist of three components that: 1) let the loss function optimize the classifier's output probability for at least one other class position that is not specified by the label y , 2) let the loss function minimize the classifier's output probability instead of maximizing it, and 3) let the loss function robust to noisy labels. Inspired by NLNL [10] and APL [6], we use 1) complementary label learning, 2) 'vertical flipping' and 3) normalization operation as these three components respectively.

Complementary label learning [7, 8] is an indirect learning method for training CNNs, which randomly selects complementary labels and trains the CNN to recognize that 'input does not belong to this complementary label'. Unlike the usual training approach, complementary label learning focuses on the loss of classifier's predictions with complementary labels, which naturally fits with the passive loss function. Here, we only use its basic idea of letting the loss function focus on all classes { 1 , · · · , K } except the labeled class y .

'Vertical flipping' is a simple operation that can convert the loss function from 'maximizing' to 'minimizing'. As shown in the fig. 1, given an active loss function L ( f ( x ) , k ) , the new loss function A -L ( f ( x ) , k ) is obtained by flipping L ( f ( x ) , k ) vertically with axis loss = 1 2 A . It should be noted that, A -L ( f ( x ) , k ) is the opposite of L ( f ( x ) , k ) , and it focuses on optimizing p ( k | x ) to 0 .

Based on these two components, given an active loss function L , we propose Negative Loss Functions (NLFs) as follows:

Here, [ · · · , p min , · · · ] T is some probability distribution that may be output by the classifier f , where p ( y | x ) = p min , the minimum value of p ( k | x ) (e.g., 0 ). Therefore A is some constant, the maximum loss value of L . In practice, setting p min = 0 could cause some computational problems, for example, if L is CE and p min = 0 , then A = -log 0 = + ∞ . So in this paper, unless otherwise specified, we define p min = 1 × 10 -7 . This technique is similar to the clipping operation implemented by most deep learning frameworks.

Our proposed NLF can transform any active loss function into a passive loss function, where 1) (1 -q ( k | x )) ensures that the loss function focuses on classes { 1 , · · · , K }\{ y } , and 2) ( A -L ( f ( x ) , k )) ensures that the loss function aims to minimize the output probability p ( k | x ) .

Next, to make our proposed passive loss functions robust to noisy labels, we perform a normalization operation on NLFs. Given an active loss function L , we propose Normalized Negative Loss Functions (NNLFs) as follows:

where A has the same definition as eq. (7). The detailed derivation of NNLFs can be found in appendix A.2. Additionally, NNLFs have the property that L nn ∈ [0 , 1] . Accordingly, we can create NNLFs from active loss functions as follows.

The Normalized Negative Cross Entropy (NNCE) is:

where A = -log p min .

The Normalized Negative Focal Loss (NNFL) is:

where A = -(1 -p min ) γ log p min .

ANL Framework. We can now create new robust loss functions by replacing the MAE in APL with our proposed NNLF. Given an active loss function L , we propose the Active Negative Loss (ANL) functions as follows:

Here, α and β are parameters greater than 0, L norm denotes the normalized L and L nn denotes the Normalized Negative Loss Function corresponding to L . Accordingly, we can create ANL from the two mentioned active loss functions as follows.

For Cross Entropy (CE), we have ANL-CE:

For Focal Loss (FL), we have ANL-FL:

3.2 Robustness to noisy labels

NNLFs are symmetric. We first prove that our proposed Normalized Negative Loss Functions (NNLFs) are symmetric. Detailed proofs can be found in appendix B.

Theorem 1. Normalized negative loss function L nn is symmetric.

NNLFs are robust to noise. In reference to Theorem 1 and Theorem 3 from [3], it has been proven that symmetric loss functions, under some mild assumptions, exhibit noise tolerant in the face of both symmetric and asymmetric noise. Given that our NNLFs fall under the category of symmetric loss functions, they inherently possess the attribute of noise tolerant.

ANL is robust to noise. In light of Lemma 3 from [6], it is understood that the combination of two noise tolerant loss functions retains the noise tolerant attribute. It is noteworthy that both L norm and L nn within our ANL are noise tolerant, which makes ANL as a whole noise tolerant.

3.3 NNLFs focus more on well-learned samples

As shown in the fig. 2, by replacing MAE with our proposed NNCE, NCE+NNCE and ANL-CE show better fitting ability. This raises the question: why does NNLF perform better than MAE? In the following, taking NNCE as an example, we analysis the gradients of MAE and NNCE to provide a preliminary answer to this question. Detailed derivations and proofs can be found in appendix C.

The gradient of the MAE with respect to the classifier's output probability can be derived as:

̸

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

Clean Samples in Training Set

Test Set

(a) CE

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(b) MAE

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(c) NCE+RCE

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(d) ANL-CE (w/ L2)

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(e) ANL-CE (w/ L1)

0

50

100

150

200

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

(f) CE

0

50

100

150

200

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

(g) MAE

0

50

100

150

200

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

(h) NCE+RCE

0

50

100

150

200

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

(i) ANL-CE (w/ L2)

0

50

100

150

200

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

(j) ANL-CE (w/ L1)

The gradient of the NNCE with respect to the classifier's output probability can be derived as:

̸

̸

For the purpose of analysis, we consider how the gradients of NNCE would differ from MAE in the following two scenarios: 1) given the classifier's output probability of sample x , we analyze the difference in gradient for each class, 2) given the classifier's output probabilities of sample x 1 and x 2 , we analyze the difference in gradient between these two samples.

̸

Theorem 2. Given the classifier's output probability p ( ·| x ) for sample x and normalized negative cross entropy L NNCE. If p ( j 1 | x ) < p ( j 2 | x ) , j 1 = j 2 = y , then ∂ L NNCE ∂ p ( j 1 | x ) > ∂ L NNCE ∂ p ( j 2 | x ) .

̸

̸

Theorem 3. Given the classifier's output probabilities p ( ·| x 1 ) and p ( ·| x 2 ) of sample x 1 and x 2 , where p ( y | x 1 ) ≥ p ( k | x 1 ) , p ( y | x 2 ) ≥ p ( k | x 2 ) , ∀ k ∈ { 1 , · · · , K } , p ( j | x 1 ) = p ( j | x 2 ) , j = y , and normalized negative cross entropy L NNCE. If p ( y | x 1 ) > p ( y | x 2 ) and p ( k | x 1 ) ≤ p ( k | x 2 ) , ∀ k ∈ { 1 , · · · , K } \{ j, y } , then ∂ L NNCE ∂ p ( j | x 1 ) > ∂ L NNCE ∂ p ( j | x 2 ) .

theorem 2 and theorem 3 demonstrate that, for the gradient of the non-labeled classes, our NNCE focuses more on the classes and samples that have been well learned compared to MAE, which treats every class and sample equally This property may enhance the model's performance in noisy label learning. Some studies [11] have shown that during the training process, DNNs would first memorize clean samples and then noisy samples. According to the property revealed by theorem 2 and theorem 3, apart from robustness, our NNLF may potentially help the model to continuously learn the clean samples that the model has memorized in the previous stages of training and ignore the unmemorized noisy samples.

4 Experiments

In this section, we empirically investigate our proposed ANL functions on benchmark datasets, including MNIST [12], CIFAR-10/CIFAR-100 [13] and a real-world noisy dataset WebVision [14].

4.1 Empirical Understandings

In this subsection, we explore some properties of our proposed loss functions. Unless otherwise specified, all detailed experimental settings are the same as those in section 4.2. More experiments and discussions about gradient, parameter analysis, and p min can be found in the appendix D.1.

0

20

40

60

80

100

120

Epochs

0.25

0.35

0.45

0.55

0.65

Test Accuracy

(a) w/o regularizar

0

20

40

60

80

100

120

Epochs

0.25

0.35

0.45

0.55

0.65

Test Accuracy

=1

e

7

=5

e

7

=1

e

6

=5

e

6

=1

e

5

=5

e

5

=1

e

4

(b) L2

0

20

40

60

80

100

120

Epochs

0.25

0.35

0.45

0.55

0.65

Test Accuracy

=1

e

7

=5

e

7

=1

e

6

=5

e

6

=1

e

5

=5

e

5

=1

e

4

(c) Jacobian

0

20

40

60

80

100

120

Epochs

0.25

0.35

0.45

0.55

0.65

Test Accuracy

=1

e

7

=5

e

7

=1

e

6

=5

e

6

=1

e

5

=5

e

5

=1

e

4

(d) L1

Overfitting problem. In practice, we find that ANL can lead to overfitting in some experimental settings. To motivate this problem, as an example, we train networks using different loss functions on CIFAR-10 under 0.8 symmetric noise and CIFAR-100 under 0.6 symmetric noise, and the experimental results are shown in fig. 2. As can be observed, in the setting of CIFAR-10 under 0.8 symmetric noise, the training set accuracy of ANL-CE (w/ L2) keeps increasing while the test set accuracy keeps decreasing. We identify this problem as an overfitting problem .

It is worth noting that although overfitting occurs, unlike CE, the gap between the clean sample accuracies and the noisy sample accuracies of the training set does not shrink, which indicates that our ANL has some robustness to noisy labels even in the case of overfitting. Moreover, we conjecture that the overfitting is caused by the property of NNLF focusing more on well-learned samples. When the noise rate is high, one might assume that the model has been trained on only a fairly small number of clean samples, with a very large gradient, which can lead to overfitting.

The choice of regularization method. We also find that the commonly used L2 regularization may struggle to mitigate the overfitting problem. To address this, we decide to try using other regularization methods. We consider 2 other regularization methods: L1 and Jacobian [15, 16]. To compare the performance of these methods, we apply them to ANL-CE for training on CIFAR-10 under 0.8 symmetric noise. For simplicity and fair comparison, we use δ as the coefficient of the regularization term and consider it as an additional parameter of ANL. We keep α and β the same as in section 4.2, tune δ for all three methods by following the parameter tuning setting in appendix D.2. We also train networks without using any regularization method. The results reported in fig. 3. As can be observed, among the three regularization methods, only L1 can somewhat better mitigate the overfitting problem. If not otherwise specified, all ANLs in this paper use L1 regularization.

Robustness and fitting ability. We conduct a set of experiments on CIFAR-10/-100 to verify the robustness and fitting ability of our proposed loss functions. We set the noise type to be symmetric and the noise rate to 0.8 for CIFAR-10 and 0.6 for CIFAR-100. In each setting, we train the network using different loss functions, including: 1) CE, 2) MAE, 3) NCE+RCE, 4) ANL-CE (w/ L2), and 5) ANL-CE (w/ L1). For ANL-CE (w/ L2), we set its parameters α and β to be the same as ANL-CE (w/ L1) and set its weight decay to be the same as NCE+RCE.

As can be observed in fig. 2: 1) CE is not robust to noise, the accuracies of clean and noisy samples in the training set are continuously close to each other, 2) MAE is robust to noise, the accuracies of clean and noisy samples in the training set keep moving away from each other, but its fitting ability is insufficient, especially when the dataset becomes complex, 3) NCE+RCE is robust to noise and has better fitting ability compared to MAE, 4) ANL-CE (w/ L2) is robust to noise and has stronger fitting ability, but suffers from over-fitting. and 5) ANL-CE is robust to noise and mitigates the impact of overfitting to achieve the best performance. To summarize, our proposed loss functions are robust to noise, NNLF shows better fitting ability than MAE, and L1 regularization addresses the overfitting problem of NNLF.

Active and passive parts separately. In table 1, we show the results of the active and passive parts separately. We separately train NCE and NNCE on CIFAR-10 with different symmetric noise rates while maintaining the same parameters as ANL-CE. Specifically, for α · NCE, we set α to 5 . 0 and for β · NNCE, we set β to 5 . 0 , while δ is set to 5 × 10 -5 for both. As indicated in the results, the test set accuracies of NNCE are very close to those of ANL-CE, except in the case with a 0.8 noise rate. This suggests that NNCE performs well on its own at low noise rates. However, at very high noise rates, a combination of active losses is needed to achieve better performance.

4.2 Evaluation on Benchmark Datasets

Baselines. We consider several state-of-the-art methods: 1) Generalized Cross Entropy (GCE) [4]; 2) Symmetric Cross Entropy (SCE) [5]; 3) Negative Learning for Noisy Labels (NLNL) [10]; 4) Active Passive Loss (APL) [6], including NCE+MAE, NCE+RCE, and NFL+RCE; 5) Asymmetric Loss Functions (AFLs) [17], including NCE+AEL, NCE+AGCE, and NCE+AUL. For our proposed ANL, we consider two loss functions: 1) ANL-CE and 2) ANL-FL. Additionally, we train networks using Cross Entropy (CE), Focal Loss (FL) [18], and Mean Absolute Error (MAE).

Experimental Details. The full experimental results and the detailed settings of noise generation, networks, training and parameters can be found in the appendix D.2.

Results. The main experimental results under symmetric and asymmetric label noise are reported in table 2. For more experimental results, please see appendix D.2. As can be observed, our ANLCE and ANL-FL show significant improvement for most label noise settings of CIFAR-10/-100, especially when the data become more complex and the noise rate becomes larger. For example, on CIFAR-10 under 0.8 symmetric noise, our ANL-CE outperform the state-of-the-art method (55.62% of NCE+AGCE) by more than 5.0%. Overall, the experimental results demonstrate that our ANL can show outstanding performance on different datasets, noise types, and noise rates, which validates the effectiveness of our proposed NNLFs and ANL.

4.3 Evaluation on Real-world Noisy Labels

Here, we evaluate our proposed ANL methods on large-scale real-world noisy dataset WebVision [14], which contains more than 2.4 million web images crawled from the internet by using queries generated from the 1,000 class labels of the ILSVRC 2012 [1] benchmark. Here, we follow the 'Mini' setting in [19], and only take the first 50 classes of the Google resized image subset. We evaluate the trained networks on the same 50 classes of both the ILSVRC 2012 validation set and the WebVision validation set, and these can be considered as clean validation sets. We compare our ANL-CE and ANL-FL with GCE, SCE, NCE+RCE, and NCE+AGCE. The experimental details can be found in the appendix D.3. The results are reported in table 3. As can be observed, our proposed methods outperform the existing loss functions. This verifies that our proposed ANL framework can help the trained model against real-world label noise.

Moreover, in addition to WebVision, to further validate the effectiveness of our method on real-world noisy datasets, we also conduct a set of experiments on CIFAR-10N/-100N [20], Animal-10N [21], and Clothing-1M [22]. The experimental details and results can be found in the appendix D.5, which demonstrate the effectiveness of our method on different real-world noisy datasets.

5 Limitations

We believe that the main limitation of our approach lies in the choice of regularization method. Although we have experimentally verified that L1 is the most efficient among the three regularization methods (L1, L2, and Jacobian), we lack further theoretical analysis of it. Furthermore, although we only consider relatively simple regularization methods for the sake of fair comparison, other regularization methods, such as dropout [23] or regmixup [24], might be more effective in mitigating the overfitting problem caused by NNLF. And we believe that a better solution to overfitting can further improve the performance of our method.

6 Related Work

In recent years, some robust loss-based methods have been proposed for robust learning with noisy labels. Here, we briefly review the relevant approaches. Ghosh et al. [3] theoretically proved that symmetric loss functions, such as MAE, are robust to label noise. Zhang and Sabuncu [4] proposed Generalized Cross Entropy (GCE), a generalization of CE and MAE. Wang et al. [5] suggested a combination of CE and scaled MAE, and proposed Symmetric Cross Entropy (SCE). Menon et al. [25] proposed composite loss-based gradient clipping and applied it to CE to obtain PHuber-CE. Ma et al. [6] proposed Active Passive Loss (APL) to create fully robust loss functions. Feng et al. [26] applied the Taylor series to derive an alternative representation of CE and proposed TaylorCE accordingly. Zhou et al. [17] proposed Asymmetric Loss Functions (ALFs) to overcome the symmetric condition. Inspired by complementary label learning, NLNL [10] and JNPL [27] use complementary labels to reduce the risk of providing the wrong information to the model.

7 Conclusion

In this paper, we propose a new class of theoretically robust passive loss functions different from MAE, which we refer to as Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with our NNLF, we propose Active Negative Loss (ANL), a robust loss function framework with stronger fitting ability. We theoretically demonstrate that our NNLFs and ANLs are robust to noisy labels and also highlight the property that NNLFs focus more on well-learned samples. We found in our experiments that NNLFs have a potential overfitting problem, and we suggest using L1 regularization to mitigate it. Experimental results verified that our ANL can outperform the state-of-the-art methods on benchmark datasets.

References

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. Proceedings of the 5th International Conference on Learning Representations , 2017.

Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust Loss Functions under Label Noise for Deep Neural Networks. Proceedings of the 31st AAAI Conference on Artificial Intelligence , pages 1919-1925, 2017.

Zhilu Zhang and Mert R. Sabuncu. Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels. Advances in Neural Information Processing Systems 31 , pages 8792-8802, 2018.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric Cross Entropy for Robust Learning with Noisy Labels. Proceedings of the 2019 IEEE International Conference on Computer Vision , pages 322-330, 2019.

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah M. Erfani, and James Bailey. Normalized Loss Functions for Deep Learning with Noisy Labels. Proceedings of the 37th International Conference on Machine Learning , pages 6543-6553, 2020.

Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from Complementary Labels. Advances in Neural Information Processing Systems 30 , pages 5639-5649, 2017.

Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with Biased Complementary Labels. 15th Proceedings of the European Conference on Computer Vision , pages 69-85, 2018.

Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with Noisy Labels. Advances in Neural Information Processing Systems 26 , pages 1196-1204, 2013.

Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. NLNL: Negative Learning for Noisy Labels. Proceedings of the 2019 IEEE International Conference on Computer Vision , pages 101-110, 2019.

Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. International conference on artificial intelligence and statistics , pages 4313-4324, 2020.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision Database: Visual Learning and Understanding from Web Data. CoRR , abs/1708.02862, 2017. URL http://arxiv.org/abs/1708. 02862 .

Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust Large Margin Deep Neural Networks. IEEE Trans. Signal Process. , 65(16):4265-4280, 2017.

Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust Learning with Jacobian Regularization. CoRR , abs/1908.02729, 2019. URL http://arxiv.org/abs/1908.02729 .

Xiong Zhou, Xianming Liu, Junjun Jiang, Xin Gao, and Xiangyang Ji. Asymmetric Loss Functions for Learning with Noisy Labels. Proceedings of the 38th International Conference on Machine Learning , pages 12846-12856, 2021.

Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. Proceedings of the 2017 IEEE International Conference on Computer Vision , pages 2999-3007, 2017.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. Proceedings of the 35th International Conference on Machine Learning , pages 2309-2318, 2018.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. Proceesdings of the 10th International Conference on Learning Representations , 2022.

Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing Unclean Samples for Robust Deep Learning. Proceedings of the 36th International Conference on Machine Learning , 2019.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition , pages 2691-2699, 2015.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(56): 1929-1958, 2014.

Francesco Pinto, Harry Yang, Ser Nam Lim, Philip H. S. Torr, and Puneet K. Dokania. Using mixup as a regularizer can surprisingly improve accuracy & out-of-distribution robustness. Advances in Neural Information Processing Systems 35 , 2022.

Aditya Krishna Menon, Ankit Singh Rawat, Sanjiv Kumar, and Sashank Reddi. Can gradient clipping mitigate label noise? Proceedings of the 8th International Conference on Learning Representations , 2020.

Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? 29th International Joint Conference on Artificial Intelligence , 2020.

Youngdong Kim, Juseung Yun, Hyounguk Shon, and Junmo Kim. Joint Negative and Positive Learning for Noisy Labels. Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition , pages 9442-9451, 2021.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition , pages 2233-2241, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations , 2015.

Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition , pages 5552-5560, 2018.

A Loss functions

A.1 NMAE, RCE, and NRCE are scaled versions of MAE

Here, inspired by [5, 6], we show how to reformulate NMAE, RCE, and NRCE as scaled versions of MAE. First, for convenience, we reformulate the Mean Absolute Error (MAE) as follows:

̸

where the last equality holds due to ∑ K k =1 p ( k | x ) = 1 .

For Normalized Mean Absolute Error (NMAE), we have:

This shows that the NMAE is a scaled version of the MAE with a factor of 1 2( K -1) .

For Reverse Cross Entropy (RCE), we have:

̸

recall that in RCE we set log 0 = A , where A < 0 is some constant (e.g., A = -4 ). This shows that the RCE is a scaled version of the MAE with a factor of -A 2 .

For Normalized Reverse Cross Entropy (NRCE), we have:

This shows that NRCE is a scaled version of the MAE with a factor of 1 2( K -1) .

̸

A.2 Normalized Negative Loss Functions

Here, we show how to derive Normalized Negative Loss Functions (NNLFs) into its proper form.

̸

B Noise tolerant

Theorem 1. Normalized negative loss function L nn is symmetric.

Proof. For all x ∈ X and all f , we have:

where K -1 is a constant and L nn satisfies the definition of the symmetric loss function.

C Gradient analysis

C.1 Gradient of MAE

The complete derivation of the Mean Absolute Error (MAE) with respect to the classifier's output probabilities is as follows:

̸

In the case of j = y :

In the case of j = y :

̸

̸

̸

̸

C.2 Gradient of NNCE

The complete derivation of our proposed Normalized Negative Cross Entropy (NNCE) with respect to the classifier's output probabilities is as follows:

̸

In the case of j = y :

In the case of j = y :

C.3 Properties of NNCE

Theorem 2. Given the classifier's output probability p ( ·| x ) for sample x and normalized negative cross entropy L NNCE. If p ( j 1 | x ) < p ( j 2 | x ) , j 1 = j 2 = y , then ∂ L NNCE ∂ p ( j 1 | x ) > ∂ L NNCE ∂ p ( j 2 | x ) .

Proof. As per the assumption, we have:

This completes the proof.

Theorem 3. Given the classifier's output probabilities p ( ·| x 1 ) and p ( ·| x 2 ) of sample x 1 and x 2 , where p ( y | x 1 ) ≥ p ( k | x 1 ) , p ( y | x 2 ) ≥ p ( k | x 2 ) , ∀ k ∈ { 1 , · · · , K } , p ( j | x 1 ) = p ( j | x 2 ) , and normalized negative cross entropy L NNCE. If p ( y | x 1 ) > p ( y | x 2 ) , j = y , and p ( k | x 1 ) ≤ p ( k | x 2 ) , ∀ k ∈ { 1 , · · · , K } \{ j, y } , then ∂ L NNCE ∂ p ( j | x 1 ) > ∂ L NNCE ∂ p ( j | x 2 ) .

̸

Proof. We first consider two functions f 1 and f 2 . The function f 1 is defined as follows:

̸

̸

̸

where the input p is a discrete probability distribution, ∑ K k =1 p k = 1 and 0 ≤ p k ≤ 1 , ∀ k ∈ { 1 , · · · , K } . Given a discrete probability distribution p and a real number D as input, the function f 2 is defined as follows:

̸

̸

̸

where p y ≥ p k , ∀ k ∈ { 1 , · · · , K } , 0 < D ≤ 1 -p y . And { d k } , k ∈ { 1 , · · · , K } \ { j, y } is a set of random variables which satisfy following conditions: 0 ≤ p k -d k ≤ p k and ∑ k = j = y d k = D . Next, we consider whether f 1 ( p ) -f 2 ( p , D ) > 0 . Given p and D , we have,

̸

̸

̸

̸

̸

̸

̸

̸

̸

The first inequality holds because for a fraction a b ≥ 1 , we always have a b ≥ a + c b + c , where c ≥ 0 .

̸

To get the maximum value of log( p y + D ) + ∑ k = j = y log( p y -d k ) , we must solve the following minimization problem subject to constraints:

̸

̸

̸

̸

̸

̸

̸

We can define the Lagrange function L as follows:

̸

̸

Now we can calculate the gradients:

Let ∂L ∂d k = 0 and ∂L ∂λ = 0 , we have:

̸

̸

̸

̸

and therefore:

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

So, the minimization value is:

Now, back to the eq. (29), we have:

̸

̸

̸

̸

̸

̸

Recall the nature of the difference, we have log p y -log( p y -K D -2 ) K D -2 = d log x dx ∣ ∣ ∣ x = p y -α = 1 p y -α , where 0 ≤ α ≤ D K -2 , and log( p y + D ) -log p y D = d log x dx ∣ ∣ ∣ x = p y + β = 1 p y + β , where 0 ≤ β ≤ D . And therefore:

So we have:

̸

Now, let p k = p ( k | x 2 ) , ∀ k ∈ { 1 , · · · , K } , D = p ( y | x 1 ) -p ( y | x 2 ) , and d k = p ( k | x 2 ) -p ( k | x 1 ) , k = j = y . Following eq. (39), we have:

̸

̸

̸

The eq. (40) holds because we have p ( j | x 1 ) = p ( j | x 2 ) , and log p ( y | x 1 ) > log p ( y | x 2 ) , where p ( y | x 1 ) > p ( y | x 2 ) . This completes the proof.

̸

̸

̸

̸

̸

̸

1

10

20

30

40

50

60

70

80

90

100

110

120

Epochs

0

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Gradient

Clean samples

Noisy samples

(a) CIFAR-10 under 0.6 symmetric noise

1

10

20

30

40

50

60

70

80

90

100

110

120

Epochs

0

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Gradient

Clean samples

Noisy samples

(b) CIFAR-10 under 0.8 symmetric noise

D Experiments

D.1 Empirical Understandings

Gradients of NNCE. We conduct experiments with ANL-CE on CIFAR-10 under 0.6 and 0.8 symmetric noise. For each sample, we calculate the gradient of our NNCE with respect to the predicted probability of each non-labeled class and take their maximum value, which is max j,j = y ( ∂ L NNCE ∂ p ( j | x ) ) . The results are shown in fig. 4 in the form of box plots. As can be observed, as the number of training epochs increases, the gradients of the clean samples are concentrated at larger values and the gradients of the noisy samples are concentrated at smaller values. The means of the gradients for noisy samples are higher than the medians and smaller than the means for clean samples' gradients. This indicates that although the model is still incorrectly learning some noisy samples, these learned noisy samples constitute only a small fraction of all noisy samples in the training set. In general, this result verifies that our NNLFs have the ability to focus more on clean samples and ignore noisy samples during the training process, and also verifies our discussion in section 3.3.

̸

Parameter Analysis. We apply different values of α and β to NCE+NNCE for training on CIFAR-10 under 0.8 symmetric noise. We test the combinations between α ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 } and β ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 } , where α is the weight of NCE and β is the weight of NNCE. The weight decay of NCE+NNCE is set to be the same as NCE+RCE. As can be observed in fig. 5, regardless of how α (the weight of NCE) varies, when β (the weight of NNCE) increases, both the robustness and the fitting ability of the model improve, although overfitting occurs. This verifies that our NNLFs are robust to noisy labels and have good fitting ability. Although in ANL we use L1 regularization instead of L2 regularization (weight decay), the effect of α and β in the loss functions created by ANL should be similar to that in NCE+NNCE.

̸

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

Clean Samples in Training Set

Test Set

(a)

α

= 0

.

1

, β

= 0

.

1

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(b)

α

= 0

.

1

, β

= 0

.

5

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(c)

α

= 0

.

1

, β

= 1

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(d)

α

= 0

.

1

, β

= 5

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(e)

α

= 0

.

1

, β

= 10

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(f)

α

= 0

.

5

, β

= 0

.

1

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(g)

α

= 0

.

5

, β

= 0

.

5

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(h)

α

= 0

.

5

, β

= 1

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(i)

α

= 0

.

5

, β

= 5

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(j)

α

= 0

.

5

, β

= 10

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(k)

α

= 1

.

0

, β

= 0

.

1

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(l)

α

= 1

.

0

, β

= 0

.

5

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(m)

α

= 1

.

0

, β

= 1

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(n)

α

= 1

.

0

, β

= 5

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(o)

α

= 1

.

0

, β

= 10

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(p)

α

= 5

.

0

, β

= 0

.

1

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(q)

α

= 5

.

0

, β

= 0

.

5

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(r)

α

= 5

.

0

, β

= 1

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(s)

α

= 5

.

0

, β

= 5

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(t)

α

= 5

.

0

, β

= 10

.

0

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

0

20

40

60

80

100

120

Epochs

0.0

0.2

0.4

0.6

0.8

(u)

α

= 10

.

0

, β

= 0

.

1

(v)

α

= 10

.

0

, β

= 0

.

5

(w)

α

= 10

.

0

, β

= 1

.

0

(x)

α

= 10

.

0

, β

= 5

.

0

(y)

α

= 10

.

0

, β

= 10

.

0

0

20

40

60

80

100

120

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Test Accuracy

ANL-CE

=1

e

7

=5

e

7

=1

e

6

=5

e

6

=1

e

5

=5

e

5

=1

e

4

(a) L2

0

20

40

60

80

100

120

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Test Accuracy

ANL-CE

=1

e

7

=5

e

7

=1

e

6

=5

e

6

=1

e

5

=5

e

5

=1

e

4

(b) L1

0

20

40

60

80

100

120

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Test Accuracy

ANL-CE

=1

e

7

=5

e

7

=1

e

6

=5

e

6

=1

e

5

=5

e

5

=1

e

4

(c) Jacobian

0

20

40

60

80

100

120

Epochs

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test Accuracy

pmin

=1

e

5

pmin

=1

e

6

pmin

=1

e

7

pmin

=1

e

8

pmin

=1

e

9

(a) CIFAR-10

0

50

100

150

200

Epochs

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Test Accuracy

pmin

=1

e

5

pmin

=1

e

6

pmin

=1

e

7

pmin

=1

e

8

pmin

=1

e

9

(b) CIFAR-100

Can APL outperform our ANL by changing the regularization method? We apply different regularization methods to NCE+RCE for training on CIFAR-10 under 0.8 symmetric noise and compare the results with our ANL-CE. Specifically, we train networks using loss functions in the form of α · NCE + β · RCE + δ · Reg, where α, β, δ > 0 are parameters and Reg is the regularization term. Similarly, we consider 3 regularization methods: 1) L2, 2) L1, and 3) Jacobian [15, 16]. We keep the α and β of NCE+RCE the same as in section 4.2 and tune δ ∈ { 1 × 10 -7 , 5 × 10 -7 , 1 × 10 -6 5 × 10 -6 1 × 10 -5 5 × 10 -5 1 × 10 -4 } for all three methods. As can be observed in fig. 6, the performance of NCE+RCE can be improved by adapting the regularization method. However, the improvement from changing the regularization method is limited, and the test accuracies of NCE+RCE are consistently lower than our ANL-CE, no matter how the regularization method is changed and how the parameter δ is varied. This verifies that APL cannot outperform our ANL by changing the regularization method and also verifies the performance of our proposed NNLFs.

Does the p min affect the performance of ANL? As shown in the definitions of NLF and NNLF, p min controls the lower bound of the classifier's output probability, and further controls the constant A . In fig. 7, we show different p min ∈ { 1 × 10 -5 , 1 × 10 -6 , 1 × 10 -7 , 1 × 10 -8 , 1 × 10 -9 } applied to ANL-CE for training on CIFAR-10/-100 under 0.6 symmetric noise. As can be observed, for both datasets, p min does not significant affect the performance of ANL.

D.2 Evaluation on Benchmark Datasets

Noise generation. The noisy labels are generated following standard approaches in previous works [28, 6, 17]. For symmetric noise, we flip the labels in each class randomly to incorrect labels of other classes. For asymmetric noise, we flip the labels within a specific set of classes. For MNIST, flipping 7 → 1, 2 → 7, 5 ↔ 6, 3 → 8. For CIFAR-10, flipping TRUCK → AUTOMOBILE, BIRD → AIRPLANE, DEER → HORSE, CAT ↔ DOG. For CIFAR-100, the 100 classes are grouped into 20 super-classes with each has 5 sub-classes, and each class are flipped within the same super-class into the next in a circular fashion. We vary the noise rate η ∈ { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } for symmetric noise and η ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } for asymmetric noise.

Networks and training. We follow the experimental setting in previous works [6, 17]. A 4-layer CNN is used for MNIST, an 8-layer CNN is used for CIFAR-10, and a ResNet-34 [29] is used for CIFAR-100. For MNIST, CIFAR-10, and CIFAR-100, the networks are trained for 50, 120, and 200 epochs, respectively. For all the training, we use SGD optimizer with momentum 0.9 and cosine learning rate annealing. Weight decay is set to 1 × 10 -3 , 1 × 10 -4 , and 1 × 10 -5 for MNIST, CIFAR-10, and CIFAR-100, respectively. Particularly, for our proposed ANL methods, weight decay is set to 0 for all datasets. The initial learning rate is set to 0 . 01 for MNIST/CIFAR-10 and 0 . 1 for CIFAR-100. Batch size is set to 128 . For all settings, we clip the gradient norm to 5 . 0 . Typical data augmentations including random width/height shift and horizontal flip are applied.

Parameter tuning. For each dataset, we tune the parameters of ANL-CE under 0.8 symmetric noise and then use them directly for all other noise settings and ANL functions. Specifically, we use 10% of the original training set as the validation set, and generate 0.8 symmetric noise on the remaining 90% of the original training set as the training set by the standard noise generation approach. We tune the parameters α ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 } , β ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 } , and δ ∈ { 1 × 10 -7 , 5 × 10 -7 , 1 × 10 -6 , 5 × 10 -6 , 1 × 10 -5 , 5 × 10 -5 , 1 × 10 -4 } .

Parameter settings. For all baseline methods, the parameters are set to match their original papers. Detailed parameter settings can be found in table 4.

Results. The experimental results of symmetric and asymmetric label noise are reported in table 6 and table 7, respectively.

D.3 Evaluation on Real-world Noisy Labels

Wetrain a ResNet-50 [29] using SGD for 250 epochs with initial learning rate 0.4, nesterov momentum 0.9 and weight decay 3 × 10 -5 (for our proposed ANL methods, the weight decay is set to 0 ) and batch size 512. The learning rate is multiplied by 0.97 after every epoch of training. For all settings, we clip the gradient norm to 5 . 0 . All the images are resized to 224 × 224 . Typical data augmentations including random width/height shift, color jittering and random horizontal flip are applied. Detailed parameter settings can be found in table 4.

D.4 More comparisons with other baseline methods

We also compare our proposed ANL with two other baseline methods, PHuber-CE [25] and TaylorCE [26]. For a fair comparison, we followed the experimental setup in the Taylor-CE paper and compared it directly with its reported experimental results. We use Adam [30] optimizer with the number of epochs set to 200 and the batch size set to 256. We use ResNet-34 for CIFAR-10/100. We tune our parameters and select the learning rate lr from { 10 -2 , 10 -3 , 10 -4 , 10 -5 } . For CIFAR-10 and CIFAR-100, we set the parameters ( α, β, δ, lr ) to (5 . 0 , 5 . 0 , 1 × 10 -7 , 1 × 10 -4 ) and (10 . 0 , 1 . 0 , 1 × 10 -7 , 1 × 10 -4 ) , respectively. The results in table 5 demonstrate that our proposed ANL can outperform both PHuber-CE and Taylor-CE on the CIFAR-10/-100 dataset with symmetric or asymmetric label noise.

D.5 Comparisons on more real-world datasets.

CIFAR-10N/-100N [20] are CIFAR-10/-100 equipped with human-annotated real-world noisy labels. CIFAR-10N contains five noisy label sets with noise rates of 9.03% (Aggregate), 17.23% (Random 1), 18.12% (Random 2), 17.64% (Random 3) and 40.21% (Worst). CIFAR-100N contains one noisy label set with noise rate of 40.20% (Noisy).

We use the same experimental settings and parameters as CIFAR-10 and CIFAR-100 for CIFAR-10N and CIFAR-100N, since the only difference is the noise label distribution. The results are reported in Table 8 and Table 9.

Animal-10N [21] is a real-world noisy data set of human-labeled online images for 10 confusing animals, with 50 , 000 training and 5 , 000 testing images, and its noise rate was estimated at 8%.

We follow the experimental setting in previous works [21]. We use VGG-19 with batch normalization. The SGD optimizer is employed. We train the network for 100 epochs and use an initial learning rate of 0.1, which is divided by 5 at 50% and 75% of the total number of epochs. Batch size is set to 128. Typical data augmentations including random horizontal flip are applied.

We compare our ANL-CE with CE and GCE. We use L2 regularization (weight decay) for GCE, and L1 regularization for ANL-CE. We denote the regularization coefficient by δ . We tune the parameters { δ } , { q, δ } , { α, β, δ } for CE, GCE and ANL-CE respectively. We use the best parameters { 1 × 10 -3 } , { 0 . 5 , 1 × 10 -4 } , { 0 . 5 , 1 . 0 , 1 × 10 -6 } for each method in our experiments. The results are reported in Table 10.

Moreover, we experiment with NCE+RCE on this dataset and tune the parameters { α, β, δ } , but we find that the performance is very poor for some unknown reason. The best test accuracy we achieve is 28.28% with { 10 . 0 , 0 . 1 , 5 × 10 -6 } . Since this result is too low and inconsistent with its performance on other datasets, we do not include it in the table for comparison.

Clothing-1M [22] is a large-scale clothing dataset contains 14 categories and 1 million training samples with nearly 40% mislabeled samples.

We follow the experimental setting in previous works [31]. We use the 14 k and 10 k clean data for validation and test, respectively, and we do not use the 50 k clean training data. We use ResNet-50 pre-trained on ImageNet. For preprocessing, we resized the images to 256 × 256 , performed mean subtraction, and cropped the middle 224 × 224 . We use SGD with a momentum of 0.9, a weight decay of 1 × 10 -3 , and batch size of 32. We train the network for 10 epochs with learning rate 1 × 10 -3 and 1 × 10 -4 for 5 epochs each. Typical data augmentations including random horizontal flip are applied.

Wecompare our ANL-CE with CE, GCE and NCE+RCE. In this experiment, we use L2 regularization (weight decay) for ANL-CE and set the coefficient the same as those for CE, GCE and NCE+RCE. We tune the parameters { q } , α, β and { α, β } for GCE, NCE+RCE and ANL-CE respectively. We use the best parameters { 0 . 6 } , { 10 . 0 , 1 . 0 } and { 5 . 0 , 0 . 1 } for each method in our experiments. The results are reported in Table 11.

Peer reviews:
comments: SUMMARY:
In this work, the authors propose a new class of theoretically robust passive loss functions named Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with the proposed NNLFs, this paper improves APL and proposes a new framework called Active Negative Loss (ANL).

STRENGTHS:
1. The motivation is clearly stated. The writing of this paper is also clear.
2. The theoretical study in this paper is solid.

WEAKNESSES:
1. More real-world datasets are needed for algorithm verification. 
2. This paper contains many typos.

QUESTIONS:
1. Some mathematical notations should be clearly defined, such as N in Section 2.1.
2. The authors should carefully proofread their paper. Some typos should be corrected, such as "at at least one other class position...". In supplementary material, the equation index is missing in "So, from Eq () we have..." above Eq. 25.
3. This paper only investigates the proposed method on one real-world dataset, namely WebVision, which I think is not adequate. In fact, there are many benchmark datasets with real-world noise, such as Animal-10N, Clothing-1M, etc.

LIMITATIONS:
See my comments above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a robust loss function for learning from label noise. The authors first find that negative losses proposed in APL are scaled versions of MAE, which is not training-friendly. To solve this problem, the authors propose the Active Negative Loss (ANL) framework which improves the APL loss by replacing the passive losses with a newly designed set of negative losses. Theoretically, the ANL losses are proved to be noise tolerant and the designed negative loss is shown to focus more on the well-learned classes and samples compared with MAE. Experimentally, the ANL framework outperforms the state-of-the-art robust losses on multiple datasets.\n\nSTRENGTHS:\n- This paper is well-written and easy to read.\n- This paper provides new insights into the negative losses proposed in APL.\n- The proposed loss framework is provably robust and its gradient is theoretically understood.\n- The experiments are extensive and sound.\n\nWEAKNESSES:\n- In Section 3.1, the motivation for incorporating the three components in NNLF is lacking. Why do we need the three components in NNLF? And why do the three components make NNLF a good robust loss function?\n- The theoretical analysis is less original. Most discussions of the relationship between symmetric loss function and noise tolerance can be found in previous works, e.g. APL. Accordingly, the proofs of Theorems 2, 3, and 4 could be simplified.\n- Typos. Lines 196, 201, 220. \"theorem\" should start with a capital T.\n\nQUESTIONS:\n- What motivates the three components of NNLF?\n- In Eq. (7), is $A$ a constant over all $p(k|x)$? If not, the derivation of gradients in Eq. (15) should consider the dependency of $A$ on $p(k|x)$. \n- The proposed ANL framework performs less well on the MNIST dataset (Tables 2, 6, 7). Are there any explanations?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes robust loss function to improve training of DNNs using noisy labels. Previously proposed work of Active Passive Loss functions is improved.\n\nSTRENGTHS:\nPaper is well written and contains theoretical foundation of the proposed work. There are a lot of derivations which give a broad understanding of loss functions. Table 2 shows improved performance of the proposed loss functions compared to [6] and [18] in some cases. Though the improvement is not consistent.\n\nWEAKNESSES:\n1. The  terminology used in this paper is mainly based on previous work of reference [6]. The terms like active and passive loss functions do not clearly separate the loss functions. As shown in the appendix, MAE=1(1-p(y|x)), shows that MAE also aims to maximise p(y|x) therefore it is also an active loss function. In [6], MAE was considered as a passive loss function which is not correct Mathematically in cases when we are dealing with a prediction as \\sum_k{p(k|x)}=1. This paper also inherits the same definitions from [6]. \n\n2. Changing the sign of a loss function also changes the max to min. If we are maximising L, we should be minimising -L. Therefore change of sign may not be considered making any fundamental difference. The term `Vertical Flip' is. derived from the graph. Mathematically, there is no thing like a vertical flip. \n\n3. Definition of A in Eq (7) is not clear. How a loss can be computed between a vector [ . .........p_min, .........] and y ? In the appendix, A is defined as a constant value. \n\n4. As shown in the appendix, the proposed normalisation  by [6] is  scaling and the same is inherited in the current work. If all loss values are scaled by the same number then how it will be robust to noise? The loss generated by the noisy and the clean samples will be scaled equally and noisy labels are not known at the time of training. \n\n5. In Eq 9, denominator can be simplified as: \\sum_k{A+ log p(k|x)}=KA+1, reducing Eq 9 to: NNFL=1-A/(KA+1)-logp(y|x)/(1+KA) . It is again a scaled version of the original function added a constant value. What fundamental change makes it robust to noise? \n\n6. The range of asymmetric noise is kept quite small {.2,.3.4} compared to symmetric noise {.4,.6,.8} in Table 2. A larger range would have revealed more insights.\n\nQUESTIONS:\nThe same as above.\n\nLIMITATIONS:\nMore theoretical insights are required for the proposed loss functions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel type of loss function called Active Negative Loss (ANL), which builds upon the Active Passive loss function (APL) framework. The authors identify a limitation in APL, where the passive loss function, being a scaled version of Mean Absolute Error (MAE), can lead to slower convergence and underfitting issues. To address this, the authors propose replacing the passive loss in APL with a normalized negative loss, drawing inspiration from negative learning and vertical flipping techniques. The paper also provides theoretical justifications for the superiority of ANL over APL. The proposed ANL framework is evaluated through experiments conducted on CIFAR10 and CIFAR100 datasets with synthetic label noise, as well as ILSVRC12 and WebVision datasets with real-world label noise.\n\nSTRENGTHS:\n1. The motivation is clearly articulated and accompanied by well-explained reasoning.\n\n2. The proposed ANL framework can be considered an enhanced version of the APL framework, offering improved performance with theoretical backing. Specifically, it demonstrates robustness against both symmetric and asymmetric label noise under certain conditions.\n\n3. Code is provided to facilitate reproducibility.\n\nWEAKNESSES:\n1. Although the proposed loss is theoretically sound, many of the techniques or analyses employed are not very new. The ANL framework appears to be a simple combination of the NL loss [R1] and APL loss [R2]. While I acknowledge the validity of this combination, I do not believe it constitutes a significant contribution to the LNL community.\n\n2. The authors mention that regularization, such as L1 regularization, is applied to the ANL framework. I am curious to know if other approaches in your experiments also utilize this regularization. Additionally, the absence of ablation studies investigating the impact of regularization on the ANL framework is worth considering.\n\n3. I believe conducting further experiments to verify the effectiveness of the ANL framework is necessary. For instance, experiments involving CIFAR with controlled instance-dependent label noise, CIFAR-10N, CIFAR-100N, and the Clothing1m dataset would provide valuable insights.\n\n\n[R1] NLNL: Negative Learning for Noisy Labels\n\n[R2] Normalized Loss Functions for Deep Learning with Noisy Labels\n\nQUESTIONS:\nSee **weaknesses**\n\nLIMITATIONS:\nI think this work does not bring negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}