SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xichen Ye
Shanghai University
yexichen0930@shu.edu.cn
Xiaoqiang Li *
xqli@shu.edu.cn
Songmin Dai
laodar@shu.edu.cn
Tong Liu
tong.liu@shu.edu.cn
Yan Sun
yansun@shu.edu.cn
Weiqin Tong
wqtong@shu.edu.cn
Robust loss functions are essential for training deep neural networks in the presence of noisy labels. Some robust loss functions use Mean Absolute Error (MAE) as its necessary component. For example, the recently proposed Active Passive Loss (APL) uses MAE as its passive loss function. However, MAE treats every sample equally, slows down the convergence and can make training difficult. In this work, we propose a new class of theoretically robust passive loss functions different from MAE, namely Normalized Negative Loss Functions (NNLFs), which focus more on memorized clean samples. By replacing the MAE in APL with our proposed NNLFs, we improve APL and propose a new framework called Active Negative Loss (ANL). Experimental results on benchmark and real-world datasets demonstrate that the new set of loss functions created by our ANL framework can outperform state-of-the-art methods. The code is available at https://github.com/Virusdoll/Active-Negative-Loss.
Relying on large-scale datasets with high quality annotations, such as ImageNet [1], deep neural networks (DNNs) achieve good performance in various supervised classification tasks. However, in practice, the process of labeling large-scale datasets is costly and inevitably introduces noisy (mislabeled) samples. Moreover, empirical studies show that over-parameterized DNNs can easily fit a randomly labeled dataset [2], which implies that DNNs may have a poor evaluation performance when trained on a noisy dataset. As a result, noisy label learning has received a lot of attention.
Different approaches have been proposed to solve the noisy label learning problem, and one popular research line is to design noise-robust loss functions, which is also the main focus of this paper. Ghosh et al. [3] have theoretically proved that, symmetric loss functions such as Mean Absolute Error (MAE), are robust to noise, while others like commonly used Cross Entropy (CE) are not. However, MAE treats every sample equally, leading to significantly longer training time before convergence and even making learning difficult, which suggests that MAE is not suitable for training DNNs with challenging datasets [4]. Motivated by this, several works proposed partially robust loss functions, including Generalized Cross Entropy (GCE) [4], a generalized mixture of CE and MAE, and Symmetric Cross Entropy (SCE) [5], a combination of CE and a scaled MAE, Reverse Cross
*Corresponding Author
Entropy (RCE). Recently, Active Passive Loss (APL) [6] framework has been proposed to create fully robust loss functions.
APL is one of the state-of-the-art methods, and shows that any loss function can be made robust to noisy labels by a simple normalization operation. Moreover, to address the underfitting problem of normalized loss functions, APL first characterizes existing loss functions into two types and then combines them. The two types are 1) "Active" loss, which only explicitly maximizes the probability of being in the labeled class, and 2) "Passive" loss, which also explicitly minimizes the probabilities of being in other classes. However, by investigating several robust loss functions created by the APL framework, we find that their passive loss functions are always scaled versions of MAE. As we mentioned before, MAE is not conducive to training. As a result, to address the underfitting problem, APL combines active loss functions with MAE, which again may lead to difficulty in training and further limits its performance.
The fact that APL still struggles with MAE motivates us to investigate new robust passive loss functions. In this paper, we propose a new class of passive loss functions different from MAE, called Negative Loss Functions (NLFs). We show that, by combining 1) complementary label learning [7, 8] and 2) a simple "vertical flipping" operation, any active loss function can be made into a passive loss function. Moreover, to make it theoretically robust to noisy labels, we further apply the normalization operation on NLFs to obtain Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with NNLF, we propose a novel framework called Active Negative Loss (ANL). ANL combines a normalized active loss function and a NNLF to build a new set of noise-robust loss functions, which can be seen as an improvement of APL. We show that under our proposed ANL framework, several commonly-used loss functions can be made robust to noisy labels while ensuring sufficient fitting ability to achieve state-of-the-art performance for training DNNs with noisy datasets. Our key contributions are highlighted as follows:
- We provide a method to build a new class of robust passive loss functions called Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with our proposed NNLFs, we propose a novel framework, Active Negative Loss (ANL), to construct a new set of robust loss functions.
- We demonstrate the theoretical robustness of our proposed NNLFs and ANL to noisy labels, and discuss how replacing the MAE in APL with our NNLFs enhances performance in noisy label learning.
- Our empirical results show that the new set of loss functions, created using our proposed ANL framework, outperform existing state-of-the-art methods.
2.1 Risk Minimization and Label Noise Model
Consider a typical K-class classification problem. Let  $\mathcal{X} \subset \mathbb{R}^d$  be the  $d$ -dimensional feature space from which the samples are drawn, and  $\mathcal{Y} = [k] = \{1, \dots, K\}$  be the label space. Given a clean training dataset,  $\mathcal{S} = \{(x_n, y_n)\}_{n=1}^N$ , where each  $(x_n, y_n)$  is drawn i.i.d. from an unknown distribution,  $\mathcal{D}$ , over  $\mathcal{X} \times \mathcal{Y}$ . We denote the distribution over different labels for sample  $x$  by  $\pmb{q}(k|x)$ , and  $\sum_{k=1}^{K} \pmb{q}(k|x) = 1$ . Since there is only one corresponding label  $y$  for a  $x$ , we have  $\pmb{q}(y|x) = 1$  and  $\pmb{q}(k \neq y|x) = 0$ .
A classifier,  $h(\pmb{x}) = \arg \max_{i} f(\pmb{x})_i$ , where  $f: \mathcal{X} \to \mathcal{C}, \mathcal{C} \subseteq [0,1]^K, \forall \pmb{c} \in \mathcal{C}, \mathbf{1}^T\pmb{c} = 1$ , is a function that maps feature space to label space. In this work, we consider  $f$  as a DNN ending with a softmax output layer. For each sample  $\pmb{x}$ ,  $f(\pmb{x})$  computes its probability  $p(k|\pmb{x})$  of each label  $k \in \{1,\dots ,K\}$  and  $\sum_{k=1}^{K} p(k|\pmb{x}) = 1$ . Throughout this paper, as a notation, we call  $f$  itself the classifier. Training a classifier  $f$  is to find an optimal classifier  $f^*$  that minimizes the empirical risk defined by a loss function:  $\sum_{n=1}^{N}\mathcal{L}(f(\pmb{x}_n),y_n)$ , where  $\mathcal{L}: \mathcal{C} \times \mathcal{V} \to \mathbb{R}^+$  is a loss function, and  $\mathcal{L}(f(\pmb{x}),k)$  is the loss of  $f(\pmb{x})$  with respect to label  $k$ .
When label noise is present, our model can only access a corrupted dataset  $S_{\eta} = \{(x_{n},\hat{y}_{n})\}_{n = 1}^{N}$ , where each sample is drawn i.i.d. from an unknown distribution,  $\mathcal{D}_{\eta}$ . In this paper, we consider a popular approach for modeling label noise, which simply assumes that, given the true label  $y$ , the
corruption process is conditionally independent of input features  $\pmb{x}$  [9]. So we can formulate noisy label  $\hat{y}$  as:
$$
\hat {y} = \left\{ \begin{array}{l l} y & \text {w i t h p r o b a b i l i t y} (1 - \eta_ {y}) \\ j, j \in [ k ], j \neq y & \text {w i t h p r o b a b i l i t y} \eta_ {y j} \end{array} , \right. \tag {1}
$$
where  $\eta_{yj}$  denotes the probability that true label  $y$  is corrupted into label  $j$ , and  $\eta_y = \sum_{j\neq y}\eta_{yj}$  denotes the noise rate of label  $y$ . Under our assumption of label noise model, label noise can be either symmetric or asymmetric. The noise is called symmetric, if  $\eta_{ij} = \frac{\eta_i}{K - 1},\forall j\neq y$  and  $\eta_{i} = \eta ,\forall i\in [k]$  where  $\eta$  is a constant. And for asymmetric noise,  $\eta_{ij}$  is conditioned on both the true label  $i$  and corrupted label  $j$ .
2.2 Active Passive Loss Functions
Ghosh et al. [3] have shown, under some mild assumptions, a loss function  $\mathcal{L}$  is noise tolerant if it is symmetric:  $\sum_{k=1}^{K} \mathcal{L}(f(\boldsymbol{x}), k) = C, \forall \boldsymbol{x} \in \mathcal{X}$ , where  $C$  is some constant. Based on this, Ma et al. [6] proposed the normalized loss functions, which normalize a loss function  $\mathcal{L}$  by:
$$
\mathcal {L} _ {\text {n o r m}} = \frac {\mathcal {L} (f (\boldsymbol {x}) , y)}{\sum_ {k = 1} ^ {K} \mathcal {L} (f (\boldsymbol {x}) , k)}. \tag {2}
$$
This simple normalization operation can make any loss function robust to noisy labels, since we always have  $\sum_{k}^{K}\mathcal{L}_{\mathrm{norm}} = 1$ . For example, the Normalized Cross Entropy (NCE) is:
$$
N C E = \frac {\sum_ {k = 1} ^ {K} \boldsymbol {q} (k | \boldsymbol {x}) (- \log \boldsymbol {p} (k | \boldsymbol {x}))}{\sum_ {j = 1} ^ {K} \sum_ {k = 1} ^ {K} \boldsymbol {q} (y = j | \boldsymbol {x}) \log \boldsymbol {p} (k | \boldsymbol {x})}. \tag {3}
$$
Similarly, we can normalize FL, MAE, and RCE to obtain Normalized Focal Loss (NFL), Normalized Mean Absolute Error (NMAE), and Normalized Reverse Cross Entropy (NRCE), respectively.
But a normalized loss function alone suffers from the underfitting problem. To address this, Ma et al. [6] characterize existing loss functions into two types: Active and Passive. Denote the function of loss  $\mathcal{L}(f(\pmb{x}),y)$  by  $\ell(f(\pmb{x}),k)$ , that is  $\mathcal{L}(f(\pmb{x}),y) = \sum_{k=1}^{K}\ell(f(\pmb{x}),k)$  (e.g., let  $\mathcal{L}$  be CE, then  $\mathcal{L}(f(\pmb{x}),y) = \sum_{k=1}^{K}\pmb{q}(k|\pmb{x})(-\log\pmb{p}(k|\pmb{x}))$ , and  $\ell(f(\pmb{x}),k) = \pmb{q}(k|\pmb{x})(-\log\pmb{p}(k|\pmb{x}))$ ), we have the following definitions:
Definition 1 (Active loss function).  $\mathcal{L}_{\mathrm{Active}}$  is an active loss function if  $\forall (\pmb {x},y)\in \mathcal{D},\forall k\neq y,\ell (f(\pmb {x}),k) = 0.$
Definition 2 (Passive loss function).  $\mathcal{L}_{\mathrm{Passive}}$  is a passive loss function if  $\forall (\pmb {x},y)\in \mathcal{D},\exists k\neq y,\ell (f(\pmb {x}),k)\neq 0.$
Active loss functions only explicitly maximize  $\pmb{p}(y|\pmb{x})$ , the classifier's output probability at the class position specified by the label  $y$ . In contrast, passive loss functions also explicitly minimize  $\pmb{p}(k \neq y|\pmb{x})$ , the probability at least one other class positions. Accordingly, the active loss functions include CE, FL, NCE, and NFL, while the passive loss functions include MAE, RCE, NMAE, and NRCE. These two types of loss functions can mutually boost each other to mitigate underfitting, and we refer the reader to [6] for more detailed discussions. By combining them, Ma et al. proposed the Active Passive Loss (APL):
$$
\mathcal {L} _ {\mathrm {A P L}} = \alpha \cdot \mathcal {L} _ {\text {A c t i v e}} + \beta \cdot \mathcal {L} _ {\text {P a s s i v e}}, \tag {4}
$$
where  $\alpha, \beta > 0$  are parameters. As an example, by combining NCE and RCE, Ma et al get NCE+RCE, one of the state-of-the-art methods.
2.3 APL struggles with MAE
As shown in the previous subsection, there are four passive loss functions available to APL, including MAE, RCE, NMAE, and NRCE. However, we can show that all these passive loss functions are scaled versions of MAE. Specifically,  $NMAE = \frac{1}{2(K - 1)}\cdot MAE$ ,  $RCE = -\frac{A}{2}\cdot MAE$ , and  $NRCE = \frac{1}{2(K - 1)}\cdot MAE$  (detailed derivations can be found in appendix A.1). Thus, we can rewrite APL as follows:
$$
\mathcal {L} _ {\mathrm {A P L}} = \alpha \cdot \mathcal {L} _ {\text {A c t i v e}} + \beta \cdot M A E. \tag {5}
$$
Figure 1: "Vertical flipping" operation.  $\mathcal{L}(f(\pmb{x}), k)$  is an active loss function, e.g., CE. Loss function  $A - \mathcal{L}(f(\pmb{x}), k)$  is obtained by flipping  $\mathcal{L}(f(\pmb{x}), k)$  vertically with axis loss  $= \frac{1}{2} A$ .
This indicates that MAE is a necessary component of the current APL. However, as we mentioned before, MAE requires longer training time and even makes learning difficult. Thus, on the one hand, APL needs passive loss functions to mitigate active loss functions underfitting, yet on the other hand, MAE is not training friendly, which may limit the performance of APL. This motivates us to investigate new robust passive loss functions.
3 Active Negative Loss Functions
3.1 Method
Normalized Negative Loss Functions. Our goal is to find a method that creates robust passive loss functions from existing active loss functions. This method must consist of three components that: 1) let the loss function optimize the classifier's output probability for at least one other class position that is not specified by the label  $y$ , 2) let the loss function minimize the classifier's output probability instead of maximizing it, and 3) let the loss function robust to noisy labels. Inspired by NLNL [10] and APL [6], we use 1) complementary label learning, 2) "vertical flipping" and 3) normalization operation as these three components respectively.
Complementary label learning [7, 8] is an indirect learning method for training CNNs, which randomly selects complementary labels and trains the CNN to recognize that "input does not belong to this complementary label". Unlike the usual training approach, complementary label learning focuses on the loss of classifier's predictions with complementary labels, which naturally fits with the passive loss function. Here, we only use its basic idea of letting the loss function focus on all classes  $\{1,\dots ,K\}$  except the labeled class  $y$
"Vertical flipping" is a simple operation that can convert the loss function from "maximizing" to "minimizing". As shown in the fig. 1, given an active loss function  $\mathcal{L}(f(\pmb{x}), k)$ , the new loss function  $A - \mathcal{L}(f(\pmb{x}), k)$  is obtained by flipping  $\mathcal{L}(f(\pmb{x}), k)$  vertically with axis loss =  $\frac{1}{2} A$ . It should be noted that,  $A - \mathcal{L}(f(\pmb{x}), k)$  is the opposite of  $\mathcal{L}(f(\pmb{x}), k)$ , and it focuses on optimizing  $\pmb{p}(k|\pmb{x})$  to 0.
Based on these two components, given an active loss function  $\mathcal{L}$ , we propose Negative Loss Functions (NLFs) as follows:
$$
\mathcal {L} _ {\text {n e g}} (f (\boldsymbol {x}), y) = \sum_ {k = 1} ^ {K} (1 - \boldsymbol {q} (k | \boldsymbol {x})) (A - \mathcal {L} (f (\boldsymbol {x}), k)), \tag {6}
$$
$$
A = \mathcal {L} \left(\left[ \dots , \boldsymbol {p} _ {\min }, \dots \right] ^ {T}, y\right). \tag {7}
$$
Here,  $[\dots ,\pmb{p}_{\min},\dots ]^T$  is some probability distribution that may be output by the classifier  $f$ , where  $\pmb {p}(y|\pmb {x}) = \pmb{p}_{\mathrm{min}}$ , the minimum value of  $\pmb {p}(k|\pmb {x})$  (e.g., 0). Therefore  $A$  is some constant, the maximum loss value of  $\mathcal{L}$ . In practice, setting  $p_{\mathrm{min}} = 0$  could cause some computational problems, for example, if  $\mathcal{L}$  is CE and  $p_{\mathrm{min}} = 0$ , then  $A = -\log 0 = +\infty$ . So in this paper, unless otherwise specified, we define  $\pmb{p}_{\mathrm{min}} = 1\times 10^{-7}$ . This technique is similar to the clipping operation implemented by most deep learning frameworks.
Our proposed NLF can transform any active loss function into a passive loss function, where 1)  $(1 - q(k|x))$  ensures that the loss function focuses on classes  $\{1,\dots ,K\} \setminus \{y\}$ , and 2)  $(A - \mathcal{L}(f(\pmb {x}),k))$  ensures that the loss function aims to minimize the output probability  $p(k|x)$ .
Next, to make our proposed passive loss functions robust to noisy labels, we perform a normalization operation on NLFs. Given an active loss function  $\mathcal{L}$ , we propose Normalized Negative Loss Functions (NNLFs) as follows:
$$
\mathcal {L} _ {\mathrm {n n}} (f (\boldsymbol {x}), y) = 1 - \frac {A - \mathcal {L} (f (\boldsymbol {x}) , y)}{\sum_ {k = 1} ^ {K} A - \mathcal {L} (f (\boldsymbol {x}) , k)}, \tag {8}
$$
where  $A$  has the same definition as eq. (7). The detailed derivation of NNLFs can be found in appendix A.2. Additionally, NNLFs have the property that  $\mathcal{L}_{\mathrm{nn}} \in [0,1]$ . Accordingly, we can create NNLFs from active loss functions as follows.
The Normalized Negative Cross Entropy (NNCE) is:
$$
N N C E = 1 - \frac {A - (- \log \boldsymbol {p} (y | \boldsymbol {x}))}{\sum_ {k = 1} ^ {K} A - (- \log \boldsymbol {p} (k | \boldsymbol {x}))}, \tag {9}
$$
where  $A = -\log p_{\mathrm{min}}$
The Normalized Negative Focal Loss (NNFL) is:
$$
N N F L = 1 - \frac {A - \left(- (1 - \boldsymbol {p} (y | \boldsymbol {x})) ^ {\gamma} \log \boldsymbol {p} (y | \boldsymbol {x})\right)}{\sum_ {k = 1} ^ {K} A - \left(- (1 - \boldsymbol {p} (k | \boldsymbol {x})) ^ {\gamma} \log \boldsymbol {p} (k | \boldsymbol {x})\right)}, \tag {10}
$$
where  $A = -(1 - \pmb{p}_{\mathrm{min}})^{\gamma}\log \pmb{p}_{\mathrm{min}}$
ANL Framework. We can now create new robust loss functions by replacing the MAE in APL with our proposed NNLF. Given an active loss function  $\mathcal{L}$ , we propose the Active Negative Loss (ANL) functions as follows:
$$
\mathcal {L} _ {\mathrm {A N L}} = \alpha \cdot \mathcal {L} _ {\text {n o r m}} + \beta \cdot \mathcal {L} _ {\mathrm {n n}}. \tag {11}
$$
Here,  $\alpha$  and  $\beta$  are parameters greater than 0,  $\mathcal{L}_{\mathrm{norm}}$  denotes the normalized  $\mathcal{L}$  and  $\mathcal{L}_{\mathrm{nn}}$  denotes the Normalized Negative Loss Function corresponding to  $\mathcal{L}$ . Accordingly, we can create ANL from the two mentioned active loss functions as follows.
For Cross Entropy (CE), we have ANL-CE:
$$
\mathcal {L} _ {\mathrm {A N L - C E}} = \alpha \cdot \mathcal {L} _ {\mathrm {N C E}} + \beta \cdot \mathcal {L} _ {\mathrm {N N C E}}. \tag {12}
$$
For Focal Loss (FL), we have ANL-FL:
$$
\mathcal {L} _ {\mathrm {A N L - F L}} = \alpha \cdot \mathcal {L} _ {\mathrm {N F L}} + \beta \cdot \mathcal {L} _ {\mathrm {N N F L}}. \tag {13}
$$
3.2 Robustness to noisy labels
NNLFs are symmetric. We first prove that our proposed Normalized Negative Loss Functions (NNLFs) are symmetric. Detailed proofs can be found in appendix B.
Theorem 1. Normalized negative loss function  $\mathcal{L}_{nn}$  is symmetric.
NNLFs are robust to noise. In reference to Theorem 1 and Theorem 3 from [3], it has been proven that symmetric loss functions, under some mild assumptions, exhibit noise tolerant in the face of both symmetric and asymmetric noise. Given that our NNLFs fall under the category of symmetric loss functions, they inherently possess the attribute of noise tolerant.
ANL is robust to noise. In light of Lemma 3 from [6], it is understood that the combination of two noise tolerant loss functions retains the noise tolerant attribute. It is noteworthy that both  $\mathcal{L}_{\mathrm{norm}}$  and  $\mathcal{L}_{\mathrm{nn}}$  within our ANL are noise tolerant, which makes ANL as a whole noise tolerant.
3.3 NNLFs focus more on well-learned samples
As shown in the fig. 2, by replacing MAE with our proposed NNCE, NCE+NNCE and ANL-CE show better fitting ability. This raises the question: why does NNLF perform better than MAE? In the following, taking NNCE as an example, we analysis the gradients of MAE and NNCE to provide a preliminary answer to this question. Detailed derivations and proofs can be found in appendix C.
The gradient of the MAE with respect to the classifier's output probability can be derived as:
$$
\frac {\partial \mathcal {L} _ {\mathrm {M A E}}}{\partial \boldsymbol {p} (j | \boldsymbol {x})} = \left\{ \begin{array}{c c} 1, & j \neq y \\ - 1, & j = y. \end{array} \right. \tag {14}
$$
(a) CE
(b) MAE
(c) NCE+RCE
(d) ANL-CE (w/ L2)
(e) ANL-CE (w/L1)
(f) CE
(g) MAE
(h) NCE+RCE
Figure 2: Training and test accuracies of different loss functions. (a) - (e): CIFAR-10 under 0.8 symmetric noise. (f) - (j): CIFAR-100 under 0.6 symmetric noise. The accuracies of noisy samples in training set should be as low as possible, since they are mislabeled.
(i) ANL-CE (w/ L2)
(j) ANL-CE (w/ L1)
The gradient of the NNCE with respect to the classifier's output probability can be derived as:
$$
\frac {\partial \mathcal {L} _ {\mathrm {N N C E}}}{\partial \boldsymbol {p} (j | \boldsymbol {x})} = \left\{ \begin{array}{c c} \frac {1}{\boldsymbol {p} (j | \boldsymbol {x})} \cdot \frac {A + \log \boldsymbol {p} (y | \boldsymbol {x})}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x})\right) ^ {2}}, & j \neq y \\ - \frac {1}{\boldsymbol {p} (y | \boldsymbol {x})} \cdot \frac {\sum_ {k \neq y} A + \log \boldsymbol {p} (k | \boldsymbol {x})}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x})\right) ^ {2}}, & j = y. \end{array} \right. \tag {15}
$$
For the purpose of analysis, we consider how the gradients of NNCE would differ from MAE in the following two scenarios: 1) given the classifier's output probability of sample  $\pmb{x}$ , we analyze the difference in gradient for each class, 2) given the classifier's output probabilities of sample  $\pmb{x}_1$  and  $\pmb{x}_2$ , we analyze the difference in gradient between these two samples.
Theorem 2. Given the classifier's output probability  $\pmb{p}(\cdot|\pmb{x})$  for sample  $\pmb{x}$  and normalized negative cross entropy  $\mathcal{L}_{NNCE}$ . If  $\pmb{p}(j_1|\pmb{x}) < \pmb{p}(j_2|\pmb{x})$ ,  $j_1 \neq j_2 \neq y$ , then  $\frac{\partial\mathcal{L}_{NNCE}}{\partial\pmb{p}(j_1|\pmb{x})} > \frac{\partial\mathcal{L}_{NNCE}}{\partial\pmb{p}(j_2|\pmb{x})}$ .
Theorem 3. Given the classifier's output probabilities  $\pmb{p}(\cdot|\pmb{x}_1)$  and  $\pmb{p}(\cdot|\pmb{x}_2)$  of sample  $\pmb{x}_1$  and  $\pmb{x}_2$ , where  $\pmb{p}(y|\pmb{x}_1) \geq \pmb{p}(k|\pmb{x}_1)$ ,  $\pmb{p}(y|\pmb{x}_2) \geq \pmb{p}(k|\pmb{x}_2)$ ,  $\forall k \in \{1, \dots, K\}$ ,  $\pmb{p}(j|\pmb{x}_1) = \pmb{p}(j|\pmb{x}_2)$ ,  $j \neq y$ , and normalized negative cross entropy  $\mathcal{L}_{NNCE}$ . If  $\pmb{p}(y|\pmb{x}_1) > \pmb{p}(y|\pmb{x}_2)$  and  $\pmb{p}(k|\pmb{x}_1) \leq \pmb{p}(k|\pmb{x}_2)$ ,  $\forall k \in \{1, \dots, K\} \setminus \{j, y\}$ , then  $\frac{\partial\mathcal{L}_{NNCE}}{\partial\pmb{p}(j|\pmb{x}_1)} > \frac{\partial\mathcal{L}_{NNCE}}{\partial\pmb{p}(j|\pmb{x}_2)}$ .
theorem 2 and theorem 3 demonstrate that, for the gradient of the non-labeled classes, our NNCE focuses more on the classes and samples that have been well learned compared to MAE, which treats every class and sample equally. This property may enhance the model's performance in noisy label learning. Some studies [11] have shown that during the training process, DNNs would first memorize clean samples and then noisy samples. According to the property revealed by theorem 2 and theorem 3, apart from robustness, our NNLF may potentially help the model to continuously learn the clean samples that the model has memorized in the previous stages of training and ignore the unmemorized noisy samples.
In this section, we empirically investigate our proposed ANL functions on benchmark datasets, including MNIST [12], CIFAR-10/CIFAR-100 [13] and a real-world noisy dataset WebVision [14].
4.1 Empirical Understandings
In this subsection, we explore some properties of our proposed loss functions. Unless otherwise specified, all detailed experimental settings are the same as those in section 4.2. More experiments and discussions about gradient, parameter analysis, and  $p_{\mathrm{min}}$  can be found in the appendix D.1.
(a) w/o regularizar
(b) L2
Figure 3: Test accuracies of ANL-CE on CIFAR-10 under 0.8 symmetric noise with different regularization methods and different parameters.  $\delta$  is the weight of regularization term of ANL-CE.
(c) Jacobian
(d) L1