SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Test accuracies  $(\%)$  of different methods on CIFAR-10 datasets with clean and symmetric  $(\eta \in \{0.2,0.4,0.6,0.8\})$  label noise. The top-1 best results are in bold.
Methods	Clean (η = 0.0)	η = 0.2	η = 0.4	η = 0.6	η = 0.8
NCE	88.68	81.65	74.80	63.14	37.52
NNCE	91.51	90.09	86.91	82.16	57.06
ANL-CE	91.66±0.04	90.02±0.23	87.28±0.02	81.12±0.30	61.27±0.55
Overfitting problem. In practice, we find that ANL can lead to overfitting in some experimental settings. To motivate this problem, as an example, we train networks using different loss functions on CIFAR-10 under 0.8 symmetric noise and CIFAR-100 under 0.6 symmetric noise, and the experimental results are shown in fig. 2. As can be observed, in the setting of CIFAR-10 under 0.8 symmetric noise, the training set accuracy of ANL-CE (w/ L2) keeps increasing while the test set accuracy keeps decreasing. We identify this problem as an overfitting problem.
It is worth noting that although overfitting occurs, unlike CE, the gap between the clean sample accuracies and the noisy sample accuracies of the training set does not shrink, which indicates that our ANL has some robustness to noisy labels even in the case of overfitting. Moreover, we conjecture that the overfitting is caused by the property of NNLF focusing more on well-learned samples. When the noise rate is high, one might assume that the model has been trained on only a fairly small number of clean samples, with a very large gradient, which can lead to overfitting.
The choice of regularization method. We also find that the commonly used L2 regularization may struggle to mitigate the overfitting problem. To address this, we decide to try using other regularization methods. We consider 2 other regularization methods: L1 and Jacobian [15, 16]. To compare the performance of these methods, we apply them to ANL-CE for training on CIFAR-10 under 0.8 symmetric noise. For simplicity and fair comparison, we use  $\delta$  as the coefficient of the regularization term and consider it as an additional parameter of ANL. We keep  $\alpha$  and  $\beta$  the same as in section 4.2, tune  $\delta$  for all three methods by following the parameter tuning setting in appendix D.2. We also train networks without using any regularization method. The results reported in fig. 3. As can be observed, among the three regularization methods, only L1 can somewhat better mitigate the overfitting problem. If not otherwise specified, all ANLs in this paper use L1 regularization.
Robustness and fitting ability. We conduct a set of experiments on CIFAR-10/-100 to verify the robustness and fitting ability of our proposed loss functions. We set the noise type to be symmetric and the noise rate to 0.8 for CIFAR-10 and 0.6 for CIFAR-100. In each setting, we train the network using different loss functions, including: 1) CE, 2) MAE, 3) NCE+RCE, 4) ANL-CE (w/ L2), and 5) ANL-CE (w/ L1). For ANL-CE (w/ L2), we set its parameters  $\alpha$  and  $\beta$  to be the same as ANL-CE (w/ L1) and set its weight decay to be the same as NCE+RCE.
As can be observed in fig. 2: 1) CE is not robust to noise, the accuracies of clean and noisy samples in the training set are continuously close to each other, 2) MAE is robust to noise, the accuracies of clean and noisy samples in the training set keep moving away from each other, but its fitting ability is insufficient, especially when the dataset becomes complex, 3) NCE+RCE is robust to noise and has better fitting ability compared to MAE, 4) ANL-CE (w/ L2) is robust to noise and has stronger fitting ability, but suffers from over-fitting. and 5) ANL-CE is robust to noise and mitigates the impact of overfitting to achieve the best performance. To summarize, our proposed loss functions are robust to noise, NNLF shows better fitting ability than MAE, and L1 regularization addresses the overfitting problem of NNLF.
Table 2: Test accuracies  $(\%)$  of different methods on benchmark datasets with clean, symmetric  $(\eta \in \{0.4,0.6,0.8\})$  or asymmetric  $(\eta \in \{0.2,0.3,0.4\})$  label noise. The results (mean±std) are reported over 3 random runs under different random seeds  $(1,2,3)$ . The top-2 best results are in bold.
Datasets	Methods	Clean (η=0.0)	Symmetric Noise Rate (η)	Asymmetric Noise Rate (η)
0.4	0.6	0.8	0.2	0.3	0.4
MNIST	CE	99.20±0.02	74.46±0.28	49.19±0.05	22.51±0.23	94.02±0.18	88.90±0.07	81.79±0.34
MAE	99.16±0.03	98.80±0.02	97.69±0.20	70.35±1.16	99.11±0.03	98.42±0.09	87.40±4.01
GCE [4]	99.18±0.01	96.81±0.13	80.86±0.31	33.59±0.48	96.59±0.07	88.99±0.27	81.91±0.58
SCE [5]	99.30±0.07	97.48±0.16	88.35±0.77	48.28±0.81	97.95±0.23	94.00±0.41	84.54±0.14
NLNL [10]	98.61±0.13	97.17±0.09	95.42±0.30	86.34±1.43	98.35±0.01	97.51±0.15	95.84±0.26
NCE+RCE [6]	99.43±0.02	98.53±0.09	95.61±0.12	74.04±1.83	98.79±0.10	95.16±0.08	91.36±0.22
NCE+AGCE [17]	99.10±0.03	98.91±0.04	98.50±0.07	96.93±0.13	99.04±0.02	98.94±0.03	98.41±0.04
ANL-CE	99.08±0.05	98.84±0.05	98.42±0.08	96.62±0.12	99.04±0.04	98.91±0.07	98.01±0.10
ANL-FL	99.13±0.05	98.90±0.05	98.46±0.12	95.73±0.22	99.05±0.09	98.93±0.02	98.18±0.01
CIFAR-10	CE	90.38±0.11	58.19±0.21	38.75±0.19	19.09±0.35	83.00±0.33	78.15±0.17	73.69±0.20
MAE	89.15±0.27	81.76±3.17	76.82±0.84	46.42±3.66	79.63±0.74	67.35±3.41	57.36±2.37
GCE [4]	89.66±0.20	82.44±0.26	68.62±0.35	25.45±0.51	85.55±0.24	79.32±0.52	72.83±0.17
SCE [5]	91.38±0.12	79.96±0.25	62.16±0.33	27.98±0.98	86.22±0.44	80.20±0.20	74.01±0.52
NLNL [10]	90.73±0.20	63.90±0.44	50.68±0.47	29.53±1.55	84.74±0.08	81.26±0.43	76.97±0.52
NCE+RCE [6]	90.94±0.01	86.03±0.13	79.89±0.11	55.52±2.74	88.36±0.13	84.84±0.16	77.75±0.37
NCE+AGCE [17]	91.08±0.06	86.16±0.10	80.14±0.27	55.62±4.78	88.48±0.09	84.79±0.15	78.60±0.41
ANL-CE	91.66±0.04	87.28±0.02	81.12±0.30	61.27±0.55	89.13±0.11	85.52±0.24	77.63±0.31
ANL-FL	91.79±0.19	87.25±0.11	81.67±0.19	61.22±0.85	89.09±0.31	85.81±0.23	77.73±0.31
CIFAR-100	CE	71.14±0.38	40.72±0.74	22.98±0.07	7.55±0.21	58.25±1.00	50.30±0.19	41.53±0.34
MAE	7.35±1.19	3.61±0.21	3.63±0.35	2.83±1.35	6.19±0.42	5.82±0.96	3.96±0.35
GCE [4]	61.62±0.43	56.46±0.95	46.27±1.30	19.51±0.86	59.06±0.46	53.88±0.96	41.51±0.52
SCE [5]	70.80±0.37	39.84±0.19	21.97±0.92	7.87±0.48	57.78±0.83	50.15±0.12	41.33±0.86
NLNL [10]	68.72±0.60	30.29±1.64	16.60±0.90	11.01±2.48	50.19±0.56	42.81±1.13	35.10±0.20
NCE+RCE [6]	68.22±0.28	57.97±0.30	46.26±1.07	25.65±0.51	62.77±0.53	55.62±0.56	42.46±0.42
NCE+AGCE [17]	68.61±0.12	59.74±0.68	47.96±0.44	24.13±0.07	64.05±0.25	56.36±0.59	44.90±0.62
ANL-CE	70.68±0.23	61.80±0.50	51.52±0.53	28.07±0.28	66.27±0.19	59.76±0.34	45.41±0.68
ANL-FL	70.40±0.15	61.73±0.48	51.32±0.34	27.97±0.58	66.26±0.44	59.68±0.86	46.65±0.04
Table 3: Top-1 validation accuracies(%) on clean ILSVRC12 and WebVision validation set of ResNet-50 models trained on WebVision using different methods. The top-2 best results are in bold.
Methods	CE	GCE [4]	SCE [5]	NCE+RCE [6]	NCE+AGCE [17]	ANL-CE	ANL-FL
ILSVRC12 Val	58.64	56.56	62.60	62.40	60.76	65.00	65.56
WebVision Val	61.20	59.44	68.00	64.92	63.92	67.44	68.32
Active and passive parts separately. In table 1, we show the results of the active and passive parts separately. We separately train NCE and NNCE on CIFAR-10 with different symmetric noise rates while maintaining the same parameters as ANL-CE. Specifically, for  $\alpha \cdot$  NCE, we set  $\alpha$  to 5.0 and for  $\beta \cdot$  NNCE, we set  $\beta$  to 5.0, while  $\delta$  is set to  $5 \times 10^{-5}$  for both. As indicated in the results, the test set accuracies of NNCE are very close to those of ANL-CE, except in the case with a 0.8 noise rate. This suggests that NNCE performs well on its own at low noise rates. However, at very high noise rates, a combination of active losses is needed to achieve better performance.
4.2 Evaluation on Benchmark Datasets
Baselines. We consider several state-of-the-art methods: 1) Generalized Cross Entropy (GCE) [4]; 2) Symmetric Cross Entropy (SCE) [5]; 3) Negative Learning for Noisy Labels (NLNL) [10]; 4) Active Passive Loss (APL) [6], including NCE+MAE, NCE+RCE, and NFL+RCE; 5) Asymmetric Loss Functions (AFLs) [17], including NCE+AEL, NCE+AGCE, and NCE+AUL. For our proposed ANL, we consider two loss functions: 1) ANL-CE and 2) ANL-FL. Additionally, we train networks using Cross Entropy (CE), Focal Loss (FL) [18], and Mean Absolute Error (MAE).
Experimental Details. The full experimental results and the detailed settings of noise generation, networks, training and parameters can be found in the appendix D.2.
Results. The main experimental results under symmetric and asymmetric label noise are reported in table 2. For more experimental results, please see appendix D.2. As can be observed, our ANL-CE and ANL-FL show significant improvement for most label noise settings of CIFAR-10/-100, especially when the data become more complex and the noise rate becomes larger. For example, on CIFAR-10 under 0.8 symmetric noise, our ANL-CE outperform the state-of-the-art method (55.62% of NCE+AGCE) by more than 5.0%. Overall, the experimental results demonstrate that our ANL can show outstanding performance on different datasets, noise types, and noise rates, which validates the effectiveness of our proposed NNLFs and ANL.
4.3 Evaluation on Real-world Noisy Labels
Here, we evaluate our proposed ANL methods on large-scale real-world noisy dataset WebVision [14], which contains more than 2.4 million web images crawled from the internet by using queries generated from the 1,000 class labels of the ILSVRC 2012 [1] benchmark. Here, we follow the "Mini" setting in [19], and only take the first 50 classes of the Google resized image subset. We evaluate the trained networks on the same 50 classes of both the ILSVRC 2012 validation set and the WebVision validation set, and these can be considered as clean validation sets. We compare our ANL-CE and ANL-FL with GCE, SCE, NCE+RCE, and NCE+AGCE. The experimental details can be found in the appendix D.3. The results are reported in table 3. As can be observed, our proposed methods outperform the existing loss functions. This verifies that our proposed ANL framework can help the trained model against real-world label noise.
Moreover, in addition to WebVision, to further validate the effectiveness of our method on real-world noisy datasets, we also conduct a set of experiments on CIFAR-10N/-100N [20], Animal-10N [21], and Clothing-1M [22]. The experimental details and results can be found in the appendix D.5, which demonstrate the effectiveness of our method on different real-world noisy datasets.
We believe that the main limitation of our approach lies in the choice of regularization method. Although we have experimentally verified that L1 is the most efficient among the three regularization methods (L1, L2, and Jacobian), we lack further theoretical analysis of it. Furthermore, although we only consider relatively simple regularization methods for the sake of fair comparison, other regularization methods, such as dropout [23] or regmixup [24], might be more effective in mitigating the overfitting problem caused by NNLF. And we believe that a better solution to overfitting can further improve the performance of our method.
In recent years, some robust loss-based methods have been proposed for robust learning with noisy labels. Here, we briefly review the relevant approaches. Ghosh et al. [3] theoretically proved that symmetric loss functions, such as MAE, are robust to label noise. Zhang and Sabuncu [4] proposed Generalized Cross Entropy (GCE), a generalization of CE and MAE. Wang et al. [5] suggested a combination of CE and scaled MAE, and proposed Symmetric Cross Entropy (SCE). Menon et al. [25] proposed composite loss-based gradient clipping and applied it to CE to obtain PHuber-CE. Ma et al. [6] proposed Active Passive Loss (APL) to create fully robust loss functions. Feng et al. [26] applied the Taylor series to derive an alternative representation of CE and proposed Taylor-CE accordingly. Zhou et al. [17] proposed Asymmetric Loss Functions (ALFs) to overcome the symmetric condition. Inspired by complementary label learning, NLNL [10] and JNPL [27] use complementary labels to reduce the risk of providing the wrong information to the model.
In this paper, we propose a new class of theoretically robust passive loss functions different from MAE, which we refer to as Normalized Negative Loss Functions (NNLFs). By replacing the MAE in APL with our NNLF, we propose Active Negative Loss (ANL), a robust loss function framework with stronger fitting ability. We theoretically demonstrate that our NNLFs and ANLs are robust to noisy labels and also highlight the property that NNLFs focus more on well-learned samples. We found in our experiments that NNLFs have a potential overfitting problem, and we suggest using L1 regularization to mitigate it. Experimental results verified that our ANL can outperform the state-of-the-art methods on benchmark datasets.
This work is supported in part by Shanghai Science and Technology Committee under grant No. 21511100600 and No. 22511106005.
[19] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. Proceedings of the 35th International Conference on Machine Learning, pages 2309–2318, 2018.
[20] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. Proceedings of the 10th International Conference on Learning Representations, 2022.
[21] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing Unclean Samples for Robust Deep Learning. Proceedings of the 36th International Conference on Machine Learning, 2019.
[22] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, pages 2691-2699, 2015.
[23] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929-1958, 2014.
[24] Francesco Pinto, Harry Yang, Ser Nam Lim, Philip H. S. Torr, and Puneet K. Dokania. Using mixup as a regularizer can surprisingly improve accuracy & out-of-distribution robustness. Advances in Neural Information Processing Systems 35, 2022.
[25] Aditya Krishna Menon, Ankit Singh Rawat, Sanjiv Kumar, and Sashank Reddi. Can gradient clipping mitigate label noise? Proceedings of the 8th International Conference on Learning Representations, 2020.
[26] Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? 29th International Joint Conference on Artificial Intelligence, 2020.
[27] Youngdong Kim, Juseung Yun, Hyounguk Shon, and Junmo Kim. Joint Negative and Positive Learning for Noisy Labels. Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, pages 9442-9451, 2021.
[28] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 2233-2241, 2017.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations, 2015.
[31] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 5552-5560, 2018.
A Loss functions
A.1 NMAE, RCE, and NRCE are scaled versions of MAE
Here, inspired by [5, 6], we show how to reformulate NMAE, RCE, and NRCE as scaled versions of MAE. First, for convenience, we reformulate the Mean Absolute Error (MAE) as follows:
$$
\begin{array}{l} M A E = \sum_ {k = 1} ^ {K} | \boldsymbol {p} (k | \boldsymbol {x}) - \boldsymbol {q} (k | \boldsymbol {x}) | \\ = (1 - \boldsymbol {p} (y | \boldsymbol {x})) + \sum_ {k \neq y} \boldsymbol {p} (k | \boldsymbol {x}) \tag {16} \\ = 2 (1 - \boldsymbol {p} (y | \boldsymbol {x})), \\ \end{array}
$$
where the last equality holds due to  $\sum_{k=1}^{K} \pmb{p}(k|\pmb{x}) = 1$ .
For Normalized Mean Absolute Error (NMAE), we have:
$$
\begin{array}{l} N M A E = \frac {\sum_ {k = 1} ^ {K} | \boldsymbol {p} (k | \boldsymbol {x}) - \boldsymbol {q} (k | \boldsymbol {x}) |}{\sum_ {j = 1} ^ {K} \sum_ {k = 1} ^ {K} | \boldsymbol {p} (k | \boldsymbol {x}) - \boldsymbol {q} (y = j | \boldsymbol {x}) |} \\ = \frac {2 (1 - \boldsymbol {p} (y | \boldsymbol {x}))}{\sum_ {j = 1} ^ {K} 2 (1 - \boldsymbol {p} (j | \boldsymbol {x}))} \tag {17} \\ = \frac {1}{K - 1} (1 - \boldsymbol {p} (y | \boldsymbol {x})) \\ = \frac {1}{2 (K - 1)} \cdot M A E. \\ \end{array}
$$
This shows that the NMAE is a scaled version of the MAE with a factor of  $\frac{1}{2(K - 1)}$ .
For Reverse Cross Entropy (RCE), we have:
$$
\begin{array}{l} R C E = \sum_ {k = 1} ^ {K} \boldsymbol {p} (x | \boldsymbol {x}) (- \log \boldsymbol {q} (k | \boldsymbol {x})) \\ = \boldsymbol {p} (y | \boldsymbol {x}) (- \log 1) + \sum_ {k \neq y} \boldsymbol {p} (k | \boldsymbol {x}) (- \log 0) (18) \\ = \sum_ {k \neq y} \boldsymbol {p} (k | \boldsymbol {x}) (- A) (18) \\ = - A (1 - \boldsymbol {p} (y | \boldsymbol {x})) \\ = - \frac {A}{2} \cdot M A E, \\ \end{array}
$$
recall that in RCE we set  $\log 0 = A$ , where  $A < 0$  is some constant (e.g.,  $A = -4$ ). This shows that the RCE is a scaled version of the MAE with a factor of  $-\frac{A}{2}$ .
For Normalized Reverse Cross Entropy (NRCE), we have:
$$
\begin{array}{l} N R C E = \frac {\sum_ {k = 1} ^ {K} \boldsymbol {p} (k | \boldsymbol {x}) (- \log \boldsymbol {q} (k | \boldsymbol {x}))}{\sum_ {j = 1} ^ {K} \sum_ {k = 1} ^ {K} \boldsymbol {p} (k | \boldsymbol {x}) (- \log \boldsymbol {q} (y = j | \boldsymbol {x}))} \\ = \frac {- A (1 - \boldsymbol {p} (y | \boldsymbol {x}))}{\sum_ {j = 1} ^ {K} (- A (1 - \boldsymbol {p} (j | \boldsymbol {x})))} \tag {19} \\ = \frac {1}{K - 1} (1 - \boldsymbol {p} (y | \boldsymbol {x})) \\ = \frac {1}{2 (K - 1)} \cdot M A E. \\ \end{array}
$$
This shows that NRCE is a scaled version of the MAE with a factor of  $\frac{1}{2(K - 1)}$ .
A.2 Normalized Negative Loss Functions
Here, we show how to derive Normalized Negative Loss Functions (NNLFs) into its proper form.
$$
\begin{array}{l} N N L F = \frac {\sum_ {k = 1} ^ {K} (1 - \boldsymbol {q} (k | \boldsymbol {x})) (A - \mathcal {L} (f (\boldsymbol {x}) , k))}{\sum_ {j = 1} ^ {K} \sum_ {k = 1} ^ {K} (1 - \boldsymbol {q} (k = j | \boldsymbol {x})) (A - \mathcal {L} (f (\boldsymbol {x}), k))} \\ = \frac {\sum_ {k \neq y} A - \mathcal {L} (f (\boldsymbol {x}) , k)}{\sum_ {j = 1} ^ {K} \sum_ {k \neq j} A - \mathcal {L} (f (\boldsymbol {x}) , k)} \\ = \frac {\sum_ {k \neq y} A - \mathcal {L} (f (\boldsymbol {x}) , k)}{(K - 1) \sum_ {k = 1} ^ {K} A - \mathcal {L} (f (\boldsymbol {x}), k)} \tag {20} \\ = \frac {1}{K - 1} \cdot \left(1 - \frac {A - \mathcal {L} (f (\boldsymbol {x}) , y)}{\sum_ {k = 1} ^ {K} A - \mathcal {L} (f (\boldsymbol {x}) , k)}\right) \\ \propto 1 - \frac {A - \mathcal {L} (f (\boldsymbol {x}) , y)}{\sum_ {k = 1} ^ {K} A - \mathcal {L} (f (\boldsymbol {x}) , k)}. \\ \end{array}
$$
B Noise tolerant
Theorem 1. Normalized negative loss function  $\mathcal{L}_{mn}$  is symmetric.
Proof. For all  $\pmb{x} \in \mathcal{X}$  and all  $f$ , we have:
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \mathcal {L} _ {\mathrm {n n}} (f (\boldsymbol {x}), k) = \sum_ {k = 1} ^ {K} \left(1 - \frac {A - \mathcal {L} (f (\boldsymbol {x}) , k)}{\sum_ {j = 1} ^ {K} A - \mathcal {L} (f (\boldsymbol {x}) , j)}\right) \\ = K - \frac {\sum_ {k = 1} ^ {K} A - \mathcal {L} (f (\boldsymbol {x}) , k)}{\sum_ {j = 1} ^ {K} A - \mathcal {L} (f (\boldsymbol {x}) , j)} \tag {21} \\ = K - 1, \\ \end{array}
$$
where  $K - 1$  is a constant and  $\mathcal{L}_{\mathrm{nn}}$  satisfies the definition of the symmetric loss function.
C Gradient analysis
C.1 Gradient of MAE
The complete derivation of the Mean Absolute Error (MAE) with respect to the classifier's output probabilities is as follows:
In the case of  $j\neq y$
$$
\begin{array}{l} \frac {\partial \mathcal {L} _ {\mathrm {M A E}}}{\partial \boldsymbol {p} (j | \boldsymbol {x})} = \frac {\partial}{\partial \boldsymbol {p} (j | \boldsymbol {x})} \sum_ {k = 1} ^ {K} | \boldsymbol {p} (k | \boldsymbol {x}) - \boldsymbol {q} (k | \boldsymbol {x}) | \\ = \frac {1}{\partial \boldsymbol {p} (j | \boldsymbol {x})} \left((1 - \boldsymbol {p} (y | \boldsymbol {x})) + \sum_ {k \neq y} \boldsymbol {p} (k | \boldsymbol {x})\right) \tag {22} \\ = 1. \\ \end{array}
$$
In the case of  $j = y$ :
$$
\begin{array}{l} \frac {\partial \mathcal {L} _ {\mathrm {M A E}}}{\partial \boldsymbol {p} (j | \boldsymbol {x})} = \frac {\partial}{\partial \boldsymbol {p} (j | \boldsymbol {x})} \sum_ {k = 1} ^ {K} | \boldsymbol {p} (k | \boldsymbol {x}) - \boldsymbol {q} (k | \boldsymbol {x}) | \\ = \frac {1}{\partial \boldsymbol {p} (j | \boldsymbol {x})} \left((1 - \boldsymbol {p} (y | \boldsymbol {x})) + \sum_ {k \neq y} \boldsymbol {p} (k | \boldsymbol {x})\right) \tag {23} \\ = - 1. \\ \end{array}
$$
C.2 Gradient of NNCE
The complete derivation of our proposed Normalized Negative Cross Entropy (NNCE) with respect to the classifier's output probabilities is as follows:
In the case of  $j \neq y$ :
$$
\begin{array}{l} \frac {\partial \mathcal {L} _ {\mathrm {N N C E}}}{\partial \boldsymbol {p} (j | \boldsymbol {x})} = \frac {\partial}{\partial \boldsymbol {p} (j | \boldsymbol {x})} \Big (1 - \frac {A - (- \log \boldsymbol {p} (y | \boldsymbol {x}))}{\sum_ {k = 1} ^ {K} A - (- \log \boldsymbol {p} (k | \boldsymbol {x}))} \Big) \\ = - \frac {\partial}{\partial \boldsymbol {p} (j | \boldsymbol {x})} \left(\frac {A + \log \boldsymbol {p} (y | \boldsymbol {x})}{\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x})}\right) \\ = - \frac {0 - (A + \log \boldsymbol {p} (y | \boldsymbol {x})) \frac {1}{\boldsymbol {p} (j | \boldsymbol {x})}}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x})\right) ^ {2}} \tag {24} \\ = \frac {1}{\boldsymbol {p} (j | \boldsymbol {x})} \cdot \frac {A + \log \boldsymbol {p} (y | \boldsymbol {x})}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x})\right) ^ {2}} \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial \mathcal {L} _ {\mathrm {N N C E}}}{\partial \boldsymbol {p} (j | \boldsymbol {x})} = \frac {\partial}{\partial \boldsymbol {p} (j | \boldsymbol {x})} \left(1 - \frac {A - (- \log \boldsymbol {p} (y | \boldsymbol {x}))}{\sum_ {k = 1} ^ {K} A - (- \log \boldsymbol {p} (k | \boldsymbol {x}))}\right) \\ = - \frac {\partial}{\partial \boldsymbol {p} (j | \boldsymbol {x})} \left(\frac {A + \log \boldsymbol {p} (y | \boldsymbol {x})}{\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x})}\right) \\ = - \frac {\frac {1}{\boldsymbol {p} (y \mid \boldsymbol {x})} \left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k \mid \boldsymbol {x})\right) - \left(A + \log \boldsymbol {p} (y \mid \boldsymbol {x})\right) \frac {1}{\boldsymbol {p} (y \mid \boldsymbol {x})}}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k \mid \boldsymbol {x})\right) ^ {2}} \tag {25} \\ = - \frac {1}{\boldsymbol {p} (y | \boldsymbol {x})} \cdot \frac {\sum_ {k \neq y} A + \log \boldsymbol {p} (k | \boldsymbol {x})}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x})\right) ^ {2}} \\ \end{array}
$$
C.3 Properties of NNCE
Proof. As per the assumption, we have:
$$
\boldsymbol {p} \left(j _ {1} \mid \boldsymbol {x}\right) <   \boldsymbol {p} \left(j _ {2} \mid \boldsymbol {x}\right)
$$
$$
\frac {1}{\boldsymbol {p} (j _ {1} | \boldsymbol {x})} > \frac {1}{\boldsymbol {p} (j _ {2} | \boldsymbol {x})}
$$