SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1}{\boldsymbol {p} \left(j _ {1} \mid \boldsymbol {x}\right)} \cdot \frac {A + \log \boldsymbol {p} (y \mid \boldsymbol {x})}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k \mid \boldsymbol {x})\right) ^ {2}} > \frac {1}{\boldsymbol {p} \left(j _ {2} \mid \boldsymbol {x}\right)} \cdot \frac {A + \log \boldsymbol {p} (y \mid \boldsymbol {x})}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k \mid \boldsymbol {x})\right) ^ {2}} \tag {26}
$$
$$
\frac {\partial \mathcal {L} _ {\text {N N C E}}}{\partial \boldsymbol {p} (j _ {1} | \boldsymbol {x})} > \frac {\partial \mathcal {L} _ {\text {N N C E}}}{\partial \boldsymbol {p} (j _ {2} | \boldsymbol {x})}.
$$
Theorem 3. Given the classifier's output probabilities  $\pmb{p}(\cdot|\pmb{x}_1)$  and  $\pmb{p}(\cdot|\pmb{x}_2)$  of sample  $\pmb{x}_1$  and  $\pmb{x}_2$ , where  $\pmb{p}(y|\pmb{x}_1) \geq \pmb{p}(k|\pmb{x}_1)$ ,  $\pmb{p}(y|\pmb{x}_2) \geq \pmb{p}(k|\pmb{x}_2)$ ,  $\forall k \in \{1,\dots,K\}$ ,  $\pmb{p}(j|\pmb{x}_1) = \pmb{p}(j|\pmb{x}_2)$ , and normalized negative cross entropy  $\mathcal{L}_{NNCE}$ . If  $\pmb{p}(y|\pmb{x}_1) > \pmb{p}(y|\pmb{x}_2)$ ,  $j \neq y$ , and  $\pmb{p}(k|\pmb{x}_1) \leq \pmb{p}(k|\pmb{x}_2)$ ,  $\forall k \in \{1,\dots,K\} \setminus \{j,y\}$ , then  $\frac{\partial\mathcal{L}_{NNCE}}{\partial\pmb{p}(j|\pmb{x}_1)} > \frac{\partial\mathcal{L}_{NNCE}}{\partial\pmb{p}(j|\pmb{x}_2)}$ .
Proof. We first consider two functions  $f_{1}$  and  $f_{2}$ . The function  $f_{1}$  is defined as follows:
$$
f _ {1} (\boldsymbol {p}) = \sum_ {k = 1} ^ {K} \log \boldsymbol {p} _ {k}, \tag {27}
$$
where the input  $\pmb{p}$  is a discrete probability distribution,  $\sum_{k=1}^{K} p_k = 1$  and  $0 \leq p_k \leq 1$ ,  $\forall k \in \{1, \dots, K\}$ . Given a discrete probability distribution  $\pmb{p}$  and a real number  $D$  as input, the function  $f_2$  is defined as follows:
$$
f _ {2} (\boldsymbol {p}, D) = \log \left(\boldsymbol {p} _ {y} + D\right) + \log \boldsymbol {p} _ {j} + \sum_ {k \neq j \neq y} \log \left(\boldsymbol {p} _ {k} - d _ {k}\right), \tag {28}
$$
where  $\pmb{p}_y \geq \pmb{p}_k, \forall k \in \{1, \dots, K\}, 0 < D \leq 1 - \pmb{p}_y$ . And  $\{d_k\}, k \in \{1, \dots, K\} \setminus \{j, y\}$  is a set of random variables which satisfy following conditions:  $0 \leq \pmb{p}_k - d_k \leq \pmb{p}_k$  and  $\sum_{k \neq j \neq y} d_k = D$ .
Next, we consider whether  $f_{1}(\pmb{p}) - f_{2}(\pmb{p}, D) > 0$ . Given  $\pmb{p}$  and  $D$ , we have,
$$
\begin{array}{l} f _ {1} (\boldsymbol {p}) - f _ {2} (\boldsymbol {p}, D) = \left(\sum_ {k = 1} ^ {K} \log \boldsymbol {p} _ {k}\right) - \left(\log \left(\boldsymbol {p} _ {y} + D\right) + \log \boldsymbol {p} _ {j} + \sum_ {k \neq j \neq y} \log \left(\boldsymbol {p} _ {k} - d _ {k}\right)\right) \\ = \log \boldsymbol {p} _ {y} - \log (\boldsymbol {p} _ {y} + D) + \sum_ {k \neq j \neq y} \log \boldsymbol {p} _ {k} - \log (\boldsymbol {p} _ {k} - d _ {k}) \\ = \log \frac {\boldsymbol {p} _ {\boldsymbol {y}}}{\boldsymbol {p} _ {\boldsymbol {y}} + D} + \sum_ {k \neq j \neq y} \log \frac {\boldsymbol {p} _ {k}}{\boldsymbol {p} _ {k} - d _ {k}} \\ \geq \log \frac {\boldsymbol {p} _ {\boldsymbol {y}}}{\boldsymbol {p} _ {\boldsymbol {y}} + D} + \sum_ {k \neq j \neq y} \log \frac {\boldsymbol {p} _ {\boldsymbol {y}}}{\boldsymbol {p} _ {\boldsymbol {y}} - d _ {k}} \\ = \log \boldsymbol {p} _ {y} - \log (\boldsymbol {p} _ {y} + D) + \sum_ {k \neq j \neq y} \log \boldsymbol {p} _ {y} - \log (\boldsymbol {p} _ {y} - d _ {k}) \\ = \left(\log \boldsymbol {p} _ {y} + \sum_ {k \neq j \neq y} \log \boldsymbol {p} _ {y}\right) - \left(\log (\boldsymbol {p} _ {y} + D) + \sum_ {k \neq j \neq y} \log (\boldsymbol {p} _ {y} - d _ {k})\right) \\ \geq \left(\log \boldsymbol {p} _ {y} + \sum_ {k \neq j \neq y} \log \boldsymbol {p} _ {y}\right) - \sup  _ {\{d _ {k} \}, k \neq j \neq y} \left(\log \left(\boldsymbol {p} _ {y} + D\right) + \sum_ {k \neq j \neq y} \log \left(\boldsymbol {p} _ {y} - d _ {k}\right)\right). \tag {29} \\ \end{array}
$$
The first inequality holds because for a fraction  $\frac{a}{b} \geq 1$ , we always have  $\frac{a}{b} \geq \frac{a + c}{b + c}$ , where  $c \geq 0$ .
To get the maximum value of  $\log (\pmb{p}_y + D) + \sum_{k\neq j\neq y}\log (\pmb{p}_y - d_k)$ , we must solve the following minimization problem subject to constraints:
$$
\min  _ {\left\{d _ {k} \right\}, k \neq j \neq y} - \left(\log \left(\boldsymbol {p} _ {y} + D\right) + \sum_ {k \neq j \neq y} \log \left(\boldsymbol {p} _ {y} - d _ {k}\right)\right). \tag {30}
$$
$$
\text {s . t .} \quad \sum_ {k \neq j \neq y} d _ {k} = D. \tag {31}
$$
We can define the Lagrange function  $L$  as follows:
$$
L \left(d _ {1}, \dots , d _ {K}, \lambda\right) = - \left(\log \left(\boldsymbol {p} _ {y} + D\right) + \sum_ {k \neq j \neq y} \log \left(\boldsymbol {p} _ {y} + d _ {k}\right)\right) + \lambda \cdot \left(\sum_ {k \neq j \neq y} d _ {k} - D\right). \tag {32}
$$
Now we can calculate the gradients:
$$
\left\{ \begin{array}{l} \frac {\partial L}{\partial d _ {k}} = - \frac {1}{\boldsymbol {p} _ {y} + d _ {k}} + \lambda , k \neq j \neq y \\ \frac {\partial L}{\partial \lambda} = \sum_ {k \neq j \neq y} d _ {k} - D. \end{array} \right. \tag {33}
$$
Let  $\frac{\partial L}{\partial d_k} = 0$  and  $\frac{\partial L}{\partial \lambda} = 0$ , we have:
$$
\left\{ \begin{array}{l} - \frac {1}{p _ {y} + d _ {k}} + \lambda = 0, k \neq j \neq y \\ \sum_ {k \neq j \neq y} d _ {k} - D = 0, \end{array} \right. \tag {34}
$$
and therefore:
$$
d _ {k} = \frac {D}{K - 2}, k \neq j \neq y. \tag {35}
$$
So, the minimization value is:
$$
- \left(\log \left(\boldsymbol {p} _ {y} + D\right) + \sum_ {k \neq j \neq y} \log \left(\boldsymbol {p} _ {y} - \frac {D}{K - 2}\right)\right). \tag {36}
$$
Now, back to the eq. (29), we have:
$$
\begin{array}{l} f _ {1} (\boldsymbol {p}) - f _ {2} (\boldsymbol {p}, D) \geq \log \boldsymbol {p} _ {y} + \sum_ {k \neq j \neq y} \log \boldsymbol {p} _ {y} - \sup  _ {\{d _ {k} \}, k \neq j \neq y} \left(\log (\boldsymbol {p} _ {y} + D) + \sum_ {k \neq j \neq y} \log (\boldsymbol {p} _ {y} - d _ {k})\right) \\ = \log \boldsymbol {p} _ {y} + \sum_ {k \neq j \neq y} \log \boldsymbol {p} _ {y} - \left(\log (\boldsymbol {p} _ {y} + D) + \sum_ {k \neq j \neq y} \log (\boldsymbol {p} _ {y} - \frac {D}{K - 2})\right) \\ = \log \boldsymbol {p} _ {y} - \log (\boldsymbol {p} _ {y} + D) + (K - 2) \left(\log \boldsymbol {p} _ {y} - \log (\boldsymbol {p} _ {y} - \frac {D}{K - 2})\right) \\ = (K - 2) \left(\log \boldsymbol {p} _ {y} - \log (\boldsymbol {p} _ {y} - \frac {D}{K - 2})\right) - \left(\log (\boldsymbol {p} _ {y} + D) - \log \boldsymbol {p} _ {y}\right) \\ = D \cdot \left(\frac {\log \boldsymbol {p} _ {y} - \log \left(\boldsymbol {p} _ {y} - \frac {D}{K - 2}\right)}{\frac {D}{K - 2}} - \frac {\log \left(\boldsymbol {p} _ {y} + D\right) - \log \boldsymbol {p} _ {y}}{D}\right). \tag {37} \\ \end{array}
$$
Recall the nature of the difference, we have  $\left.\frac{\log p_y - \log(p_y - \frac{D}{K - 2})}{\frac{D}{K - 2}} = \frac{d\log x}{dx}\right|_{x = p_y - \alpha} = \frac{1}{p_y - \alpha}$ , where  $0 \leq \alpha \leq \frac{D}{K - 2}$ , and  $\left.\frac{\log(p_y + D) - \log p_y}{D} = \frac{d\log x}{dx}\right|_{x = p_y + \beta} = \frac{1}{p_y + \beta}$ , where  $0 \leq \beta \leq D$ . And therefore:
$$
\frac {\log \boldsymbol {p} _ {y} - \log \left(\boldsymbol {p} _ {y} - \frac {D}{K - 2}\right)}{\frac {D}{K - 2}} = \frac {1}{\boldsymbol {p} _ {y} - \alpha} > \frac {1}{\boldsymbol {p} _ {y}} > \frac {1}{\boldsymbol {p} _ {y} + \beta} = \frac {\log \left(\boldsymbol {p} _ {y} + D\right) - \log \boldsymbol {p} _ {y}}{D}. \tag {38}
$$
$$
\frac {\log \boldsymbol {p} _ {y} - \log \left(\boldsymbol {p} _ {y} - \frac {D}{K - 2}\right)}{\frac {D}{K - 2}} > \frac {\log \left(\boldsymbol {p} _ {y} + D\right) - \log \boldsymbol {p} _ {y}}{D} \tag {39}
$$
$$
f _ {1} (\boldsymbol {p}) > f _ {2} (\boldsymbol {p}, D).
$$
Now, let  $\pmb{p}_k = \pmb{p}(k|\pmb{x}_2), \forall k \in \{1, \dots, K\}$ ,  $D = \pmb{p}(y|\pmb{x}_1) - \pmb{p}(y|\pmb{x}_2)$ , and  $d_k = \pmb{p}(k|\pmb{x}_2) - \pmb{p}(k|\pmb{x}_1)$ ,  $k \neq j \neq y$ . Following eq. (39), we have:
$$
f _ {1} (\boldsymbol {p}) > f _ {2} (\boldsymbol {p}, D)
$$
$$
\sum_ {k = 1} ^ {K} \log \boldsymbol {p} _ {k} > \log (\boldsymbol {p} _ {y} + D) + \log \boldsymbol {p} _ {j} + \sum_ {k \neq j \neq y} \log (\boldsymbol {p} _ {k} - d _ {k})
$$
$$
\sum_ {k = 1} ^ {K} \log \boldsymbol {p} (k | \boldsymbol {x} _ {2}) > \sum_ {k = 1} ^ {K} \log \boldsymbol {p} (k | \boldsymbol {x} _ {1}) \tag {40}
$$
$$
\frac {1}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x} _ {1})\right) ^ {2}} > \frac {1}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x} _ {2})\right) ^ {2}} \tag {40}
$$
$$
\frac {1}{\boldsymbol {p} (j | \boldsymbol {x} _ {1})} \cdot \frac {A + \log \boldsymbol {p} (y | \boldsymbol {x} _ {1})}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x} _ {1})\right) ^ {2}} > \frac {1}{\boldsymbol {p} (j | \boldsymbol {x} _ {2})} \cdot \frac {A + \log \boldsymbol {p} (y | \boldsymbol {x} _ {2})}{\left(\sum_ {k = 1} ^ {K} A + \log \boldsymbol {p} (k | \boldsymbol {x} _ {2})\right) ^ {2}}
$$
$$
\frac {\partial \mathcal {L} _ {\text {N N C E}}}{\partial \boldsymbol {p} (j | \boldsymbol {x} _ {1})} > \frac {\partial \mathcal {L} _ {\text {N N C E}}}{\partial \boldsymbol {p} (j | \boldsymbol {x} _ {2})}.
$$
The eq. (40) holds because we have  $\pmb{p}(j|\pmb{x}_1) = \pmb{p}(j|\pmb{x}_2)$ , and  $\log \pmb{p}(y|\pmb{x}_1) > \log \pmb{p}(y|\pmb{x}_2)$ , where  $\pmb{p}(y|\pmb{x}_1) > \pmb{p}(y|\pmb{x}_2)$ . This completes the proof.
(a) CIFAR-10 under 0.6 symmetric noise
(b) CIFAR-10 under 0.8 symmetric noise
Figure 4: The gradient  $\max_{j,j\neq y}\left(\frac{\partial\mathcal{L}_{\mathrm{NNCE}}}{\partial p(j|\boldsymbol{x})}\right)$  for clean and noisy samples. Blue boxes indicate clean samples, red boxes indicate noisy samples, blue triangles indicate the means of clean samples, and red triangles indicate the means of noisy samples. The outliers are ignored.
D Experiments
D.1 Empirical Understandings
Gradients of NNCE. We conduct experiments with ANL-CE on CIFAR-10 under 0.6 and 0.8 symmetric noise. For each sample, we calculate the gradient of our NNCE with respect to the predicted probability of each non-labeled class and take their maximum value, which is  $\max_{j,j\neq y}\left(\frac{\partial\mathcal{L}_{\mathrm{NNCE}}}{\partial p(j|x)}\right)$ . The results are shown in fig. 4 in the form of box plots. As can be observed, as the number of training epochs increases, the gradients of the clean samples are concentrated at larger values and the gradients of the noisy samples are concentrated at smaller values. The means of the gradients for noisy samples are higher than the medians and smaller than the means for clean samples' gradients. This indicates that although the model is still incorrectly learning some noisy samples, these learned noisy samples constitute only a small fraction of all noisy samples in the training set. In general, this result verifies that our NNLFs have the ability to focus more on clean samples and ignore noisy samples during the training process, and also verifies our discussion in section 3.3.
Parameter Analysis. We apply different values of  $\alpha$  and  $\beta$  to NCE+NNCE for training on CIFAR-10 under 0.8 symmetric noise. We test the combinations between  $\alpha \in \{0.1, 0.5, 1.0, 5.0, 10.0\}$  and  $\beta \in \{0.1, 0.5, 1.0, 5.0, 10.0\}$ , where  $\alpha$  is the weight of NCE and  $\beta$  is the weight of NNCE. The weight decay of NCE+NNCE is set to be the same as NCE+RCE. As can be observed in fig. 5, regardless of how  $\alpha$  (the weight of NCE) varies, when  $\beta$  (the weight of NNCE) increases, both the robustness and the fitting ability of the model improve, although overfitting occurs. This verifies that our NNLFs are robust to noisy labels and have good fitting ability. Although in ANL we use L1 regularization instead of L2 regularization (weight decay), the effect of  $\alpha$  and  $\beta$  in the loss functions created by ANL should be similar to that in NCE+NNCE.
(a)  $\alpha = 0.1, \beta = 0.1$
(b)  $\alpha = 0.1, \beta = 0.5$
(c)  $\alpha = 0.1, \beta = 1.0$
(d)  $\alpha = 0.1, \beta = 5.0$
(e)  $\alpha = 0.1, \beta = 10.0$
(f)  $\alpha = 0.5, \beta = 0.1$
(g)  $\alpha = 0.5, \beta = 0.5$
(h)  $\alpha = 0.5, \beta = 1.0$
(i)  $\alpha = 0.5, \beta = 5.0$
(j)  $\alpha = 0.5, \beta = 10.0$
(k)  $\alpha = 1.0, \beta = 0.1$
(1)  $\alpha = 1.0, \beta = 0.5$
(m)  $\alpha = 1.0, \beta = 1.0$
(n)  $\alpha = 1.0, \beta = 5.0$
(o)  $\alpha = 1.0, \beta = 10.0$
(p)  $\alpha = 5.0, \beta = 0.1$
(q)  $\alpha = 5.0, \beta = 0.5$
(r)  $\alpha = 5.0, \beta = 1.0$
(s)  $\alpha = 5.0, \beta = 5.0$
(t)  $\alpha = 5.0, \beta = 10.0$
(u)  $\alpha = 10.0, \beta = 0.1$
(V)  $\alpha = 10.0, \beta = 0.5$
(W)  $\alpha = 10.0, \beta = 1.0$
Figure 5: Training and test accuracies of NCE+NNCE on CIFAR-10 under 0.8 symmetric noise with different parameters.  $\alpha$  is the weight of NCE and  $\beta$  is the weight of NNCE. The accuracies of noisy samples in training set (red dashed line) should be as low as possible, since they are mislabeled.
(x)  $\alpha = 10.0, \beta = 5.0$
y)  $\alpha = 10.0, \beta = 10.0$
(a) L2
(b) L1
(c) Jacobian
Figure 6: Test accuracies of NCE+RCE on CIFAR-10 under 0.8 symmetric noise with different regularization methods and different parameters.  $\delta$  is the weight of regularization term of NCE+RCE. As a comparison, the blue line shows the test accuracy of our ANL-CE with the same data set and noise setting.
(a) CIFAR-10
(b) CIFAR-100
Figure 7: Test accuracies of different  $p_{\mathrm{min}}$  applied to ANL-CE on CIFAR-10/CIFAR-100 under 0.6 symmetric noise.
Can APL outperform our ANL by changing the regularization method? We apply different regularization methods to NCE+RCE for training on CIFAR-10 under 0.8 symmetric noise and compare the results with our ANL-CE. Specifically, we train networks using loss functions in the form of  $\alpha \cdot NCE + \beta \cdot RCE + \delta \cdot$  Reg, where  $\alpha ,\beta ,\delta >0$  are parameters and Reg is the regularization term. Similarly, we consider 3 regularization methods: 1) L2, 2) L1, and 3) Jacobian [15, 16]. We keep the  $\alpha$  and  $\beta$  of NCE+RCE the same as in section 4.2 and tune  $\delta \in \{1\times 10^{-7},5\times 10^{-7}$ $1\times 10^{-6}5\times 10^{-6}1\times 10^{-5}5\times 10^{-5}1\times 10^{-4}\}$  for all three methods. As can be observed in fig. 6, the performance of NCE+RCE can be improved by adapting the regularization method. However, the improvement from changing the regularization method is limited, and the test accuracies of NCE+RCE are consistently lower than our ANL-CE, no matter how the regularization method is changed and how the parameter  $\delta$  is varied. This verifies that APL cannot outperform our ANL by changing the regularization method and also verifies the performance of our proposed NNLFs.
Does the  $p_{\mathrm{min}}$  affect the performance of ANL? As shown in the definitions of NLF and NNLF,  $p_{\mathrm{min}}$  controls the lower bound of the classifier's output probability, and further controls the constant  $A$ . In fig. 7, we show different  $p_{\mathrm{min}} \in \{1 \times 10^{-5}, 1 \times 10^{-6}, 1 \times 10^{-7}, 1 \times 10^{-8}, 1 \times 10^{-9}\}$  applied to ANL-CE for training on CIFAR-10/-100 under 0.6 symmetric noise. As can be observed, for both datasets,  $p_{\mathrm{min}}$  does not significant affect the performance of ANL.
D.2 Evaluation on Benchmark Datasets
Noise generation. The noisy labels are generated following standard approaches in previous works [28, 6, 17]. For symmetric noise, we flip the labels in each class randomly to incorrect labels of other classes. For asymmetric noise, we flip the labels within a specific set of classes. For MNIST, flipping  $7 \rightarrow 1$ ,  $2 \rightarrow 7$ ,  $5 \leftrightarrow 6$ ,  $3 \rightarrow 8$ . For CIFAR-10, flipping TRUCK  $\rightarrow$  AUTOMOBILE, BIRD  $\rightarrow$  AIRPLANE, DEER  $\rightarrow$  HORSE, CAT  $\leftrightarrow$  DOG. For CIFAR-100, the 100 classes are grouped into 20 super-classes with each has 5 sub-classes, and each class are flipped within the same super-class into the next in a circular fashion. We vary the noise rate  $\eta \in \{0.2, 0.4, 0.6, 0.8\}$  for symmetric noise and  $\eta \in \{0.1, 0.2, 0.3, 0.4\}$  for asymmetric noise.
Networks and training. We follow the experimental setting in previous works [6, 17]. A 4-layer CNN is used for MNIST, an 8-layer CNN is used for CIFAR-10, and a ResNet-34 [29] is used for CIFAR-100. For MNIST, CIFAR-10, and CIFAR-100, the networks are trained for 50, 120, and 200 epochs, respectively. For all the training, we use SGD optimizer with momentum 0.9 and cosine learning rate annealing. Weight decay is set to  $1 \times 10^{-3}$ ,  $1 \times 10^{-4}$ , and  $1 \times 10^{-5}$  for MNIST, CIFAR-10, and CIFAR-100, respectively. Particularly, for our proposed ANL methods, weight decay is set to 0 for all datasets. The initial learning rate is set to 0.01 for MNIST/CIFAR-10 and 0.1 for CIFAR-100. Batch size is set to 128. For all settings, we clip the gradient norm to 5.0. Typical data augmentations including random width/height shift and horizontal flip are applied.
Parameter tuning. For each dataset, we tune the parameters of ANL-CE under 0.8 symmetric noise and then use them directly for all other noise settings and ANL functions. Specifically, we use  $10\%$  of the original training set as the validation set, and generate 0.8 symmetric noise on the remaining  $90\%$  of the original training set as the training set by the standard noise generation approach. We tune the parameters  $\alpha \in \{0.1, 0.5, 1.0, 5.0, 10.0\}$ ,  $\beta \in \{0.1, 0.5, 1.0, 5.0, 10.0\}$ , and  $\delta \in \{1 \times 10^{-7}, 5 \times 10^{-7}, 1 \times 10^{-6}, 5 \times 10^{-6}, 1 \times 10^{-5}, 5 \times 10^{-5}, 1 \times 10^{-4}\}$ .
Table 4: Parameters settings for different methods.
Method	MNIST	CIFAR-10	CIFAR-100	WebVision
CE (-)	(-)	(-)	(-)	(-)
FL (γ)	(0.5)	(0.5)	(0.5)	-
MAE (-)	(-)	(-)	(-)	-
GCE (q)	(0.7)	(0.7)	(0.7)	(0.7)
SCE (α, β)	(0.01, 1.0)	(0.1, 1.0)	(6.0, 0.1)	(10.0, 1.0)
NCE+MAE (α, β)	(1.0, 10.0)	(1.0, 1.0)	(10.0, 0.1)	-
NCE+RCE (α, β)	(1.0, 10.0)	(1.0, 1.0)	(10.0, 0.1)	(50.0, 0.1)
NFL+RCE (α, β, γ)	(1.0, 10.0, 0.5)	(1.0, 1.0, 0.5)	(10.0, 0.1, 0.5)	-
NCE+AEL (α, β, α)	(0.0, 1.0, 3.5)	(1.0, 4.0, 5.0)	(10.0, 0.1, 1.5)	-
NCE+AGCE (α, β, a, q)	(0.0, 1.0, 4.0, 0.2)	(1.0, 4.0, 6.0, 1.5)	(10.0, 0.1, 1.8, 3.0)	(50.0, 0.1, 2.5, 3.0)
NCE+AUL (α, β, a, p)	(0.0, 1.0, 3.0, 0.1)	(1.0, 4.0, 6.3, 1.5)	(10.0, 0.015, 6.0, 3.0)	-
ANL-CE (α, β, δ)	(1.0, 1.0, 1 × 10-6)	(5.0, 5.0, 5 × 10-5)	(10.0, 1.0, 5 × 10-7)	(20.0, 1.0, 6 × 10-6)
ANL-FL (α, β, δ, γ)	(1.0, 1.0, 1 × 10-6, 0.5)	(5.0, 5.0, 5 × 10-5, 0.5)	(10.0, 1.0, 5 × 10-7, 0.5)	(20.0, 1.0, 6 × 10-6, 0.5)
Table 5: Test accuracies  $(\%)$  of different methods on benchmark datasets with symmetric  $(\eta \in \{0.2, 0.4, 0.6, 0.8\})$  or asymmetric  $(\eta \in \{0.1, 0.2, 0.3, 0.4\})$  label noise. The results (mean±std) of our methods are reported over 3 random runs under different random seeds  $(1, 2, 3)$ . The reported results of PHuber-CE and Taylor-CE are directly taken from the original paper of Taylor-CE. The top-2 best results are in bold.
Datasets	Methods	Symmetric Noise Rate (η)	Asymmetric Noise Rate (η)
0.2	0.4	0.6	0.8	0.1	0.2	0.3	0.4
CIFAR-10	PHuber-CE [25]	85.81±0.21	80.25±0.22	67.71±0.19	32.97±0.32	87.91±0.13	84.87±0.26	79.01±0.38	71.87±0.36
Taylor-CE [26]	85.96±0.09	80.51±0.11	66.36±0.32	33.48±0.44	87.34±0.12	85.02±0.11	79.37±0.12	72.65±0.11
ANL-CE (ours)	87.43±0.24	83.18±0.54	73.80±0.24	45.18±1.27	88.29±0.07	86.54±0.27	83.12±0.58	75.07±0.43
ANL-FL (ours)	86.75±0.70	82.97±0.46	74.47±0.21	45.52±2.24	88.42±0.16	86.36±0.54	83.26±0.14	74.95±1.21
CIFAR-100	PHuber-CE [25]	58.11±0.11	50.89±0.13	35.85±0.29	13.83±0.25	60.07±0.09	53.30±0.10	44.39±0.14	35.36±0.13
Taylor-CE [26]	59.11±0.11	50.99±0.09	38.31±0.12	15.96±0.31	60.96±0.21	55.45±0.12	45.81±0.19	35.45±0.25
ANL-CE (ours)	59.20±0.36	53.42±0.18	44.42±0.09	26.27±0.81	60.58±0.68	58.11±0.60	49.91±0.42	38.57±0.72
ANL-FL (ours)	59.23±0.17	53.79±0.50	44.71±0.64	26.89±0.62	61.51±0.33	58.05±0.11	50.27±0.54	39.14±0.98
Parameter settings. For all baseline methods, the parameters are set to match their original papers. Detailed parameter settings can be found in table 4.
Results. The experimental results of symmetric and asymmetric label noise are reported in table 6 and table 7, respectively.
D.3 Evaluation on Real-world Noisy Labels
We train a ResNet-50 [29] using SGD for 250 epochs with initial learning rate 0.4, nesterov momentum 0.9 and weight decay  $3 \times 10^{-5}$  (for our proposed ANL methods, the weight decay is set to 0) and batch size 512. The learning rate is multiplied by 0.97 after every epoch of training. For all settings, we clip the gradient norm to 5.0. All the images are resized to  $224 \times 224$ . Typical data augmentations including random width/height shift, color jittering and random horizontal flip are applied. Detailed parameter settings can be found in table 4.
D.4 More comparisons with other baseline methods
We also compare our proposed ANL with two other baseline methods, PHuber-CE [25] and Taylor-CE [26]. For a fair comparison, we followed the experimental setup in the Taylor-CE paper and compared it directly with its reported experimental results. We use Adam [30] optimizer with the number of epochs set to 200 and the batch size set to 256. We use ResNet-34 for CIFAR-10/100. We tune our parameters and select the learning rate lr from  $\{10^{-2}, 10^{-3}, 10^{-4}, 10^{-5}\}$ . For CIFAR-10 and CIFAR-100, we set the parameters  $(\alpha, \beta, \delta, \mathrm{lr})$  to  $(5.0, 5.0, 1 \times 10^{-7}, 1 \times 10^{-4})$  and  $(10.0, 1.0, 1 \times 10^{-7}, 1 \times 10^{-4})$ , respectively. The results in table 5 demonstrate that our proposed ANL can outperform both PHuber-CE and Taylor-CE on the CIFAR-10/-100 dataset with symmetric or asymmetric label noise.