$\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise
Authors: Jialiang Wang
Xiong Zhou
Deming Zhai
Junjun Jiang
Xiangyang Ji
Xianming Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Learning with Noisy Labels
Robust Loss Function
Excess Risk Bound
Primary area: optimization_for_deep_networks
TL;DR: We propose a simple yet effective method for mitigating label noise, which can be implemented with just two lines of code.

Abstract: Noisy labels pose a common challenge for training accurate deep neural networks. To mitigate label noise, prior studies have proposed various robust loss functions to achieve noise tolerance in the presence of label noise, particularly symmetric losses. However, they usually suffer from the underfitting issue due to the overly strict symmetric condition. In this work, we propose a simple yet effective approach for relaxing the symmetric condition, namely **$\epsilon$-softmax**, which simply modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $\epsilon$. Essentially, ***$\epsilon$-softmax** not only acts as an alternative for the softmax layer, but also implicitly plays the crucial role in modifying the loss function.* We  prove theoretically that **$\epsilon$-softmax** can achieve noise-tolerant learning with controllable excess risk bound for almost any loss function. Recognizing that **$\epsilon$-softmax**-enhanced losses may slightly reduce fitting ability on clean datasets, we further incorporate them with one symmetric loss, thereby achieving a better trade-off between robustness and effective learning. Extensive experiments demonstrate the superiority of our method in mitigating synthetic and real-world label noise.

e-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise
Jialiang Wang'*
Xiong Zhou
Deming Zhai'
Junjun Jiang'
Xiangyang Ji²
Xianming Liu
Faculty of Computing, Harbin Institute of Technology Department of Automation, Tsinghua University cswjl@stu.hie hit. edu. cn', cszx0hit.o www.hhiteedu.crr. csxm@hit edu. cn
Abstract
Noisy labels pose a common challenge for training accurate deep neural networks. To mitigate label noise, prior studies have proposed various robust loss functions to achieve noise tolerance in the presence of label noise, particularly symmetric losses. However, they usually suffer from the underfitting issue due to the overly strict symmetric condition. this work, we propose simple yet effective approach for relaxing the symmetric condition, namely e-softmax, which simply modifies the outputs of the softmax layer to approximate one-hot vectors with controllable error c. Essentially, c-softmax not only acts as alternative for the softmax layer; but also implicitly plays the crucial role modifying the loss function. We prove theoretically that e-softmax can achieve noise-tolerant learning with controllable excess risk bound for almost any loss function. Recognizing that e-sfttmax- enhanced losses may slightly reduce fitting ability on clean datasets, we further incorporate them with one symmetric loss, thereby achieving a better trade-off between robustness and effective learning. Extensive experiments demonstrate the superiority of our method mitigating synthetic and real-world label noise. The code is available https: /ttp:/giittt giihhu..... com/ com/cswji htwpw//eppp //ppss- tmax.
Introduction
In recent years, deep neural networks (DNNs) have achieved remarkable advancements across various machine learning tasks [1, 2]. Despite its significant success, the prevalence of noisy labels in real- world datasets is pervasive issue, often stemming from human bias lack of relevant professional knowledge [2]. The application of supervised learning methods directly to data with noisy labels consistently results in a decline in model performance [3]. Moreover, the ability to generalize from weak learners plays pivotal role in the alignment of large language models [4]. Consequently, the pursuit of noise-tolerant learning has emerged as a compelling and significant challenge within domain of weakly supervised learning, garnering increased attention in recent years [5, 6, 8].
The literature presents several strategies for remedying this issue, with the design of robust loss functions standing out as particularly popular approach due to its simplicity and broad applicability. Some previous works [9, 10, 5] theoretically proved that loss function noise-tolerant label noise under mild conditions if it is symmetric:
L((()),)) V EH
(1.1)
where & [K] is the label corresponding each class, C is constant, and H is the hypothesis class. Furthermore, Asymmetric Loss Functions (ALFs) [7] are proposed as an extension of symmetric losses, which are designed for clean-label-dominant noise. However, both symmetric and asymmetric
"Equal contribution
Cooresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

losses, such as Mean Absolute Error (MAE) [5] and Asymmetric Unhinged Loss (AUL) [7], encounter the underfitting problem and prove challenging to optimize [5, 6, 7]. The fitting ability of existing symmetric loss functions is constrained by the overly strict symmetric condition in Equation 1.1 [7]. Some approaches aim to improve the classical symmetric loss MAF by incorporating the robustness of the MAE and the rapid convergence of the Cross Entropy (CE). Examples include Generalized Cross Entropy (GCE) [11], Symmetric Cross Entropy (SCE) [12], and Jensen-Shannon Divergence Loss (IS) [13]. However, these loss functions often mechanically select intermediate value between the derivatives of CE and MAE, essentially representing a trade-off between fitting ability and robustness.
This prompts crucial question: How learning?
simultaneousty achieve both robustness and effective
Zhou et al. [14] proposed an alternative approach to achieve the symmetric condition, diverging from the development of new robust loss function. By restricting the hypothesis class H. which restricts the outputs of the prediction function f to one-hot vectors, any loss function can inherently become symmetric, i.e., Σ k=1 L(f(x),k arises from the fact that directly mapping outputs to one-hot vectors constitutes a non-differentiable operation. Accordingly, the crux of the matter lies in formulating an effective method constrain the outputs to one-hot vectors. Previous attempts, such as temperature-dependee softmax [14), sparseness constraint [15], sparse regularization [14], and variance enlargement [16], have aimed to approximate one-hot vectors through the application of regularization methods. Nevertheless, these methods lack predictability, fail to achieve a quantitative approximation one-hot vectors, and exhibit limited effectiveness, particularly at higher noise rates. Up to this point, reliable approach for rigorously enforcing one-hot vector outputs remains elusive. Addressing this gap continues to pose a significant challenge in realizing the symmetric condition.
C, Vx VL E L. However, notable challenge
In this paper, we present simple yet effective and theoretically sound approach for approximating outputs to one-hot vectors, which we term e-softmax. This method serves as a valuable alternative to the conventional softmax function in mitigating label noise. The distinctive attribute of e-softmax lies in its guarantee to possess a controllable approximation error to one-hot vectors, thus achieving perfect constraint for the hypothesis class. This approach is universally applicable across diverse mod- els and loss functions, as it only needs to implement a simple layer resembling softmax. Specifically, the process of applying our e-softmax is outlined as follows:
Step 1. p( [x) <- softmax(((xx),,
Step 2. Pt Ps m, where t arg max Pk kE(K)
Step 3. p(-xx) + p(1x))((m 1).
Herein, p(- xx) represents the prediction probabilities, Pk denotes the k-th element of the vector p(-Ix), and h(x) denotes the logits. Step 1 obtains the original predictions by the softmax function. Step 2 involves a hyperparameter m 2 0 to amplify the maximum term in the predictions with controllable approximation error to one-hot vectors. Step 3 performs a normalization to make predictions sum to one, which also reduces the values of non-maximum terms.
The above description underscores that e-softmax as a plug-and-play module applicable to any classifier incorporating a softmax layer. Through the adjustment of the parameter m, our approach allows for the quantitative approximation of output to one-hot vectors, and thus owns the ability for mitigating label noise in classification. The main contributions of our work are highlighted as follows:
We propose simple yet effective scheme, e-softmax, for mitigating label noise. This scheme operates as auug-and--lay module, seamlessly integrating with any classifier that incorporates softmax layer through just two additional lines of code.
We offer rigorous theoretical analyses, which indicate that e-softmax is capable of control- lably approximating one-hot vectors. Consequently, (-softmma--enhance loss functions can achieve noise-tolerant learning and Bayes optimal top-k error.
We develop practical loss functions that enhance noise-tolerant learning. These include integration with MAE, achieving a better trade-off between robustness and effective learning. Extensive experimental results demonstrate the superiority of our method.

---

2 Preliminary
Problem Formulation. In a typical supervised classification scenario, let X' C. R represent the d-dimensional imput space, and y [K] {1, .... K} is the label space, where is the number of classes. We are provided with a labeled dataset S = {(xn, Un) Vn) == where each (xn; Mo) is drawn i.i.d. from an underlying distribution D over X' y. The classifier / a mapping from the sample space to the label space, the prediction label y arg max ((x)x. Here, the prediction function f .' + Ax estimates the probability p(-|x), and Δк {u [0, 1]
= 1} represents the probability simplex. Typically, the function f is expressed as f softmax o h, where denotes the logits input to the softmax layer. In the context of deep learning, h is commonly neural network. The objective or loss function is defined as measure of distance L A x A + R. For classification problem, the loss function is characterized by L(u, ey)) where By represents the one-hot vector with its y-th element set to In this study, we consider the loss functional L, where VL E L. L(u, v) = Σ k= E(Ux, Uk) with a basic loss function f. For brevity, we slightly abuse notation by defining Lu k) ct).
the context of learning with noisy labels, the accessible training set is the noisy counterpart S = {(x.. Yn } rather than the clean set S. We characterize the noise corruption process as the flipping of the clean label of into its noisy version with probability denoted as " = pl x. y). Txx Lkzp lx, denotes the noise rate for x. Our focus is on two prevalent types of label noise [6. 7]:
Label Noise Model.
- Symmetric or uniform noise: T/x.y = 1 and 1/x,kty K K-I
1)
- Asymmetric or class-conditional noise: Tx, = 1 Th and Enjty 1x.k Tyy
where 1/x for symmetric noise, Txx = my denotes the noise rate for the y-th class, and 7x, is not necessarily equal to Tx..j i + for asymmetric noise.
We also empirically consider learning with human-annotated noisy labels.
Expected Risk and Noise Tolerance. In learning with clean labels, given loss function L E L and a prediction function f. the expected risk with respect to f is defined as: RR(() E(x,y)) (f y)]. The objective learn an optimal classifier f" that minimizes the expected risk, i.e., f E arg min JEF (f).
In the case learning with noisy labels, the corresponding noisy expected risk with respect to f is defined as:
RI(() = -ED ((1 L +
Σ (fx))
(2.1)
kzy
where Exety 1x. [x) is the noisy part that usually poses challenges in training accurate DNNs.
loss function is claimed to be noise-tolerant the global minimizer f of RT) also minimizes Rf),), that is, f E arg min, (f).
All-k Consistency. Consistency is an important property of a loss function. A standard consistency for achieving Bayes optimal top-1 error. We consider much stronger consistency for achieving Bayes optimal top-k error for any k E [K]. To this end, we introduce some definitions about top-k: consistency [17,8] 8).
For any vector f E TRKK we let ra (f) denote a top-k selector that selects the & indices of the largest entries of by breaking ties arbitrarily. Given a data (x, y), its top-k error is defined as err (f,x, x, y) I(y & Tu (x))) The goal of a classification algorithm under the top-k error metric learn predictor that minimizes the err, expected risk: Rer,, (f) (xxD) Jerry (f, x, v)].
For a fixed A: [K]. loss function L is top-k consistent if for any sequence of measurable functions x AR, we have the global minimizer I of (f) also minimizes Rem, (f), that is, f" E arg min, Rony (f). If the above holds for all & [K], it is referred to as All-k consistency.
3

---

Methodology and Theoretical Investigation
The symmetry condition in Equation 1.1, theoretically ensures that symmetric loss function can be noise-tolerant [5]. Existing methods primarily focus on designing new loss functions. Those derived based on this design principle exhibit drawbacks, such as being challenging optimize [5, and prone to encounter the gradient explosion problem [7]. In this work, we take an alternative approach by proposing to constrain the hypothesis class H. such that any loss functions will be approximately symmetric thereby rendering them robust to label noise.
3.1 Robustness
We introduce e-softmax to make the output fx) approximate one-hot vectors. The implementation of e-softmax is easy to follow, as outlined in the gray box of the Introduction Section 1, requiring just two additional lines of code alongside the standard softmax layer. This underscores that softmax is plug-and this following, we investigate theory how e-softmax realizes the controllable approximation of outputs to one-hot vectors, thereby enhancing the noise tolerance of any loss function.
module applicable to any classifier that incorporates softmax layer. In
Approximating One-Hot Vectors. We first introduce the concept of e-relaxation for hypoth- esis class and then prove 6-softmax can strictly approximate outputs to one-hot vectors with a controllable error.
Definition 1 (e-relaxation). Given a fixed vector v and its permutation set PP'', the e-relaxation of Py defined as the hypothesis class Hy,er which any hypothesis f E Hv, outputs vectors the e-ball of Pv, i.e. Hv.e {f minueP, [xx ull S e, V.).
Without loss generality, we consider v as a one-hot vector, which is common in machine learning, to facilitate the implementation and analysis. We then denote the permutation set of the one-hot vector as Pey. where all elements are also one-hot vectors. In accordance with Definition 1, we can further derive that:
Lemma 1. e-softmax can achieve e-relaxation for one-hot vectors:
V1-1/K
min 1ʃ(x) u112 se=
(3.1)
uEPa
m+1
where (xx e-softmax hxx).
Lemma suggests that e-softmax effectively enables f(x) to approximate one-hot vectors with
controllable error VI-1/K
m+1
Robustness Guarantee. We then establish theoretical guarantees for the robustness in mitigating
label noise, where the constrained hypothesis class Hey, is considered.
Zhou et al. [14] established the excess risk bound [18] under symmetric noise, which holds when outputs fall within an e-relaxation of permutation set. We prove more comprehensive conclusion by considering asymmetric noise, of which symmetric noise is a special case.
Theorem (Excess Risk Bound under Asymmetric Noise). In a multi-class classification problem, (L(u,, k)) I( k))l & ō when |lll -u2l2 S and
if the loss function I E L. satisfies 8 O - O, then for asymmetric label noise Ts.kk (1 myy) Vk # y. RR(() 0, the
excess risk bound f E Hv, can be expressed as
2c0
Re(S)) S +
(3.2)
a
where c ED (1-1)), a min "ly 1x,k), 8ú and 8 denote the global minimum of (f) and (f),), respectively.
Theorem demonstrate that under mild conditions for symmetric and asymmetric label noise, any loss function can be made noise-tolerant when the function f(x) increasingly approximates a permutation set Py (i.e., 5 + 0 as 0).
For example, consider the vector |ee, v2], its permutation set is defined Pv ([ei, v2). [v2, e1]}.

---

--Softmax-Enaance Loss Functions. Lemma enable f(x) = e-softmax o h(x) in closely approximating one-hot vector, aligns with the principle outlined in Theorem within the framework of the hypothesis class Hey.c- Hence, e-sottmax progressively enhances the noise tolerance of any loss function as the hyperparameter m approaches infinity -> m - oc and the discrepancy 5 0).
In this paper we consider CE loss and Focal loss (FL) [19]. We combine them with e-softmax, denoted as CE, and FL.. e-softmax approach is effective in adapting them to become more resilient noise, ensuring better performance the presence of label noise.
3.2 Consistency
Fundamentally, e-softmax not only acts as an alternative for the softmax layer. but also plays the crucial role in modifying the loss function. Consistency is important property of loss function. A standard consistency for achieving Bayes optimal top-1 error. We show much stronger consistency for achieving Bayes optimal top-k error for any k [K] of the CE loss when combined with softmax. To establish the All-k consistency, we first introduce some existing results of sufficient condition of top-k consistency by top-k calibration [17. 8].
Let Ps (f, q) denote that f top-k preserving with respect to the underlying label distribution q. i.e.,
if for all E [K] > 9[K+1] = fi /[R+1]- and Q < qμ = Ji S(x]- Here, G[k] denotes he k-th
greatest entry of For example, if [0.2, 0.4, 0.4), then quj = 0.4, 922) = 0.4, 93 0.2.
Definition 2 (All-k calibrated). For a fixed & [K], loss function L is called top-k calibrated if for all q E k it holds that:
inf
RL()> inf Ry(f).
(3.3)
FERK:-Pa(1
FERK
A loss fanction is called All-k calibrated if the loss function is top-k calibrated for all k E [K].
Yang and Koyejo [17] demonstrate that suppose a nonnegative top-k calibrated loss function, then L is top-k consistent. Furthermore, Zhu et al. [8] show that if f arg miny Rz(f) f) is rank preserving with respect to q, then is All-k calibrated. f called rank preserving w.r.t q, i... if for any pair q. qj it holds that f. < fg.
Then we establish comprehensive All-k consistency for CE, as follows:
Lemma 2. For one-hot label eyy. CE2 is All-k calibrated and All-k consistency. Theorem 2. For any label E A, let = arg max qe and = arg maXke[K] Ps if and 9v maXt4y qk
CE. is All-k calibrated and All-k consistency.
Lemma 2 and Theorem 2 mean that CE, performs well not only on the top-1 prediction, but also on the top-k predictions for any k [K]. We show the All-k consistency property of different losses in Table 1, the consistency of other losses refer to [8].
Table 1: All-k consistency between different loss functions.
Loss
CE
MAE
NCE
GCE
SCE
AUL
AGCE
AEL
LDR-KL
CE,
All-A Consistency
X
X
3.3
Gradient Analysis of e-Softmax.
To provide a comprehensive understanding of e-softmax in mitigating label noise, we further analyze the gradient of the CE loss when combined with e-softmax. The gradient of LCE, (f(x), with respect to the model h(x) can derived as follows:
OLCE, (f(x).v) αh(x)
t=
mm
(3.4)
Op
1 1
Dh(x)
where f = e-softmax o h, p(x) softmax(h((s)) (x))) denotes the probabilities by standard softmax, and arg max Pk is the class with the largest value in prediction probabilities.
Remark. The gradient in Equation 3.4 shows that CE, will be equivalent the standard CE if the maximum prediction is not the target class (i.c., * y). in which the division of m + 1 in probabilities
5

---

(a) CEc (1) 0)
(b) CE2 (1) 0.8)
(c) CE&+MAB (17 = 0) (d) CE& (1) 0.8)
Figure/ 1: Test accuracies CIFAR-10 under symmetric noise with different m, where the red box represents the zoomed-in accuracies of the last 20 epochs. (a) and (b) illustrate CE. with 0 (clean)
and 0.8 noise rates, respectively. (c) and (d) illustrate CMA+ (a 0.01, 3 5) similarly.
is omitted due to the partial deviation. Conversely, when the prediction class t matches the target _y__. class y, the gradient undergoes dynamic scaling by P This scaling results in smaller gradients, akin to a form of soft early-stopping [20]. which facilitates the mitigation of overfitting to noisy labels. Such characteristic enables Deep Neural Networks (DNNs) to efficiently fit clean samples in the early phases of training [21. 20], while simultaneously preventing the overfitting of noisy labels in the later stages of the training process. As illustrated in Figure (b), CE2 achieves stable test accuracy curve, even in the challenging scenario with 0.8 symmetric label noise, without overfitting noisy labels. On the contrary, CE with the standard softmax tends to rapidly overfit to noisy labels after the early phase of training, leading to poor performance.
3.4 Better Trade-off between Robustness and Effective Learning
It can be noted that the incorporation of e-softmax somewhat sacrifices the fitting ability of the CE loss on clean datasets, as shown in Figure I(a). Therefore, we need to enhance the fitting ability using additional techniques. Inspired by the Active Passive Loss [6], we propose to accommodate with the symmetric loss MAE. For instance, we formulate the combination CE, and MAE (a.k.a., CE,+MAR as follows
LCE,+MAE = a LCE, + B- IMAE,
(3.5)
ditto for FLA+
Lemma 3. For any loss function Le with e-softmax and symmetric loss function Legmmetric defined in Equation 1.1, the excess risk bound of Tue 3 Issymeerii is equivalent that of o Tee.
Lemma 3 suggests that the e-softmax-enhancee loss function Lg can be scamlessly integrated with any symmetric loss function while not modifying the inherent robustness. As can be noticed in Figure ((c) and Figure 1(d). CE,+MAE not only depicts strong fitting capabilities but also achieves better noise tolerance. More interestingly, the test accuracy on clean datasets obtained by CE,+MAE even exceeds that of the standard CE loss.
Striet Convexity of CE,+MAE. To elaborate on how the combination of CE, and MAE can overcome the underfitting issue, we conduct an in-depth analysis from the optimization perspective. When the prediction / = y, the gradients of CE,, CE and MAE w.r.t. Py (0. 1]. are tm Py and 2. respectively. As can be seen, CE and CE, are strictly convex, while MAE exhibits linearity. Morcover, CE has stronger convexity compared to CE. (specifically, the gradient of CE changes more rapidly as 1/p² > 1/(pg + ))),, rendering CE more susceptible to overfitting noisy labels while CE. suffering from underfitting for large as illustrated in Figure ((a) and Figure 1(b). Conversely, owing to the linearity, MAE treats every sample equally, making robust to label noise but leading to more training time for convergence [1]]. Hence, the combination of CE, and MAE, which notably forms a strictly convex function (where the convexity can controlled by m), can provide better trade-off between robustness and effective learning.
Association with APL. Additionally, our proposed MEA+MAE coincides with the concept of active and passive losses in [6]. Specifically, for loss function denoted as (ff(),,)) E, (x y) Ensy (x(f(x), k). L is active if (x(x), k) for any l A y, and L is passive if (2( k) + for some + y. Active losses only explicitly maximize the target probability s(x),. while passive (x losses also explicitly minimize non-target probabilities (f(x)x Jayy. For example, CE is an active
loss, while MAE passive. Based on these two loss terms, Ma [6] proposed to combine a robust active loss and a robust passive loss into an "Active Passive Loss" (APL) framework for improving

---

Table 2: Last epoch test accuracies (%) of different methods on CIFAR-I0/100 symmetric and asymmetric noise. The results "sd" are reported over 3 random runs and the top-2 best results are boldfaced.
Symmetric Noise Rate (1))
Asymmetric Noise Rate
CIFAR-10
Clean
0.2
0.4
0.6
0.8
0.2
0.3
0.4
CE
30.50g0
75.47
28.46.0.21
69116
18.95
86, 38
83 83.82a004
79.35
75.28
FL
29,70=024
74.50a0.18
58.23+0.0
38.69an.06
19.47
86.64s0.12
83.0 33.08=007
74.68+0.31 39.34
GCE
89.42-021
86,87
82,24=0.5
68.4344.26
25.82+1.08 03
88.43em.20
86.17 29
80.72+042
54.01+0:3
NLNL
20.73-0.0
73.70=n.05
63.90+0444
50.68+0,47
29.53+1.55
88.54+0.25
84.74-008
81. 81,26+043
76.97+0.52
SCE
91. 01.30.008
87.5880005
79.47.0.48
59. 1420.07
25.88
89, 87
86.48.0225
81.30a0.18
74.99
NCE+MAE
89. 89.0220.10
86.79
83.6020.14
75.93an.41
46.9620.67
88. 08.0322
85.53
81.10an.52
74.98
NCE+RCE
91.03=028
88.41
85.13
79. 79.20sm.06
25.28e1.26
90. 25.205
88.11s011
85.35a0.18
79.43
NFL+RCE
91.08-029
29.00.0.23
85,90.0.19
79.7920.52
55.47a2.73
89. 99.0.35
88.3 26
85.27a0.13
79.0510.35
NCE+AUL
91.062021
89. 19.077
85.7910.16
79.57a0.21
57.59.0.84
90.18
88, 30,0.0
85.2810.04
79. 19.144
NCE+AGCE
91.13.0.11
89.00 39.00s0.29
85,91
80.3629.36
49.98s4.1
89 9020.09
88, 36260.1
85.7340.12
79.281a.37
NCE+AEL
88.43
86.462022
83.061021
75.15
43.2240.46
37.59
85 98..4
82.8740.16
75.78
LDR-KL
91.38-0035
89. -9.01
85,46+0.11
74.93+0.38
34,78-0.67
10.24
88. 38.080
85.03+0.16
77.68+0.37
CE+LC
90.06.0.41
85.66.0.32
79. 18.
53.87a0.57
21.04.0.47
87.9920.06
84.01a001
19.71
74.3410.30
CE +
12
89.29.0.10
85.93:0.19
79.52s0.14
58.9620.70
90.30aa.11
88.62a0.18
85.5610.12
78.91
MAE
91.11
89.13
86.15-029
79.81
58.02a1.12
90.392a.15
88.40.0.07
85.31
79.0420.10
Symmetric Noise Rate (o))
Asymmetric Noise Rate (1)
CIFAR-100
Clean
0.2
0,4
0.6
0.8
0.1
0.2
0.3
0.4
CE
50
56.21+204
39.31
22.38Hl.74
7.33+0.10
65. 10H0.4
58 38.336
49.99e0.54
41.15+1.04
FL.
30.58-034
56.32+1.43
50.83+0.52
22.44+0.54
7.68+0.37
65.00+0.46
58. 12-0.4
51.16+1.32
4 41.46+0.3
GCF
25
64.55-036
56.60+161
45. 19..99
19.85-088
63 63.94
60.89-006
40.82 53.36+1,58
NLNL
68.72.0.00 46.992091
30.29:164
16.60an.90
11.01a2.48
59,55at.22
50.19
35.10a0.0 42.81
SCE
70.41=020 55.23
40.23
21.44aa..2
7.63+0.24
64.54ee0.0
57. 37.62s000
50.17
41.01
NCE+MAE
67.69=0os
63.21
17.91
45,26H0.44
23.72e0.99
65. 15.7001.04
62.87=0.2
55.82+0.19
41.86e0.27
NCE+RCE
67. 47
64.60a0.22
58.6420.19
45.25a0.50
24.8720.52
66. 2010.8
63.188027
55.05a0.32
41.21
NFL+RCE
68. 68.28.030
64.5720.52
57.6420.74
45.47a0.59
24.3520.32
66. 1820.8
63.632020
55.3320.25
40. 8220276677
NCE+AUL
69.5520.40
15.1240:36
55.8620.0
37.88an.32
12.6920.14
67.0620.23
58.1620.17
48.06a0.16
38.30a0.12
NCE+AGCE
68.78.024
65.302046
59.95=0.15
47.63a1994
67. 07.15 24.13
64.21s017
56.18
45.15
NCE+AEL
64.47.0.19 18.07.0.16
32.29
19.78s1.03
10.50.0.51
58. 38.20
50.19a061
43.82a0.32
35. 13.023
LDR-KL
71.03.028 56.692006
40.6910.66
22.5920.23
7.4920.3
65.9310.01
58.4 28.472001
50.9240.15
41,9410.37
CE+LC
71,80.034
56.2620.09
37.3620.9
17.4620.62
6.3240.16
65.8520.30
58. 28.842002
50,4610.12
40.9720.39
CE. +MAE
70.8320.18
65.45:0.31
59.2010.42
48.15
26.30s0.46
67.5828.04
64.52a0.18
58.4720.12
48.51:0.36
FL, +LA
70.58.0.8
65.4551.39
59.5820.30
48.09220.35
26.7320.45
67.73
64.80s029
58.88.0.30
48.1020.23
sufficient learning with underfitting losses. Note that CE, is also active, thus CE, +MAE coincides with the API. framework and further mitigates the underfitting issue.
To further validate CE,+MAR we incorporate it with sample selection, pseudo-label prediction [22], and MixUp [23], culminating in a semi-supervised learning algorithm we term CE,+MAE (Semi). The algorithm details can be found in the Appendix C. In our experiments, we use "CE, MMAE (Semi)" to ensure a fair comparison with other hybrid methods with sample selection and semi-supervised learning (SSL). No additional techniques are utilized for "ME,+MAE".
4 Experiments
In this section, we conduct extensive experiments to validate the superiority of e-softmax in mitigating label noise. Complete experimental setting and results can be found in the Appendix D and E.
4.1 Evaluation on Benchmark Datasets
We evaluate our proposed methods on benchmark datasets CIFAR- CIFAR-10/CIFAR-IIO label noise, following [6, 7].
CIFAR-100 [24] with synthetic
Baselines. We consider several baseline methods for comparison, including Standard CE and
FL [19]; MAE; GCE [11]; NLNL [25]; SCE [12]; APL [6], including NCE+MAE, NCE+RCE, and NFL+RCE; AFL. [7], including NCE+AEL, AGE+ and NCE+AUL; LDR-KL. [8]; and LogitClip [26], including CE+LC.
7

---

Table 3: Ablation experiments CIFAR-100 The results "mean=std" reported over 3 random runs and the best results are boldfaced. If m = O, CE, EMAE equals CE+MAE.
Symmetric
Asymmetric
CIFAR-100
Clean
0.4
0.8
0.4
CB
70.79
39.3 74
7,33-0 10
41.15s1.04
MAE
5,31
2.78a1.68
2.13
3.022
CE2 +MAE (m 0)
69.33=0.51
37.00=0.40
11.65e0.18
41.533a097
CE, (m 1e2)
70.55=0.77
39.39
13.05+0.58
48.51
MEA lc4)
70.83a0.8
59.20s0.42
26.3010.46
40.3620.66
CE, +MAE (m 1e5)
67.72a038
56.41=0222
22.14
7.56
10
0.8
0.6
0.6
0.0
0.4
0.3
0.2
0.5
05 0.8 1.0
0.0
02
64
(a) CE (17 0.2)
(b) CE (1y 0.4)
(c) CE, +MAE (17 = 0.2) (d) CE 4EA+ (1) 0.4)
Figure 2: Visualizations of learned representations on CIFAR-10 with symmetric label noise. The
x-axis and y-axis represent the first and second dimensions of the 2D embeddings, respectively.
Results. Table 2 presents the test accuracy of various loss functions under symmetric and asym- metric label noise. As can be seen, our proposed ssoftmax-enhannee loss functions, CE,+MAE and MMAE, demonstrate remarkable performance, ranking among the top-2 in most cases across both datasets. These methods consistently outperform others such as GCE, SCE, NLNL, NCE+MAE and LDR-KL, regardless of the noise rates. In scenarios of clean labels, CE, and MAAE also exhibit strong fitting abilities, outperforming NCE+RCE and AGE+ CIFAR- 100 with 0.4 asymmetric noise, most robust loss functions have no effect, but our methods achieve over 48% accuracy, significantly outperforming all other methods. These findings underscore the robustness and effectiveness of performance in various noise scenarios.
particular, on
loss functions, delivering their excellent
Ablation Experiments. We perform detailed ablation experiments to further explore the role of each component and hyperparameter m in our CEA+ experimental results are shown in Table 3. We can observe that CE will severely fit the noise label, and the symmetric loss MAE is difficult to optimize. CE+MAE (i.e., m 0) is a trade-off between robustness and fitting ability, increasing noise tolerance at the cost of reducing fitting ability on clean labels, consistent with previous works [11, 12, 13]. In particular, our CE, +MAE shows remarkable properties. As the parameter m experiences moderate increase, CE.+MAB not only achieves noise tolerance for symmetric and asymmetric noise, but also achieves effective learning for the clean scenario. Additionally, the experimental results suggest that strict constraints are better suited for symmetric noise, while looser constraints are more effective for asymmetric noise.
Visualization. We conduct further analysis to compare the effectiveness of CE,+MAE and traditional CE in learning representations. We train models with different label noise and use the trained models to extract feature representations of the test set by t-SNE [27]. The visualizations for CIFAR- 10 symmetric noise are depicted in Figure 2. Notably, the embeddings generated by CE show evident overfitting to label noise, as seen in the blending of embeddings from distinct classes. In sharp contrast, embeddings from the CE,+MA +MAE method consistently form clear, well-separated clusters, demonstrating its superior ability to learn robust and distinct representations under noisy label conditions.
4.2 Evaluation on Human-Annotated Datasets
We further conduct comparison studies on human-annotated datasets CIFAR-ION/CIFAL [28], following the experiment setting in [28].
8

---

Table 4: Best epoch test accuracies (%) of different methods on CIFAR-N datasets. We compare meth- ods without and with semi-supervised learning (SSL) and sample selection. The results "standisdd" are reported over 5 random runs and the best results are boldfaced.
Method
CIFAR-ION
CIFAR-10ON
Without SSL
Aggregate
Random
Random
Random 3
Worst
Noisy
CR
87.77+0.38
85.02-065
86.46-1.79
85.16-
77.69+1.55
55.50+0.66
Forward T
88. 28..24
86.8820.50
86. 1420.24
87 37.04g0.35
79.7920.46
57.01
GCE
87. 85
87.61
87.7020.56
87. $7.58
80.66a0.3
56. 36.73
T-Rovision
88.5 1
88.33+00.2
11.02
87. 37.79
80.48+120
51.55e0.31
Peer Loss
20
89.06+0.11
88. 19.7690.19
88.57
82,00=0.c
57.59
F-Div
91. 11.6440.34
89. 79.70..040
89.79 19.79
89.5520.19
82,53,0.55 52555
57.10 10...0
Negative-LS
91.9720.46
90.29.0.32
90.37
90.13a0.r
82,9940.6
58.5920.98
VolMinNet
89. 19.70
88.3020.12
88.27 38.2720009
88, 18.1990.41
80.53 20
57. 87.001
AGCE
88. 81
87.88.0.43
88.01
87.97
81.43a0 30
N/A
CEA+MAE
91.00m333
90.43sm.29
90.53+0.28
90.64:0.35
83.74a0.43
61.78tb.14
Method With SSL
CIFAR-10N
CIFAR-100N
Aggregate
Random
Random
Randorn 3
Worst
Noisy
Co-teaching+
90.61
89.70s0.27
89.47.0.18
89.54.0.22
83.26.0.1 17
57.88.0.24
JoCoR
91.4420.05
90.30.0.20
90.21
90.1 1
83.37.0.2 30
59,9720.24
ELR+
94.8 83
94.44++041
94.20+0.24
94.34=0.22
91.09+1.00
66.72+9.07
Divide-Mix
95.01+0.1
05.16+0.19
95.23-0.07
95.21-0.14
92.56+0
71.13++.48
CORES
95.25+0.09
94,45=0.14
94.88-0.31
94,74+0.03
1.66+0.00
55.72+0.4
CAL
91.97a0.32
90.931031
90.75.0.30
90.74:0.24
85.36.0.h
61.7310.4
PES (Semi)
94.66
95.06#0.15
95.192023
95.2240.13
92.68=022
70.36
SOP+
95.61
95.28+0.13
95.31
95.39en.11
93.24s021
67.81
Proto-semi
95.03+0.14
95.48 + 0.17
95.48-o.21
95.67-n.10
92.97-0.33
67.73+0.67
CE.+MAE (Semi)
95.95ta.06
95.79sa.13
95.91
95.96=0.19
95.12+0.10
71.97s0.18
Table 5: Last epoch accuracies (%) on the WebVision and ILSVRC12 validation sets and the Clothing IM test set. The best results are boldfaced.
Method
CE
GCE
SCE
AGCE
NCE+RCE
NCE+AGCE
LDR-KL
CE, +MAE
Top-1
66.08
61.96
67.92
69.48
66.88
66.00
69.64
71.32
WebVision
Top-5
84.76
76.80
86.36
87.28
86.48
85.20
87.16
88.48
Top-1
60.72
60.52
63.28
65.12
63.96
62,68
65.24
67.20
ILSVRC12
Top-5
84.76
76.56
85.16
86.12
84.68
84.96
86.12
87.48
ClothingIM
67.38
69.03
67.40
68.43
68.67
67.52
66.88
69.85
Baselines. For a fair comparison, we divide the baselines into those without and those with semi- supervised learning (SSL) and sample selection:
Without SSL: Standard loss CE, Forward T [29], GCE [11], T-Revision [30], Peer Loss [31],
F-Div [32], Negative-LS [33], VolMinNet [34], and AGCE [35].
With SSL: Co-teaching+ [36], JoCoR [37], ELR+ [38], DivideMix [39], CORES [40], CAL [41],
PES (Semi) [20]. SOP+ [42], and Proto-semi [43].
Results. Table 4 reports the accuracy results of each method on the human-annotated datasets. The results show that the proposed CE,+MAE and CE,+MAE (Semi) provide significant improve- ments in handling human-annotated label noise, especially at high noise rates. Among the methods without SSL, CE, stands out on the CIFAR-10ON "Noisy" case as the only method to exceed 61% accuracy. Within the methods with SSL, CE,+MAE (Semi) shows a pronounced superiority in all scenarios, especially in the most difficult CIFAR- "Worst" case and CIFAR-10OM "Noisy" case. In the CIFAR- "Worst" case, MEA+ (Semi) achieves an impressive accuracy rate of over 95%, significantly outperforming competing methods. These results underscore the effectiveness of the (-softmax-enhareeee loss function in counteracting label noise for human-annotated scenarios.
4.3 Evaluation on the
Datasets
We perform experiments on massively real-world noisy datasets, including WebVision [44), ILSVRCI2 (ImageNet) [45] and Clothing IM [46], following the experiment setting in [7].

---

Results. In Table 5, we showcase the accuracies achieved on WebVision, ILSVRCI2 and Cloth- ingIM by various leading methods. Notably, our CMA+ method outshines others, achieving the highest results on all real-world datasets. It surpasses CF by approximately 5.5% on WehVision and 6.5% on IISVRR12. For Clothing we finetune a pretrained ResNet-50, so the differences between the methods are relatively small, but our method still achieves the best accuracy. These results underline the robustness and efficacy of the e-softmax-enbanced loss function in real- world scenarios.
Conclusion
In this paper, we introduced e-softmax, simple yet effective and theoretically sound scheme for noise-tolerant learning. Our method is not only easy to implement but also can be seamlessly integrated with any softmax-based DNNs, requiring just additional lines of code. Our rigorous and comprehensive theoretical analysis reveals that e-softmax effectively alleviates the common issue of overfitting noisy labels. Furthermore. we propose to incorporate e-softmax-enhancee loss functions with MAE, achieving better trade-off between effective learning and robustness. Extensive experimental results demonstrate the superior performance of our method in mitigating label noise.
Broader Impacts
This work has the potential to advance the development of machine learning methods that can be deployed in contexts where it is costly to gather accurate annotations. This is an important issue in applications such as medicine, where machine learning has great potential societal impact. This work will not have negative social impacts.
Acknowledgements
This work was supported by National Natural Science Foundation of China under Grants 922701 16, 62071 and 632B2031, and in part by the Fundamental Research Funds for the Central Universities (Grant No. HIT.DZJJ.0023775
References
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444 2015.
[2] Bo Han, Quanming Yao. Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey label-noise representation learning: Past, present and future. arXiv preprint arXiv:201 1.04466, 2020.
[3] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMLR, 2017.
[4] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390. 2023.
[5] Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[6] Xingjun Ma. Hanxun Huang, Yisen Wang. Simone Romano, Sarah Erfani, and James Bailey, Normalized loss functions for deep learning with noisy labels. In International conference on machine learning, pages 6543-6553. PMLR, 2020.
10

---

[7] Xiong Zhou, Xianming Liu, Junjun Jiang. Xin Gao, and Xiangyang Ji. Asymmetric loss functions for learning with noisy labels. In International conference on machine learning, pages 12846-12856. PMLR, 2021.
[8] Dixian Zhu, Yiming Ying, and Tianbao Yang. Label distributionally robust losses for multi-clast classification: Consistency, robustness and adaptivity. In International Conference on Machine Learning, pages 09229-433322 PMLR, 2023.
[9] Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics, 43(3): 146-1151, 2013.
[10] Brendan Van Rooyen, Aditya Menon, and Robert C. Williamson. Learning with symmetric label noise: The importance of being unhinged. Advances in neural information processing systems,
28, 2015.
[11] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[12] Yisen Wang. Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision, pages 322-330, 2019.
[13] Erik Englesson and Hossein Azizpour. Generalized jensen-shannon divergence loss for learning with noisy labels. Advances in Neural Information Processing Systems, 0..30284-30297,, 2021.
[14] Xiong Zhou, Xianming Liu, Chenyang Wang, Deming Zhai, Junjun Jiang, and Xiangyang
Ji. Learning with noisy labels via sparse regularization. In Proceedings of the IEEE/CVF international conference computer vision, pages 72-81,
[15] Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learming research, 5()), 2004.
[16] Xiong Zhou, Xianming Liu, Hao Yu, Jialiang Wang, Zeke Xie, Junjun Jiang, and Xiangyang Ji. Variance-enlarged poisson learning for graph-based semi-supervised learning with extremely sparse labeled data. The Twelfth International Conference on Learning Representations, pages 1-, 2024.
[17] Forest Yang and Sanmi Koyejo. Conference on Machine Learning, pages 10727-10735. PMLR, 2020.
On the consistency of top-k surrogate losses. In International
[18] Peter L Bartleu, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473): 2006.
[19] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[20] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24392-24400 2021.
[21] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[22] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-600 2020.
[23] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
11

---

[25] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Ninl: Negative learning for noisy labels. Proceedings of the IEEE/CVF international conference on computer vision, pages 101-110, 2019.
[26] Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, and Yixuan Li. Mitigating memorization of noisy labels by clipping the model prediction. In International Conference Machine Learning, pages 36868-36886. PMLR, 2023.
[27] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). 2008.
[28] Jiaheng Wei, Zhaowei Zhu, Hao Cheng. Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2021.
[29] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944-1952, 2017.
[30] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable label-noise learning? Advances in neural information processing systems, 32, 2019.
[31] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International conference on machine learning, pages 0226-6236. PMLR, 2020.
[32] Jiaheng Wei and Yang Liu. When optimizing f-divergence is robust with label noise. In International Conference on Learning Representations, 2021.
[33] Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In International Conference on Machine Learning, pages
PMLR, 2022.
[34] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In International conference on machine learning, pages 6403-6413. PMLR, 2021.
[35] Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, and Xiangyang Ji. Asymmetric loss functions for noise-tolerant learning: Theory and applications. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[36] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? International Conference on Machine Learning, pages 17777173. PMLR, 2019.
[37] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13726-13735, 2020.
[38] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Grande Early- learning regularization prevents memorization of noisy labels. Advances in neural information processing systems,
2020.
[39] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020.
[40] Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. In International Conference on Learning Representations, 2021.
[41] Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. Proceedings of the IEEF/CV conference on computer vision and pattern recognition, pages 10113-10123, 2021.
12

---

[42] Sheng Liu, Zhihui Zhu, Qing Qu and Chong You. Robust training under label noise by over- parameterization. In International Conference on Machine Learning, pages 14153-14172. PMLR, 2022.
[43] Renyu Zhu, Haoyu Liu, Runze Wu, Minmin Lin, Tangjee Lv, Changjie Fan, and Haobo Wang. Rethinking noisy label learning in real-world annotation scenarios from the noise-type perspective. arXiv preprint
2023.
[44] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: 122800220,,, 2017.
[45] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 148-255, 2009. doi: 1001
[46] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 2691-2699, 2015.
[47] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pages 702-703, 2020.
[48] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7610,, 2020.
13

---

A Limitation and Discussion
The limitation of e-softmax is that it may slightly reduce fitting ability on clean case. Therefore, we propose combine the (-softmax-enhanced loss with the symmetric loss MAE, Consequently, our practical loss functions utilized for noise-tolerant learning exhibit a hybrid form similar to GCE and SCE, but their meanings are significantly different.
Comparing with GCE and SCE. GCE a hybrid of CE and MAE var the negative Box-Cox transformation [11]. SCE combines CE with Reverse CE (RCE), where the RCE component actually acts as a scaled version of the MAE. This relationship is unveiled through the following derivation,
adapted from Section 4.3 in SCE [12]: LRCE = - p(k x) log q(k x) = -p(y x) log -
ExSy pkk x)A - E k p(k x) = -A(1 p(y x)) - A LMAE- Consequently, SCE essentially translates to CE+MAE. Hence, GCE and SCE increases the fitting ability but reduces the robustness because of the CE term. Conversely, our CE, inherently robust. The combination of CE, and MAE does not reduce the robustness, as demonstrated by Lemma 3, and also improves the fitting ability. We perform further experiments cimparing with GCE and CE+MAE (SCE), the results can be seen in Table 6 Our +MAE obtains obviously the best results at all noise rates, significantly outperforming GCE and CE+MAE (SCE).
Meanwhile, we further compare our e-softmax with lemperature-dependennt softmax.
Comparing with Temperature- Demperrauure-eeendeett Softmma. softmax h(a)) where T is the temperature
parameter, useful technique for making outputs sparse [14]. Compared to our e-softmax, lemperature-dependent softmax does not achieve a quantitative approximation to a one-hot vector for each output, and therefore cannot achieve a controllable excess risk bound. We also perform further experiments cimparing with coperature-dependde with cemperature-dependee softmax as CE,, the results can be seen in Table 6. Our CE,+MAE obtains obviously the best results at all noise rates, significantly outperforming temperaturr-ddepeddent softmax.
softmax. For simplicity, we refer to CE
Table 6: Last epoch test accuracies (%) of ablation and comparetion experiments on CIFAR- 100. The results "sd are reported over 3 random runs. The best results are boldfaced and the best results of each method are underlined. If m 0, CE,+MAE equals CE+MAE.
Symmetric
Asymmetric 0.4
CIFAR-100
Clean
0.4
0.8
CE
70.79-0.58
39.31 19.31
7.33-0 10
41.15+1.04
MAE
5.31
2.78
2. 2.1390
3. 3.122
GCE (q 0.3)
70.3
38.72 88.772088
6.43
38.7921.47
GCE (q 0.5)
20.57=0
50.6 60064
38
GCE (g 0.7)
65.22+1.57
56,60 56.60-161
18.2340.25
40.82-085
GCE (q 0.9)
18.27-241
17.6 1-2.25
19.85+0.8 88
13.96-1.d
CE,+MAE 0.3)
70.00
36.87=2.12
10.41
40.37
CMA 0.5)
69.57=0.6
47.99s0.48
13.62en.24
45.53e1.19
CE,+MAE 0.7)
70.1
36.08=221
10.58+0.20
46.92+0.5
CE,+MAE (7 0.9)
69.32-0.7
36.34-147
11.19
42.27+092
CR&+MAR (m 0)
69.33-o.1
39.72-067
11.65+0.18
41.53=097
CE,+MAE (m 1c2)
40.55+047
48.39-0
13.05+0.58
48.51
CE++MAE le4)
18
59.20g9.2
26.30
40.362096
CE,+MAE (m- 1e5)
67.72
54.99=1.05
22. 14a.6
7.56x1.10
B Proof of Theorems
Lemma 1. e-softmax can achieve e-relaxation for one-hot vectors:
u1/2 S < 11-1/K
min lx(x) PeP
(B.1)
where f(x) e-softmax (xx).
14

---

Proof.
v - 2pr + _k==1 P
min [1(x)) u/22-
uEPa
m
Ex= Ph (pe Pk)
- Pt-
m +
Pt
VI-
m +
m +1
Theorem (Excess Risk Bound under Asymmetric Noise). In multi-class classification problem. the loss function L C satisfies ) (L(u,, k)) L(u,, k))1 < à when I|u uzlla S and
ō + 0 as + O, then for asymmetric label noise 1x < (1 T7g) Ak # y. if 0, the
excess risk bound for Hv,e can be expressed as
2c²
RL(F)) - 26
(B.2)
a
where e = ED (1 - (1))))) a minx, "By kx,,), fūj and S denote the global minimum of (f) and (f),), respectively.
Proof.
R)(()= ED [(1 m1y L(J(X).)) v\1 ED
Σ L((x)L((
]
< ED (1 nw)
C C+ Σ [L((())))))
+ EED
Σ Mx.M L(xx)((()), Ketten
(C 5)) -mn) ED
Σ (1 To (.x,))
A
where
C'= EK L(v, k), ditto
RT 2 (C - 5))Ep(1 7my) - -D
Σ
my mxxx)) [(xx))((((()))))
AマU
hence,
<2SED (1 -my)+
ED Σ (1
f/y T/x,k)
According to the assumption R2( = O, we have I(f'(x),,)) y) = then [xff((x,,)) - where + y. Since (x),, k) - Lx)* k) < 0 where k + y, the second term on the right of the inequality is a non-positive value. And R7 (f*) - R27 (f:)) 2 So we have
Ep
Σ (1 Tl Tx. (L(f) k) - (x)) 20d, ke
where Ep (1 ny)-
Let = mi Mly - k), we have Ep kzy - < that fú,, fr Hy,s means that Ex (L(ff - (x)((x) 5 25, then we obtain (x)
2cb Note
2cd
Ep LL) s 28
a
15

---

that is. RL(f,) S R2(ff) 28 2cd = 28 + 2cd
Lemma 2. For one-hot label Cy. CE. is All-k calibrated and All-k consistency.
Proof. Here c-softmax o h, p(-|x) softmax(h(x)) denotes the probabilities by standard softmax, E (0, 1] and / = arg ma] Pse is the class with the largest value in prediction probabilities.
if =
OLCE, (L (f(x).
log Px m-1
Opy
m +
apy
ah(ylx)
Opy
Oh(y)x)
+ 1 Py + m dh(y)x)
1
Opy @h(yxx)
Py (1 Py).
Py
Py
HLce (f(x).) 0h(x)
By the first-order optimality condition we have ex = 0 < Ey and pk Py
O, we have: Py = Hence, for any lo + y.
if t y:
aLEE, (f(x))
θ log Pv m+1
Opy Oh(y)x)
+ 1
Opy (h(y)x)
Oh(ylx)
0p,
m +
Pu
1 0py
-11 Pyy)
ah(y(x)
Py
By the first-order optimality condition aLen, (f(x)..) dh(x)
O, we have: Pg = 1. Hence, for any k + y.
we have Gj: = 0 < by and Pke < Ps Hence, CE, All-k calibrated. Since CE, is nonnegative, CE, is All-k consistency.
Theorem 2. For any label AR, let arg ma[ qi and l arg max Pk. if and qu maXk4y Qs
mti CE, is All-k calibrated and All-k consistency.
m
Proof. For 0Le, OLe, (f(x)..) we have: Oh(y)/x) θLer, (f(x), Oh(ylx) Pt m
9t m + 1
0p.
Σ lipht
Op: (h())x)
qb pk
m + ah(t/x)
= pt m
Pc(1 pc) Σ yh
(- PPPP).
Ps
By the first-order optimality condition
aLen, (f(x).9) Oh(ylx)
O, have:
Pc(1 1 Pt)) Σ
qsPr
Pr m
keft
+ 9t pt + m
(1 Pt)
P: gr (1 m)
m < 9t < 1, we can get 0 < < 1. Since, m+1
For 0eee, (f(x).g) (h()
we have:
0Ler, (f(x),
1 Op.
Σ
apc
1
Op;
Øh(j 4 y|x)
+ Oh(jlx)
Th Pk:
dh(jlx)
Oh(ixx)
Dj
ktt.j
1
= Pr + ( (-p,ppc)
Σ
q
(-P,pk) + g, (p: - 1)
Pk
k
16

---

By the first-order optimality condition
0Len, (f(x).g) (
O, we have:
PjPt
Σ
91
+
TePj + 4:Pp Lj
Pr.
q
qj
Pj RePt
Zktte + 4j
0.Pr + 1 qt.
Substituting Pt = 9:(1 + m) m, we can get Pj oj (m + 1). Since > m m+I 50 qj
m+1 and
0 < Pj
For i.j # t, if di < Φj. we have p; < Consider lettt and qt. because of the condition by 9kpy> - have ql qt 9: 9h qe m) 9x (m 1) 0.
Hence, CE, is All-k calibrated. Since CE, is nonnegative, so CE, is All-k consistency.
The gradient of CE..
OLCE, (f(x). αh(x)
Py
(
Dp.
1=
(B.3)
Bpy
t +
Py
(
where f = e-softmax o h, p(x) softmax (xx) denotes the probabilities by standard softmax, and arg max Pk: is the class with the largest value in prediction probabilities.
Proof. The proof is similar to Theorem 2.
if t y
OlCE, (f(x).v)
a log Pu m+1
Opy 0h(x)
m + 1
Opy
0h(x)
Op,
m + 1 Py + m 0h(x)
Pp
Py 0h(x)
if * y:
dLCE, (f(x),
- loog m+11
Ps
Opy
m Opy
@h(x)
Öpy
ah(x)
m
+ 1
Py
ah(x)
1 Opy
Py 0h(x)
Lemma 3. For any loss function L. with e-softmax and symmetric loss function Leymmenic defined in Equation 1.1, the excess risk bound of a Le + 3 Lyymmetric is equivalent to that of a Le.
Proof. For u1, u2 E Hr. and ug. U4 AK. we have
K
K
(a L.(u,,h) + 3 Lsymmetrie (us,, k))
Σ
(a L. k) + B Lsymmetrie (n k))
K
K
L. kk) -
Σ
a Le k1
k=
k=1
-0.
L.(ui,k)
Le (u k) Σ
k=1
k=1
<a. 5
17

---

C The Algorithm of CE,+MAR (Semi)
Algorithm 1 CE,+ +MAE (Semi)
1: Input:
The noisy labeled dataset S = {X., Fn.. 1.... .N). initialized model f. loss function +MA, total epochs Tal, robust learning epochs Trobust and tradc-off parameter À
2: for epoch to T robuss do:
3: Train f on S var LCE, +MAE
4: end for
5: for epoch Tahust to Tal do:
6:
Sample selection: Divide the dataser S into labeled (clean) dataset ,...,(((]) and unlabeled (noisy) dataset S, {(x.), = (x |SS.]}
S = ((x,,,
7:
for each minibatch Dj E Si and Du E S. do:
8:
On arzmax(f(zx)), In E D # #sseudo-label prediction
9:
D. (,,, Augment
W shuffle(concat(D); 1Dw))
D MixUp(D,, Wn) n 1,. [Di]
12:
D a = MixUp(Du, Wn+ID.) n = 1, 1, (Dul
13:
Loss| LCE, Dj)
Loss, LCE, +MAE
15:
Loss = Loss: - X Loss,
16:
Train f on Loss
17: end for
18:
end for
19: return f
Algorithm Details and Parameters. Reference to [20], we set Trabus 65, Trobust 300 and learning rate decay 0.1 at [60, 160, 260] epochs. Other experimental settings are the same as the CIFAR-N experiment [28] the Appendix D.
For sample selection: We simply select k samples from each class with the least loss as clean samples. For CIFAR-ION, we set 2500 for "Worst" case and 3500 for others. For CIFAR-1OON, we set t: = 250 for "Noisy" case and 350 for others. In practice, if > |sample_num), we set /ample num 20.
For pscudo-label prediction: In the actual training. we do the pseudo-label prediction using two standard augment versions from the sample. We add the probabilities and divide by to make the pseudo-label prediction. At the same time, we set the threshold a 0.2 and discard the samples whose prediction probability is less than the threshold.
For the Augment to D., we employ RandAugment [47]. set the trade-off parameter to grow linearly from 0 over 200 epochs. The MixUp parameter is set to 0.75 for epochs less than 100, and adjusted to for epochs greater than 100. CE,+MAB m = 1e4, a 0.5, B 1 is same as 10, 1, B for the semi-supervised
CIFAR-N experiment for the robust learning stage and learning stage. In CE,+MAE (Semi). we ensemble the outputs of two networks during inference and exchange the samples selected by the two networks during training. is customary for methods that train two networks simultancously [21, 36, 39, 38].
D Experiments
D.1 Evaluation on Benchmark Datasets
Noise Generation. We follow the approach previous studies [6, 7] to experiment with two types of synthetic label noise: symmetric (uniform) noise and asymmetric (class-conditional) noise. In the case of symmetric label noise, we intentionally corrupt the training labels by randomly flipping labels within each class to incorrect labels in other classes. As for asymmetric label noise, flip the labels
18

---

within a specific sets of classes: For CIFAR- 10, the flips occur from TRUCK - AUTOMOBILE, BIRD + AIRPLANE, DEER - HORSE, and CAT ++ DOG. For CIFAR- 100, the 100 classes are grouped into 20 super-classes, each containing 5 sub-classes, and we flip the labels within the same super-class into the next.
Experimental Setting. We follow the same experimental settings in [6. 7]: An 8-layer CNN is used for CIFAR-IO and a ResNet-34 for CIFAR-I00) The networks are trained for 120 and 200 epochs for CIFAR-10 and CIFAR-100 with batch size 128. We use the SGD optimizer with momentum 0.9 and cosine learning rate annealing. The weight decay is set to 1 x 10-4 and x 10- 5 for CIFAR-10
and CIFAR- 100. The initial learning rate is set to 0.01 for CIFAR-10 10 and 0.1 for CIFAR- 100. Clipping the gradient norm 5.0 and the minimum allowable value for log to 1 x 10-8 Typical data augmentations including random shift and horizontal flip are applied to CIFAR- 10; random shift, horizontal flip and random rotation are applied to CIFAR-100.
Parameters Setting. We set the parameter settings which match their original papers for all baseline methods [6, 7]. Specifically, for FL, we set 7 = 0.5. For GCE, we set g = 0.7 for CIFAR- 10, and q [0.5, 0.5, 0.7, 0.7, 0.9] for CIFAR-100 clean and symmetric noise (17 E [0, 0.2, 0.4, 0.6, 0.8), = 0.7 asymmetric noise. For SCE, we A 4, CIFAR-100. For APL (NCE+MAE, NCE+RCE and NFL+RCE), we set a 1, = 1 for CIFAR-10, and a = 10, = 0.1 for CIFAR- 100.. For NCE+AUL, we set a 6.3.q 115, 1,
0.1,
for CIFAR-10, and a
0.1 for
for 10, and = 6.9 3. a = 10, 5 = 0.015 for CIFAR-100. For NCE+AGCE, we set a 6. 1.5, Q 1, for CIFAR-10, and a = 1.8, q = 3, = 10, = 0.1 for CIFAR-100. For NCE,AEL, we set a 5,a 1, 4 for CIFAR-10, and a 1.5, = 1 = 0.1 for CIFAR-100. For CE+LC, we set 5 1,, 1.5] for CIFAR-10 clean and symmetric noise (1) E 00 0.2, 0.4, 0.6, 0.8) and 8 = -2.5 for CIFAR-10 asymmetric noise. We set 8 2.5 for CIFAR- 100 asymmetric noise and 0 = 0.5 for others. For LDR-KL, We set X = 10 for CIFAR-10 and 1 for CIFAR- 100. For our CE,+MAE, we set 3 = 5, m. 1e5, a = 0.01 for CIFAR-I0 symmetric, and m = 1e3, a = 0.02 for asymmetric. For CIFAR- we set 8 1, m = le4 and (0.1, 0.05, 0.03, 0.0125, 0.0075) for clean and symmetric noise (1) E 00, 0.2 0.6, 0.8). and m- = Le2, a = [0.015, 0.007, 0.005, 0.004] for asymmetric noise (1) E [0.1, 0.2, 0.3, 0.4)) For our FL, +MAE, we 7 0.1 and others same as CE, For NLNL, use the results in [7] directly.
D.2 Evaluation on Human-Annotated Datasets
Experimental Setting. We follow the experimental settings in [28]: Train a Resnet-34 using SGD for 100 epochs with initial learning rate 0.1, momentum 0.9, and weight decay 0.0005. Set the learning rate decay 0. at 60 epochs. Standard data augmentation including random shift and horizontal flip are applied. Best epoch test accuracies are compared. The results of the comparison
methods are taken directly from [28] and the original papers [35, 43].
Parameters Setting, For our CE,+MAE, we set m = 1e4, a = 0.5, B = 1 for CE,+MAB (Semi) has been covered in detail in the previous section C.
D.3 Evaluation on Real-World Dataset Web Vision
Experimental Setting. For WebVision, the training details follow [7]: We use the mini WebVision setting [6, 7] and train a ResNet-50 using SGD for 250 epochs with initial learning rate 0.4, nesterov momentum 0.9 and weight decay 3 x 10-5 and batch size 256. The learning rate is multiplied by 0.97 after each epoch of training. All the images are resized to 224 x 224. Typical data augmentations including random width/height shift, color jittering, and horizontal flip are applied. We train the model on Webvision and evaluate the trained model on the same 50 concepts on the corresponding WebVision and ILSVRC12 validation sets.
For Clothing M,, we use ResNet-50 pre-trained on ImageNet similar to [46]. All the images are resized to 224 x 224. We use SGD with momentum of 0.9, weight decay of 1 x 10-3 and batch size of 256. We train the network for 10 epochs with a learning rate of 5 x 10 and a decay of 0.1 at 5 epochs. Typical data augmentations including random shift and horizontal flip are applied.
Parameters Setting. We set the best parameter settings which match their original papers for all baseline methods [6, 7J. Specifically, for GCE, we set 0.7 for Vision and 0.6 for Clothing IM.
19

---

For SCE, we set A = - 10, 8 1. For NCE+RCE, we set a 50, = 0.1 for WebVision
and a 10, 8 for Clothing IM. For AGCE, we set le 5. 0.5. For NCE+AGCE.
we set 2.5.q a.a = 50, 8 = 0.1. For LDR-KI,, we set X = 1. For our CE&+MAE, we set m 1e3,a = 0.015, = 0.3 for WebVison and a 0.012, 3 0.1 for Clothing
E More Experimental Results
Visualization.
We show more visualizations of learned representations in Figure 3.
Detailed Experimental Results of CE,+MAR (Semi) The more detailed results are reported in Table 7.
Instance- Destance-Dependentt Noise. We follow the method in PDN [48] to generate iustance-dependen noise. The experimental setting is the same as DEFAR-O//IFAR-II 100. For CE +MAE on CIFAR-10. 10, set 0.045, 3 10, 1c5. For CIFAR- 1000 we use the same parameters symmetric noise. The results are reported Table 8.
0.8
0.9
0.6
0.5
0.4
0.2
0.3
0.2
0.2
(a)
CE (17 = 0)
(b) CE (1) = 0.2)
(c) CE (1 0.4)
(d) CE (7) = 0.6)
0.6
0.6
04
0.2
0.2
(e) CE, MAA (1) = 0)
(f) CE, MAAE (n 0.2) (g) CE,+MAE (1 = 0.4) (h) CE, +MAE (1) = 0.6)
Figure 3: Visualizations of learned representations on CIFAR-IO with different symmetric label noise
(n [0. 0.2, 0.4, 0.6)). The x-axis and y-axis represent the first and second dimensions of the 2D embeddings, respectively.
Table Last and best epoch test accuracies (%) of MEA (Semi) on CIFAR-N datasets. The results "teandstd" are reported over 5 random runs.
CIEAR-1ON
CIFAR-10ON
CE. +MAE (Semi)
clean
Aggregate
Random
Random 2 Random
Worst
clean
Noisy
Last
96.062015
95.832014
95.76.0.12
95.832012
95.87a0.11
95.012016
78.5420.33
71.78.02
Best
18
95.95-006
13.79%0
95.91
95.
95.12.0.10
78.79.024
71.97-0.8
Table 8:
Last epoch test accuracies (%) on CIFAR-10/100 instance dentedeed noise (IDN). The results "tdd are reported over random runs and the best results are boldfaced.
CIFAR-10 IDN
CIFAR-100 IDN
Method
0.2
0.4
0.6
0.2
0.4
0.6
CE
75.0520.31
57.27
37. 27.622002
54.4621.73
40.9 20.25
25.57.0.00
GCE
86. 25.39
79.35
52,30
61.9521.27
56.99an.42
44.192036
SCE
86. 17
74.56+0.9
49.6 19.63-014
55.58-074
40.31
25.63+076
NCE+RCE
89. 39.06s0.31
85.07
70.45:0.2
64, 49
15.15
43. 43.22a2311
NCE+AGCE
88. 90
85, 16.0.2
72.68
65.33.a.18
$8.59.0.68
43. 43,424024
LDR-KL
88. 15
84. 10H124
63. 1
59.19
43.74
16e.0
CE, +MAE
89.27:9.42
85.26
74.32
67.44.0.19
60.8040.30
46.5320.54
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope. Specifically, we provide a simple yet effective method for mitigating label noise with elaborated descriptions and theoretical results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work the Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaaiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: We provide the full set of assumptions the main paper and all proofs in Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe the experiment details in the Appendix and submit the code for reproducibility in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have submitted the code for with sufficient instructions to faithfully reproduce the main experimental results, And the datasets are obtained from open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public/guides
cesssooooicyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cc/ ctc//pblicc/ae /gwii//g////////eeeeeee Suoeee/mmddSaaii sssisssnn iisoPolicyy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have specified all the training and test details in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For all experiments, we include error bars for added clarity.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information in the experiment details. All experiments are implemented by Py PoTch and are conducted on NVIDIA GeForce RTX 4090.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tp:://////ueee htt.//nuripp.pee wwwpblic/Ethissuiieeeeo
Answer: [Yes]
Justification: We promise that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed broader impact of this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do pretrained language models, image generators, etc.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The existing asserts in this paper are properly credited an the license and terms of use explicitly mentioned and properly respected with appropriate citations.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "FS3mp6lxaN", "reviewer": "Reviewer_5cpw", "summary": "The author proposes the epsilon-softmax technique as a method to address label noise. Epsilon-softmax facilitates peaky predictions by increasing the value of the highest prediction, and it also functions to reduce the magnitude of the gradient when the prediction aligns with the given label. The author introduces the concept of All-k consistency to interpret this paradigm and presents experiments on prominent real-world benchmark datasets in the field of label noise learning, specifically WebVision and Clothing1M.", "questions": "1. Based on the gradient analysis, it appears that when m is sufficiently large, the sensitivity of performance to m would not be significant. However, as shown in Table 3, the difference in performance is quite notable. Does the author have an explanation for this discrepancy? Additionally, has the author investigated the results when m is infinite, meaning the CE loss is not used when the prediction matches the label?\n2. Has the author ever checked the results of using only the CE_(epslion) loss function? Sections 3.1 to 3.3 and Lemma 2 suggest the importance of the single CE_(epslion) loss function.", "limitations": "-", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed epsilon softmax by the author takes a different approach compared to existing symmetric-like functions, which aim to reduce the gradient value for entirely incorrect predictions. From my understanding, the author’s approach reduces the gradient value for predictions that match the given label. By providing the value and interpretation of this novel approach, the author has significantly broadened the scope of the label noise learning field with their straightforward yet impactful idea. The proposed method has the advantage of simple gradient computation without requiring additional high-cost operations, making it applicapable to other Label Noise Learning (LNL) methods. The author demonstrates the experimental value of this approach by applying it to both the cross-entropy loss function and the focal loss function.", "weaknesses": "This section addresses two major concerns. For minor concerns, please refer to the \"Questions\" part.\n1. The author mentions in line 40 the necessity for a method that can achieve both effective learning and robustness. While the proposed method offers a different perspective compared to symmetric-like loss methods, it is challenging to assert that it fully meets this necessity. Ironically, to balance the trade-off between effective learning and robustness, the author combines CE_(epsilon) loss and MAE loss. This ability to manage trade-offs is also found in other symmetric-like loss-based methods. In this context, I am interested in understanding why the proposed method might offer a better trade-off and whether it truly provides a better trade-off. I attempted to verify this through experimental comparison, but several issues arose: (1) There are no experiments that allow for a comparison of trade-offs. Experiments demonstrating the trade-off by varying alpha and beta are necessary. (2) The performance of existing methods is reported to be lower. For example, refer to the SCE paper.\n2. The proposed method introduces two additional hyperparameters: m and alpha / beta. Unfortunately, based on the recorded experimental results, the proposed method appears to be sensitive to these hyperparameters. If this is not true, providing comprehensive experimental results that show the effects of varying these hyperparameters would enhance the perceived value of the proposed method. \nAnd if the proposed method is indeed sensitive to changes in hyperparameters, I would like to see evidence that it is not sensitive to hyperparameter variations within the in-distribution domain. I recommend performing validation and test processes to identify optimal hyperparameters (refer to the processes outlined in the GJS and JS papers).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your positive comments. We would like to offer the following responses to your concerns.\n\n**1. Response to Weakness 1** \n\nThanks for your insightful comments.\n\n--- About the better trade-off\n\nThe better trade-off means that we achieve better performance on both fitting ability and robustness compared to previous works.\nPrevious works like GCE and SCE [1] increase fitting ability but reduce robustness due to the CE term. In contrast, our combination retains robustness, as demonstrated by Lemma 2, and also improves fitting ability. To verify this, we present comparison results on CIFAR-10 symmetric noise:\n\n|CIFAR-10|Clean|0.2|0.4|0.6|0.8|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|GCE|89.42±0.21|86.87±0.06|82.24±0.25|68.43±0.26|25.82±1.03|\n|SCE|91.30±0.08|87.58±0.05|79.47±0.48|59.14±0.07|25.88±0.49|\n|CE$_\\epsilon$+MAE|**91.40±0.12**|**89.29±0.10**|**85.93±0.19**|**79.52±0.14**|**58.96±0.70**|\n\nAs can be seen, CE$_\\epsilon$+MAE achieves better results on both the clean and noisy cases. \n\n--- About issue (1)\n\nWe offer the ablation experiments for $\\alpha$ and $\\beta$. For simplicity, we fix $\\alpha$ and then adjust $\\beta$. $\\beta=5$ for CIFAR-10 and 1 for CIFAR-100 is used in the paper. \"S\" is symmetric noise, and \"AS\" is asymmetric noise.\n\n|CIFAR-10|Clean|S (0.2)|S (0.4)|S (0.6)|S (0.8)|AS (0.1)|AS (0.2)|AS (0.3)|AS (0.4)|\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|$\\beta=1$|90.19±0.13|87.89±0.25|84.71±0.09|76.90±0.30|44.26±0.78|89.20±0.17|86.85±0.17|82.74±0.32|75.75±0.42|\n|$\\beta=5$|**91.40±0.12**|**89.29±0.10**|85.93±0.19|**79.52±0.14**|**58.96±0.70**|**90.30±0.11**|**88.62±0.18**|**85.56±0.12**|**78.91±0.25**|\n|$\\beta=10$|91.31±0.21|89.08±0.19|**86.06±0.08**|77.78±3.32|43.00±3.86|90.14±0.11|88.58±0.39|83.42±4.61|72.87±0.55|\n|**CIFAR-100**|**Clean**|**S (0.2)**|**S (0.4)**|**S (0.6)**|**S (0.8)**|**AS (0.1)**|**AS (0.2)**|**AS (0.3)**|**AS (0.4)**|\n|$\\beta=0.5$|**71.01±0.05**|64.31±0.32|52.07±0.76|43.32±0.76|15.42±0.88|65.50±0.84|62.22±0.24|53.27±0.31|41.48±0.32|\n|$\\beta=1$|70.83±0.18 |**65.45±0.31**|**59.20±0.42**|**48.15±0.79**|**26.30±0.46**|**67.58±0.04**|**64.52±0.18**|**58.47±0.12**|**48.51±0.36**|\n|$\\beta=5$|67.87±0.88|60.05±0.84|51.19±1.62|26.93±1.80|10.30±2.83|63.22±0.46|51.67±0.66|40.34±5.45|25.69±0.77|\n\n--- About issue (2)\n\nOur experimental setup precisely follows that of previous work [2], ensuring a completely fair comparison. The results we reproduced are consistent with [2]. We note that the key difference between our results and those in SCE paper [1] lies in the recording of epoch accuracy. [1] recorded the best epoch accuracy, while we recorded the last epoch accuracy. Consequently, [1] avoided overfitting noisy labels in the later stage of training and reported better results. For example, for CIFAR-10 with 0.8 symmetric noise, CE achieved about 39\\% accuracy at the best epoch and about 18\\% at the last epoch. \n\n**2. Response to Weakness 2**\n\nThanks for your kind comment. Ours, and most robust loss functions, are indeed somewhat sensitive to hyperparameters. For instance, NCE+AGCE [2] involves four hyperparameters. \n\nAs suggested and in line with [3], we provide a more extensive hyperparameter search for learning rate and weight decay. The results are available in the global rebuttal (please see response to Q2). As can be seen, our method achieves better results in many cases. \n\n**3. Response to Question 1**\n\nThanks for your insightful comment. Due to the excellent fitting ability of the neural network, even if $m$ is about $1e4$, the gradient will still play a guiding role in optimization.\nAccording to your suggestion, we conducted experiments on sufficiently large and infinity $m$. The results on CIFAR-100 symmetric noise are as follows:\n\n| CIFAR-100 | 0.2 | 0.4 | 0.6 | 0.8 |\n|---|:---:|:---:|:---:|:---:|\n| m=1e10 | 53.63±0.41 |36.94±0.16|24.56±1.31|11.24±0.46|\n| m=1e15 | 53.45±0.53 |36.54±0.47|24.51±0.71|11.42±0.53|\n| m=1e20 | 53.64±0.94 |36.75±0.01|25.47±0.60|10.95±0.17|\n|m=$\\infty$|54.07±0.59|36.54±1.44|24.94±0.98|11.12±0.76|\n\nAs can be seen, when $m$ is sufficiently large, there is no obvious\n difference between the results. \n\n**4. Response to Question 2**\n\nThanks for your kind comment. As suggested, the results of  CE$_\\epsilon$ are offered as follows, where \"S\" is symmetric noise and \"AS\" is asymmetric noise:\n\n| CIFAR-10 | Clean | S (0.2) | S (0.4) | S (0.6) | S (0.8) | AS (0.1) | AS (0.2) | AS (0.3) | AS (0.4) |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| CE | 90.50±0.35  | 75.47±0.27  | 58.46±0.21  | 39.16±0.50 | 18.95±0.38| 86.98±0.31|83.82±0.04 | 79.35±0.66 | 75.28±0.58  |\n|CE$_\\epsilon$|88.10±0.06 |**86.88±0.18**|**83.51±0.39**|**74.68±0.40**|**29.54±0.89**|**88.06±0.30**|**85.57±0.15**|**80.82±0.41**|**76.74±0.22**|\n|**CIFAR-100**|**Clean**|**S (0.2)**|**S (0.4)**|**S (0.6)**|**S (0.8)**|**AS (0.1)**| **AS (0.2)**|**AS (0.3)**|**AS (0.4)**|\n| CE | 70.79±0.58 | 56.21±2.04 | 39.31±0.74 | 22.38±0.74 | 7.33±0.10 | 65.10±0.74 | 58.26±0.31 |49.99±0.54|41.15±1.04 |\n|CE$_\\epsilon$|69.28±0.09 |**64.35±0.20**|**58.21±0.24**|**40.27±1.62**|**19.32±1.09**|**65.89±0.12**|**60.11±0.13**|**52.74±0.11**|**42.05±1.72**|\n\n[1] Symmetric cross entropy for robust learning with noisy labels, CVPR, 2019.\n\n[2] Asymmetric loss functions for noise-tolerant learning: Theory and applications, TPAMI, 2023.\n\n[3] Generalized jensen-shannon divergence loss for learning with noisy labels, NeurIPS, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cwEb4uunDn", "reviewer": "Reviewer_d71F", "summary": "The paper introduces “ϵ-softmax,” a method to adjust softmax outputs for better approximation to one-hot vectors, thereby mitigating the impact of label noise in classification tasks. The approach modifies the softmax layer outputs to include a controllable error term ϵ, aiming to improve noise robustness without extensive alteration to the network architecture. The authors provide theoretical backing for the effectiveness of ϵ-softmax in achieving noise-tolerant learning across various loss functions. Extensive experiments with both synthetic and real-world noisy datasets are conducted to validate the claims.", "questions": "1.This paper seems to focus on classification tasks. Does ε-Softmax also work well for regression tasks?\n\n2.Is ε-Softmax computationally efficient in terms of training time compared with baseline methods?\n\n3.This paper shows the excellent performance of ε-Softmax for label noise. Does ε-Softmax work for input (features) noise as well?\n\n4.I assume that in this paper, for the label noise models, both training and testing labels are noisy. I am curious about the performance when the training labels are clean and the testing labels are noisy.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.ϵ-softmax is presented as a plug-and-play module compatible with any existing classifier that uses a softmax layer, enhancing its practical utility.\n\n2.The paper proves that ϵ-softmax can achieve a controlled approximation to one-hot vectors, which is significant for learning with noisy labels.\n\n3.The methodology is backed by extensive experimental results showing its superiority over existing methods in handling label noise, with detailed results across multiple datasets and noise configurations.", "weaknesses": "This paper should pay attention to the axis labels of its figures. In Figure 1, the x-label is Epoch and y-label is Test Accuracy. In Figures 2 and 3, the axis labels are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your positive comments. We would like to offer the following responses to your concerns.\n\n**1. Response to Weakness**\n\nThanks for your kind comment. For Figures 2 and 3, we extract the high-dimensional features of the test set at the second-to-last fully connected layer, then project them into 2D embeddings using t-SNE. The x-label and y-label represent the first and second dimensions of the 2D embeddings, respectively. We will add the x-label and y-label in Figures 2 and 3 in the revised version.\n\n\n**2. Response to Question 1**\n\nThanks for your kind comment. For regression tasks, the goal is to predict a continuous numerical output rather than a categorical probability. Therefore, the softmax function and our $\\epsilon$-softmax are not suitable for regression tasks.\n\n**3. Response to Question 2**\n\nThanks for your insightful comment. We evaluate the training time of different loss functions on CIFAR100 using ResNet34, and record the average training time of an epoch on NVIDIA RTX 4090. The results are summarized in the table below.\n\n|   Loss   | Train time (s) |\n|:--------:|:--------------:|\n| CE  |      10.17|\n| GCE |      10.24|\n| SCE |      10.29|\n| NCE+MAE |10.39  |\n| NCE+AGCE|10.46  |\n| LDR-KL |10.38   |\n|CE$_\\epsilon$|10.26|\n|CE$_\\epsilon$+MAE|10.38|\n\nAs can be seen, the training time required for all methods is almost the same.\n\n**4. Response to Question 3**\n\nThanks for your insightful comment.  We use random masking, Gaussian blur and solarisation to  synthesize symmetric feature noise. The experimental results with 3 random trials are as follows:\n\n| CIFAR-10 | 0.2 | 0.4 | 0.6 | 0.8 |\n|---|---|---|---|---|\n| CE |       90.76±0.21 | 89.42±0.30 | 85.86±0.29 | 77.80±0.49 |\n| GCE |      88.78±0.25 | 86.52±0.05 | 82.12±0.36 | 71.37±0.29 |\n| NCE+RCE |  89.45±0.30 | 85.93±0.08 | 79.11±0.39 | 61.06±1.09 |\n| NCE+AGCE | 89.26±0.03 | 85.22±0.10 | 77.53±0.51 | 57.76±1.61 |\n| LDR-KL |   90.17±0.10 | 87.50±0.20 | 82.62±0.19 | 71.22±0.30 |\n|CE$_\\epsilon$+MAE|**91.41±0.29**|**90.27±0.15**|**87.30±0.04**|**80.03±0.23**|\n| **CIFAR-100**|**0.2** | **0.4** | **0.6** | **0.8** |\n| CE |       69.82±0.36 | 66.29±0.17 | 63.47±0.39 | 57.69±1.11 |\n| GCE |      67.49±0.79 | 65.82±0.19 | 59.70±0.54 | 48.35±0.86 |\n| NCE+RCE |  62.63±0.54 | 52.96±0.40 | 38.42±0.21 | 23.73±1.01 |\n| NCE+AGCE | 63.94±0.40 | 55.66±0.50 | 43.13±0.41 | 26.68±0.36 |\n| LDR-KL |**70.47±0.32**|**67.83±0.67**| 63.61±0.34 | 57.93±0.27 |\n|CE$_\\epsilon$+MAE|70.12±0.27 | 67.11±0.20 |**64.06±0.56**|**58.34±0.56**|\n\nWe believe that the feature noise primarily tests the fitting ability of the loss function, and we can see that our CE$_\\epsilon$+MAE also performs very well for  feature noise.\n\n\n**5. Response to Question 4**\n\nThanks for your kind comment. In the area of learning with noisy labels, it is commonly assumed that the training labels are noisy and the test labels are clean, which aligns with real-world applications of machine learning models. Having clean training labels and noisy test labels doesn't make sense in practical scenarios, and we don't recommend considering this scenario.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bEFTq3DDkf", "reviewer": "Reviewer_NkMU", "summary": "This manuscript proposes a novel method to approximate the symmetric condition of the loss function, which is necessary for robustness to label noise. Specifically, the proposed method, named \\\\( \\\\epsilon \\\\)-softmax, can adjust the model output to approximate one-hot vector. However, the proposed method alone suffers from underfitting, so the authors combined it with MAE to achieve better performance. The authors evaluated the proposed method on datasets with different noise types and rates.", "questions": "1. Is \\\\( \\\\epsilon \\\\)-softmax + MAE still All-\\\\( k \\\\) calibrated and All-\\\\( k \\\\) consistency?\n2. Can the proposed method perform better compared to sparse regularization?", "limitations": "The authors have acknowledged the limitations and societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This manuscript proposes a novel and simple method to approximate the symmetric condition of loss function.\n2. The theoretical analysis focuses not only on robustness to label noise but also on the top-k consistency of the loss function.\n3. The proposed method was evaluated on various noise types and rates, including class-dependent noise and real-world noise.\n4. It good to see the authors compare their proposed method with temperature-dependent softmax combined with MAE. The experimental results demonstrate the superiority of their proposed method compared to temperature-dependent softmax.", "weaknesses": "1. The theoretical discussion with temperature-dependent softmax is missing. As the authors mentioned in L42 to L53, there are other output restriction-based methods in the literature, such as temperature-dependent softmax. Although the authors claim that these methods “lack predictability, fail to achieve a quantitative approximation to one-hot vectors, and exhibit limited effectiveness,” there is no detailed discussion on why the proposed \\\\( \\\\epsilon \\\\)-softmax has superior properties.\n2. A direct comparison with sparse regularization [1] is missing. Sparse regularization utilizes temperature- dependent softmax, which this manuscript has already compared, to approximate one-hot vector output. However, sparse regularization also employs an additional regularization term, \\\\( \\\\ell_p \\\\)-norm \\\\( \\\\| p(\\\\cdot | x) \\\\|^p_p \\\\) to enhance performance, and this regularization term is equivalent to MAE only if \\\\( p = 1 \\\\). It’s necessary to highlight the advantages of the proposed method compared to this highly relevant approach.\n3. There is no ablation study on \\\\( \\\\alpha \\\\) and \\\\( \\\\beta \\\\). As the authors mentioned in L218, \\\\( \\\\epsilon \\\\)-softmax alone suffers from a loss in fitting ability, and they combined it with MAE to balance the robustness and effective learning. However, without the relevant ablation study, it’s unclear how this “trade-off” is achieved.\n4. The theoretical discussions and experiments regarding instance-dependent label noise are overlooked. In recent years, the instance-dependent label noise has attracted increasing attention [2,3,4]. Experimenting the proposed method on instance-dependent label noise can provide a better understanding of how the proposed method performs with different types of label noise. I encourage the authors to include related discussion in the revised manuscript.\n\n[1] Learning with noisy labels via sparse regularization, ICCV, 2021.\n\n[2] Part-dependent Label Noise: Towards Instance-dependent Label Noise, NeurIPS, 2020.\n\n[3] Learning with Instance-Dependent Label Noise: A Sample Sieve Approach, ICLR, 2021.\n\n[4] Instance-Dependent Label-Noise Learning With Manifold-Regularized Transition Matrix Estimation, CVPR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your positive comments. We would like to offer the following responses to your concerns.\n\n**1. Response to Weakness 1**\n\nThanks for your insightful comment. \nIn the following,  we  give the  theoretical discussion for temperature-dependent softmax.\n\nFor model with temperature-dependent softmax, i.e., $f(\\mathbf{x})= \\tau$-softmax$ \\circ h(\\mathbf x)$, we have: \n$$\\min _{\\mathbf u \\in \\mathcal{P} _{\\mathbf e_1}} \\\\|f(\\mathbf x)-\\mathbf u \\\\|_2=\\sqrt{1-2p _{t}+\\sum _{k=1}^K p_k^2} = \\sqrt{1-p _{t}-\\sum _{k=1}^K p_k(p_t-p_k)} \\le \\sqrt{1-p_t}\\le \\sqrt{1-1/K},$$\nwhere $\\mathbf p(\\cdot|\\mathbf x) = f(\\mathbf{x})$ and $t=\\arg\\max _{k\\in[K]}p _k$. The equality holds when each term of $h(\\mathbf x)$ is equal, no matter what value the temperature\n $\\tau$ takes, we have $p_k = 1/K, \\forall k$. \n\nFor $\\epsilon$-softmax, we have $\\min_{\\mathbf u\\in\\mathcal{P}_{\\mathbf e_1}}\\\\|f(\\mathbf x)-\\mathbf u\\\\|_2\\le \\epsilon = \\tfrac{\\sqrt{1 - 1/K}}{m+1}$ (Lemma 1). Our $\\epsilon$-softmax make every output approximating one-hot vectors with a smaller error $\\epsilon$. Thus, a smaller excess risk bound can be derived. \n\n**2. Response to Weakness 2 and Question 2**\n\nThanks  for your kind suggestion.\nSparse regularization (SR) is a regularization method for probabilities, independent of labels and distinct from a loss function. We tend to compare with robust loss functions in this paper.\nFollowing the suggestion, we compared our method with SR. The results for 0.8 symmetric and real-world noise are provided.\n\n|Method|CIFAR-10|CIFAR-100|WebVision|\n|---|---|---|---|\n|CE+SR|51.13±0.51|17.35±0.13|69.12|\n|CE$_\\epsilon$+MAE|**58.96±0.70**|**26.30±0.46**|**71.32**|\n\nAs can be seen, our approach significantly outperforms SR in difficult and real-world situations. In addition, SR requires dynamic adjustment of hyperparameters at each epoch, making it difficult to train. \n\n**3. Response to Weakness 3**\n\nThanks for your kind suggestion. \n\n--- About ablation study on $\\alpha$ and $\\beta$.\n\nWe offer the ablation experiments. For simplicity, we fix $\\alpha$ and then adjust $\\beta$,  $\\beta=5$ for CIFAR-10 and 1 for CIFAR-100 is used in the paper. \"S\" is symmetric noise, and \"AS\" is asymmetric noise.\n\n|CIFAR-10|Clean|S (0.2)|S (0.4)|S (0.6)|S (0.8)|AS (0.1)|AS (0.2)|AS (0.3)|AS (0.4)|\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|$\\beta=1$|90.19±0.13|87.89±0.25|84.71±0.09|76.90±0.30|44.26±0.78|89.20±0.17|86.85±0.17|82.74±0.32|75.75±0.42|\n|$\\beta=5$|**91.40±0.12**|**89.29±0.10**|85.93±0.19|**79.52±0.14**|**58.96±0.70**|**90.30±0.11**|**88.62±0.18**|**85.56±0.12**|**78.91±0.25**|\n|$\\beta=10$|91.31±0.21|89.08±0.19|**86.06±0.08**|77.78±3.32|43.00±3.86|90.14±0.11|88.58±0.39|83.42±4.61|72.87±0.55|\n|**CIFAR-100**|**Clean**|**S (0.2)**|**S (0.4)**|**S (0.6)**|**S (0.8)**|**AS (0.1)**|**AS (0.2)**|**AS (0.3)**|**AS (0.4)**|\n|$\\beta=0.5$|**71.01±0.05**|64.31±0.32|52.07±0.76|43.32±0.76|15.42±0.88|65.50±0.84|62.22±0.24|53.27±0.31|41.48±0.32|\n|$\\beta=1$|70.83±0.18 |**65.45±0.31**|**59.20±0.42**|**48.15±0.79**|**26.30±0.46**|**67.58±0.04**|**64.52±0.18**|**58.47±0.12**|**48.51±0.36**|\n|$\\beta=5$|67.87±0.88|60.05±0.84|51.19±1.62|26.93±1.80|10.30±2.83|63.22±0.46|51.67±0.66|40.34±5.45|25.69±0.77|\n\n--- About the better trade-off\n\nThe better trade-off means that we achieve better performance on both fitting ability and robustness compared to previous works.\nPrevious works like GCE and SCE increase fitting ability but reduce robustness due to the CE term. In contrast, our combination retains robustness, as demonstrated by Lemma 2, and also improves fitting ability. To verify this, we present comparison results on CIFAR-10 symmetric noise:\n\n|CIFAR-10|Clean|0.2|0.4|0.6|0.8|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|GCE|89.42±0.21|86.87±0.06|82.24±0.25|68.43±0.26|25.82±1.03|\n|SCE|91.30±0.08|87.58±0.05|79.47±0.48|59.14±0.07|25.88±0.49|\n|CE$_\\epsilon$+MAE|**91.40±0.12**|**89.29±0.10**|**85.93±0.19**|**79.52±0.14**|**58.96±0.70**|\n\nAs can be seen, CE$_\\epsilon$+MAE achieves better results on both the clean and noisy cases. \n\n**4. Response to Weakness 4**\n\nThanks for your comment. The results for instance-dependent noise are available in the global rebuttal (please see response to Q1).\nFurthermore, we provide the excess risk bound for instance-dependent noise: \n$$\\mathcal{R} _L(f  _\\eta^*)\\le 2\\delta + \\frac{2c\\delta}{a},$$\n\nwhere $c = \\mathbb{E} _\\mathcal D\\left(1-\\eta _{\\mathbf{x}}\\right)$, $a=\\min _{\\mathbf x,k}(1-\\eta _y-\\eta _{\\mathbf x,k})$,  $f^* _\\eta$ and $f^*$ denote the global minimum of $\\mathcal R_L^\\eta(f)$ and $\\mathcal R_L(f)$, respectively.\n\nThe proof is similar to that for asymmetric noise. We will add the proof in the revised version, as there is not enough space.\n\n**5. Response to Question 1**\n\nThanks for your kind comment. The answer is yes. \nWe rename the $\\alpha, \\beta$ for CE$ _\\epsilon$+MAE loss as  $1-\\lambda$ and $\\lambda$ for a better presentation, where $0 < 1 - \\lambda \\le 1$.\n\nFor $L=$ CE$_ \\epsilon$+MAE, label is $\\mathbf q$, and $f(\\mathbf x)=\\epsilon$-softmax$ \\circ h(\\mathbf x)$, we have\n$L = \\sum _{k=1}^K q _k(-(1-\\lambda))\\log f(\\mathbf x) _k + \\lambda(1-f(\\mathbf x) _k)).$ By Lagrangian multiplier, we have\n\n$$\\min _{f(\\mathbf x)} \\max _{\\alpha, \\beta _k \\mathcal \\geq 0} \\sum _{k=1}^K q _k \\left( -(1-\\lambda) \\log f(\\mathbf x) _k + \\lambda (1-f(\\mathbf x) _k) \\right) + \\alpha \\left( \\sum _{k=1}^K f(\\mathbf x) _k - 1 \\right) - \\sum _{k=1}^K \\beta_k f(\\mathbf x)_k$$\n\nConsider the stationary condition for $f(\\mathbf x)$, we have $\\frac{1}{f(\\mathbf x)_k^*} + \\frac{\\lambda}{1-\\lambda} = \\frac{\\alpha - \\beta_k}{q_k(1-\\lambda)}, 0<f(\\mathbf x)_k <1$. By complementary slackness, we can get $\\beta_k^* =0$ and $\\alpha >0$. Hence, we have $\\left( \\frac{1}{f(\\mathbf x)_k^*} + \\frac{\\lambda}{1-\\lambda} \\right) \\propto \\frac{1}{q_k}$ and consequently $f(\\mathbf x)^*$ is rank consistent with $\\mathbf q$, i.e., $L$ is All-$k$ calibrated. Since $L$ is non-negative, so $L$ is All-$k$ consistency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Py8fejDZWi", "reviewer": "Reviewer_MvPF", "summary": "This submission proposes a enhanced softmax layer for label-noise learning, namely $\\epsilon$-softmax. By incorporating with the well-known $\\epsilon$-relaxation, the proposed $\\epsilon$-softmax can regularize the outputs of the model and avoid fitting the label-noise sample. This simple and plug-and-play method theoretically bounds the output logits to be an approximated one-hot vector. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "- Why choose the MAE as the loss to incorporate with ${CE_\\epsilon}$? Following GCE and SCE, the MAE is indeed a practical and evaluated choice. Do there have any alternatives?\n- What is the connection or relationship between logit adjustment and $\\epsilon$-softmax?", "limitations": "No obvious limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is simple, plug-and-play, and effective. Unlike other label-noise robust losses, the proposed method not only works well by itself, but also can be integrated with other label-noise robust method such as DivideMix. To the best of my knowledge, this could one of the first works endow such property.\n- The theoretical analysis is comprehensive and make sense. The theoretical results suggests the proposed method possesses the Top-K error consistency and label-noise robustness.\n- The analysis between the most related previous works is in reason. The basic idea that balances the label-noise robustness and learning effectiveness has been researched for a long time, e.g., GCE. This submission clearly presents the connection between the proposed method and other symmetric losses.\n- The empirical results is effective and enough. This submission presents the comparison results between many label-noise robust losses and the proposed method achieves the best performance in most cases. Additionally, this submission provides the experimental results that demonstrated the plug-and-play property of the proposed method on sample-selection based method and loss-correction based method.", "weaknesses": "- The ablation studies on the gradient clipping should be conducted and providing experimental results with different backbones would be better.\n- It is exhaustive and labor-expensive to find the optimal $m$ for diverse datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your positive comments. We would like to offer the following responses to your concerns.\n\n**1. Response to Weakness 1**\n\nThanks for you kind comment. \n\n--- About ablation study on gradient clipping\n\nWe fully followed the experimental setup of previous work [1], and we used gradient clipping because it was used in [1].  This PyTorch component stabilizes training without impacting the method comparison. We report the results without using gradient clipping as follows,  where \"S\" is symmetric noise and \"AS\" is asymmetric noise.\n\n|CIFAR-10 w/o|Clean|S (0.2)|S (0.4)|S (0.6)|S (0.8)|AS (0.1)|AS (0.2)|AS (0.3)|AS (0.4)|\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|CE|90.63±0.18|74.67±0.25|57.77±0.69|38.84±0.66|19.41±0.49|87.17±0.23|83.58±0.56|79.69±0.07|74.76±0.11|\n|NCE+AGCE|90.85±0.24|88.91±0.11|85.96±0.14|**79.97±0.12**|43.85±3.98 |90.13±0.09|88.40±0.16|85.27±0.36|**79.79±0.37**|\n|LDR-KL|91.07±0.11|89.15±0.12|84.98±0.52|74.77±0.31|32.20±0.78|90.22±0.27|88.50±0.47|85.43±0.17|77.86±0.07|\n|CE$\\epsilon$+MAE|91.20±0.12|**89.22±0.25**|**86.10±0.21**|79.75±0.24 |**57.43±0.48**|**90.34±0.04**|**88.59±0.29**|**85.84±0.22**|78.72±1.09|\n|**CIFAR-100 w/o**|**Clean**|**S (0.2)**|**S (0.4)**|**S (0.6)**|**S (0.8)**|**AS (0.1)**|**AS (0.2)**|**AS (0.3)**|**AS (0.4)**|\n|CE|69.11±1.11|53.63±0.47|35.23±2.35|20.04±1.43|7.25±0.35|58.46±3.88|56.45±2.19|47.72±2.78|39.46±0.55|\n|NCE+AGCE|69.08±0.34|65.36±0.20|58.65±0.63|46.19±1.03|23.89±0.79|67.13±0.21|63.82±0.23|55.90±0.87|43.68±0.45|\n|LDR-KL|71.67±0.30|56.54±0.52|40.17±0.87|22.18±0.44|7.02±0.28|65.42±0.15|58.20±0.24|50.24±0.28|41.23±0.09|\n|CE$\\epsilon$+MAE|70.20±0.78|**65.90±0.18**|**59.09±0.66**|**47.03±1.04**|**26.47±0.62**|**67.48±0.62**|**64.19±0.31**|**58.12±1.11**|**48.24±0.36**|\n\nAs can be seen, whether gradient clipping is used or not does not affect the comparison of methods, and our method still shows excellent performance.\n\n--- About different backbone\n\nWe conduct more experiments using VGG-13.\nThe training settings are listed as follows: batch size 256, learning rate 0.1 with cosine annealing, weight decay 5e-4, and 200 epochs. The results with 3 random trials are as follows:\n\n|CIFAR-10|Clean|S (0.2)|S (0.4)|S (0.6)|S (0.8)|AS (0.1)|AS (0.2)|AS (0.3)|AS (0.4)|\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|CE|93.91±0.32|83.21±0.08 |65.96±0.53 |43.30±0.60|21.46±0.82 |91.53±0.08|87.75±0.14|83.11±0.38|77.18±0.31|\n|GCE|93.90±0.09|91.15±0.19|82.57±0.21 |57.75±0.65|24.67±0.94|92.28±0.10|87.87±0.32|81.87±0.30|76.99±0.51 |\n|NCE+AGCE|90.42±0.09|88.24±0.33|83.61±0.86|50.63±5.62|11.68±2.92|89.98±0.14|88.03±0.45|86.46±0.53|69.02±0.45|\n|LDR-KL|92.51±0.13|90.53±0.19|87.91±0.03|79.24±0.13|39.49±0.58|91.64±0.17|89.86±0.35|87.50±0.40|67.63±0.24|\n|CE$\\epsilon$+MAE|93.61±0.05|**91.62±0.24**|**88.38±0.39**|**80.31±0.47**|**48.05±1.50**|**92.67±0.38**|**90.13±0.19**|**87.99±0.07**|**81.69±0.48**|\n|**CIFAR-100**|**Clean**|**S (0.2)**|**S (0.4)**|**S (0.6)**|**S (0.8)**|**AS (0.1)**|**AS (0.2)**|**AS (0.3)**|**AS (0.4)**|\n|CE|71.71±0.18|59.96±0.47|47.26±0.40|30.50±0.14|9.68±0.53|66.62±0.59|60.47±0.33|53.35±1.09|43.26±0.24|\n|GCE|70.59±0.30|66.79±0.08|58.90±0.08|41.06±0.42|10.89±0.09|68.99±0.16|59.85±0.17|48.44±0.97|38.68±0.94|\n|NCE+AGCE|71.40±0.41|67.25±0.26|59.80±0.23|30.82±2.42|4.90±1.07|**69.85±0.01**|65.63±0.38|53.27±0.14|41.83±0.80|\n|LDR-KL|68.68±0.21|54.83±0.33|40.84±0.59|24.71±0.34|8.11±0.07|62.73±0.62|55.65±0.19|48.03±0.53|40.01±0.18|\n|CE$\\epsilon$+MAE|71.07±0.21|**67.53±0.13**|**60.76±0.24**|**44.49±0.19**|**23.29±0.68**|69.28±0.09|**65.84±0.26**|**57.80±0.10**|**47.09±0.80**|\n\nAs can be seen, our approach still has outstanding performance.\n\n**2. Response to Weakness 2**\n\nThanks for your kind comment. Our method, like most robust loss functions, requires different parameters for different datasets. Many approachs have been proposed  to find optimal parameters quickly and economically. One efficient approach is to random select a subset of the training set as a smaller training set, for instance, 1/5 of the original set. We can then train on this smaller set to search for parameters efficiently.\n\n**3. Response to Question 1**\n\nThanks for your insightful question. We chose MAE because it is the most classic symmetric loss, and there is no other reason. We recommend using any symmetric loss in combination with CE$_\\epsilon$ rather than non-symmetric loss, because combining with symmetric loss does not increase the excess risk bound, as demonstrated by Lemma 2.\n\n**4. Response to Question 2**\n\nThanks for your insightful question. Logit adjustment [2] modifies logits to encourage a large margin between rare and dominant labels for long-tail learning. However, logit adjustment requires prior knowledge of label distributions, which is not available for noisy labels. Therefore, it is not suitable for learning with noisy labels.\n\n$\\epsilon$-Softmax mitigates overfitting to noisy labels by adjusting the probabilities. We find that $\\epsilon$-softmax is also effective for long-tail learning, where we can use larger $m$ for dominant labels. Specifically, we set $m_k$ for the k-th class as $m \\cdot p_k^{label}$, where $p_k^{label}$ is the frequency of the k-th class labels among all labels. We use ResNet-32 for CIFAR-10-Lt and CIFAR-100-LT, following the setting in [2] with long-tail imbalance ratio 100. The average class accuracies with 3 random trials are as follows:\n\n|Method|CIFAR-10-LT|CIFAR-100-LT|\n|---|---|---|\n|CE|70.21±0.34|39.18±0.27|\n|CE$_\\epsilon$ (m=0.1)|71.75±0.46|40.08±0.25|\n|CE$_\\epsilon$ (m=0.5)|**72.53±0.69**|40.32±0.06|\n|CE$_\\epsilon$ (m=1)|72.50±0.40|**40.42±0.26**|\n\nThanks again for your kind comment, which has inspired us to explore the application of $\\epsilon$-softmax in other areas. We are pleased to see the flexibility of our plug-and-play method.\n\n[1] Asymmetric loss functions for noise-tolerant learning: Theory and applications, TPAMI, 2023.\n\n[2] Long-tail learning via logit adjustment, ICLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S9UoASaNhu", "reviewer": "Reviewer_4jLX", "summary": "This paper proposes $\\epsilon$-softmax to deal with label noise.  $\\epsilon$-softmax modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $\\epsilon$. Both theoretical and empirical studies show the effectiveness of the proposed method.", "questions": "1. The term \"symmetric condition\" in abstract needs further explanation.\n2. In the implementation steps in Line 61, is $\\mathbf{p}(\\cdot)$ the same as ${p}(\\cdot)$? Or what is the relationship between these two notations? I think the mathematical notations should be strictly used and defined.\n3. The robustness of the proposed $\\epsilon$-softmax loss is theoretically justified. However, I'm curious to know the insight, or the underlying reason for its robustness. The authors wrote that \"The distinctive attribute of $\\epsilon$-softmax lies in its guarantee to possess a controllable approximation error $\\epsilon$ to one-hot vectors, thus achieving perfect constraint for the hypothesis class.\" But, I cannot figure out why controlling approximation error $\\epsilon$ to one-hot vectors and achieving perfect constraint for the hypothesis class are useful for handling label noise? What is the direct reason? It would be better if the authors can provide some intuitive explanations.\n4. From the experiments, I note that the performance is a bit sensitive to different selection of m. Therefore, is it possible to give some guidance in choosing m for practical use? Besides, will better performance be obtained if we add the $\\epsilon$-softmax loss functions with different m?\n5. From the experimental results in Table 2, 4, I can see that the improvement of the proposed method over other methods is quite marginal. I guess if we do statistical significance test, maybe such improvement will not be statistically significant.", "limitations": "I think this work will not have negative social impacts.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The writing of this paper is good.\n2. The robustness of the proposed loss is theoretically proved.", "weaknesses": "1. I think the motivation or the underlying reason for the effectiveness needs further explanation.\n2. In experiment, the advantage of the proposed method over the competitors is probably not statistically significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your valuable comments. We would like to offer the following responses to your concerns.\n\n**1. Response to Weakness 1 and Question 3**\n\nThanks for your kind comment.\n\nPrevious work indicates that, for a fixed vector $\\mathbf{v}$ and $\\forall L \\in \\mathcal{L}$, we have\n$$\\sum _{k=1}^K L(\\mathbf{u}, k) = C, \\forall \\mathbf{u} \\in \\mathcal{P} _{\\mathbf{v}},$$\n\nwhere $k \\in [K]$ denotes the label corresponding to each class, $C$ is a constant. This lemma suggests that when the network output $\\mathbf{u}$ is restricted to a permutation set $\\mathcal{P}_{\\mathbf{v}}$ of a fixed vector $\\mathbf{v}$, any loss function will satisfy the symmetric condition.\n\nIn this paper, we consider the fixed vector as a one-hot vector $\\mathbf{e} _1$. The key challenge is that directly mapping outputs to a permutation set $\\mathcal{P} _{\\mathbf{v}}$ is a non-differentiable operation. Based on these motivations, we propose a simple yet effective scheme, $\\epsilon$-softmax, to approximate one-hot vectors $\\mathcal{P} _{\\mathbf{e}_1}$. Thus, any loss function using $\\epsilon$-softmax can mitigate label noise with a theoretical guarantee. We will include this detailed explanation in the revised version.\n\n**2. Response to Weakness 2 and Question 5**\n\nThanks for your feedback.  We would like to emphasize that our experiments include dozens of  baselines and follow the same setting with previous works. On difficult noise and real world noise, our method achieves remarkable results.  These comprehensive evaluations aims to provide a robust assessment of our method. \n\nFor Table 2, as suggested by other reviewer, we search more carefully about\nhyperparameters and obtain better performance. More details are available in the global rebuttal (see response to Q2). The new results are provided as follows, where \"S\" is symmetric noise,  \"AS\" is asymmetric noise, and * denotes using better learning rate and weight decay.\n\n|CIFAR-10|Clean|S (0.2)|S (0.4)|S (0.6)|S (0.8)|AS (0.1)|AS (0.2)|AS (0.3)|AS (0.4)|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|NCE+AGCE|91.13±0.11|89.00±0.29|85.91±0.15|80.36±0.36|49.98±4.81|89.90±0.09|88.36±0.11|85.73±0.12|79.28±0.37|\n|CE$_\\epsilon$+MAE*|**91.94±0.18**|**89.76±0.08**|**86.77±0.16**|**80.47±0.87**|**58.96±0.70**|**91.06±0.13**|**89.58±0.29**|**87.78±0.23**|**82.47±0.56**|\n|**CIFAR-100**|**Clean**|**S (0.2)**|**S (0.4)**|**S (0.6)**|**S (0.8)**|**AS (0.1)**|**AS (0.2)**|**AS (0.3)**|**AS (0.4)**|\n|NCE+AGCE|68.78±0.24|65.30±0.46|59.95±0.15|47.63±0.94|24.13±0.06|67.15±0.40|64.21±0.17|56.18±0.24|44.15±0.08|\n|CE$_\\epsilon$+MAE*|**75.62±0.23**|**70.96±0.20**|**64.22±0.50**|**50.69±0.25**|**26.30±0.46**|**72.86±0.06**|**66.70±0.20**|**58.47±0.12**|**48.51±0.36**|\n\nAs can be seen, we significantly outperform the SOTA approaches.\n\nFor Table 4, we used the t-test to determine if our CE$\\epsilon$+MAE (Semi) statistically significantly exceeds the SOTA approaches. The p-values are as follows:\n\n|CIFAR-N|Aggregate|Random 1|Random 2|Random 3|Worst|Noisy100|\n|---|---|---|---|---|---|---|\n|Divide-Mix|0.018|0.000|0.000|0.000|0.000|0.006|\n|SOP+|0.001|0.000|0.000|0.000|0.000|0.000|\n|Proto-semi|0.000|0.012|0.002|0.001|0.000|0.000|\n\nAs can be seen, in all cases, our approach statistically significantly exceeds the previous sota approaches (p-value $<$ 0.05).  This additional analyses will be included in the revised manuscript.\n\n**2. Response to Question 1**\n\nA loss function $L$  satisfies the symmetric condition is defined as (Eq. 1.1 in our Introduction):\n\n$$\n\\sum_{k=1}^K L(f(\\mathbf{x}), k) = C, \\forall \\mathbf{x} \\in \\mathcal{X}, \\forall f \\in \\mathcal{H},\n$$\nwhere $k \\in [K]$ denotes the label corresponding to each class, $C$ is a constant, and $\\mathcal{H}$ is the hypothesis class. We will include this explanation in the abstract of the revised version to clarify this detail.\n\n**3. Response to Question 2**\n\nThanks for your comment. $\\mathbf{p(\\cdot|x)}$ denotes the prediction probability **vector** for the sample $\\mathbf{x}$. $p_k$ denotes the prediction probability **value** for the $k$-th class, i.e., the $k$-th element in the vector $\\mathbf{p(\\cdot|x)}$. We have followed common definitions of mathematical notations in academic paper, i.e., vectors are typically represented in boldface while scalars are usually represented in italicized font.\n\n**4. Response to Question 4**\n\nThanks for your suggestions. \n\n--- Guidance on Choosing $m$\n\nAn effective guideline is to use a larger $m$ for symmetric noise and a smaller $m$ for asymmetric noise. A larger $m$ imposes tighter symmetric constraints, which enhances robustness. However, if $m$ is too large, optimization on asymmetric noise can become challenging. For instance, on CIFAR-10, the symmetric MAE and CE achieve 45.36\\% and 18.95\\% accuracy under 0.8 symmetric noise, respectively. Conversely, MAE becomes difficult to optimize under asymmetric noise, resulting in 55.88\\% and 75.28\\% accuracy under 0.4 asymmetric noise.\n\n--- More experiments on different $m$\n\nWe conducted further experiments on CIFAR-10 with various values of $m$. The results are summarized below. \"S\" is symmetric noise, and \"AS\" is asymmetric noise. We use m=1e5 for symmetric noise and 1e3 for asymmetric noise in the paper.\n\n|CIFAR-10|Clean|S (0.2)|S (0.4)|S (0.6)|S (0.8)|AS (0.1)|AS (0.2)|AS (0.3)|AS (0.4)|\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|m=1e2|91.24±0.13|**89.33±0.09**|86.06±0.06|78.92±0.53|52.28±0.86|90.33±0.10|8.44±0.41 |85.16±0.18|78.36±0.16|\n|m=1e3|**91.49±0.22**|89.08±0.36|85.99±0.02|79.29±0.29|54.24±1.24|90.30±0.11|**88.62±0.18**|**85.56±0.12**|**78.91±0.25**|\n|m=1e4|91.02±0.21|89.32±0.40|**86.14±0.40**|**79.72±0.06**|57.41±1.32|90.20±0.04|88.49±0.39|82.73±4.26|71.65±0.66|\n|m=1e5|91.40±0.12|89.29±0.10|85.93±0.19|79.52±0.14|**58.96±0.70**|**90.44±0.10**|80.53±0.91|68.27±3.30|56.52±0.11|\n\nAs shown above, using different values of $m$ can sometimes yield better performance, while the differences are not substantial.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers for their valuable time and insightful comments. We have carefully considered the suggestions and will revise our manuscript accordingly.  We conducted some additional experiments in this global rebuttal space. Responses to other specific comments can be found in the individual rebuttal sections for each reviewer.

**Q1: Evaluation on Instance-Dependent Noise**

Following the previous work [1], we generate noise and set noise rates for instance-dependent noise. We use the same experimental settings and parameters as those described in our paper. The best results are highlighted in bold.

| CIFAR-10 IDN | 0.2 | 0.4 | 0.6 |
|---|:---:|:---:|:---:|
| CE | 75.05±0.31 | 57.27±0.96 | 37.62±0.02 |
| GCE| 86.95±0.38 | 79.35±0.30 | 52.30±0.12 |
| SCE| 86.79±0.17 | 74.56±0.49 | 49.63±0.14 |
| NCE+RCE | 89.06±0.31 | 85.07±0.17 | 70.45±0.26 |
| NCE+AGCE | 88.90±0.22 |85.16±0.26| 72.68±0.21 |
| LDR-KL | 88.99±0.15 | 84.10±0.24 | 63.11±0.23 |
|CE$_\epsilon$+MAE|**89.27±0.42**|**85.26±0.29**|**74.32±0.89**|
| **CIFAR-100 IDN**|**0.2**|**0.4**|**0.6**|
| CE | 54.46±1.73 | 40.81±0.25 | 25.57±0.03 |
| GCE | 61.95±1.37 | 56.99±0.42 | 44.19±0.36 |
| SCE | 55.58±0.74 | 39.71±0.39 | 25.63±0.76 |
| NCE+RCE | 64.13±0.49 | 57.15±0.24 | 43.22±2.31 |
| NCE+AGCE | 65.33±0.18 | 58.59±0.68 | 43.42±0.24 |
| LDR-KL | 59.19±0.34 | 43.74±0.12 | 26.10±0.16 |
|CE$_\epsilon$+MAE|**67.44±0.19**|**60.80±0.20**|**46.53±0.54**|

As can be seen, our method achieves the best performance among compared methods for instance-dependent noise.


**Q2: More Hyperparameter Search**

We conduct more hyperparameter search about learning rate and weight decay for our method. We search for learning rates (lr) in [0.01, 0.1] and weight decays (wd) in [1e-5, 1e-4, 5e-4]. lr=0.01, wd=1e-4 for CIFAR-10 and lr=0.1, wd=1e-5 for CIFAR-100 are used in the paper. The best results are highlighted in bold.


|CIFAR-10|Clean|S(0.2)|S (0.4)|S (0.6)|S (0.8)|AS (0.1)|AS (0.2)|AS (0.3)|AS (0.4)|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|lr=0.01, wd=1e-5|90.81±0.29|89.05±0.32|85.59±0.54|80.24±0.43|57.20±0.23|90.03±0.08|88.24±0.13|85.38±0.07|74.82±3.79|
|**lr=0.01, wd=1e-4**|91.40±0.12|89.29±0.10|85.93±0.19|79.52±0.14|**58.96±0.70**|90.30±0.11|88.62±0.18|85.56±0.12|78.91±0.25|
|lr=0.01, wd=5e-4|**91.94±0.18**|**89.76±0.08**|86.01±0.30|78.87±0.18|56.54±0.71|90.68±0.21|88.76±0.36|85.02±0.37|77.78±0.29|
|lr=0.1, wd=1e-5|61.47±1.08|57.09±5.70|50.60±10.72|35.75±1.47|27.88±10.13|80.38±3.68|74.63±7.77|76.36±3.62|59.72±10.09|
|lr=0.1, wd=1e-4|89.00±0.49|87.96±0.31|80.03±4.28|58.34±1.87|42.41±2.65|89.28±0.18|88.27±0.23|86.61±0.30|74.49±0.30|
|lr=0.1, wd=5e-4|91.11±0.36|89.44±0.29|**86.77±0.16**|**80.47±0.87**|35.56±5.59|**91.06±0.13**|**89.58±0.29**|**87.78±0.23**|**82.47±0.56**|
|**CIFAR-100**|**Clean**|**S (0.2)**|**S (0.4)**|**S (0.6)**|**S (0.8)**|**AS (0.1)**|**AS (0.2)**|**AS (0.3)**|**AS (0.4)**|
|lr=0.01, wd=1e-5|67.62±0.50|61.21±0.23|53.00±0.01|42.29±1.25|20.03±0.93|64.62±0.71|61.69±0.13|55.10±0.52|41.98±1.12|
|lr=0.01, wd=1e-4|69.31±0.49|62.66±0.19|54.69±0.35|43.67±0.22|18.49±0.64|65.13±0.28|63.05±0.30|55.45±0.33|41.52±1.27|
|lr=0.01, wd=5e-4|72.89±0.11|64.62±0.24|55.81±0.11|37.43±1.01|10.57±0.57|70.08±0.13|64.85±0.18|50.25±0.71|31.05±0.42|
|**lr=0.1, wd=1e-5**|70.83±0.18|65.45±0.31|59.20±0.42|48.15±0.79|**26.30±0.46**|67.58±0.04|64.52±0.18|**58.47±0.12**|**48.51±0.36**|
|lr=0.1, wd=1e-4|73.63±0.12|66.89±0.22|59.69±0.42|**50.69±0.25**|13.53±1.28|70.62±0.26|**66.70±0.20**|57.97±0.60|45.78±0.61|
|lr=0.1, wd=5e-4|**75.62±0.23**|**70.96±0.20**|**64.22±0.50**|14.07±0.51|1.10±0.00|**72.86±0.06**| 52.45±0.26|23.77±1.95|12.93±1.41|

As can be seen, we achieve better results in many cases.




[1] Part-dependent Label Noise: Towards Instance-dependent Label Noise, NeurIPS, 2020.

Meta-review: All reviewers provided positive recommendations (including 3 borderline accepts) for this paper. All reviewer recognized that the proposed method is simple and effective. They also suggested several aspects where the paper needs improvement, including hyperparameter sensitivity analysis, comparison with related works, and some ablation studies. In particular, Reviewer 4jLX pointed out the math should be rigorously defined and explained, which should be improved in the final version. Additionally, it might be better for the authors to add comparison or discussion with those most-related methods that transform the labels, logits or softmax outputs, such as label smoothing [1] and logit clipping [2].  

[1] Lukasik, Michal, et al. "Does label smoothing mitigate label noise?." International Conference on Machine Learning. PMLR, 2020.

[2] Wei, Hongxin, et al. "Mitigating memorization of noisy labels by clipping the model prediction." International Conference on Machine Learning. PMLR, 2023.
Decision: Accept (poster)