Title: $\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise
Venue: NeurIPS 2024
Accepted: yes
Keywords: Learning with Noisy Labels

Robust Loss Function

Excess Risk Bound
Abstract: Noisy labels pose a common challenge for training accurate deep neural networks. To mitigate label noise, prior studies have proposed various robust loss functions to achieve noise tolerance in the presence of label noise, particularly symmetric losses. However, they usually suffer from the underfitting issue due to the overly strict symmetric condition. In this work, we propose a simple yet effective approach for relaxing the symmetric condition, namely **$\epsilon$-softmax**, which simply modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $\epsilon$. Essentially, ***$\epsilon$-softmax** not only acts as an alternative for the softmax layer, but also implicitly plays the crucial role in modifying the loss function.* We  prove theoretically that **$\epsilon$-softmax** can achieve noise-tolerant learning with controllable excess risk bound for almost any loss function. Recognizing that **$\epsilon$-softmax**-enhanced losses may slightly reduce fitting ability on clean datasets, we further incorporate them with one symmetric loss, thereby achieving a better trade-off between robustness and effective learning. Extensive experiments demonstrate the superiority of our method in mitigating synthetic and real-world label noise.

ϵ -Softmax: Approximating One-Hot Vectors for Mitigating Label Noise

Abstract

Noisy labels pose a common challenge for training accurate deep neural networks. To mitigate label noise, prior studies have proposed various robust loss functions to achieve noise tolerance in the presence of label noise, particularly symmetric losses. However, they usually suffer from the underfitting issue due to the overly strict symmetric condition. In this work, we propose a simple yet effective approach for relaxing the symmetric condition, namely ϵ -softmax , which simply modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error ϵ . Essentially, ϵ -softmax not only acts as an alternative for the softmax layer, but also implicitly plays the crucial role in modifying the loss function. We prove theoretically that ϵ -softmax can achieve noise-tolerant learning with controllable excess risk bound for almost any loss function. Recognizing that ϵ -softmax -enhanced losses may slightly reduce fitting ability on clean datasets, we further incorporate them with one symmetric loss, thereby achieving a better trade-off between robustness and effective learning. Extensive experiments demonstrate the superiority of our method in mitigating synthetic and real-world label noise. The code is available at https://github.com/cswjl/eps-softmax .

1 Introduction

In recent years, deep neural networks (DNNs) have achieved remarkable advancements across various machine learning tasks [1, 2]. Despite its significant success, the prevalence of noisy labels in realworld datasets is a pervasive issue, often stemming from human bias or a lack of relevant professional knowledge [2]. The application of supervised learning methods directly to data with noisy labels consistently results in a decline in model performance [3]. Moreover, the ability to generalize from weak learners plays a pivotal role in the alignment of large language models [4]. Consequently, the pursuit of noise-tolerant learning has emerged as a compelling and significant challenge within the domain of weakly supervised learning, garnering increased attention in recent years [5, 6, 7, 8].

The literature presents several strategies for remedying this issue, with the design of robust loss functions standing out as a particularly popular approach due to its simplicity and broad applicability. Some previous works [9, 10, 5] theoretically proved that a loss function is noise-tolerant to label noise under mild conditions if it is symmetric:

where k ∈ [ K ] is the label corresponding to each class, C is a constant, and H is the hypothesis class.

Furthermore, Asymmetric Loss Functions (ALFs) [7] are proposed as an extension of symmetric losses, which are designed for clean-label-dominant noise. However, both symmetric and asymmetric

1 ∗

losses, such as Mean Absolute Error (MAE) [5] and Asymmetric Unhinged Loss (AUL) [7], encounter the underfitting problem and prove challenging to optimize [5, 6, 7]. The fitting ability of existing symmetric loss functions is constrained by the overly strict symmetric condition in Equation 1.1 [7]. Some approaches aim to improve the classical symmetric loss MAE by incorporating the robustness of the MAE and the rapid convergence of the Cross Entropy (CE). Examples include Generalized Cross Entropy (GCE) [11], Symmetric Cross Entropy (SCE) [12], and Jensen-Shannon Divergence Loss (JS) [13]. However, these loss functions often mechanically select an intermediate value between the derivatives of CE and MAE, essentially representing a trade-off between fitting ability and robustness. This prompts a crucial question: How can we simultaneously achieve both robustness and effective learning?

Zhou et al. [14] proposed an alternative approach to achieve the symmetric condition, diverging from the development of a new robust loss function. By restricting the hypothesis class H , which restricts the outputs of the prediction function f to one-hot vectors, any loss function can inherently become symmetric, i.e., ∑ K k =1 L ( f ( x ) , k ) = C, ∀ x ∈ X , ∀ L ∈ L . However, a notable challenge arises from the fact that directly mapping outputs to one-hot vectors constitutes a non-differentiable operation. Accordingly, the crux of the matter lies in formulating an effective method to constrain the outputs to one-hot vectors. Previous attempts, such as temperature-dependent softmax [14], sparseness constraint [15], sparse regularization [14], and variance enlargement [16], have aimed to approximate one-hot vectors through the application of regularization methods. Nevertheless, these methods lack predictability, fail to achieve a quantitative approximation to one-hot vectors, and exhibit limited effectiveness, particularly at higher noise rates. Up to this point, a reliable approach for rigorously enforcing one-hot vector outputs remains elusive. Addressing this gap continues to pose a significant challenge in realizing the symmetric condition.

In this paper, we present a simple yet effective and theoretically sound approach for approximating outputs to one-hot vectors, which we term ϵ -softmax . This method serves as a valuable alternative to the conventional softmax function in mitigating label noise. The distinctive attribute of ϵ -softmax lies in its guarantee to possess a controllable approximation error ϵ to one-hot vectors, thus achieving perfect constraint for the hypothesis class. This approach is universally applicable across diverse models and loss functions, as it only needs to implement a simple layer resembling softmax. Specifically, the process of applying our ϵ -softmax is outlined as follows:

Herein, p ( ·| x ) represents the prediction probabilities, p k denotes the k -th element of the vector p ( ·| x ) , and h ( x ) denotes the logits. Step 1 obtains the original predictions by the softmax function. Step 2 involves a hyperparameter m ≥ 0 to amplify the maximum term in the predictions with a controllable approximation error to one-hot vectors. Step 3 performs a normalization to make predictions sum to one, which also reduces the values of non-maximum terms.

The above description underscores that ϵ -softmax as a plug-and-play module applicable to any classifier incorporating a softmax layer. Through the adjustment of the parameter m , our approach allows for the quantitative approximation of output to one-hot vectors, and thus owns the ability for mitigating label noise in classification. The main contributions of our work are highlighted as follows:

We propose a simple yet effective scheme, ϵ -softmax , for mitigating label noise. This scheme operates as a plug-and-play module, seamlessly integrating with any classifier that incorporates a softmax layer through just two additional lines of code.

We offer rigorous theoretical analyses, which indicate that ϵ -softmax is capable of controllably approximating one-hot vectors. Consequently, ϵ -softmax -enhanced loss functions can achieve noise-tolerant learning and Bayes optimal topk error.

We develop practical loss functions that enhance noise-tolerant learning. These include integration with MAE, achieving a better trade-off between robustness and effective learning. Extensive experimental results demonstrate the superiority of our method.

2 Preliminary

Problem Formulation. In a typical supervised classification scenario, let X ⊂ R d represent the d -dimensional input space, and Y = [ K ] = { 1 , 2 , ..., K } is the label space, where K is the number of classes. We are provided with a labeled dataset S = { ( x n , y n ) } N n =1 , where each ( x n , y n ) is drawn i.i.d. from an underlying distribution D over X × Y . The classifier f is a mapping from the sample space to the label space, the prediction label ˆ y = arg max k f ( x ) k . Here, the prediction function f : X → ∆ K estimates the probability p ( ·| x ) , and ∆ K = { u ∈ [0 , 1] K : 1 ⊤ u = 1 } represents the probability simplex. Typically, the function f is expressed as f = softmax ◦ h , where h denotes the logits input to the softmax layer. In the context of deep learning, h is commonly a neural network. The objective or loss function is defined as a measure of distance L : ∆ K × ∆ K → R . For a classification problem, the loss function is characterized by L ( u , e y ) , where e y represents the one-hot vector with its y -th element set to 1. In this study, we consider the loss functional L , where ∀ L ∈ L , L ( u , v ) = ∑ K k =1 ℓ ( u k , v k ) with a basic loss function ℓ . For brevity, we slightly abuse notation by defining L ( u , k ) = L ( u , e k ) .

̸

Label Noise Model. In the context of learning with noisy labels, the accessible training set is the noisy counterpart ˜ S = { ( x n , ˜ y n ) } N n =1 rather than the clean set S . We characterize the noise corruption process as the flipping of the clean label of x into its noisy version ˜ y with a probability denoted as η x , ˜ y = p (˜ y | x , y ) . η x = ∑ k = y η x ,k denotes the noise rate for x . Our focus is on two prevalent types of label noise [6, 7] :

̸

Symmetric or uniform noise : η x ,y = 1 -η and η x ,k = y = η K -1 ,

̸

Asymmetric or class-conditional noise: η x ,y = 1 -η y and ∑ k = y η x ,k = η y ,

̸

where η x = η for symmetric noise, η x = η y denotes the noise rate for the y -th class, and η x ,i is not necessarily equal to η x ,j , i = j for asymmetric noise.

We also empirically consider learning with human-annotated noisy labels.

Expected Risk and Noise Tolerance. In learning with clean labels, given a loss function L ∈ L and a prediction function f , the expected risk with respect to f is defined as: R L ( f ) = E ( x ,y ) ∼D [ L ( f ( x ) , y )] . The objective is to learn an optimal classifier f ∗ that minimizes the expected risk, i.e., f ∗ ∈ arg min f ∈F R L ( f ) .

In the case of learning with noisy labels, the corresponding noisy expected risk with respect to f is defined as:

̸

̸

where ∑ k = y η x ,k L ( f ( x ) , k ) is the noisy part that usually poses challenges in training accurate DNNs.

A loss function L is claimed to be noise-tolerant if the global minimizer f ∗ η of R η L ( f ) also minimizes R L ( f ) , that is, f ∗ η ∈ arg min f R L ( f ) .

Allk Consistency. Consistency is an important property of a loss function. A standard consistency is for achieving Bayes optimal top-1 error. We consider much stronger consistency for achieving Bayes optimal topk error for any k ∈ [ K ] . To this end, we introduce some definitions about topk consistency [17, 8].

For any vector f ∈ R K , we let r k ( f ) denote a topk selector that selects the k indices of the largest entries of f by breaking ties arbitrarily. Given a data ( x , y ) , its topk error is defined as err k ( f, x , y ) = I ( y / ∈ r k ( f ( x ))) . The goal of a classification algorithm under the topk error metric is to learn a predictor f that minimizes the err k expected risk: R err k ( f ) = E ( x ,y ) ∼D [ err k ( f, x , y )] .

For a fixed k ∈ [ K ] , a loss function L is topk consistent if for any sequence of measurable functions f : X → ∆ K , we have the global minimizer f ∗ of R L ( f ) also minimizes R err k ( f ) , that is, f ∗ ∈ arg min f R err k ( f ) . If the above holds for all k ∈ [ K ] , it is referred to as Allk consistency .

3 Methodology and Theoretical Investigation

The symmetry condition in Equation 1.1, theoretically ensures that a symmetric loss function can be noise-tolerant [5]. Existing methods primarily focus on designing new loss functions. Those derived based on this design principle exhibit drawbacks, such as being challenging to optimize [5, 6] and prone to encounter the gradient explosion problem [7]. In this work, we take an alternative approach by proposing to constrain the hypothesis class H such that any loss functions will be approximately symmetric thereby rendering them robust to label noise.

3.1 Robustness

We introduce ϵ -softmax to make the output f ( x ) approximate one-hot vectors. The implementation of ϵ -softmax is easy to follow, as outlined in the gray box of the Introduction Section 1, requiring just two additional lines of code alongside the standard softmax layer. This underscores that ϵ -softmax is a plug-and-play module applicable to any classifier that incorporates a softmax layer. In this following, we investigate in theory how ϵ -softmax realizes the controllable approximation of outputs to one-hot vectors, thereby enhancing the noise tolerance of any loss function.

Approximating One-Hot Vectors. We first introduce the concept of ϵ -relaxation for a hypothesis class and then prove ϵ -softmax can strictly approximate outputs to one-hot vectors with a controllable error.

Definition 1 ( ϵ -relaxation) . Given a fixed vector v and its permutation set P v 1 , the ϵ -relaxation of P v is defined as the hypothesis class H v ,ϵ , in which any hypothesis f ∈ H v ,ϵ outputs vectors in the ϵ -ball of P v , i.e., H v ,ϵ = { f : min u ∈P v ∥ f ( x ) -u ∥ 2 ≤ ϵ, ∀ x } .

Without loss of generality, we consider v as a one-hot vector, which is common in machine learning, to facilitate the implementation and analysis. We then denote the permutation set of the one-hot vector as P e 1 , where all elements are also one-hot vectors. In accordance with Definition 1, we can further derive that:

Lemma 1. ϵ -softmax can achieve ϵ -relaxation for one-hot vectors:

where f ( x ) = ϵ -softmax ◦ h ( x ) .

Lemma 1 suggests that ϵ -softmax effectively enables f ( x ) to approximate one-hot vectors with a controllable error √ 1 -1 /K m +1 .

Robustness Guarantee. We then establish theoretical guarantees for the robustness in mitigating label noise, where the constrained hypothesis class H e 1 ,ϵ is considered.

Zhou et al. [14] established the excess risk bound [18] under symmetric noise, which holds when outputs fall within an ϵ -relaxation of a permutation set. We prove a more comprehensive conclusion by considering asymmetric noise, of which symmetric noise is a special case.

̸

Theorem 1 (Excess Risk Bound under Asymmetric Noise) . In a multi-class classification problem, if the loss function L ∈ L satisfies | ∑ K k =1 ( L ( u 1 , k ) -L ( u 2 , k )) | ≤ δ when ∥ u 1 -u 2 ∥ 2 ≤ ϵ , and δ → 0 as ϵ → 0 , then for asymmetric label noise η x ,k < (1 -η y ) , ∀ k = y , if R L ( f ∗ ) = 0 , the excess risk bound for f ∈ H v ,ϵ can be expressed as

where c = E D (1 -η y ) , a = min x ,k (1 -η y -η x ,k ) , f ∗ η and f ∗ denote the global minimum of R η L ( f ) and R L ( f ) , respectively.

Theorem 1 demonstrate that under mild conditions for symmetric and asymmetric label noise, any loss function can be made noise-tolerant when the function f ( x ) increasingly approximates a permutation set P v (i.e., δ → 0 as ϵ → 0 ).

ϵ -Softmax-Enhanced Loss Functions. Lemma 1 enable f ( x ) = ϵ -softmax ◦ h ( x ) in closely approximating a one-hot vector, aligns with the principle outlined in Theorem 1 within the framework of the hypothesis class H e 1 ,ϵ . Hence, ϵ -softmax progressively enhances the noise tolerance of any loss function as the hyperparameter m approaches infinity ( ϵ → 0 as m →∞ and the discrepancy δ → 0 ).

In this paper we consider CE loss and Focal loss (FL) [19]. We combine them with ϵ -softmax , denoted as CE ϵ and FL ϵ . ϵ -softmax approach is effective in adapting them to become more resilient to noise, ensuring better performance in the presence of label noise.

3.2 Consistency

Fundamentally, ϵ -softmax not only acts as an alternative for the softmax layer, but also plays the crucial role in modifying the loss function . Consistency is an important property of a loss function. A standard consistency is for achieving Bayes optimal top-1 error. We show much stronger consistency for achieving Bayes optimal topk error for any k ∈ [ K ] of the CE loss when combined with ϵ -softmax . To establish the Allk consistency, we first introduce some existing results of sufficient condition of topk consistency by topk calibration [17, 8].

Let P k ( f , q ) denote that f is topk preserving with respect to the underlying label distribution q , i.e., if for all l ∈ [ K ] , q l > q [ k +1] ⇒ f l > f [ k +1] , and q l < q [ k ] ⇒ f l < f [ k ] . Here, q [ k ] denotes he k -th greatest entry of q . For example, if q = [0 . 2 , 0 . 4 , 0 . 4] , then q [1] = 0 . 4 , q [2] = 0 . 4 , q [3] = 0 . 2 .

Definition 2 (Allk calibrated) . For a fixed k ∈ [ K ] , a loss function L is called topk calibrated if for all q ∈ ∆ K it holds that:

A loss function is called Allk calibrated if the loss function L is topk calibrated for all k ∈ [ K ] .

Yang and Koyejo [17] demonstrate that suppose L is a nonnegative topk calibrated loss function, then L is topk consistent. Furthermore, Zhu et al. [8] show that if f ∗ = arg min f R L ( f ) is rank preserving with respect to q , then L is Allk calibrated. f is called rank preserving w.r.t q , i.e., if for any pair q i < q j it holds that f i < f j .

Then we establish comprehensive Allk consistency for CE ϵ as follows:

Lemma 2. For one-hot label e y , CE ϵ is Allk calibrated and Allk consistency.

̸

Theorem 2. For any label q ∈ ∆ K , let y = arg max k ∈ [ K ] q k and t = arg max k ∈ [ K ] p k , if t = y and q y -max k = y q k > m m +1 , CE ϵ is Allk calibrated and Allk consistency.

Lemma 2 and Theorem 2 mean that CE ϵ performs well not only on the top-1 prediction, but also on the topk predictions for any k ∈ [ K ] . We show the Allk consistency property of different losses in Table 1, the consistency of other losses refer to [8].

3.3 Gradient Analysis of ϵ -Softmax.

To provide a comprehensive understanding of ϵ -softmax in mitigating label noise, we further analyze the gradient of the CE loss when combined with ϵ -softmax . The gradient of L CE ϵ ( f ( x ) , y ) with respect to the model h ( x ) can be derived as follows:

̸

where f = ϵ -softmax ◦ h , p ( x ) = softmax ( h ( x )) denotes the probabilities by standard softmax, and t = arg max k ∈ [ K ] p k is the class with the largest value in prediction probabilities.

̸

Remark. The gradient in Equation 3.4 shows that CE ϵ will be equivalent to the standard CE if the maximum prediction is not the target class (i.e., t = y ), in which the division of m +1 in probabilities

0

20

40

60

80

100

120

Test Accuracy / Epoch

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

CE

m=1e1

m=1e4

m=1e6

(a) CE

ϵ

(

η

= 0

)

0

20

40

60

80

100

120

Test Accuracy / Epoch

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

CE

m=1e1

m=1e4

m=1e6

(b) CE

ϵ

(

η

= 0

.

8

)

0

20

40

60

80

100

120

Test Accuracy / Epoch

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

CE

m=1e1

m=1e4

m=1e6

(c) CE

ϵ

+MAE (

η

= 0

)

0

20

40

60

80

100

120

Test Accuracy / Epoch

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

CE

m=1e1

m=1e4

m=1e6

(d) CE

ϵ

+MAE (

η

= 0

.

8

)

is omitted due to the partial deviation. Conversely, when the prediction class t matches the target class y , the gradient undergoes dynamic scaling by p y p y + m . This scaling results in smaller gradients, akin to a form of soft early-stopping [20], which facilitates the mitigation of overfitting to noisy labels. Such a characteristic enables Deep Neural Networks (DNNs) to efficiently fit clean samples in the early phases of training [21, 20], while simultaneously preventing the overfitting of noisy labels in the later stages of the training process. As illustrated in Figure 1(b), CE ϵ achieves a stable test accuracy curve, even in the challenging scenario with 0.8 symmetric label noise, without overfitting to noisy labels. On the contrary, CE with the standard softmax tends to rapidly overfit to noisy labels after the early phase of training, leading to poor performance.

3.4 Better Trade-off between Robustness and Effective Learning

It can be noted that the incorporation of ϵ -softmax somewhat sacrifices the fitting ability of the CE loss on clean datasets, as shown in Figure 1(a). Therefore, we need to enhance the fitting ability using additional techniques. Inspired by the Active Passive Loss [6], we propose to accommodate with the symmetric loss MAE. For instance, we formulate the combination of CE ϵ and MAE (a.k.a., CE ϵ +MAE) as follows ditto for FL ϵ +MAE.

Lemma 3. For any loss function L ϵ with ϵ -softmax and symmetric loss function L symmetric defined in Equation 1.1, the excess risk bound of α · L ϵ + β · L symmetric is equivalent to that of α · L ϵ .

Lemma 3 suggests that the ϵ -softmax -enhanced loss function L ϵ can be seamlessly integrated with any symmetric loss function while not modifying the inherent robustness. As can be noticed in Figure 1(c) and Figure 1(d), CE ϵ +MAE not only depicts strong fitting capabilities but also achieves better noise tolerance. More interestingly, the test accuracy on clean datasets obtained by CE ϵ +MAE even exceeds that of the standard CE loss.

Strict Convexity of CE ϵ +MAE. To elaborate on how the combination of CE ϵ and MAE can overcome the underfitting issue, we conduct an in-depth analysis from the optimization perspective. When the prediction t = y , the gradients of CE ϵ , CE and MAE w.r.t. p y ∈ (0 , 1] , are -1 p y + m , -1 p y and -2 , respectively. As can be seen, CE and CE ϵ are strictly convex, while MAE exhibits linearity. Moreover, CE has stronger convexity compared to CE ϵ (specifically, the gradient of CE changes more rapidly as 1 /p 2 y > 1 / ( p y + m ) 2 ), rendering CE more susceptible to overfitting noisy labels while CE ϵ suffering from underfitting for large m , as illustrated in Figure 1(a) and Figure 1(b). Conversely, owing to the linearity, MAE treats every sample equally, making it robust to label noise but leading to more training time for convergence [11]. Hence, the combination of CE ϵ and MAE, which notably forms a strictly convex function (where the convexity can be controlled by m ), can provide better trade-off between robustness and effective learning.

̸

̸

̸

Association with APL. Additionally, our proposed CE ϵ +MAE coincides with the concept of active and passive losses in [6]. Specifically, for a loss function denoted as L ( f ( x ) , y ) = ℓ 1 ( f ( x ) , y ) + ∑ k = y ℓ 2 ( f ( x ) , k ) , L is active if ℓ 2 ( f ( x ) , k ) = 0 for any k = y , and L is passive if ℓ 2 ( f ( x ) , k ) = 0 for some k = y . Active losses only explicitly maximize the target probability f ( x ) y , while passive losses also explicitly minimize non-target probabilities { f ( x ) k } k = y . For example, CE is an active loss, while MAE is passive. Based on these two loss terms, Ma et al. [6] proposed to combine a robust active loss and a robust passive loss into an 'Active Passive Loss' (APL) framework for improving

̸

̸

sufficient learning with underfitting losses. Note that CE ϵ is also active, thus CE ϵ +MAE coincides with the APL framework and further mitigates the underfitting issue.

To further validate CE ϵ +MAE, we incorporate it with sample selection, pseudo-label prediction [22], and MixUp [23], culminating in a semi-supervised learning algorithm we term CE ϵ +MAE (Semi). The algorithm details can be found in the Appendix C. In our experiments, we use "CE ϵ +MAE(Semi)" to ensure a fair comparison with other hybrid methods with sample selection and semi-supervised learning (SSL). No additional techniques are utilized for "CE ϵ +MAE".

4 Experiments

In this section, we conduct extensive experiments to validate the superiority of ϵ -softmax in mitigating label noise. Complete experimental setting and results can be found in the Appendix D and E.

4.1 Evaluation on Benchmark Datasets

Weevaluate our proposed methods on benchmark datasets CIFAR-10 / CIFAR-100 [24] with synthetic label noise, following [6, 7].

Baselines. We consider several baseline methods for comparison, including Standard CE and FL [19]; MAE; GCE [11]; NLNL [25]; SCE [12]; APL [6], including NCE+MAE, NCE+RCE, and NFL+RCE; AFLs [7], including NCE+AEL, NCE+AGCE, and NCE+AUL; LDR-KL [8]; and LogitClip [26], including CE+LC.

(a) CE (

η

= 0

.

2

)

(b) CE (

η

= 0

.

4

)

(c) CE

ϵ

+MAE (

η

= 0

.

2

)

(d) CE

ϵ

+MAE (

η

= 0

.

4

)

1.0

0.8

0.8

0.8

0.7

0.8

0.6

0.6

0.6

0.6

0.5

0.4

0.4

0.4

0.4

0.3

0.2

0.2

0.2

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Results. Table 2 presents the test accuracy of various loss functions under symmetric and asymmetric label noise. As can be seen, our proposed ϵ -softmax -enhanced loss functions, CE ϵ +MAE and FL ϵ +MAE, demonstrate remarkable performance, ranking among the top-2 in most cases across both datasets. These methods consistently outperform others such as GCE, SCE, NLNL, NCE+MAE and LDR-KL, regardless of the noise rates. In scenarios of clean labels, CE ϵ +MAE and FL ϵ +MAE also exhibit strong fitting abilities, outperforming NCE+RCE and NCE+AGCE. In particular, on CIFAR-100 with 0.4 asymmetric noise, most robust loss functions have no effect, but our methods achieve over 48% accuracy, significantly outperforming all other methods. These findings underscore the robustness and effectiveness of ϵ -softmax -enhanced loss functions, delivering their excellent performance in various noise scenarios.

Ablation Experiments. We perform detailed ablation experiments to further explore the role of each component and hyperparameter m in our CE ϵ +MAE, experimental results are shown in Table 3. We can observe that CE will severely fit the noise label, and the symmetric loss MAE is difficult to optimize. CE+MAE (i.e., m = 0 ) is a trade-off between robustness and fitting ability, increasing noise tolerance at the cost of reducing fitting ability on clean labels, consistent with previous works [11, 12, 13]. In particular, our CE ϵ +MAE shows remarkable properties. As the parameter m experiences a moderate increase, CE ϵ +MAE not only achieves noise tolerance for symmetric and asymmetric noise, but also achieves effective learning for the clean scenario. Additionally, the experimental results suggest that strict constraints are better suited for symmetric noise, while looser constraints are more effective for asymmetric noise.

Visualization. We conduct a further analysis to compare the effectiveness of CE ϵ +MAE and traditional CE in learning representations. We train models with different label noise and use the trained models to extract feature representations of the test set by t-SNE [27]. The visualizations for CIFAR-10 symmetric noise are depicted in Figure 2. Notably, the embeddings generated by CE show evident overfitting to label noise, as seen in the blending of embeddings from distinct classes. In sharp contrast, embeddings from the CE ϵ +MAE method consistently form clear, well-separated clusters, demonstrating its superior ability to learn robust and distinct representations under noisy label conditions.

4.2 Evaluation on Human-Annotated Datasets

We further conduct comparison studies on human-annotated datasets CIFAR-10N/CIFAR-100N [28], following the experiment setting in [28].

Baselines. For a fair comparison, we divide the baselines into those without and those with semisupervised learning (SSL) and sample selection:

- Without SSL : Standard loss CE, Forward T [29], GCE [11], T-Revision [30], Peer Loss [31], F-Div [32], Negative-LS [33], VolMinNet [34], and AGCE [35].

- With SSL : Co-teaching+ [36], JoCoR [37], ELR+ [38], DivideMix [39], CORES* [40], CAL [41], PES (Semi) [20], SOP+ [42], and Proto-semi [43].

Results. Table 4 reports the test accuracy results of each method on the human-annotated datasets. The results show that the proposed CE ϵ +MAE and CE ϵ +MAE (Semi) provide significant improvements in handling human-annotated label noise, especially at high noise rates. Among the methods without SSL, CE ϵ +MAE stands out on the CIFAR-100N "Noisy" case as the only method to exceed 61% accuracy. Within the methods with SSL, CE ϵ +MAE (Semi) shows a pronounced superiority in all scenarios, especially in the most difficult CIFAR-10N "Worst" case and CIFAR-100N "Noisy" case. In the CIFAR-10N "Worst" case, CE ϵ +MAE (Semi) achieves an impressive accuracy rate of over 95%, significantly outperforming competing methods. These results underscore the effectiveness of the ϵ -softmax -enhanced loss function in counteracting label noise for human-annotated scenarios.

4.3 Evaluation on the Real-World Datasets

We perform experiments on massively real-world noisy datasets, including WebVision [44], ILSVRC12 (ImageNet) [45] and Clothing1M [46], following the experiment setting in [7].

Results. In Table 5, we showcase the accuracies achieved on WebVision, ILSVRC12 and Clothing1M by various leading methods. Notably, our CE ϵ +MAE method outshines others, achieving the highest results on all real-world datasets. It surpasses CE by approximately 5.5% on WebVision and 6.5% on ILSVRC12. For Clothing1M, we finetune a pretrained ResNet-50, so the differences between the methods are relatively small, but our method still achieves the best accuracy. These results underline the robustness and efficacy of the ϵ -softmax -enhanced loss function in real-world scenarios.

5 Conclusion

In this paper, we introduced ϵ -softmax , a simple yet effective and theoretically sound scheme for noise-tolerant learning. Our method is not only easy to implement but also can be seamlessly integrated with any softmax-based DNNs, requiring just two additional lines of code. Our rigorous and comprehensive theoretical analysis reveals that ϵ -softmax effectively alleviates the common issue of overfitting to noisy labels. Furthermore, we propose to incorporate ϵ -softmax -enhanced loss functions with MAE, achieving better trade-off between effective learning and robustness. Extensive experimental results demonstrate the superior performance of our method in mitigating label noise.

Broader Impacts

This work has the potential to advance the development of machine learning methods that can be deployed in contexts where it is costly to gather accurate annotations. This is an important issue in applications such as medicine, where machine learning has great potential societal impact. This work will not have negative social impacts.

References

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature , 521(7553):436-444, 2015.

Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:2011.04406 , 2020.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390 , 2023.

Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International conference on machine learning , pages 6543-6553. PMLR, 2020.

Xiong Zhou, Xianming Liu, Junjun Jiang, Xin Gao, and Xiangyang Ji. Asymmetric loss functions for learning with noisy labels. In International conference on machine learning , pages 12846-12856. PMLR, 2021.

Dixian Zhu, Yiming Ying, and Tianbao Yang. Label distributionally robust losses for multi-class classification: Consistency, robustness and adaptivity. In International Conference on Machine Learning , pages 43289-43325. PMLR, 2023.

Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics , 43(3):1146-1151, 2013.

Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. Advances in neural information processing systems , 28, 2015.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems , 31, 2018.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision , pages 322-330, 2019.

Erik Englesson and Hossein Azizpour. Generalized jensen-shannon divergence loss for learning with noisy labels. Advances in Neural Information Processing Systems , 34:30284-30297, 2021.

Xiong Zhou, Xianming Liu, Chenyang Wang, Deming Zhai, Junjun Jiang, and Xiangyang Ji. Learning with noisy labels via sparse regularization. In Proceedings of the IEEE/CVF international conference on computer vision , pages 72-81, 2021.

Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learning research , 5(9), 2004.

Xiong Zhou, Xianming Liu, Hao Yu, Jialiang Wang, Zeke Xie, Junjun Jiang, and Xiangyang Ji. Variance-enlarged poisson learning for graph-based semi-supervised learning with extremely sparse labeled data. In The Twelfth International Conference on Learning Representations , pages 1-19, 2024.

Forest Yang and Sanmi Koyejo. On the consistency of top-k surrogate losses. In International Conference on Machine Learning , pages 10727-10735. PMLR, 2020.

Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems , 34:24392-24403, 2021.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems , 31, 2018.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Nlnl: Negative learning for noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision , pages 101-110, 2019.

Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, and Yixuan Li. Mitigating memorization of noisy labels by clipping the model prediction. In International Conference on Machine Learning , pages 36868-36886. PMLR, 2023.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations , 2021.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1944-1952, 2017.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in neural information processing systems , 32, 2019.

Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International conference on machine learning , pages 6226-6236. PMLR, 2020.

Jiaheng Wei and Yang Liu. When optimizing f-divergence is robust with label noise. In International Conference on Learning Representations , 2021.

Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In International Conference on Machine Learning , pages 23589-23614. PMLR, 2022.

Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In International conference on machine learning , pages 6403-6413. PMLR, 2021.

Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, and Xiangyang Ji. Asymmetric loss functions for noise-tolerant learning: Theory and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning , pages 7164-7173. PMLR, 2019.

Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13726-13735, 2020.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Earlylearning regularization prevents memorization of noisy labels. Advances in neural information processing systems , 33:20331-20342, 2020.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations , 2020.

Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. In International Conference on Learning Representations , 2021.

Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instancedependent label noise. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10113-10123, 2021.

Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by overparameterization. In International Conference on Machine Learning , pages 14153-14172. PMLR, 2022.

Renyu Zhu, Haoyu Liu, Runze Wu, Minmin Lin, Tangjie Lv, Changjie Fan, and Haobo Wang. Rethinking noisy label learning in real-world annotation scenarios from the noise-type perspective. arXiv preprint arXiv:2307.16889 , 2023.

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862 , 2017.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009. doi: 10.1109/CVPR.2009.5206848.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2691-2699, 2015.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems , 33:7597-7610, 2020.

A Limitation and Discussion

The limitation of ϵ -softmax is that it may slightly reduce fitting ability on clean case. Therefore, we propose to combine the ϵ -softmax -enhanced loss with the symmetric loss MAE. Consequently, our practical loss functions utilized for noise-tolerant learning exhibit a hybrid form similar to GCE and SCE, but their meanings are significantly different.

̸

Comparing with GCE and SCE. GCE is a hybrid of CE and MAE var the negative Box-Cox transformation [11]. SCE combines CE with Reverse CE (RCE), where the RCE component actually acts as a scaled version of the MAE. This relationship is unveiled through the following derivation, adapted from Section 4.3 in SCE [12]: L RCE = -∑ K k =1 p ( k | x ) log q ( k | x ) = -p ( y | x ) log 1 -∑ k = y p ( k | x ) A = -A ∑ k = y p ( k | x ) = -A (1 -p ( y | x )) = -A 2 L MAE. Consequently, SCE essentially translates to CE+MAE. Hence, GCE and SCE increases the fitting ability but reduces the robustness because of the CE term. Conversely, our CE ϵ is inherently robust. The combination of CE ϵ and MAE does not reduce the robustness, as demonstrated by Lemma 3, and also improves the fitting ability. We perform further experiments cimparing with GCE and CE+MAE (SCE), the results can be seen in Table 6. Our CE ϵ +MAE obtains obviously the best results at all noise rates, significantly outperforming GCE and CE+MAE (SCE).

̸

Meanwhile, we further compare our ϵ -softmax with temperature-dependent softmax.

Comparing with Temperature-Dependent Softmax. softmax ( h ( x ) τ ) , where τ is the temperature parameter, is a useful technique for making outputs sparse [14]. Compared to our ϵ -softmax , temperature-dependent softmax does not achieve a quantitative approximation to a one-hot vector for each output, and therefore cannot achieve a controllable excess risk bound. We also perform further experiments cimparing with temperature-dependent softmax. For simplicity, we refer to CE with temperature-dependent softmax as CE τ , the results can be seen in Table 6. Our CE ϵ +MAE obtains obviously the best results at all noise rates, significantly outperforming temperature-dependent softmax.

B Proof of Theorems

Lemma 1. ϵ -softmax can achieve ϵ -relaxation for one-hot vectors:

where f ( x ) = ϵ -softmax ◦ h ( x ) .

Proof.

̸

Theorem 1 (Excess Risk Bound under Asymmetric Noise) . In a multi-class classification problem, if the loss function L ∈ L satisfies | ∑ K k =1 ( L ( u 1 , k ) -L ( u 2 , k )) | ≤ δ when ∥ u 1 -u 2 ∥ 2 ≤ ϵ , and δ → 0 as ϵ → 0 , then for asymmetric label noise η x ,k < (1 -η y ) , ∀ k = y , if R L ( f ∗ ) = 0 , the excess risk bound for f ∈ H v ,ϵ can be expressed as

where c = E D (1 -η y ) , a = min x ,k (1 -η y -η x ,k ) , f ∗ η and f ∗ denote the global minimum of R η L ( f ) and R L ( f ) , respectively.

Proof.

where C = ∑ K k =1 L ( v , k ) , ditto

hence,

̸

̸

According to the assumption R L ( f ∗ ) = 0 , we have L ( f ∗ ( x ) , y ) = 0 then L ( f ∗ ( x ) , k ) = C k -1 where k = y . Since L ( f ∗ η ( x ) , k ) -L ( f ∗ ( x ) , k ) ≤ 0 where k = y , the second term on the right of the inequality is a non-positive value. And R η L ( f ∗ ) -R η L ( f ∗ η ) ≥ 0 . So we have

̸

̸

where c = E D (1 -η y ) .

Let a = min x ,k (1 -η y -η x ,k ) , we have ∣ ∣ ∣ E D ∑ k = y ( L ( f ∗ η ( x ) , k ) -L ( f ∗ ( x ) , k ) ) ∣ ∣ ∣ ≤ 2 cδ a . Note that f ∗ η , f ∗ ∈ H v ,ϵ means that | ∑ k ( L ( f ∗ η ( x ) , k ) -L ( f ∗ ( x ) , k ) ) | ≤ 2 δ , then we obtain

̸

̸

̸

̸

̸

̸

Lemma 2. For one-hot label e y , CE ϵ is Allk calibrated and Allk consistency.

Proof. Here f = ϵ -softmax ◦ h , p ( ·| x ) = softmax ( h ( x )) denotes the probabilities by standard softmax, p k ∈ (0 , 1] and t = arg max k ∈ [ K ] p k is the class with the largest value in prediction probabilities.

if t = y :

̸

By the first-order optimality condition ∂L CE ϵ ( f ( x ) ,y ) ∂h ( x ) = 0 , we have: p y = 1 . Hence, for any k = y , we have e k = 0 < e y and p k < p y .

̸

if t = y :

̸

By the first-order optimality condition ∂L CE ϵ ( f ( x ) ,y ) ∂h ( x ) = 0 , we have: p y = 1 . Hence, for any k = y , we have e k = 0 < e y and p k < p y .

Hence, CE ϵ is Allk calibrated. Since CE ϵ is nonnegative, so CE ϵ is Allk consistency.

̸

Theorem 2. For any label q ∈ ∆ K , let y = arg max k ∈ [ K ] q k and t = arg max k ∈ [ K ] p k , if t = y and q y -max k = y q k > m m +1 , CE ϵ is Allk calibrated and Allk consistency.

Proof. For ∂L CE ϵ ( f ( x ) , q ) ∂h ( y | x )

̸

By the first-order optimality condition ∂L CE ϵ ( f ( x ) , q ) ∂h ( y | x ) = 0 , we have:

̸

Since, m m +1 < q t ≤ 1 , we can get 0 < p t ≤ 1 .

̸

For ∂L CE ϵ ( f ( x ) , q ) ∂h ( j = y | x ) , we have:

̸

̸

̸

̸

̸

̸

By the first-order optimality condition ∂L CE ϵ ( f ( x ) , q ) ∂h ( j = y | x ) = 0 , we have:

̸

Substituting p t = q t (1 + m ) -m , we can get p j = q j ( m +1) . Since q t > m m +1 , so q j < 1 m +1 and 0 < p j < 1 .

̸

̸

For i, j = t , if q i < q j , we have p i < p j . Consider q k = t and q t , because of the condition q y -q k = y> m m +1 , we have q k < q t , q t -q k = q t (1 + m ) -m -q k ( m +1) > 0 .

̸

Hence, CE ϵ is Allk calibrated. Since CE ϵ is nonnegative, so CE ϵ is Allk consistency.

The gradient of CE ϵ .

̸

where f = ϵ -softmax ◦ h , p ( x ) = softmax ( h ( x )) denotes the probabilities by standard softmax, and t = arg max k ∈ [ K ] p k is the class with the largest value in prediction probabilities.

Proof. The proof is similar to Theorem 2.

if t = y :

̸

if t = y : p y

Lemma 3. For any loss function L ϵ with ϵ -softmax and symmetric loss function L symmetric defined in Equation 1.1, the excess risk bound of α · L ϵ + β · L symmetric is equivalent to that of α · L ϵ .

Proof. For u 1 , u 2 ∈ H v ,ϵ and u 3 , u 4 ∈ ∆ K , we have

C The Algorithm of CE ϵ +MAE (Semi)

Algorithm 1 CE ϵ +MAE (Semi)

Algorithm Details and Parameters. Reference to [20], we set T robust = 65 , T robust = 300 and learning rate decay 0.1 at [60, 160, 260] epochs. Other experimental settings are the same as the CIFAR-N experiment [28] in the Appendix D.

For sample selection: We simply select k samples from each class with the least loss as clean samples. For CIFAR-10N, we set k = 2500 for 'Worst' case and 3500 for others. For CIFAR-100N, we set k = 250 for 'Noisy' case and 350 for others. In practice, if k > | sample_num | , we set k = | sample_num | -20 .

For pseudo-label prediction: In the actual training, we do the pseudo-label prediction using two standard augment versions from the sample. We add the probabilities and divide by 2 to make the pseudo-label prediction. At the same time, we set the threshold σ = 0 . 2 and discard the samples whose prediction probability is less than the threshold.

For the Augment to D u , we employ RandAugment [47]. We set the trade-off parameter λ to grow linearly from 0 to 1 over 200 epochs. The MixUp parameter α is set to 0.75 for epochs less than 100, and adjusted to 4 for epochs greater than 100. CE ϵ +MAE m = 1 e 4 , α = 0 . 5 , β = 1 is the same as the CIFAR-N experiment for the robust learning stage and m = 10 , α = 1 , β = 1 for the semi-supervised learning stage. In CE ϵ +MAE (Semi), we ensemble the outputs of two networks during inference and exchange the samples selected by the two networks during training, as is customary for methods that train two networks simultaneously [21, 36, 39, 38].

D Experiments

D.1 Evaluation on Benchmark Datasets

Noise Generation. We follow the approach of previous studies [6, 7] to experiment with two types of synthetic label noise: symmetric (uniform) noise and asymmetric (class-conditional) noise. In the case of symmetric label noise, we intentionally corrupt the training labels by randomly flipping labels within each class to incorrect labels in other classes. As for asymmetric label noise, we flip the labels

within a specific sets of classes: For CIFAR-10, the flips occur from TRUCK → AUTOMOBILE, BIRD → AIRPLANE, DEER → HORSE, and CAT ↔ DOG. For CIFAR-100, the 100 classes are grouped into 20 super-classes, each containing 5 sub-classes, and we flip the labels within the same super-class into the next.

Experimental Setting. We follow the same experimental settings in [6, 7]: An 8-layer CNN is used for CIFAR-10 and a ResNet-34 for CIFAR-100. The networks are trained for 120 and 200 epochs for CIFAR-10 and CIFAR-100 with batch size 128. We use the SGD optimizer with momentum 0.9 and cosine learning rate annealing. The weight decay is set to 1 × 10 -4 and 1 × 10 -5 for CIFAR-10 and CIFAR-100. The initial learning rate is set to 0.01 for CIFAR-10 and 0.1 for CIFAR-100. Clipping the gradient norm to 5.0 and the minimum allowable value for log to 1 × 10 -8 . Typical data augmentations including random shift and horizontal flip are applied to CIFAR-10; random shift, horizontal flip and random rotation are applied to CIFAR-100.

Parameters Setting. Weset the parameter settings which match their original papers for all baseline methods [6, 7]. Specifically, for FL, we set γ = 0 . 5 . For GCE, we set q = 0 . 7 for CIFAR-10, and q = [0 . 5 , 0 . 5 , 0 . 7 , 0 . 7 , 0 . 9] for CIFAR-100 clean and symmetric noise ( η ∈ [0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8] ), q = 0 . 7 asymmetric noise. For SCE, we set A = -4 , α = 0 . 1 , β = 1 for CIFAR-10, and α = 6 , β = 0 . 1 for CIFAR-100. For APL (NCE+MAE, NCE+RCE and NFL+RCE), we set α = 1 , β = 1 for CIFAR-10, and α = 10 , β = 0 . 1 for CIFAR-100. For NCE+AUL, we set a = 6 . 3 , q = 1 . 5 , α = 1 , β = 4 for CIFAR-10, and a = 6 , q = 3 , α = 10 , β = 0 . 015 for CIFAR-100. For NCE+AGCE, we set a = 6 , q = 1 . 5 , α = 1 , β = 4 for CIFAR-10, and a = 1 . 8 , q = 3 , α = 10 , β = 0 . 1 for CIFAR-100. For NCE+AEL, we set a = 5 , α = 1 , β = 4 for CIFAR-10, and a = 1 . 5 , α = 10 , β = 0 . 1 for CIFAR-100. For CE+LC, we set δ = [1 , 1 , 1 , 1 . 5 , 1 . 5] for CIFAR-10 clean and symmetric noise ( η ∈ [0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8] ) and δ = 2 . 5 for CIFAR-10 asymmetric noise. We set δ = 2 . 5 for CIFAR-100 asymmetric noise and δ = 0 . 5 for others. For LDR-KL, We set λ = 10 for CIFAR-10 and 1 for CIFAR-100. For our CE ϵ +MAE, we set β = 5 , m = 1 e 5 , α = 0 . 01 for CIFAR-10 symmetric, and m = 1 e 3 , α = 0 . 02 for asymmetric. For CIFAR-100, we set β = 1 , m = 1 e 4 and α = [0 . 1 , 0 . 05 , 0 . 03 , 0 . 0125 , 0 . 0075] for clean and symmetric noise ( η ∈ [0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8] ), and m = 1 e 2 , α = [0 . 015 , 0 . 007 , 0 . 005 , 0 . 004] for asymmetric noise ( η ∈ [0 . 1 , 0 . 2 , 0 . 3 , 0 . 4] ). For our FL ϵ +MAE, we set γ = 0 . 1 and others are same as CE ϵ +MAE. For NLNL, we use the results in [7] directly.

D.2 Evaluation on Human-Annotated Datasets

Experimental Setting. We follow the experimental settings in [28]: Train a Resnet-34 using SGD for 100 epochs with initial learning rate 0.1, momentum 0.9, and weight decay 0.0005. Set the learning rate decay 0.1 at 60 epochs. Standard data augmentation including random shift and horizontal flip are applied. Best epoch test accuracies are compared. The results of the comparison methods are taken directly from [28] and the original papers [35, 43].

Parameters Setting. For our CE ϵ +MAE, we set m = 1 e 4 , α = 0 . 5 , β = 1 for CIFAR-10N/100N. CE ϵ +MAE (Semi) has been covered in detail in the previous section C.

D.3 Evaluation on Real-World Dataset WebVision

Experimental Setting. For WebVision, the training details follow [7]: We use the mini WebVision setting [6, 7] and train a ResNet-50 using SGD for 250 epochs with initial learning rate 0.4, nesterov momentum 0.9 and weight decay 3 × 10 -5 and batch size 256. The learning rate is multiplied by 0.97 after each epoch of training. All the images are resized to 224 × 224 . Typical data augmentations including random width/height shift, color jittering, and horizontal flip are applied. We train the model on Webvision and evaluate the trained model on the same 50 concepts on the corresponding WebVision and ILSVRC12 validation sets.

For Clothing1M, we use ResNet-50 pre-trained on ImageNet similar to [46]. All the images are resized to 224 × 224 . We use SGD with a momentum of 0.9, a weight decay of 1 × 10 -3 , and batch size of 256. We train the network for 10 epochs with a learning rate of 5 × 10 -3 and a decay of 0.1 at 5 epochs. Typical data augmentations including random shift and horizontal flip are applied.

Parameters Setting. We set the best parameter settings which match their original papers for all baseline methods [6, 7]. Specifically, for GCE, we set q = 0 . 7 for WebVision and 0.6 for Clothing1M.

For SCE, we set A = -4 , α = 10 , β = 1 . For NCE+RCE, we set α = 50 , β = 0 . 1 for WebVision and α = 10 , β = 1 for Clothing1M. For AGCE, we set a = 1 e -5 , q = 0 . 5 . For NCE+AGCE, we set a = 2 . 5 , q = 3 , α = 50 , β = 0 . 1 . For LDR-KL, we set λ = 1 . For our CE ϵ +MAE, we set m = 1 e 3 , α = 0 . 015 , β = 0 . 3 for WebVison and α = 0 . 012 , β = 0 . 1 for Clothing1M.

E More Experimental Results

Visualization. We show more visualizations of learned representations in Figure 3.

Detailed Experimental Results of CE ϵ +MAE (Semi) The more detailed results are reported in Table 7.

Instance-Dependent Noise. We follow the method in PDN [48] to generate instance-dependent noise. The experimental setting is the same as CIFAR-10/CIFAR-100. For CE ϵ +MAE on CIFAR-10, we set α = 0 . 045 , β = 10 , m = 1 e 5 . For CIFAR-100, we use the same parameters as symmetric noise. The results are reported in Table 8.

(a) CE (

η

= 0

)

(b) CE (

η

= 0

.

2

)

(c) CE (

η

= 0

.

4

)

(d) CE (

η

= 0

.

6

)

(e) CE

ϵ

+MAE (

η

= 0

)

(f) CE

ϵ

+MAE (

η

= 0

.

2

)

(g) CE

ϵ

+MAE (

η

= 0

.

4

)

(h) CE

ϵ

+MAE (

η

= 0

.

6

)

0

0.8

1.0

0.8

1

0.8

2

0.7

0.8

0.6

3

4

0.6

0.6

0.6

5

0.5

0.4

6

7

0.4

0.4

0.4

8

0.3

0.2

0.2

9

0.2

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. Specifically, we provide a simple yet effective method for mitigating label noise with elaborated descriptions and theoretical results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of the work in the Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions in the main paper and all proofs in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the experiment details in the Appendix D and submit the code for reproducibility in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have submitted the code for with sufficient instructions to faithfully reproduce the main experimental results. And the datasets are obtained from open source.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have specified all the training and test details in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For all experiments, we include error bars for added clarity.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information in the experiment details. All experiments are implemented by PyTorch and are conducted on NVIDIA GeForce RTX 4090.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We promise that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed broader impact of this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not use pretrained language models, image generators, etc.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The existing asserts in this paper are properly credited an are the license and terms of use explicitly mentioned and properly respected with appropriate citations.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes $\\epsilon$-softmax to deal with label noise.  $\\epsilon$-softmax modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $\\epsilon$. Both theoretical and empirical studies show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The writing of this paper is good.\n2. The robustness of the proposed loss is theoretically proved.\n\nWEAKNESSES:\n1. I think the motivation or the underlying reason for the effectiveness needs further explanation.\n2. In experiment, the advantage of the proposed method over the competitors is probably not statistically significant.\n\nQUESTIONS:\n1. The term \"symmetric condition\" in abstract needs further explanation.\n2. In the implementation steps in Line 61, is $\\mathbf{p}(\\cdot)$ the same as ${p}(\\cdot)$? Or what is the relationship between these two notations? I think the mathematical notations should be strictly used and defined.\n3. The robustness of the proposed $\\epsilon$-softmax loss is theoretically justified. However, I'm curious to know the insight, or the underlying reason for its robustness. The authors wrote that \"The distinctive attribute of $\\epsilon$-softmax lies in its guarantee to possess a controllable approximation error $\\epsilon$ to one-hot vectors, thus achieving perfect constraint for the hypothesis class.\" But, I cannot figure out why controlling approximation error $\\epsilon$ to one-hot vectors and achieving perfect constraint for the hypothesis class are useful for handling label noise? What is the direct reason? It would be better if the authors can provide some intuitive explanations.\n4. From the experiments, I note that the performance is a bit sensitive to different selection of m. Therefore, is it possible to give some guidance in choosing m for practical use? Besides, will better performance be obtained if we add the $\\epsilon$-softmax loss functions with different m?\n5. From the experimental results in Table 2, 4, I can see that the improvement of the proposed method over other methods is quite marginal. I guess if we do statistical significance test, maybe such improvement will not be statistically significant.\n\nLIMITATIONS:\nI think this work will not have negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes a enhanced softmax layer for label-noise learning, namely $\\epsilon$-softmax. By incorporating with the well-known $\\epsilon$-relaxation, the proposed $\\epsilon$-softmax can regularize the outputs of the model and avoid fitting the label-noise sample. This simple and plug-and-play method theoretically bounds the output logits to be an approximated one-hot vector. Extensive experiments demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n- The proposed method is simple, plug-and-play, and effective. Unlike other label-noise robust losses, the proposed method not only works well by itself, but also can be integrated with other label-noise robust method such as DivideMix. To the best of my knowledge, this could one of the first works endow such property.\n- The theoretical analysis is comprehensive and make sense. The theoretical results suggests the proposed method possesses the Top-K error consistency and label-noise robustness.\n- The analysis between the most related previous works is in reason. The basic idea that balances the label-noise robustness and learning effectiveness has been researched for a long time, e.g., GCE. This submission clearly presents the connection between the proposed method and other symmetric losses.\n- The empirical results is effective and enough. This submission presents the comparison results between many label-noise robust losses and the proposed method achieves the best performance in most cases. Additionally, this submission provides the experimental results that demonstrated the plug-and-play property of the proposed method on sample-selection based method and loss-correction based method.\n\nWEAKNESSES:\n- The ablation studies on the gradient clipping should be conducted and providing experimental results with different backbones would be better.\n- It is exhaustive and labor-expensive to find the optimal $m$ for diverse datasets.\n\nQUESTIONS:\n- Why choose the MAE as the loss to incorporate with ${CE_\\epsilon}$? Following GCE and SCE, the MAE is indeed a practical and evaluated choice. Do there have any alternatives?\n- What is the connection or relationship between logit adjustment and $\\epsilon$-softmax?\n\nLIMITATIONS:\nNo obvious limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes a novel method to approximate the symmetric condition of the loss function, which is necessary for robustness to label noise. Specifically, the proposed method, named \\\\( \\\\epsilon \\\\)-softmax, can adjust the model output to approximate one-hot vector. However, the proposed method alone suffers from underfitting, so the authors combined it with MAE to achieve better performance. The authors evaluated the proposed method on datasets with different noise types and rates.\n\nSTRENGTHS:\n1. This manuscript proposes a novel and simple method to approximate the symmetric condition of loss function.\n2. The theoretical analysis focuses not only on robustness to label noise but also on the top-k consistency of the loss function.\n3. The proposed method was evaluated on various noise types and rates, including class-dependent noise and real-world noise.\n4. It good to see the authors compare their proposed method with temperature-dependent softmax combined with MAE. The experimental results demonstrate the superiority of their proposed method compared to temperature-dependent softmax.\n\nWEAKNESSES:\n1. The theoretical discussion with temperature-dependent softmax is missing. As the authors mentioned in L42 to L53, there are other output restriction-based methods in the literature, such as temperature-dependent softmax. Although the authors claim that these methods “lack predictability, fail to achieve a quantitative approximation to one-hot vectors, and exhibit limited effectiveness,” there is no detailed discussion on why the proposed \\\\( \\\\epsilon \\\\)-softmax has superior properties.\n2. A direct comparison with sparse regularization [1] is missing. Sparse regularization utilizes temperature- dependent softmax, which this manuscript has already compared, to approximate one-hot vector output. However, sparse regularization also employs an additional regularization term, \\\\( \\\\ell_p \\\\)-norm \\\\( \\\\| p(\\\\cdot | x) \\\\|^p_p \\\\) to enhance performance, and this regularization term is equivalent to MAE only if \\\\( p = 1 \\\\). It’s necessary to highlight the advantages of the proposed method compared to this highly relevant approach.\n3. There is no ablation study on \\\\( \\\\alpha \\\\) and \\\\( \\\\beta \\\\). As the authors mentioned in L218, \\\\( \\\\epsilon \\\\)-softmax alone suffers from a loss in fitting ability, and they combined it with MAE to balance the robustness and effective learning. However, without the relevant ablation study, it’s unclear how this “trade-off” is achieved.\n4. The theoretical discussions and experiments regarding instance-dependent label noise are overlooked. In recent years, the instance-dependent label noise has attracted increasing attention [2,3,4]. Experimenting the proposed method on instance-dependent label noise can provide a better understanding of how the proposed method performs with different types of label noise. I encourage the authors to include related discussion in the revised manuscript.\n\n[1] Learning with noisy labels via sparse regularization, ICCV, 2021.\n\n[2] Part-dependent Label Noise: Towards Instance-dependent Label Noise, NeurIPS, 2020.\n\n[3] Learning with Instance-Dependent Label Noise: A Sample Sieve Approach, ICLR, 2021.\n\n[4] Instance-Dependent Label-Noise Learning With Manifold-Regularized Transition Matrix Estimation, CVPR, 2022.\n\nQUESTIONS:\n1. Is \\\\( \\\\epsilon \\\\)-softmax + MAE still All-\\\\( k \\\\) calibrated and All-\\\\( k \\\\) consistency?\n2. Can the proposed method perform better compared to sparse regularization?\n\nLIMITATIONS:\nThe authors have acknowledged the limitations and societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces “ϵ-softmax,” a method to adjust softmax outputs for better approximation to one-hot vectors, thereby mitigating the impact of label noise in classification tasks. The approach modifies the softmax layer outputs to include a controllable error term ϵ, aiming to improve noise robustness without extensive alteration to the network architecture. The authors provide theoretical backing for the effectiveness of ϵ-softmax in achieving noise-tolerant learning across various loss functions. Extensive experiments with both synthetic and real-world noisy datasets are conducted to validate the claims.\n\nSTRENGTHS:\n1.ϵ-softmax is presented as a plug-and-play module compatible with any existing classifier that uses a softmax layer, enhancing its practical utility.\n\n2.The paper proves that ϵ-softmax can achieve a controlled approximation to one-hot vectors, which is significant for learning with noisy labels.\n\n3.The methodology is backed by extensive experimental results showing its superiority over existing methods in handling label noise, with detailed results across multiple datasets and noise configurations.\n\nWEAKNESSES:\nThis paper should pay attention to the axis labels of its figures. In Figure 1, the x-label is Epoch and y-label is Test Accuracy. In Figures 2 and 3, the axis labels are missing.\n\nQUESTIONS:\n1.This paper seems to focus on classification tasks. Does ε-Softmax also work well for regression tasks?\n\n2.Is ε-Softmax computationally efficient in terms of training time compared with baseline methods?\n\n3.This paper shows the excellent performance of ε-Softmax for label noise. Does ε-Softmax work for input (features) noise as well?\n\n4.I assume that in this paper, for the label noise models, both training and testing labels are noisy. I am curious about the performance when the training labels are clean and the testing labels are noisy.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes the epsilon-softmax technique as a method to address label noise. Epsilon-softmax facilitates peaky predictions by increasing the value of the highest prediction, and it also functions to reduce the magnitude of the gradient when the prediction aligns with the given label. The author introduces the concept of All-k consistency to interpret this paradigm and presents experiments on prominent real-world benchmark datasets in the field of label noise learning, specifically WebVision and Clothing1M.\n\nSTRENGTHS:\nThe proposed epsilon softmax by the author takes a different approach compared to existing symmetric-like functions, which aim to reduce the gradient value for entirely incorrect predictions. From my understanding, the author’s approach reduces the gradient value for predictions that match the given label. By providing the value and interpretation of this novel approach, the author has significantly broadened the scope of the label noise learning field with their straightforward yet impactful idea. The proposed method has the advantage of simple gradient computation without requiring additional high-cost operations, making it applicapable to other Label Noise Learning (LNL) methods. The author demonstrates the experimental value of this approach by applying it to both the cross-entropy loss function and the focal loss function.\n\nWEAKNESSES:\nThis section addresses two major concerns. For minor concerns, please refer to the \"Questions\" part.\n1. The author mentions in line 40 the necessity for a method that can achieve both effective learning and robustness. While the proposed method offers a different perspective compared to symmetric-like loss methods, it is challenging to assert that it fully meets this necessity. Ironically, to balance the trade-off between effective learning and robustness, the author combines CE_(epsilon) loss and MAE loss. This ability to manage trade-offs is also found in other symmetric-like loss-based methods. In this context, I am interested in understanding why the proposed method might offer a better trade-off and whether it truly provides a better trade-off. I attempted to verify this through experimental comparison, but several issues arose: (1) There are no experiments that allow for a comparison of trade-offs. Experiments demonstrating the trade-off by varying alpha and beta are necessary. (2) The performance of existing methods is reported to be lower. For example, refer to the SCE paper.\n2. The proposed method introduces two additional hyperparameters: m and alpha / beta. Unfortunately, based on the recorded experimental results, the proposed method appears to be sensitive to these hyperparameters. If this is not true, providing comprehensive experimental results that show the effects of varying these hyperparameters would enhance the perceived value of the proposed method. \nAnd if the proposed method is indeed sensitive to changes in hyperparameters, I would like to see evidence that it is not sensitive to hyperparameter variations within the in-distribution domain. I recommend performing validation and test processes to identify optimal hyperparameters (refer to the processes outlined in the GJS and JS papers).\n\nQUESTIONS:\n1. Based on the gradient analysis, it appears that when m is sufficiently large, the sensitivity of performance to m would not be significant. However, as shown in Table 3, the difference in performance is quite notable. Does the author have an explanation for this discrepancy? Additionally, has the author investigated the results when m is infinite, meaning the CE loss is not used when the prediction matches the label?\n2. Has the author ever checked the results of using only the CE_(epslion) loss function? Sections 3.1 to 3.3 and Lemma 2 suggest the importance of the single CE_(epslion) loss function.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}