Title: Label Noise: Ignorance Is Bliss
Venue: NeurIPS 2024
Accepted: yes
Keywords: label noise; multi-class classification; learning theory; domain adaptation; minimax rate; self-supervised learning
Abstract: We establish a new theoretical framework for learning under multi-class, instance-dependent label noise. 
    This framework casts learning with label
    noise as a form of domain adaptation, in particular, domain adaptation
    under posterior drift. 
    We introduce the concept of \emph{relative signal strength} (RSS), a pointwise measure that quantifies the transferability from noisy to clean posterior. 
    Using RSS, we establish nearly matching upper and lower bounds on the excess risk. 
    Our theoretical findings support 
    the simple
    \emph{Noise Ignorant Empirical Risk Minimization (NI-ERM)} principle,
    which minimizes empirical risk while ignoring label noise.
    Finally, we translate this theoretical insight into practice: by
    using NI-ERM to fit a linear classifier on top of a self-supervised
    feature extractor, we achieve state-of-the-art performance on the
    CIFAR-N data challenge.

Label Noise: Ignorance Is Bliss

Abstract

We establish a new theoretical framework for learning under multi-class, instancedependent label noise. This framework casts learning with label noise as a form of domain adaptation, in particular, domain adaptation under posterior drift. We introduce the concept of relative signal strength (RSS), a pointwise measure that quantifies the transferability from noisy to clean posterior. Using RSS, we establish nearly matching upper and lower bounds on the excess risk. Our theoretical findings support the simple Noise Ignorant Empirical Risk Minimization (NI-ERM) principle, which minimizes empirical risk while ignoring label noise. Finally, we translate this theoretical insight into practice: by using NI-ERM to fit a linear classifier on top of a self-supervised feature extractor, we achieve state-of-the-art performance on the CIFAR-N data challenge.

1 Introduction

The problem of classification with label noise can be stated in terms of random variables X , Y , and ˜ Y , where X is the feature vector, Y ∈ { 1 , . . . , K } is the true label associated to X , and ˜ Y ∈ { 1 , . . . , K } is a noisy version of Y . The learner has access to i.i.d. realizations of ( X, ˜ Y ) , and the objective is to learn a classifier that optimizes the risk associated with ( X,Y ) .

In recent years, there has been a surge of interest in the challenging setting of instance (i.e., feature) dependent label noise, in which ˜ Y can depend on both Y and X . While several algorithms have been developed, there remains relatively little theory regarding algorithm performance and the fundamental limits of this learning paradigm.

This work develops a theoretical framework for learning under multi-class, instance-dependent label noise. Our framework hinges on the concept of relative signal strength , which is a point-wise measure of 'noisiness' in a label noise problem. Using relative signal strength to charachterize the difficulty of a label noise problem, we establish nearly matching upper and lower bounds for excess risk. We further identify distributional assumptions that ensure that the lower and upper bounds tend to zero as the sample size n grows, implying that consistent learning is possible.

Surprisingly, Noise Ignorant Empirical Risk Minimization (NI-ERM) principle, which conducts empirical risk minimization as if no label noise exists, is (nearly) minimax optimal. To translate this insight into practice, we use NI-ERM to fit a linear classifier on top of a self-supervised feature extractor, achieving state-of-the-art performance on the CIFAR-N data challenge.

2 Literature review

Theory and algorithms for classification with label noise are often based on different probabilistic models. Such models may be categorized according on how ˜ Y depends on Y and X . The simplest model is symmetric noise, where the distribution of ˜ Y is independent of Y and X [Angluin and Laird, 1988]. In this case, the probability that ˜ Y = k is the same for all k = Y , regardless of Y and X . In this setting, it is easy to show that minimizing the noisy excess risk (associated to the 0/1 loss) implies minimizing the clean excess risk, a property known as immunity . When immunity holds, there is no need to modify the learning algorithm on account of noisy labels. In other words, the learner may be ignorant of the label noise and still learn consistently.

̸

A more general model is classification with label dependent noise, in which the distribution of ˜ Y depends on Y , but not X . Many practical algorithms have been developed over the years, based on principles including data re-weighting [Liu and Tao, 2015], robust training [Han et al., 2018, Liu et al., 2020, Hu et al., 2020, Foret et al., 2021, Liu et al., 2022] and data cleaning [Brodley and Friedl, 1999, Northcutt et al., 2021]. Consistent learning algorithms still exist, such as those based on loss correction [Natarajan et al., 2013, Patrini et al., 2017, Van Rooyen and Williamson, 2018, Liu and Guo, 2020, Zhang et al., 2022]. These approaches assume knowledge of the noise transition probabilities, which can be estimated under some identifiability assumptions [Scott et al., 2013, Zhang et al., 2021b].

In the most general setting, that of instance dependent label noise, the distribution of ˜ Y depends on both Y and X . While algorithms are emerging [Cheng et al., 2021, Zhu et al., 2021, Wang et al., 2022, Yang et al., 2023], theory has primarily focused on the binary setting. Scott [2019] establishes immunity for a Neyman-Pearson-like performance criterion under a posterior drift model, discussed in more detail below. Cannings et al. [2020] establish an upper bound for excess risk under the strong assumption that the optimal classifiers for the clean and noisy distributions are the same.

Closest to our work, Im and Grigas [2023] derive excess risk upper and lower bounds, and reach a similar conclusion, that noise-ignorant ERM attains the lower bound up to a constant factor. Our results, based on the new concept of relative signal strength, provide a more refined analysis.

Additional connections between our contributions and prior work are made throughout the paper.

3 Problem statement

Notation. X denotes the feature space and Y = { 1 , 2 , . . . , K } denotes the label space, with K ∈ N . The K -simplex is ∆ K := { p ∈ R K : ∀ i, p i ≥ 0 , ∑ p i = 1 } . A K × K matrix is row stochastic if all of its rows are in ∆ K . Denote the i -th element of a vector v as [ v ] i , and the ( i, j ) -th element of a matrix M as [ M ] i,j .

̸

We consider the setting where, instead of the true class label Y , a noisy label ˜ Y is observed. The training data ( X 1 , ˜ Y 1 ) , . . . , ( X n , ˜ Y n ) can be viewed as an i.i.d. sample drawn from a 'noisy' distribution P X ˜ Y . We define P ˜ Y | X = x , η , R and ˜ f ∗ analogously to the 'clean' distribution P XY .

In conventional multiclass classification, we observe training data ( X 1 , Y 1 ) , . . . , ( X n , Y n ) drawn i.i.d. from a joint distribution P XY . The marginal distribution of X is denoted by P X , and the class posterior probabilities P Y | X = x are captured by a K -simplex-valued vector η : X → ∆ K , where the j -th component of the vector is [ η ( x )] j = P ( Y = j | X = x ) . A classifier f : X → Y maps an instance x to a class f ( x ) ∈ Y . Denote the risk of a classifier f with respect to distribution P XY as R ( f ) = E ( X,Y ) ∼ P XY [ ✶ { f ( X ) = Y } ] . The Bayes optimal classifier for P XY is f ∗ ( x ) ∈ arg max η ( x ) . The Bayes risk, which is the minimum achievable risk, is denoted as R ∗ = R ( f ∗ ) = inf f R ( f ) .

˜ ˜ The goal of learning from label noise is to find a classifier that is able to minimize the 'clean test error,' that is, the risk R defined w.r.t. P XY , even though the learner has access to only corrupted training data ( X i , ˜ Y i ) i.i.d. ∼ P X ˜ Y .

Noise transition perspective. Traditionally, label noise is modeled through the joint distribution of ( X,Y, ˜ Y ) . This joint distribution is governed by P X , the clean class posterior P Y | X , and a

matrix-valued function known as the noise transition matrix . The ( i, j ) -th entry of the matrix is defined as:

Domain adaptation perspective. Alternatively, label noise learning can be framed as a domain adaptation problem. In this view, P X ˜ Y represents the source domain, and P XY represents the target domain. The relationship between the two domains is characterized by 'posterior drift,' meaning that while the source and target share the same X -marginal, the class posteriors (i.e., the distribution of labels given X ) may differ [Scott, 2019, Cai and Wei, 2021, Maity et al., 2023, Liu et al., 2024]. Thus, a label noise problem can also be described by a triple ( P X , η , η ) .

This implies that the noisy and clean class posteriors are related by ˜ η ( x ) = E ( x ) ⊤ η ( x ) , where ⊤ denotes the matrix transpose.

˜ The two perspectives are equivalent, as discussed in Appendix A.1. In this work, we emphasize the domain adaptation perspective for Sections 4 and 5, and the noise transition perspective for Section 6.

4 Relative signal strength

To study label noise, we introduce the concept of relative signal strength (RSS). This is a pointwise measure of how much 'signal' (certainty about the label) is contained in the noisy distribution relative to the clean distribution. Previous work [Cannings et al., 2020, Cai and Wei, 2021] has examined a related concept within the context of binary classification, under the restriction that clean and noisy Bayes classifiers are identical. Our definition incorporates multi-class classification and relaxes the requirement that the clean and noisy Bayes classifiers agree.

Definition 1 (Relative Signal Strength) For any class probability vectors η , ˜ η , define the relative signal strength (RSS) at x ∈ X as where 0 / 0 := + ∞ . Furthermore, for κ ∈ [0 , ∞ ) , denote the set of points whose RSS exceeds κ as .

M ( x ; η , ˜ η ) is a point-wise measure of how much 'signal' the noisy posterior contains about the clean posterior. To gain some intuition, first notice that if the noisy Bayes classifier predicts a different class than the clean Bayes classifier, the RSS is 0 by taking j = arg max ˜ η (assuming for simplicity that the arg max is a singleton set). Now suppose the clean and noisy Bayes classifiers do make the same prediction at x , say i ∗ , and consider a fixed j . If is small, it means that the clean Bayes classifier is relatively certain that j is not the correct clean label, while the noisy Bayes classifier is less certain that j is not the correct noisy label. Taking the minimum over j gives the relative signal strength at x . As we formalize in the next section, a large RSS at x ensures that a small (pointwise) noisy excess risk at x implies a small (pointwise) clean excess risk. To gain more intuition, consider the following examples.

Here, first of all, arg max η = arg max ˜ η = 2 , i.e., the clean and noisy Bayes classifier give the same prediction. What's more, M ( x ; η , ˜ η ) < 1 because the clean Bayes classifier is absolutely certain about its prediction, while the noisy Bayes classifier is much less certain.

Example 2 When η ( x ) = [0 1 0] ⊤ and ˜ η ( x ) = [0 0 1] ⊤ , M ( x ; η , ˜ η ) = min j ∈Y max i [ ˜ η ( x )] i -[ ˜ η ( x )] j max i [ η ( x )] i -[ η ( x )] j = [ ˜ η ( x )] 3 -[ ˜ η ( x )] 3 [ η ( x )] 2 -[ η ( x )] 3 = 1 -1 1 -0 = 0 . The zero signal strength results from η and η leading to different predictions about arg max .

Example 3 (Comparison to KL divergence) When η ( x ) = [0 . 05 0 . 7 0 . 25] ⊤ , and η (1) ( x [0 . 25 0 . 7 0 . 05] ⊤ , η (2) ( x ) = [0 . 1 0 . 6 0 . 3] ⊤ ,

∥ ˜ ∥ ˜ Here, ˜ η (2) is 'closer' to η in terms of KL divergence, but ˜ η (1) provides more information in terms of predicting the arg max of η . There is no conflict: KL divergence considers the similarity between two (whole) distributions, while the task of classification only focuses on predicting the arg max .

This also illustrates why our notion of RSS is better suited for the label noise problem than other general-purpose distance measures between distributions.

A desirable learning scenario would be if A κ ( η , ˜ η ) = X for some large κ , indicating that the signal strength is big across the entire space. Unfortunately, this ideal situation is generally not achievable. To gain some insight, consider the following result, proved in Appendix A.2.1.

̸

˜ ˜ If we assume that both arg max sets are singletons, this result indicates that A 0 , the region with positive RSS, is the region where the true and noisy Bayes classifiers agree. Accordingly, X \ A 0 , the zero signal region, is the region where the clean and noisy Bayes decision rules differ. The 'region of strong signal,' A κ , is a subset of A 0 . Since the clean and noisy Bayes classifiers will typically disagree for at least some x , A 0 = X in general. We note that the strong assumption that A 0 = X has been made in prior studies [Cannings et al., 2020, Cai and Wei, 2021]. Our notion of RSS relaxes this assumption and provides a unified view.

4.1 RSS in binary classification

We can express relative signal strength more explicitly in the binary setup. Let η ( x ) := [ η ( x )] 1 = P ( Y = 1 | X = x ) and ˜ η ( x ) := [ ˜ η ( x )] 1 = P ( ˜ Y = 1 | X = x ) . In standard binary classification, the margin [Tsybakov, 2004, Massart and Nédélec, 2006], defined as ∣ ∣ η ( x ) -1 2 ∣ ∣ , serves as a pointwise measure of signal strength. Our notion of relative signal strength (RSS) can be interpreted as an extension of this concept in the context of label noise learning.

Proposition 2 In the binary setting, for κ ≥ 0 ,

In other words, RSS can be viewed as a 'relative' margin.

Example 4 Illustration of relative signal strength in a binary classification setup (Figure 1).

4.2 Posterior Drift Model Class.

Now putting definitions together, we consider the posterior drift model Π defined over the triple ( P X , η , ˜ η ) . Let ϵ ∈ [0 , 1] , κ ∈ [0 , + ∞ ) , and define

˜ ˜ This is a set of triples (label noise problems) such that A κ , the region with RSS at least κ , covers at least 1 -ϵ of the probability mass. In the next section, we will demonstrate that the performance within A κ can be guaranteed, whereas learning outside the region A κ is provably challenging.

-

10

.

0

-

7

.

5

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Class

Probability

P

(

Y

= 1

|

X

=

x

)

P

(

˜

Y

= 1

|

X

=

x

)

threshold

=

0.5

X \ A

0

-

10

.

0

-

7

.

5

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

0

2

4

6

8

10

12

Relative

Signal

Strength

M

(

x

;

η

,

˜

η

)

X \ A

0

A

0

.

4

x

x

5 Upper and lower bounds

In this section, we establish both upper and lower bounds for excess risk under multi-class instancedependent label noise.

5.1 Minimax lower bound

Our first theorem reveals a fundamental limit: no classifier trained using noisy data can surpass the constraints imposed by relative signal strength in a minimax sense. To state the theorem, we employ the following notation and terminology. Denote the noisy training data by Z n = { ( X i , ˜ Y i ) } n i =1 i.i.d. ∼ P X ˜ Y . A learning rule ˆ f is an algorithm that takes Z n and outputs a classifier. The risk R ( ˆ f ) of a learning rule is a random variable, where the randomness is due to the draw Z n .

Theorem 1 (Minimax Lower Bound) Let ϵ ∈ [0 , 1] , κ > 0 . Then

where the inf is over all learning rules.

The proof in Appendix A.2.3 offers insights into how label noise impacts the learning process: On the low RSS region ( X\A κ ), learning is difficult if not impossible, and the learner incurs an irreducible error of (1 -1 /K ) ϵ . On the high RSS region ( A κ ), the standard nonparametric rate [Devroye et al., 1996] is scaled by 1 /κ . These aspects determine fundamental limits that no classifier trained only on noisy data can overcome without additional assumptions.

5.2 Upper bound

This subsection establishes an upper bound for Noise Ignorant Empirical Risk Minimizer (NI-ERM) , the empirical risk minimizer trained on noisy data. This result implies that NI-ERM is (nearly) minimax optimal, a potentially surprising result given that NI-ERM is arguably the simplest approach one might consider. We begin by presenting a general result on the excess risk of any classifier, which is proved in Appendix A.2.4.

Lemma 1 (Oracle Inequality) For any label noise problem ( P X , η , η ) and any classifier f ,

For ( P X , η , ˜ η ) ∈ Π( ϵ, κ ) , the first term is bounded by ϵ . When f is selected by ERM over the noisy training data, conventional learning theory implies a bound on the second term. This leads to the following upper bound for NI-ERM, whose proof is in Appendix A.2.5.

̸

Theorem 2 (Excess Risk Upper Bound of NI-ERM) Let ϵ ∈ [0 , 1] , κ > 0 . Consider any ( P X , η , ˜ η ) ∈ Π( ϵ, κ ) , assume function class F has Natarajan dimension V , and the noisy Bayes classifier ˜ f ∗ belongs to F . Let ˆ f ∈ F be the ERM trained on Z n = { ( X i , ˜ Y i ) } n i =1 , i.e., ˆ f = arg min f ∈F 1 n ∑ n i =1 ✶ { f ( X i ) = ˜ Y i } . Then

˜ O denotes bigO notation ignoring logarithmic factors. The Natarajan dimension is a multiclass analogue of the VC dimension. The upper bound in Theorem 2 aligns with the minimax lower bound (Theorem 1) in both terms. For the irreducible error ϵ , there is a small gap of 1 /K . This gap arises because, in the lower bound construction, the low signal region X \ A κ is known to the learner, whereas knowledge of X \ A κ is not provided to NI-ERM. If A κ were known to the learner (an unrealistic assumption), then a mixed strategy that preforms NI-ERM on A κ and randomly guesses on X \ A κ would have an upper bound with first term of (1 -1 /K ) ϵ , exactly matching the lower bound. Regarding the second term, there is a universal constant and a logarithmic factor between the lower and upper bounds, which is a standard outcome in learning theory.

This result is surprising as it indicates that the simplest possible approach, which ignores the presence of noise, is nearly optimal. No learning rule could perform significantly better in this minimax sense.

5.3 A smooth margin-condition on the relative signal strength

The previous sections have analyzed learning with label noise over the class Π( ϵ, κ ) = { ( P X , η , ˜ η ) : P X ( A κ ) ≥ 1 -ϵ } . Now, the set X \ A κ equals ( X \ A 0 ) ∪ ( A 0 \ A κ ) . The first part of this decomposition is the region where the Bayes classifiers under the noisy and clean distributions differ, while the second is a region where these match, but the RSS is small. Naturally, while P X ( X \ A 0 ) must be incurred as an irreducible error, one may question why the class Π also limits the mass of A 0 \ A κ . After all, with enough data, the optimal prediction in this region can be learned.

This issue would be resolved if there existed a κ 0 > 0 such that P X ( A 0 ) = P X ( A κ 0 ) , i.e., if P X ( A 0 \ A κ 0 ) = 0 . In fact, our lower bound from Theorem 1 uses precisely such a construction. An interesting point of comparison to this condition lies in Massart's hard-margin condition from standard supervised learning theory, which, for binary problems, demands that P X ( | [ η ( x )] 1 -[ η ( x )] 2 | < h ) = 0 for some h > 0 , under which one obtains minimax excess risk bounds of O ( V / nh ) [Massart and Nédélec, 2006]. With this lens, we can view the condition P X ( A 0 \ A κ 0 ) = 0 as a type of hard-margin condition on the relative signal strength M . This naturally motivates a smoothened version of this condition, inspired by Tsybakov's soft-margin condition [Tsybakov, 2004].

Definition 2 A triple ( P X , η , ˜ η ) satisfies an ( ϵ, α, C α ) -smooth relative signal margin condition with ϵ ∈ [0 , 1] , α > 0 , C α > 0 if

We show in Appendix A.2.7 that the techniques of Section 5.2 yield the following result for Π ′ .

∀ κ > 0 , P X ( M ( x ; η , ˜ η ) ≤ κ ) ≤ C α κ α + ϵ. Further, we define Π ′ ( ϵ, α, C α ) as the set of triples ( P X , η , ˜ η ) that satisfy an ( ϵ, α, C α ) -smooth relative signal margin condition.

Theorem 3 Let ϵ ∈ [0 , 1] , α > 0 , C α > 0 . Consider any ( P X , η , ˜ η ) ∈ Π ′ ( ϵ, α, C α ) , assume function class F has Natarajan dimension V , and the noisy Bayes classifier ˜ f ∗ belongs to F . Let ˆ f ∈ F be the ERM trained on Z n = { ( X i , Y i ) } n i =1 . Then

Compared to Theorem 2, we see that the rate of the second term is slightly slower, which is consistent with standard learning theory where Massart's hard margin assumption leads to faster rates than Tsybakov's. The advantage of the smooth relative margin is that the irreducible term in the above theorem is exactly P X ( X \ A 0 ) , which has a clearer meaning as it measures the mismatch between clean and noisy Bayes classifiers. Further, notice that the NI-ERM algorithm does not need information about α , and thus the result is adaptive to both α , and to the optimal κ for each value of α , as a consequence of the oracle inequality of Lemma 1.

More broadly, Theorem 3 illustrates the flexibility of our conceptualization of label noise problems through RSS. The RSS M characterizes the irreducible error in label noise learning, similar to how the regression function η characterizes excess risk in standard learning. Thus, standard theoretical frameworks can be adapted to the noisy label problem via the relative signal .

6 Conditions that ensure noise immunity

The minimax lower bound in the previous section revealed a negative outcome, indicating that no method can do well in the low signal region. Nevertheless, numerous empirical successes have been observed even under significant label noise. This is not mere coincidence. In this section, we will illustrate that the high signal region A κ can indeed cover the entire input space X even under massive label noise, albeit with the constraint A κ ⊆ A 0 as stated in Proposition 1. This not only explains past empirical successes, but also gives a rigorous condition on the consistency of NI-ERM.

This section will delve into the study of noise transition matrix E and establish precise conditions that lead to A 0 = X . These conditions are linear algebraic conditions on E that ensure arg max ˜ η ( x ) = arg max η ( x ) . As a result, we can infer that in a 10 -class classification problem, even with up to 90% of training labels being incorrect, the NI-ERM can still asymptotically achieve Bayes accuracy. In the upcoming definition, we introduce the concept of noise immunity, wherein the optimal classifiers remain unaffected by label noise [Menon et al., 2015, Scott, 2019].

Definition 3 (Immunity) We say that a K -class classification problem ( P X , η , ˜ η ) is immune to label noise if ∀ x ∈ X , arg max η ( x ) = arg max η ( x ) .

˜ Notice that due to Proposition 1, if a problem is immune, then A 0 = X . We now provide necessary and sufficient conditions on noise transition matrix E that ensure noise immunity. We begin by considering distribution P XY with zero Bayes risk, that is, where η is one-hot almost surely. A matrix is defined as diagonally dominant if, for each row, the diagonal element is the unique maximum.

Theorem 4 (Immunity for Zero-error Distribution) If P XY has Bayes risk of zero, then immunity holds if and only if for all x , the noise transition matrix E ( x ) is diagonally dominant.

̸

The above result relies on strong assumptions about the distribution P XY . Now, we present a result that applies to any distribution, which, as a trade-off, turns out to impose more requirements on E .

Remark For a zero-error distribution P XY , even corrupted with instance-dependent label noise, achieving the Bayes risk is still feasible with a noise rate P ( ˜ Y = Y ) up to K -1 K . This highlights that the task of classification itself is robust to label noise, specially when the clean η is well-separated.

Theorem 5 (Universal Immunity) For any choice of P XY , immunity holds

̸

0

0

.

2

0

.

4

0

.

6

0

.

8

1

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Testing Accuracy

Binary

Linear

Theory

(a) Logistic regression trained on 2D gaussian mixture data with different levels of symmetric noise

̸

0

0

.

2

0

.

4

0

.

6

0

.

8

1

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Testing Accuracy

10-class

2-layer CNN

Theory

̸

Previous works [Ghosh and Kumar, 2017, Menon et al., 2018, Oyen et al., 2022] have established that symmetric label noise is sufficient for immunity. Our contribution advances this understanding by demonstrating that such noise conditions are not only sufficient but also necessary. Specifically, under symmetric label noise, learning towards the Bayes classifier is feasible as long as the proportion of wrong labels does not exceed K -1 K . Furthermore, this transition is abrupt: when P ( ˜ Y = Y ) < K -1 K , A 0 = X , but when P ( ˜ Y = Y ) ≥ K -1 K , A 0 = ∅ . Consequently, we expect to see a sudden drop in performance when noise rate passes the threshold.

̸

̸

The rationale behind the necessity of E ( x ) taking this specific form is that it redistributes the probability mass of η in a 'uniform' manner. This constraint arises because E ( x ) cannot favor any classes besides the true class. For instance, consider η ( x ) = [ 1 K + δ 1 K -δ 1 K · · · 1 K ] ⊤ for some small δ > 0 , a 'non-uniform' E ( x ) would alter the arg max .

̸

The above theorems demonstrate that signal strength at x can still be high even under massive label noise P ( ˜ Y = Y ) , and, in essence, it is the discrete nature of the classification problem that allows robustness to label noise. When immunity holds, the irreducible error in Theorem 2 vanishes, therefore NI-ERM becomes a consistent learning rule. We validate this through data simulations presented in Figure 2, where we systematically flip labels uniformly and observe the corresponding changes in the testing accuracy of NI-ERM. The simulation results align closely with the theoretical expectations: NI-ERM achieves near-Bayes risk performance until a certain noise threshold is reached, beyond which the testing performance sharply deteriorates.

7 Practical implication

The modern practice of machine learning often involves training a deep neural network. In complex tasks involving noisy labels, the naïve NI-ERM is often outperformed by state-of-the-art methods by a significant extent [Li et al., 2020, Xiao et al., 2023]. This is consistent with the finding that directly training a large neural network on noisy data frequently leads to overfitting [Zhang et al., 2021a].

Yet this is not grounds for abandoning NI-ERM altogether as a practical strategy. Instead of using NI-ERM for end-to-end training of a deep neural network, we instead propose the following simple, two-step procedure, termed 'feature extraction + NI-ERM'.

Perform feature extraction using any method (e.g., transfer learning or self-supervised learning) that does not require labels.

Learn a simple classifier (e.g., a linear classifier) on top of these extracted features, using the noisily labelled data, in a noise-ignorant way.

This approach has three advantages over full network training. First, it avoids the potentially negative impact of the noisy labels on the extracted features. Second, it enjoys the inherent robustness of fitting a simple model (step 2) on noisy data, which we observed in Figure 2. Third, it avoids the need to tune hyperparameters of the feature extractor using noisy labels. We note that a 'self-supervised + simple approach' to learning was previously studied by Bansal et al. [2021], although their focus was on generalisation properties without label noise. We also acknowledge that the practical idea of ignoring label noise is not new [Ghosh and Lan, 2021], but the full power of this approach has not been previously recognized. For example, prior works often combine this approach with additional steps or employ early stopping to mitigate the effects of noise [Zheltonozhskii et al., 2022, Xue et al., 2022].

Remarkably, this two-step approach attains extremely strong performance. We conducted experiments 1 on the CIFAR image data under two scenarios: synthetic label flipping (symmetric noise) and realistic human label errors [Wei et al., 2022], as shown in Figure 3. We examine three different feature extractors: the DINOv2 foundation model [Oquab et al., 2023], ResNet-50 features extracted from training on ImageNet [He et al., 2016], and self-supervised ResNet-50 using contrastive loss [Chen et al., 2020]. We also compared to a simple linear model trained on the raw pixel intensities, and a ResNet-50 trained end-to-end. We observed that ResNet-50 exhibits degrading performance with increasing noise, consistent with previous findings [Zhang et al., 2021a, Mallinar et al., 2022]. The linear model demonstrates robustness to noise, but suffers from significant approximation error.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Test Accuracy

Synthetic Label Error

Linear + DINOv2 SSL

Linear + ResNet-50 SSL

Linear + ResNet-50 TL

ResNet-50

0

0

.

2

0

.

4

0

.

6

0

.

8

1

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Test Accuracy

Human Label Error

Linear + DINOv2 SSL

Linear+ ResNet-50 SSL

Linear + ResNet-50 TL

ResNet-50

(a) CIFAR-10 with synthetic label noise

(b) CIFAR-10 with realistic label noise

Conversely, the FE+NI-ERM approach enjoys the best of both worlds. Regardless of how the feature extraction is carried out, the resulting models exhibit robustness to label noise, while the overall accuracy depends entirely on the quality of the extracted features. This is illustrated in Figure 3, where the flatness of the accuracy curves as noise increases indicates the robustness, and the intercept at zero label noise is a measure of the feature quality. Importantly, this property holds true even under realistic label noise of CIFAR-N [Wei et al., 2022]. In fact, we find that using the DINOv2 [Oquab et al., 2023] extracted features in our FE+NI-ERM approach yields state of the art results on the CIFAR-10N and CIFAR-100N benchmarks, across the noise levels, as shown in Table 1.

We emphasize that the only hyperparameters of our model are the hyperparameters of the linear classifier, which are tuned automatically using standard cross-validation on the noisy labels. This contrasts to the implementation of many methods on the CIFAR-N leaderboard ( http://noisylabels.com/ )

2 , where the hyperparameters are hard-coded. Furthermore, our approach does not rely on data augmentation. Additional experiments, detailed in Appendix A.4 , include comparisons with the 'linear probing, then fine-tuning' approach [Kumar et al., 2022], the application of different robust learning strategies on DINOv2 features, and results on synthetic instance-dependent label noise.

Overall, the strong performance, the simplicity of the approach and the lack of any untunable hyperparameters highlights the effectiveness of FE+NI-ERM, and indicates the value of further investigation into its properties.

̸

±

±

RSS for realistic human label error. To calculate the RSS under realistic human label error, we train two linear classifiers on DINOv2 features under clean and noisy labels and use the models' predictions as estimates for the class probabilities η and ˜ η . Despite the high overall noise rate in CIFAR-10N 'Worst' labels, with P ( Y = ˜ Y ) = 40 . 21% , we conjecture that the region where there is no signal, X \ A 0 , covers only a small portion of the probability mass ( ϵ ≤ 4% ). Furthermore, the cumulative distribution of the estimated RSS can be upper-bounded by a polynomial C α κ α + ϵ , supporting the validity of the smooth relative signal margin condition introduced in Section 5.3.

8 Conclusions

This work presents a rigorous theory for learning under multi-class, instance-dependent label noise. We establish nearly matching upper and lower bounds for excess risk and identify precise conditions for classifier consistency. Our theory reveals the (nearly) minimax optimality of Noise Ignorant Empirical Risk Minimizer (NI-ERM). To make this theory practical, we provide a simple modification leveraging a feature extractor with NI-ERM, demonstrating significant performance enhancements. A limitation of this work is that our methodology warrants more extensive experimental evaluation.

±

±

±

±

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Empirical

CDF

C

α

κ

α

+

/epsilon1

= 1

.

49

·

κ

1

.

35

+

/epsilon1

P

X

(

M

(

x

;

η

,

˜

η

)

κ

))

/epsilon1

= 0

.

04

κ

References

Paul Albert, Eric Arazo, Tarun Krishna, Noel E O'Connor, and Kevin McGuinness. Is your noise correction noisy? pls: Robustness to label noise with two stage detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 118-127, 2023.

Dana Angluin and Philip Laird. Learning from noisy examples. Machine learning , 2:343-370, 1988.

Yamini Bansal, Gal Kaplun, and Boaz Barak. For self-supervised learning, rationality implies generalization, provably. In International Conference on Learning Representations , 2021.

Carla E Brodley and Mark A Friedl. Identifying mislabeled training data. Journal of artificial intelligence research , 11:131-167, 1999.

T Tony Cai and Hongji Wei. Transfer learning for nonparametric classification: Minimax rate and adaptive classifier. The Annals of Statistics , 49(1):100-128, 2021.

Timothy I Cannings, Yingying Fan, and Richard J Samworth. Classification with imperfect training labels. Biometrika , 107(2):311-330, 2020.

Hao Chen, Ankit Shah, Jindong Wang, Ran Tao, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, and Bhiksha Raj. Imprecise label learning: A unified framework for learning with various imprecise label configurations. arXiv preprint arXiv:2305.12715 , 2023.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instancedependent label noise: A sample sieve approach. In International Conference on Learning Representations , 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. IEEE, 2009.

Luc Devroye, László Györfi, and Gabor Lugosi. A Probabilistic Theory of Pattern Recognition , volume 31. Springer, 1996.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=6Tm1mposlrM .

Aritra Ghosh and Himanshu Kumar. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , 2017.

Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2703-2708, 2021.

Aritra Ghosh, Naresh Manwani, and PS Sastry. Making risk minimization tolerant to label noise. Neurocomputing , 160:93-107, 2015.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems , 31, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In International Conference on Learning Representations , 2020.

Hyungki Im and Paul Grigas. Binary classification with instance and label dependent label noise. arXiv preprint arXiv:2306.03402 , 2023.

Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations , 2022.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semisupervised learning. In International Conference on Learning Representations , 2020.

Jiashuo Liu, Tianyu Wang, Peng Cui, and Hongseok Namkoong. On the need for a language describing distribution shifts: Illustrations on tabular datasets. Advances in Neural Information Processing Systems , 36, 2024.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. Advances in neural information processing systems , 33:20331-20342, 2020.

Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by overparameterization. In International Conference on Machine Learning , pages 14153-14172. PMLR, 2022.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence , 38(3):447-461, 2015.

Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International conference on machine learning , pages 6226-6236. PMLR, 2020.

Jianhao Ma and Salar Fattahi. Blessing of depth in linear regression: Deeper models have flatter landscape around the true solution. Advances in Neural Information Processing Systems , 35: 34334-34346, 2022.

Subha Maity, Diptavo Dutta, Jonathan Terhorst, Yuekai Sun, and Moulinath Banerjee. A linear adjustment based approach to posterior drift in transfer learning. Biometrika , 2023. URL https: //doi.org/10.1093/biomet/asad029 .

Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. Advances in Neural Information Processing Systems , 35:1182-1195, 2022.

Pascal Massart and Élodie Nédélec. Risk bounds for statistical learning. The Annals of Statistics , 34 (5):2326-2366, 2006.

Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class-probability estimation. In International conference on machine learning , pages 125-134. PMLR, 2015.

Aditya Krishna Menon, Brendan Van Rooyen, and Nagarajan Natarajan. Learning from binary labels with instance-dependent noise. Machine Learning , 107:1561-1595, 2018.

Balas K Natarajan. On learning sets and functions. Machine Learning , 4:67-97, 1989.

Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems , 26, 2013.

Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research , 70:1373-1411, 2021.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research , 2023.

Diane Oyen, Michal Kucer, Nicolas Hengartner, and Har Simrat Singh. Robustness to label noise depends on the shape of the noise distribution. Advances in Neural Information Processing Systems , 35:35645-35656, 2022.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1944-1952, 2017.

Clayton Scott. A generalized Neyman-Pearson criterion for optimal domain adaptation. In Algorithmic Learning Theory , pages 738-761. PMLR, 2019.

Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. In Conference on learning theory , pages 489-511. PMLR, 2013.

Alexander B Tsybakov. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics , 32(1):135-166, 2004.

Brendan Van Rooyen and Robert C Williamson. A theory of learning with corrupted labels. Journal of Machine Learning Research , 18(228):1-50, 2018.

N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability & Its Applications , 16(2):264-280, 1971. doi: 10.1137/1116025. URL https://doi.org/10.1137/1116025 .

Jialu Wang, Eric Xin Wang, and Yang Liu. Estimating instance-dependent label-noise transition matrix using a deep neural network. In International Conference on Machine Learning , 2022.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations , 2022.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems , 33:7597-7610, 2020.

Ruixuan Xiao, Yiwen Dong, Haobo Wang, Lei Feng, Runze Wu, Gang Chen, and Junbo Zhao. Promix: Combating label noise via maximizing clean sample utility. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI23 , pages 4442-4450. International Joint Conferences on Artificial Intelligence Organization, 8 2023. doi: 10.24963/ijcai.2023/494. URL https://doi.org/10.24963/ijcai.2023/494 . Main Track.

Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In International Conference on Machine Learning , pages 24851-24871. PMLR, 2022.

Shuo Yang, Songhua Wu, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. A parametrical model for instance-dependent label noise. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021a.

Jianxin Zhang, Yutong Wang, and Clayton Scott. Learning from label proportions by learning with label noise. Advances in Neural Information Processing Systems , 35:26933-26942, 2022.

Mingyuan Zhang, Jane Lee, and Shivani Agarwal. Learning from noisy labels with no change to the training process. In International Conference on Machine Learning , pages 12468-12478. PMLR, 2021b.

Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1657-1667, 2022.

Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instancedependent label noise. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10113-10123, 2021.

A Appendix / supplemental material

A.1 Equivalence of noise transition and domain adaptation perspectives

The noise transition perspective models the joint distribution of ( X,Y, ˜ Y ) , which can be characterized as:

Thus, by specifying P X , η , and E , we obtain a triple ( P X , η , ˜ η ) with ˜ η ( x ) = E ( x ) ⊤ η ( x ) . In contrast, the domain adaptation perspective views label noise problems directly as a triple ( P X , η , η ) , bypassing the explicit modeling of the noise transition matrix E .

˜ If no assumptions are made about the form of E , the domain adaptation view remains fully expressive. Given a triple ( P X , η , ˜ η ) , we can always define a noise transition matrix as: E ( x ) = 1 ˜ η ⊤ , where 1 = [1 . . . 1] ⊤ . We can verify that E is row-stochastic, and

Therefore, these two perspectives are equivalent.

A.2 Proofs

A.2.1 Proof of Proposition 1

Proposition

Proof. Notice that

M ( x ; η , ˜ η ) = 0 ⇐⇒ arg max ˜ η ( x ) ̸⊆ arg max η ( x ) . This is because M ( x ; η , ˜ η ) = 0 when the numerator is zero and the denominator is non-zero, which happens when arg max ˜ η ( x ) ̸⊆ arg max η ( x ) . An equivalent statement of this is

A.2.2 Proof of Proposition 2

Proposition In the binary setting, for κ ≥ 0 ,

Proof. In a brute-force way, we can examine the nine cases where η ( x ) , ˜ η ( x ) is greater, equal, or smaller than 1 / 2 .

If ˜ η ( x ) > 1 2 , η ( x ) > 1 2 , then

If ˜ η ( x ) < 1 2 , η ( x ) < 1 2 , the same argument holds. If ˜ η ( x ) > 1 2 , η ( x ) < 1 2 or ˜ η ( x ) < 1 2 , η ( x ) > 1 2 , take j = arg max η ( x ) , we have

̸

If ˜ η ( x ) = 1 2 , η ( x ) < 1 2 or ˜ η ( x ) = 1 2 , η ( x ) < 1 2 , take j = arg max η ( x ) , we have

If η ( x ) = 1 2 , then

Note that it makes sense for RSS to be + ∞ when η ( x ) = 1 2 , because in this case, clean excess risk R ( f ) -R ( f ∗ ) is 0 at point x for any classifier.

Therefore, we can conclude that

by definition, we have, for κ ≥ 0

A.2.3 Proof of lower bound: Theorem 1

Now we provide a more formal statement of the minimax lower bound and its proof. We begin with the scenario where the noisy distribution P X ˜ Y has zero Bayes risk as an introductory example. The proof for the general case follows a similar strategy but involves more complex bounding techniques. We recommend that interested readers first review the proof of the zero-error version to build a solid understanding before tackling the general case.

Now consider a more restricted subset of Π( ϵ, κ ) :

Theorem (Minimax Lower Bound: when ˜ R ∗ = 0 ) Let ϵ ∈ [0 , 1] , κ > 0 , V > 1 . For any learning rule ˆ f based upon Z n = { ( X i , ˜ Y i ) } n i =1 , and n > max( V -1 , 2) ,

Proof.

We will construct a triple ( P X , η , ˜ η ) that is parameterized by j, b := [ b 1 b 2 · · · b V -1 ] ⊤ , and δ . First, we define P X . Pick any V +1 distinct points x 0 , x 1 , . . . , x V , this is where we need the condition that n > V -1 .

Then, define the clean and noisy class posteriors:

where e i denotes the one-hot vector whose i -th element is one.

The triple ( P X , η , ˜ η ) is thus parameterized by j, b := [ b 1 b 2 · · · b V -1 ] ⊤ , and δ . This construction ensures ( P X , η , η ) ∈ Π( ϵ, κ, V, 0) . In particular,

and ˜ R ∗ = 0 because ˜ η ( x ) is one-hot for all x .

For any classifier f , by definition, its risk equals

Therefore, the Bayes risk and excess risk equal

̸

Under our construction of P X , R ( f ) can be decomposed into two parts and so can the excess risk

where the last equality holds because region { x 0 } only depends on j , while region { x 1 , . . . , x V } only depends on b , δ .

In the remaining part of the proof, we will examine

and

̸

̸

separately.

Let's start with the first term (5), which reflects the excess risk over the 'low signal strength' region { x 0 } . Since η is one-hot on { x 0 } , its Bayes risk over that is zero

̸

To deal with sup j , we use a technique called 'the probabilistic method' [Devroye et al., 1996]: replace j with a random variable J ∼ Uniform { 1 , 2 , . . . , K } :

̸

̸

̸

Again, notice that J is an independent draw. Even if the point x 0 is observed in Z n , the associated noisy label Y = 1 does not give any information about the clean label Y = J . Thus

̸

Now we have the minimax lower bound for the first part (5):

̸

For the second part (6), which is over { x 1 , . . . , x V } , due to the relative signal strength condition, and from our explicit construction in Eqn. (3) and (4), the excess risks w.r.t. true and noisy distribution are related by where ˜ R V ( f ) := ∫ { x 1 ,...,x V } ( 1 -[ ˜ η ( x )] f ( x ) ) dP X ( x ) . Also note that f ∗ ( x ) = ˜ f ∗ ( x ) for x ∈ { x 1 , . . . , x V } , which is a result of our construction of η , ˜ η . Then

̸

̸

This allows us to reduce the label noise problem to a standard learning problem: we have an iid sample Z n from P X ˜ Y and consider the risk evaluated on the same distribution P X ˜ Y . The remainder of the proof is similar to the proof of Devroye et al. [1996, Theorem 14.1].

Notice that by our construction, ˜ Y is a deterministic function of X . To be specific, ˜ Y = ˜ f ∗ ( X ) , where

is the noisy Bayes classifier.

We use the shorthand f b := ˜ f ∗ to denote that the noisy Bayes classifier depends on b . Since the noisy Bayes risk is zero,

Again, use the probabilistic method, replace b with B ∼ Uniform { 1 , 2 } V -1 ,

̸

Since we have B ∼ Uniform { 1 , 2 } V -1 and also Z n | B ∼ P ⊗ n X ˜ Y , then by Bayes rule (or eye-balling, since ˜ η is one-hot), we get the posterior distribution of B | Z n , to be specific: ∀ x ∈ { x 1 , · · · , x V } ,

̸

where we overload the notation P to denote conditional probability of B | Z n .

Then the optimal decision rule for predicting B based on sample Z n is:

̸

̸

̸

Therefore, the error comes from the probability of X ∈ { x 1 , . . . , x V } not being one of the observed X i : for any ˆ f ,

̸

̸

̸

Now we get the minimax risk for the second part (6)

Combine the two parts together, we get the final result, for n > max( V -1 , 2)

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

As for the general version of the lower bound, now consider the set of triples:

Proof.

Now we construct a triple ( P X , η , ˜ η ) that is parameterized by j, b := [ b 1 b 2 · · · b V -1 ] ⊤ , δ , c and p . First, we define P X . Pick any V +1 distinct points x 0 , x 1 , . . . , x V ,

This imposes the constraint ( V -1) p ≤ 1 , which will be satisfied in the end. Notice the difference compared to the previous zero-error proof: we place probability mass p , rather than 1 /n , on x 1 , . . . , x V -1 .

As for the clean and noisy class probabilities, choose

where e i denotes the one-hot vector whose i -th element is one.

The construction for class posterior is also similar to the previous proof, except that for x = x t , t ∈ { 1 , . . . , V -1 } , ˜ η is no longer a one-hot vector, rather has class probability separated by 2 c : ∣ ∣ ∣ [ ˜ η ( x )] 1 -[ ˜ η ( x )] 2 ∣ ∣ ∣ = 2 c .

Therefore, the triple ( P X , η , ˜ η ) can be parameterized by j, b := [ b 1 b 2 · · · b V -1 ] ⊤ , δ , c and p . Again, this construction ensures ( P X , η , η ) ∈ Π( ϵ, κ ) , to be specific:

For any classifier f , its risk can be decomposed into two parts

as can its excess risk

In our construction, ( P X , η , η ) is parameterized by j, b := [ b 1 b 2 · · · b V -1 ] ⊤ , δ , c and p , therefore

Note that we have used the fact that

The first part (10) is exactly the same as in the zero-error proof, and we have

where ˜ R V ( f ) := ∫ { x 1 ,...,x V } ( 1 -[ ˜ η ( x )] f ( x ) ) dP X ( x ) .

From this point forward, the procedure is similar to the proof of Devroye et al. [1996, Theorem 14.5]. For the second part (11), the noisy Bayes classifier is still

We also use the shorthand f b := ˜ f ∗ to denote that the noisy Bayes classifier depends on b . Now the noisy Bayes risk is no longer zero. In fact

What's more,

where the inequality holds from ˜ R A κ ( ˜ f ∗ ) = ˜ R V ( ˜ f ∗ ) (because ˜ η is one-hot at point x 0 ) and P X ( A κ ( η , ˜ η ) ) ≥ P X ( { x 1 , . . . , x V } ) .

Notice that in order to ensure that our construction ( P X , η , η ) ∈ Π( ϵ, κ, V, L ) , by definition

Due to the upper bound of (12), it suffices to require that

and this ensures that ( P X , η , ˜ η ) ∈ Π( ϵ, κ, V, L ) upon recalling that ( P X , η , ˜ η ) ∈ Π( ϵ, κ ) , and that P X is supported on V +1 points.

For fixed b , plugging in the definition of η , the excess risk over region { x 1 , . . . , x V } becomes

It should be noted that since ( V -1) p ≤ 1 is required, and since c > 0 , we have L < 1 · 1 / 2 . This is the origin of our condition L < 1 / 2 in the statement of the theorem. Naturally, the statement can be adjusted to min( L, 1 / 2) instead. In any case, we are left with two nontrivial constraint on our parameters ( p, c ) : (13) and ( V -1) p ≤ 1 , along with the boundary consraints p ∈ [0 , 1] and c ∈ [0 , 1 / 2] .

̸

̸

where the inequality follows from the fact that we ignore the risk on point x V .

Using the probabilistic method, replace b with B ∼ Uniform { 1 , 2 } V -1 ,

Now, we need to calculate B | Z n , which can be calculated using Bayes rule because we have B ∼ Uniform { 1 , 2 } V -1 and also Z n | B ∼ P ⊗ n X ˜ Y .

To be specific, for any x ∈ { x 0 , x 1 , . . . , x V -1 } , assume point x t is observed k times in training sample Z n ,

̸

̸

where B t denotes the t -th element of vector B (that associates with x t ).

Next we compute P ( B t = 1 | Y t 1 = y 1 , . . . , Y t k = y k ) for y 1 , . . . , y k ∈ { 1 , 2 } . Denote the numbers of ones and twos by k 1 = |{ j ≤ k : y j = 1 }| and k 2 = |{ j ≤ k : y j = 2 }| . Using Bayes rule, we get

̸

After some calculation, following the proof of Devroye et al. [1996, Theorem 14.5], we get

where the supremum is over ( p, c ) ∈ [0 , 1] × [0 , 1 / 2] such that

Now, suppose n is so large that

and further that

We choose

By our choice of c and the first condition on n above, we can conclude that L ≤ (1 / 2 -c ) , and therefore,

meaning that both the constraints required on ( p, c ) are met by the above choice.

As a consequence of this choice of c, p , we observe that

Since c ≤ 1 / 8 further implies that 1 1 -2 c ≤ 4 3 , this implies that

Thus, instantiating the bound (14), we conclude that

Putting the two parts together

A.2.4 Proof of upper bound: Lemma 1

Lemma (Oracle Inequality under Feature-dependent Label Noise) For any ( P X , η , ˜ η ) and any classifier f , we have

Proof. For any κ ≥ 0 , the input space X can be divided into two regions: X \ A κ and A κ . For any f , its risk is

̸

Therefore, its excess risk is

Now examine the two terms separately, and

A.2.5 Proof of upper bound: Theorem 2

To set the stage for the rate of convergence proof, we first introduce the concept of shattering in the multiclass setting and the Natarajan dimension [Natarajan, 1989], which serves as a multiclass counterpart to the VC dimension [Vapnik and Chervonenkis, 1971].

̸

Definition 4 (Multiclass Shattering) Let H be a class of functions from X to Y = { 1 , 2 , . . . , K } . For any set containing n distinct elements C n = { x 1 , . . . , x n } ⊂ X , denote

and therefore |H C n | is the number of distinct vectors of length n that can be realized by functions in H .

The n th shatter coefficient is defined as

We say that a set C n is shattered by H if there exists f, g : C n → Y such that for every x ∈ C n , f ( x ) = g ( x ) , and

̸

If Y = { 1 , 2 } , this definition reduces to the binary notion of shattering which says all labeling of points can be realized by some function in the hypothesis class H , i.e., H C = { 1 , 2 } | C | . Note that multiclass shattering does not mean being able to realize all K possible labels for each point x ∈ C . Instead, multiclass shattering is more like 'embed the binary cube into multiclass', where every x ∈ C is allowed to pick from two of the K labels.

Definition 5 (Natarajan Dimension) The Natarajan dimension of H , denoted Ndim( H ), is the maximal size of a shattered set C ∈ X .

Theorem (Excess Risk Upper Bound of NI-ERM) Let ϵ ∈ [0 , 1] , κ ∈ (0 , + ∞ ) . Consider any ( P X , η , ˜ η ) ∈ Π( ϵ, κ ) , assume function class F has Natarajan dimension V , and the noisy Bayes classifier ˜ f ∗ belongs to F . Let ˆ f ∈ F be the ERM trained on Z n = { ( X i , Y i ) } n i =1 , then

Proof. Following directly from Lemma 1, with ( P X , η , η ) ∈ Π( ϵ, κ ) , we already have

Now replace f with NI-ERM ˆ f . To bound the expected excess risk we employ a multiclass VC-style inequality.

Lemma 2

The binary version of this lemma is Corollary 12.1 in Devroye et al. [1996]. We prove the multiclass version below in Section A.2.6.

Next, we bound the multiclass shattering coefficient with Natarajan dimension, using the following lemma, which can be viewed as a multiclass version of Sauer's lemma.

Lemma 3 (Natarajan [1989]) Let C and Y be two finite sets and let H be a set of functions from C to Y . Then

Letting V denote Ndim ( H ) , we have that S ( H , n ) ≤ n V K 2 V , and therefore Lemma 2 can be upper bounded by

Putting things together,

A.2.6 Proof of Lemma 2

Theorem 6 Consider any set of multiclass classifiers F . Let ( X 1 , Y 1 ) , . . . , ( X n , Y n ) be iid draws from P XY . For any n , and any ϵ > 0 ,

where the probability is with respect to the draw of the data.

Proof. Apply Theorem 12.5 from Devroye et al. [1996], with the following identifications. In what follows, the left-hand side of each equation is a notation from Devroye et al. [1996], and the right-hand side is our notation.

With these identifications, we have

By Theorem 12.5 we conclude

where s ( A , n ) (note the lowercase 's') is defined to be

where the max is over points z 1 , . . . , z n , and N A ( z 1 , . . . , z n ) is the number of distinct subsets of the form as f ranges over F .

To conclude the proof, it suffices to show that s ( A , n ) ≤ S ( F , n ) , where the latter expression is the multiclass shatter coefficient defined above. We show this as follows.

Consider fixed pairs z i = ( x i , y i ) , i = 1 , . . . , n . Supposed that there are N distinct subsets of the form A f ∩ { z 1 , . . . , z n } , and let f 1 , . . . , f N be the classifiers in F that realize these distinct subsets. Consider the map that sends f i to the vector of its values at x 1 , . . . , x n :

We will show that this map is injective, from which the claim follows. To see injectivity, consider classifiers f i and f j , where i = j . Since f i and f j yield different subsets, it means there is some pair ( x k , y k ) such that one of f i and f j classifies the pair correctly, while the other does not. This implies that f i ( x k ) = f j ( x k ) , and therefore

̸

̸

̸

This concludes the proof.

Now, Lemma 2 follows from the above theorem (stated in terms of the noisy data/distribution/risk) in precisely the same way that Corollary 12.1 in Devroye et al. [1996] follows from Theorem 12.6 in the same book.

A.2.7 Proof of upper bound: Theorem 3

Theorem (Excess Risk Upper Bound of NI-ERM under smooth relative margin condition) Let ϵ ∈ [0 , 1] , α > 0 , C α > 0 . Consider any ( P X , η , ˜ η ) ∈ Π ′ ( ϵ, α, C α ) , assume function class F has Natarajan dimension V , and the noisy Bayes classifier ˜ f ∗ belongs to F . Let ˆ f ∈ F be the ERM trained on Z n = { ( X i , Y i ) } n i =1 . Then

Proof. Again, using Lemma 1, and Theorem 2, we can conclude the following, where C is some large enough constant.

Now, by definition of Π ′ ( ϵ, α, C α ) , it holds that

Thus, we can further conclude that

The final statement now comes from optimizing the above bound, which is attained by taking the derivative w.r.t. κ and set to zero, we have

This yields the bound

A.2.8 Proof of immunity results: Theorem 4 and 5

Here, we state the immunity theorems in an equivalent but different way, so that the proofs are easier to follow.

Theorem (Immunity for one-hot vector) Denote B = { e 1 , . . . , e K } to be the set of one-hot vectors.

∀ ∈ B ⇐⇒ Diagonal elements of E ( x ) maximizes its row.

Proof. Let η ( x ) = e y for some y , then

To have

for any choice of y , it is equivalent to say that the diagonal elements of E ( x ) maximizes its row.

Theorem (Universal Immunity) Consider K -class classification,

Proof.

⇐ = : Plug E ( x ) into the expression

When e ( x ) ∈ [0 , 1 K ) , we have

= ⇒ : Denote T ( x ) := E ( x ) ⊤ , then has each column sum to 1 . Let us consider several choices of η ( x ) , which pose conditions on matrix T ( x ) .

To have arg max ˜ η ( x ) = arg max η ( x ) = { 1 , 2 , . . . , k } , all elements of ˜ η ( x ) must be equal, i.e., each row of T ( x ) should sum to the same value. The sum of all elements in T ( x ) is K , since all column sum to 1 . Therefore, each row of T ( x ) also sum to 1 .

To have arg max ˜ η ( x ) = arg max η ( x ) = { 1 , 2 , . . . , K -1 } , the first K -1 elements of ˜ η ( x ) must be equal (and larger than t KK ( x ) ), then we have t 1 K ( x ) = t 2 K ( x ) = · · · = t ( K -1) K ( x ) .

In other words, all elements of the K -th column of T ( x ) are the same (except for the ( K,K ) -th element). Similarly, consider η ( x ) to be a vector that contains 0 in the i -th position and 1 K -1 in other positions, then the general condition for T ( x ) is that: all elements of the i -th column are equal, except the i -th diagonal. Written explicitly,

1

Since each row and column of T ( x ) sum to 1 , we have

Subtracting the first equation from the second, we have t 12 ( x ) = t 21 ( x ) . Repeating for all pairs of equations, we have t 21 ( x ) = t 12 ( x ) = t 13 ( x ) = · · · = t 1 K ( x ) . What's more, all diagonal elements of T ( x ) will be equal. Thus,

3) The final step is to determine what value e ( x ) can take. Take η ( x ) = e y for some y , then from Theorem 4, we know that the diagonal elements of T ( x ) maximize their column, therefore

Finally, take any η ( x ) , the arg max is preserved by multiplying this specific choice of T ( x ) . This concludes the ⇐ = part.

A.3 Experimental details

A.3.1 2D Gaussian with synthetic label noise

For 2D Gaussian mixture data, we draw from two Gaussian centered at [1 1] ⊤ and [ -1 -1] ⊤ , with covariance matrix being identity, 200 data points from each, with label Y = 1 , 2 respectively. To generate noisy labels, we flip every label uniformly with some probability. We use Sklearn's logistic regression (with no ℓ 2 regularization). The experiment was conducted on AMD Ryzen 5 3600 CPU. The goal of the simulation is to experimentally verify noise immunity results in Section 6. Notice that different trial corresponds to different draw of both instances and noisy labels.

̸

̸

Table 2: Testing accuracy of logistic regression on gaussian mixture data with uniform label noise. 'Noise rate' refers to P ( ˜ Y = Y ) , the percentage of wrong labels in the training data. As theory in Section 6 predicts, when P ( ˜ Y = Y ) reach 50% , there is a sharp decrease in performance. Noise rate 0 0.05 0.1 0.15 0.2 Trial #1 93.00 92.83 92.38 92.08 91.78

A.3.2 MNIST with synthetic label noise

We flip the clean training label of MNIST ( http://yann.lecun.com/exdb/mnist/ ) uniformly (to any of the wrong classes). We use a shallow neural network with two convolution layers and two fully connected layers. We train with stochastic gradient descent with learning rate 0 . 01 for 10 epochs, batch size equals 64 . We use the same hyperparamters for all tests. The experiments were conducted on a single NVIDIA GTX 1660S GPU. The goal of the simulation is to experimentally verify noise immunity results in Section 6. Here randomness corresponds to different realization of noisy labels and stochastic gradient descent.

̸

̸

A.3.3 CIFAR with synthetic label noise

We flip the clean training label of CIFAR-10 ( https://www.cs.toronto.edu/~kriz/cifar. html ) uniformly (to any of the wrong classes). To have a fair comparison between different methods, we fix the realization of noisy labels. Follow the 2-step procedure described in Section 7, we use different pre-trained neural networks as feature extractor: forward-passing the training image through the network and record the feature. Then use sklearn's ( https://scikit-learn.org/stable/ ) logistic regression function to fit the ( feature, noisy label ) pair in a full batch manner. We prespecify a range of values for ℓ 2 regularization ( { 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 } ) and number of iterations for lbfgs optimizer ( { 10 , 20 , 50 , 100 } ), then do cross-validation on noisy data to pick the best hyper-parameters. We use the same range of hyper-parameters in all tests. The experiments were conducted on a single NVIDIA Tesla V100 GPU. The result is deterministic.

̸

̸

A.3.4 CIFAR with human label error

We load the noisy human labels provided by http://noisylabels.com/ , then follow exact the same procedure as above.

A.4 Additional experiments

A.4.1 Linear probing, then fine tuning (LP-FT)

We study whether 'linear probing, then fine tuning' (LP-FT) [Kumar et al., 2022] works better than linear probing (LP) only, in label noise learning scenario.

A.4.2 Robust learning strategy over DINOv2 feature

This section examines how different robust learning strategy works over DINOv2 feature, compared with only training with cross entropy.

A.4.3 Synthetic instance-dependent label noise

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims are either supported by theory statements or by reproducible experiment results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations about our practical method is described.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are stated in the theorem statement. Full proofs are included in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Important information about the experiments are in main text. Details on the experimental setup is described in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is provided, common benchmark datase were used, instructions are given, the result is easily reproducible.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See appendix and attached code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have done repeated experiments for data simulation, e.g., different iid draw. For the result on CIFAR-N data challenge, our result is deterministic and thus no error bar.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have read the NeurIPS Code of Ethics and confirm that this research follows the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theory-oriented paper towards better understanding of label noise problem.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Citations and urls are included.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve with this matter.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical framework for learning under multi-class, instance-dependent label noise. It introduces the novel concept of Relative Signal Strength (RSS) to measure the impact of noise and uses it to derive upper and lower bounds on excess risk. Notably, it proves the minimax optimality of Noise Ignorant Empirical Risk Minimization (NI-ERM) and provides conditions for immunity to label noise. Bridging theory and practice, the paper proposes a two-step 'feature extraction + NI-ERM' approach, achieving state-of-the-art performance on the CIFAR-N dataset.\n\nSTRENGTHS:\n1. Theoretical Depth: \nThis paper introduces the novel concept of Relative Signal Strength (RSS) to quantify the impact of noise in label learning. Using RSS, the authors mathematically derive precise upper and lower bounds for excess risk.\n\n2. Combination of Theory and Practicality: \nThe research seamlessly bridges theory and practice by developing a two-step 'feature extraction + NI-ERM' approach based on their theoretical analysis. This method achieves high performance using simple linear classifiers instead of complex deep learning models, validating the theoretical predictions about the optimality of NI-ERM in practical applications. This successful translation of theoretical insights into effective practical methods is a significant strength of the paper.\n\n3. Surprising Results:\nOne of the most striking aspects of this work is the mathematical proof that simple Noise Ignorant Empirical Risk Minimization (NI-ERM) is minimax optimal. The authors theoretically guarantee accurate learning even under high noise levels, such as up to 90% in 10-class problems. These surprising findings challenge conventional wisdom about the necessity of complex noise handling techniques and achieve state-of-the-art performance with a remarkably simple approach.\n\n4. Experimental Validation:\nThis paper provides experimental validation of its theoretical claims by achieving top performance on real-world noisy datasets like CIFAR-10N and CIFAR-100N. They demonstrate that performance changes with increasing noise levels align closely with their theoretical predictions, providing empirical support for their theoretical framework.\n\n5. Practicality:\nThe proposed method achieves high performance without relying on data augmentation or complex hyperparameter tuning\n\nWEAKNESSES:\n1. Insufficient Introduction:\nThe introduction lacks a comprehensive overview of the paper's content and contributions. A more detailed exposition of the overall approach and key findings would better prepare readers and emphasize the paper's significance. This could include a clearer roadmap of the theoretical and practical aspects of the work.\n\n2. Validity of Relative Signal Strength (RSS) Definition:\nThe definition of RSS is not intuitive and lacks a sufficient explanation of how it represents the signal content of the noisy distribution relative to the clean distribution. While examples are provided, they primarily demonstrate RSS calculation rather than explaining the fundamental reasoning behind its definition. Since all theoretical results in the paper are based on the RSS definition, it is crucial to establish whether RSS accurately represents the degree to which the noisy distribution contains signal compared to the clean distribution. A more thorough justification and explanation would strengthen the paper's theoretical foundation. I will raise a question about the definition of RSS in the question section to seek further clarification on this point.\n\n3. Experimental Validity Concerns:\nThe two-step method employs pre-trained feature extractors. While this demonstrates the effectiveness of NI-ERM, it may not provide a fair comparison with other methods. To more convincingly demonstrate the superiority of NI-ERM, the paper could:\na) Compare NI-ERM using feature extractors from models trained with other SOTA noisy label methods.\nb) Apply SOTA methods to the same high-quality feature extractors used for NI-ERM, training only the final linear layer for a fairer comparison.\nThese additional experiments would provide a more robust validation of the NI-ERM approach in practical scenarios.\n\nQUESTIONS:\n1. Definition of RSS: \nI am curious about how RSS measures certainty about the label. The definition of RSS does not seem intuitive. Generally, in deep learning, KL divergence is used to measure the difference between two probability distributions. The definition of RSS shows a different tendency compared to KL divergence. For example, let's consider three probability vectors: p1 = [0.05, 0.7, 0.25], p2 = [0.25, 0.7, 0.05], and p3 = [0.1, 0.6, 0.3]. The RSS between p1 and p2 is 1.44, and the RSS between p1 and p3 is 0.67. If we assume p1 is the clean distribution, it can be interpreted that p2 contains more label information. However, the KL divergence between p1 and p2 is 1.96, and between p1 and p3 is 1.05, indicating that p1 and p3 are closer in terms of distribution. Is there a justification for claiming that p2 contains more label information despite having a higher KL divergence?\n\n2. Gap between Theory and Practice: \nWhile the paper bridges the gap between theory and experiments by using pretrained feature extractors, as mentioned in line 254, generally training the feature extractor with NI-ERM does not perform well. This differs from the theory presented in the paper. What do you think is the reason for this discrepancy?\n\nLIMITATIONS:\nLimited Application to Classification Problems: \nWhile noisy label problems exist in various tasks, the theoretical analysis and experimental validation in this paper focus solely on classification problems, particularly image classification tasks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work provides a new insight on how to deal with instance-dependent label noise under the context of multi-class classification problem. Under certain conditions, they prove that training a classifier as if there is no noisy labels is the best course of action.\nThis idea is presented in details supported by several theorems with finite sample analysis as well as by several experiments on synthetic and a real dataset (CIFAR-10N).\n\nSTRENGTHS:\nI enjoy reading this paper. The work presents a surprisingly simple idea to deal with noisy labels, especially instance-dependent noise, and show that it is the optimal learning strategy under some conditions.\n- The authors introduce the Relative Signal Strength (RSS) to quantify how much noisy label distribution can reveal about the clean label distribution.\n- Based on this measure, the authors suggest a class of noisy label learning problems that can be guaranteed to be solvable (Theorem 2) by just ignoring the existence of noisy labels. This is somewhat still intuitive because RSS is positive and the excessive risk is measured based on 0-1 loss.\n- They also present a min-max analysis to show that for the same class of problems, the above learning strategy achieves an optimal rate. This is really a surprising and intriguing result.\n- Lastly, they show how applicable this class of problems can be in practice (Theorem 4, 5): the label noise probability vector should have the same argmax as the clean probability vector.\n\nWEAKNESSES:\n- The key condition for ignoring the existence of noisy labels to work is to have $\\mathcal{A}_0 = \\mathcal{X}$ (or $\\mathcal{A}_0$ covers most of $\\mathcal{X}$), which in turns requires $\\text{argmax } \\widetilde{\\eta}(x)  = \\text{argmax } \\eta(x)$. This condition would be violated if certain classes are tricky and easy to mistaken one as another, for example, labeling leopard, lion,  cheetah, tiger. Therefore, the condition that \\kappa > 0 might be restrictive in this regard.\nOn the other hand, it would be intuitively feasible to still learn a classifier under this type of noise as long as one mistakes a class with another class in a consistent way. In that case, having permutation ambiguity is inevitable, but it is not a very detrimental as the cost of post-processing to disambiguating permutation is relatively cheap.\n- The experiment section is limited:\n   + How is performance of the proposal compared to baselines under various noise level?\n   + More baselines should be included, such as BLTM[1], MEIDTM[2], MaxMIG[3].\n\n[1] Yang, Shuo, et al. \"Estimating instance-dependent bayes-label transition matrix using a deep neural network.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Cheng, De, et al. \"Instance-dependent label-noise learning with manifold-regularized transition matrix estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Cao, Peng, et al. \"Max-mig: an information theoretic approach for joint learning from crowds.\" arXiv preprint arXiv:1905.13436 (2019).\n\nQUESTIONS:\n- Does the analysis depend on any particular loss used during training?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes a Label Noise Learning (LNL) method that assumes a noise transition matrix. The author introduces the concept of Relative Signal Strength (RSS), which is calculated as the ratio of the signal difference between the true prediction and the prediction under label noise. The author demonstrates that the set satisfying the condition of RSS being greater than zero is noise immune. Additionally, the author provides a method to define a noise transition matrix that upholds this condition.\n\nSTRENGTHS:\nThe author presents a novel perspective for solving the LNL problem. The interpretations of RSS proposed by the author are sufficiently insightful for peers in the LNL field. Moreover, the author's method is well-suited for use with pretrained feature extractors and aligns with current trends in the AI field.\n\nWEAKNESSES:\nThere are two major concerns. The first is the low reproducibility. Despite the robust interpretation of the proposed method, the lack of a specific algorithm makes it challenging to conceptualize a clear learning approach. Specifically, it would be beneficial to provide examples of e(x) in Theorem 5, as well as pseudocode for feature extraction and the derivation of the transition matrix. Secondly, it is challenging to interpret the advantages of the proposed method from the experiments. (1) The author conducted experiments solely on light synthetic data, making it difficult to ascertain the effectiveness of the theoretically-based method in practical noise scenarios. (2) The author employs a highly trained pretrained model, which introduces an unfair factor in comparisons with other methods. Notably, there is experimental evidence suggesting that self-supervised pre-training can enhance the performance of existing methods (https://arxiv.org/pdf/2103.13646v2). I recommend including the author's method applied to various pretrained models in Table 1, or adding the use of pretrained models for existing methods to provide a fair comparison.\n\nQUESTIONS:\nI have given high marks to the author's novel perspective and the reasonable interpretations and proof methods presented. Despite the proposed method's low reproducibility and unproven practical performance, it has the potential to positively influence other research in the LNL field. Therefore, I have assigned a rating of \"weak accept\". If the author provides more detailed treatment of the proposed method and richer empirical interpretations, I believe the paper would be strong enough for acceptance.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors use a new theoretical framework for analyzing learning under label noise in multi-class classification. \nThe proposed framework is based on relative signal strength (RSS), which measures noisiness data points in the training sets.\nBased on RSS, the authors propose new upper and lower bounds on excess risk and identify when the classifier learned from label noise is consistent. Based on the theoretical results, a new simple learning framework, called Noise Ignorant Empirical Risk Minimisation (NI-ERM), is proposed, which basically performs standard ERM learning on nosy data. To practically apply NI-ERM, the authors proposed a simple framework of learning linear classifiers on top of a feature extractor trained in an unsupervised/semi-supervised way. The effectiveness of this approach is validated in a few experiments where the method was tested on popular benchmarks under different noise distributions.\n\nSTRENGTHS:\n- The paper is sound and, despite being very theoretical, is easy to read, as authors step by step explain the thinking process.\n- Newly obtained bounds are indeed simple in form.\n- The effectiveness of the new two-step NI-ERM is confirmed by a few empirical empirical experiments.\n\nWEAKNESSES:\n- It seems to me, that the analysis conducted under the Relative Signal Strength framework does not provide a lot of new surprising insights, the conclusions more or less confirm findings from previous works. \n- Training feature extractors in an unsupervised/semi-supervised way might be more difficult in the case of some more specialized applications than, for example, general image classification used in the experiments.\n- The considered analysis and proposed approach is limited to classifier accuracy, while in many applications other task losses/utilities are often considered.\n\nQUESTIONS:\nNot really, I will be happy to read the authors comments on my points from the weakness section.\n\nLIMITATIONS:\nLimitations were discussed. I see no negative social impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates multiclass classification under label noise, specifically instance-dependent label noise in which the noise can depend on the features as well (a.k.a. local noise). A minimax result is derived that lower-bounds the misclassification probability of a classifier. The paper finds a good empirical practice to be the procedure of training a linear classifier on some separately-learned features.\n\nSTRENGTHS:\nThe paper's analysis is correct, and the results relative to the CIFAR-N leaderboard are impressive. (I wish I was able to see the leaderboard, however; several browsers failed to load more than a blank screen.) The message of the paper -- in the presence of unknown noise, learning final-layer classifiers is more robust than learning a classifier from scratch -- is an old one, known empirically to be worth using. The paper's message is clear, and it is overall clearly written.\n\nWEAKNESSES:\nThe problem addressed in this paper is very common, and people use many heuristic practical strategies it. The paper states that there are theoretical contributions leading to practical consequences. These practical consequences (recommending ignoring the noise) are not new, and the theory lends no understanding on when this might succeed and fail. \n\nThe practical consequences are to encourage linear probing (learning a linear classifier on top of a frozen feature layer) on features learned separately from possibly different data. This is possibly the most commonly used idea in practice, including in many situations where it's assumed that there is no noise by default, even when there is. This is not benchmarked thoroughly on different noise distributions. Many other papers in similar areas use Dirichlet distributions to precisely simulate different non-uniform noise, which would be a more thorough synthetic benchmark (e.g. Garg et al. \"A Unified View of Label Shift Estimation\"). On the real-data side, I would like to see different fine-tuning strategies also compared.\n\nThe theory is quite brittle in a couple of ways. It only takes into account the gap to the top class probability, making some of the definitions like A_0 brittle as well; for many classes of noise distributions A_0 could be quite small to empty. The minimax lower bound construction again uses uniform label noise, meaning that it does not adapt to the actual structure of the instance-dependent label noise; the results showing NI-ERM matching this bound are therefore fairly weak. \n\nPresentation-wise, the paper introduces many new terms without apparent reason; these are used correctly, but their combined effect is not to simplify and may be misleading as to the results. An example is \"immunity\" - it is easy to construct class-conditional noise distributions which would break the minimax results, and they tend to be non-uniform in interesting ways.\n\nQUESTIONS:\n- Why not simulate with instance-dependent label noise of some kind (e.g. simulate cluster-dependent label noise) instead of uniform label noise? Uniform label noise is easy to combat - simply being oblivious works. \n\n- The main message of the paper is that linear probing deals with label noise better than full training / fine-tuning of the network. Are the authors aware of the two-step strategy of linear probing then fine-tuning the whole network, as popularized recently as a general strategy for encouraging robustness to distribution shift? How does this do? (ref. Kumar et al. '22, \"Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution\")\n\n- How about estimating label shift and benchmarking similar methods, which all implicitly rely on the adjustments to the calibration of a classifier? The Garg et al. reference given above, and methods therein, are SOTA here, and I would think that if there is significant class-conditional noise in \\tilde{Y}, then estimating this would be key and perform better than NI-ERM.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-23"}