Title: Noisy Label Learning with Instance-Dependent Outliers: Identifiability via Crowd Wisdom
Venue: NeurIPS 2024
Accepted: yes
Keywords: Noisy label

instance-dependent label noise

sample selection

end-to-end learning

identifiability

crowdsourcing
Abstract: The generation of label noise is often modeled as a process involving a probability transition matrix (also interpreted as the _annotator confusion matrix_) imposed onto the label distribution. Under this model, learning the ``ground-truth classifier''---i.e., the classifier that can be learned if no noise was present---and the confusion matrix boils down to a model identification problem. Prior works along this line demonstrated appealing empirical performance, yet identifiability of the model was mostly established by assuming an instance-invariant confusion matrix. Having an (occasionally) instance-dependent confusion matrix across data samples is apparently more realistic, but inevitably introduces outliers to the model. Our interest lies in confusion matrix-based noisy label learning with such outliers taken into consideration. We begin with pointing out that under the model of interest, using labels produced by only one annotator is fundamentally insufficient to detect the outliers or identify the ground-truth classifier. Then, we prove that by employing a crowdsourcing strategy involving multiple annotators, a carefully designed loss function can establish the desired model identifiability under reasonable conditions. Our development builds upon a link between the noisy label model and a column-corrupted matrix factorization mode---based on which we show that crowdsourced annotations distinguish nominal data and instance-dependent outliers using a low-dimensional subspace. Experiments show that our learning scheme substantially improves outlier detection and the classifier's testing accuracy.

Noisy Label Learning with Instance-Dependent Outliers: Identifiability via Crowd Wisdom

Abstract

The generation of label noise is often modeled as a process involving a probability transition matrix (also interpreted as the annotator confusion matrix ) imposed onto the label distribution. Under this model, learning the 'ground-truth classifier'i.e., the classifier that can be learned if no noise was present-and the confusion matrix boils down to a model identification problem. Prior works along this line demonstrated appealing empirical performance, yet identifiability of the model was mostly established by assuming an instance-invariant confusion matrix. Having an (occasionally) instance-dependent confusion matrix across data samples is apparently more realistic, but inevitably introduces outliers to the model. Our interest lies in confusion matrix-based noisy label learning with such outliers taken into consideration. We begin with pointing out that under the model of interest, using labels produced by only one annotator is fundamentally insufficient to detect the outliers or identify the ground-truth classifier. Then, we prove that by employing a crowdsourcing strategy involving multiple annotators, a carefully designed loss function can establish the desired model identifiability under reasonable conditions. Our development builds upon a link between the noisy label model and a columncorrupted matrix factorization mode-based on which we show that crowdsourced annotations distinguish nominal data and instance-dependent outliers using a lowdimensional subspace. Experiments show that our learning scheme substantially improves outlier detection and the classifier's testing accuracy.

1 Introduction

Deep neural networks can easily overfit to noisy labels due to its excessive expressiveness [1,2]. Many strategies have been proposed to avoid negative impacts of label noise when training neural classifiers; see, e.g., noisy label filtering [3-8], robust loss design [9-13], and noise generation modeling (or loss correction) [14-22]. In the last genre, a noisy class label is modeled as a realization of a categorical distribution, which is the ground-truth label distribution being distorted by a probability transition matrix . The transition matrix is interpreted as the 'confusion matrix' [9, 14, 18] that can effectively

model the annotators' expertise level and the difficulty of annotating each class/sample, and thus is considered intuitive. Under this model, learning the 'label noise-free' target neural classifier boils down to identifying the confusion matrix.

The confusion matrix-based models have proven quite useful in practice-algorithms developed in this line of work often exhibits appealing empirical performance; see, e.g., [9,14-17,19-21,23-31]. In addition, these models admit interesting statistical and algebraic structures, leading to plausible results on identifiability of the confusion matrix and/or the 'ground-truth classifier'-i.e., the classifier that can be learned if no noisy annotations were present. However, most of the aforementioned early works considered an instance-invariant confusion matrix-i.e., a confusion matrix is not affected by sample features, but only classes-for analytical and computational simplicity. Considering instancedependent confusion models is more realistic, as the sample characteristics, e.g., lightening and resolution of an image, affect the annotation accuracy [32]. The existence of such (at least occasionally occurred) instance-dependent noisy labels inevitably introduces outliers to the instance-invariant confusion models, leading to performance degradation. In general, learning under instance-dependent confusion matrices is heavily ill-posed. Hence, various problem-specific structures were exploited to add regularization terms and constraints; see, e.g., [27, 28, 33-43]. Nonetheless, unlike the instanceinvariant confusion matrix case, identifiability guarantees of the target classifier have been largely under-studied. The lack of theoretical understanding also affects algorithm design-many approaches in this domain had to resort to somewhat ad-hoc treatments with multi-stage training procedures, often involving nontrivial pre- and post-processing; see [27, 28, 33-35, 37, 39, 40].

Contributions. To advance understanding, we consider a model where instance-dependent confusion matrices occur occasionally across the samples, and the rest of data share a common nominal confusion matrix. This way, the instance-dependent noisy labels can be regarded as outliers. The model is motivated by the fact that only a proportion of all instances may have a labeling difficulty that significantly deviates from the general population [36, 44, 45]. Our contributions are as follows:

(i) Identifiability via Crowdsourcing. We first show that using the sparsity prior on outliers is insufficient to identify the model of interest, if only one annotator is present. To circumvent this challenge, we propose to employ a crowd of annotators-which is the common practice in data labeling [14, 18, 20, 31]. We show that, by incorporating a carefully designed column sparsity constraint to a coupled cross-entropy (CCE) loss from crowdsourcing [20, 31] to integrate the annotators' outputs, the outliers can be provably identified. Consequently, the ground-truth classifier can be learned with generalization guarantees.

(ii) End-to-end One-Stage Implementation. Our approach features a one-stage continuous optimization-based implementation. To be specific, our proposed learning loss allows to approximate the column-sparsity constraint using a smoothed nonconvex ℓ p (where 0 < p ≤ 1 ) function based regularization (see [46]). This way, the overall loss is differentiable and can be readily tackled by off-the-shelf optimizers. This is in contrast to many existing methods that involve multiple stages (e.g., [27, 28, 33-35, 37, 39, 40])-and is arguably easier to implement.

We evaluate the proposed method over a number real datasets that are annotated by machine and human annotators under various conditions and observed nontrivial improvements of testing accuracy.

Notation. The notations are summarized in the supplementary material in Sec. A.

2 Problem Statement

The Confusion Model and Learning Goal. Consider N data items { x n } N n =1 from K classes. Here, x n ∈ R D represents the feature vector of the n th data item. Let { y n } N n =1 be the set of ground-truth labels, where y n ∈ [ K ] . Assume that y n 's are unobserved. Instead, we observe the 'noisy' version { ̂ y n } N n =1 . The label ̂ y n ∈ [ K ] is noisy due to various reasons, e.g., the lack of expertise of the annotator. In this setting, our goal is to learn a performance-guaranteed classifier using the data items { x n } N n =1 , and noisy labels { ̂ y n } N n =1 . We consider the following expression of Pr ( ̂ y n = k | x n ) :

Note that Pr ( y n = k | x n ) is the ground-truth label distribution given the sample x n . This is also the distribution that the target classifier wishes to learn from. We represent the ground-truth classifier

using a function f ♮ : R D → R K such that [ f ♮ ( x n )] k ≜ Pr ( y n = k | x n ) . We call f ♮ the ground-truth classifier as it is the function that we aim to learn-and it can be learned under ideal conditions (e.g., when N = ∞ and the learner is a universal approximator), given that no noise is present. We define T ♮ : R D → R K × K such that [ T ♮ ( x n )] k,k ′ ≜ Pr ( ̂ y n = k | y n = k ′ , x n ) , ∀ x n . That is, T ♮ ( x n ) is the label transition matrix or confusion matrix of sample x n . Let [ g ♮ n ] k ≜ Pr ( ̂ y n = k | x n ) . Accordingly, the noisy label generation process for x n i . i . d . ∼ D x is modeled as follows:

where categorical ( g ) denotes the K -dimensional categorical distribution. Per the physical meaning of T ♮ ( x n ) and f ♮ ( x n ) , we have 1 ⊤ T ♮ ( x n ) = 1 ⊤ , T ♮ ( x n ) ≥ 0 and 1 ⊤ f ♮ ( x n ) = 1 , f ♮ ( x n ) ≥ 0 , for all n . Under this model, the main goal is to learn f ♮ from { x n } N n =1 and { ̂ y n } N n =1 .

Identifiability under Instance-Invariant T ♮ ( x ) . From (2a), it becomes apparent that learning f ♮ is not a straightforward task. Even if g ♮ n is observed (which is not), it is hard to identify f ♮ ( x n ) or T ♮ ( x n ) from the product g ♮ n = T ♮ ( x n ) f ♮ ( x n ) . To tackle the identifiability challenge, a popular approach is to simplify (2) by assuming that all instances have the same confusion matrix, i.e., T ♮ ( x n ) = A ♮ with a certain A ♮ ∈ R K × K for all n [16, 18, 19, 21, 47, 48]. Under this assumption, many used the 'loss correction' based formulation, e.g., [9,16,19,21,48]. The loss correction idea modifies the training loss of the classifier by taking the confusion matrix into consideration, e.g.,

where the objective is a modified cross-entropy (CE) loss. The 'noise correction' term A and f are used to learn A ♮ and f ♮ , respectively, ̂ y n ∈ { 0 , 1 } K denotes the one-hot encoding of the noisy label ̂ y n , and A and F denote appropriate constraint sets-i.e., A = { A ∈ R K × K | 1 ⊤ A = 1 ⊤ , A ≥ 0 } and F is a certain deep neural network function class whose outputs reside in the probability simplex. Note that the objective is often used together with regularization and additional constraints of A for various purposes; see, e.g., [19, 21, 30]. When N → ∞ , the CE term enforces the learned A and f to satisfy g ♮ n = Af ( x n ) for all n [19]. Note that ˜ G = [ g ♮ 1 , . . . , g ♮ N ] can be expressed as ˜ G = A ♮ [ f ♮ ( x 1 ) , . . . , f ♮ ( x N ) ] = A ♮ F ♮ , where F ♮ = [ f ♮ ( x 1 ) , . . . , f ♮ ( x N )] . Consequently, Eq. (3) can be understood as learning A and F such that ˜ G ≈ AF . As both A ♮ and F ♮ are nonnegative matrices (due to their physical meaning), the identifiability of F ♮ can be connected to uniqueness of the nonnegative matrix factorization (NMF) model ˜ G = A ♮ F ♮ ; see [19, 21, 45].

These are interesting developments, yet the key assumption T ♮ ( x n ) = A ♮ for all n appears to be overly stringent. As mentioned, it makes sense to believe that at least a proportion of samples have instance-dependent T ♮ ( x n ) 's [36, 44, 45, 49]. Not considering such samples may cause performance degradation.

Instance-Dependent Confusion-Induced Outliers. Whenthe instance-dependent confusion happens sparingly instead of overwhelmingly, we can re-express T ♮ ( x n ) using the following decomposition:

where A ♮ ∈ R K × K represents an instance-independent (class-dependent) confusion matrix-which is the nominal confusion matrix that reflects the general annotation difficulty of the dataset. The term E ♮ ( x n ) represents the instance-dependent 'perturbation'. For many n 's, E ♮ ( x n ) = 0 . When E ♮ ( x n ) = 0 , we have ( A ♮ + E ♮ ( x n )) f ♮ ( x n ) = A ♮ f ♮ ( x n ) + e ♮ n , where e ♮ n = E ♮ ( x n ) f ♮ ( x n ) . Using (4), the model in (2a) can be expressed as follows:

̸

where I and I c represent the instance index set where E ♮ ( x n ) = 0 and its complement, respectively. In other words, I is the outlier index set.

A remark is that from this point on we will ignore the structure e ♮ n = E ♮ ( x n ) f ♮ ( x n ) of the outliers. Disregarding the structure comes with some losses. For example, this way, our method is not able

̸

to learn E ♮ ( x n ) that could be of interest. Nonetheless, considering E ♮ ( x n ) renders extra modeling and computational burdens. Our treatment simplifies the subsequent analytical and computational developments, particularly, algorithm design. In addition, other types of outliers and anomalies can also be handled by the proposed approach due to the unstructured treatment.

Challenge of Outlier Detection. Under the model in (5), the natural idea is to first identify I and remove the impacts of outliers. Our first attempt is to explicitly model e ♮ n and modify (3) as follows:

where E = { e ∈ R K | 1 ⊤ e = 0 } is the constraint set for e n 's (due to the probability simplex constraints on the model parameters in (5)) and C is a scalar that is an estimation of |I| . The idea is to use the prior knowledge that e n does not occur overwhelmingly as constraint. The hope is that the solution of (6) can detect I , thereby enabling accurate estimation of A ♮ and f ♮ . However, the following fact reveals a conflicting insight:

Fact 2.1. Assume that all f ∈ F are universal function approximators, that rank ( A ♮ ) = K , and that N →∞ . Suppose that I ̸ = ∅ . Optimal solutions of Problem (6) can attain the minimal value of the objective function and satisfy the sparsity constraint without detecting any outliers. One such trivial solution ( A ⋆ , f ⋆ , e ⋆ n ) is A ⋆ = I , f ⋆ ( x ) = A ♮ f ♮ ( x n ) + e ♮ n , and e ⋆ n = 0 for all n .

This fact is easy to show (see Appendix F), yet it highlights a somewhat unexpected issue in confusion matrix-based noisy label learning: If only one nominal confusion matrix A ♮ is present (i.e., only one annotator is employed), it does not suffice to recover the ground-truth f ♮ when there exists instancedependent outliers (or any other types of outliers) under the model in (5)-under the condition that f is a universal function approximator. The practical implication is undesirable: as deep neural networks are powerful function approximators and are usually very expressive, Fact 2.1 means that the learned neural networks easily overfit to outliers, even if the sparsity prior on outliers is explicitly used in the loss function.

3 Proposed Approach

Intuition: Exploiting Crowd-induced Subspace. To get around the issues illustrated in Fact 2.1, our idea is to 'create' a low-dimensional subspace where the nominal data reside while the outliers are likely far away from. To this end, we propose a crowdsourcing approach. Consider the scenario where M annotators label the dataset { x n } N n =1 . The noisy label provided by an annotator m for the data item x n is denoted as ̂ y ( m ) n ∈ [ K ] and follows the generation model as follows:

where A ♮ m is the annotator m 's confusion matrix and e ♮ ( m ) n is the outlier term induced by annotator m 's instance-dependent confusion.

Putting together, we have the following expression:

where F ♮ is defined as before, and

̸

Denote I ( m ) as the index set where e ♮ ( m ) n = 0 . Then I = I (1) ∪ . . . ∪ I ( M ) stands for the nonzero column support of E ♮ .

From (7), one can see that now the 'data columns', i.e., the columns of the matrix G ♮ , live in a highdimensional space, i.e., R MK . However, the nominal data part W ♮ F ♮ resides in a K -dimensional subspace range ( W ♮ ) where K ≪ MK , if multiple annotators are employed. More importantly, e ♮ j for j ∈ I is an MK -dimensional vector and is unlikely to be inside range( W ♮ ) . With this geometry, it is much more likely that one can separate the outliers from the nominal data.

Next, we will use the above intuition to build a learning loss. We will take into consideration of practical aspects, e.g., missing annotations and finite dataset size N .

Proposed Identification Criterion & Analysis. In this section, we connect our proposed idea to a practically convenient, end-to-end identification criterion and provide identifiability guarantees for the desired model parameters. We consider the following empirical risk minimization under the crowdsourcing model in (6):

where S ⊆ [ M ] × [ N ] denotes the set of observed noisy labels indexed by ( m,n ) with S = |S| (note that all data items may not be labeled by an annotator), ̂ y ( m ) n is the one-hot encoding of the annotatorprovided label ̂ y ( m ) n , and C is an estimate of |I| . Here, the constraint sets F , A , and E are as defined before. If e ( m ) n is not considered, the objective function (9a) is sometimes referred to as coupled cross-entropy minimization (CCEM) in the end-to-end crowdsourcing literature [20, 30, 31, 50]. CCEM has not been used together with the outlier detection constraint (9b)-and the theoretical characterizations of the constrained formulation is unknown.

̸

We will use the following notations in our analysis. We use I ⊆ [ N ] to denote index set of the outliers, i.e., I = { n | e ♮ n = 0 } . We also consider that the function class in our learning problem, i.e., F , has a complexity measure R F . In particular, we adopt the so-called spectral-complexity upper bound of the function class F [51]; see Lemma C.4 in the Appendix.

We first establish that the criterion (9) recovers the ground-truth g ♮ ( m ) n 's with a reasonable accuracy. Specifically, we hope to characterize the average estimation accuracy of ̂ g ( m ) n 's where ̂ g ( m ) n = ̂ A m ̂ f ( x n ) + ̂ e ( m ) n and ̂ A m 's, ̂ e ( m ) n 's and ̂ f are estimated via solving (9):

Lemma 3.1. Assume that the observed index pairs ( m,n ) ∈ S are sampled uniformly at random with replacement. Also assume f ♮ ∈ F , |I| ≤ C ≤ N/ 2 , and that each [ A m f ( x n ) + e ( m ) n ] k , ∀ A m ∈ A , ∀ e ( m ) n ∈ E , ∀ f ∈ F are lower bounded by (1 /β ) for a certain β > 0 . Then, with probability greater than 1 -δ , the following holds:

where R 2 S = K 2 log ( K ) + K ∥ X ∥ 2 F R F / S + CMK log √ NM and X = [ x n 1 , . . . , x n S ] are the annotated samples, in which ( m s , n s ) ∈ S for s = 1 , . . . , S .

The proof is in Appendix C, which uses a similar idea as [31, Proposition 1] but takes into consideration of the outliers. Lemma 3.1 reveals that the criterion (9) essentially recovers the complete matrix G ♮ [cf. Eq, (7)] from the noisy and incomplete observations of its entries, when S / C is sufficiently large. To proceed, we will need a suite of assumptions and definitions:

Assumption 3.2. The outliers satisfy e ♮ i / ∈ range ( W ♮ ) , ∀ i ∈ I , where rank ( W ♮ ) = K .

Assumption 3.2 presents the key condition to disassociate instance-dependent and instanceindependent confusions. As discussed after Eq. (8), having a larger M would decrease the possibility that any e ♮ n belongs to range ( W ♮ ) . We also characterize the 'quantity' that e ♮ n is perturbed from range ( W ♮ ) in the following definition:

outlier samples

clean samples

̸

Definition 3.3. The outlier impact score κ ( e ♮ i ) of data i ∈ I is defined as

A larger score κ ( e ♮ i ) implies it is easier for our criterion to distinguish the outliers from the nominal data. Fig. 1 illustrates the geometry that we rely on to detect outliers. One can notice that when M = 1 , we always have κ ( e ♮ i ) = 0 , as the outlier satisfies e ♮ i ∈ range ( W ♮ ) .

In addition, we consider the following assumptions to assist identifying A ♮ and f ♮ :

Assumption 3.4 ( Class Specialists and Anchor Points). Assume that the following conditions hold:

There exists a near-class specialist for each class k , i.e., ∀ k ∈ [ K ] , ∃ m k ∈ [ M ] such that ∥ A ♮ m k ( k, :) -e ⊤ k ∥ ≤ ξ 1 , where e k is a unit vector.

There exists a near-anchor point sample for each class k in the dataset, i.e., ∀ k ∈ [ K ] , ∃ n k ∈ I c such that ∥ f ♮ ( x n k ) -e ⊤ k ∥ ≤ ξ 2 , where e k is a unit vector.

Assumption 3.4.a. is sometimes used in the crowdsourcing literature (e.g., [52]) to characterize the expertise of annotators. Assumption 3.4.b. is often seen in loss correction based noisy label learning; see, e.g., [16, 21, 27]. Under Assumptions 3.2 and 3.4, we have the following result:

̸

Theorem 3.5 ( Identifiability and Generalization). Let ( { ̂ A m } , { ̂ e ( m ) n } , ̂ f ) be any optimal solution of (9) with ̂ I = { n ∈ [ N ] | ̂ e n = [( ̂ e (1) n ) ⊤ , . . . , ( ̂ e ( M ) n ) ⊤ ] ⊤ = 0 } . Suppose that the conditions in Lemma 3.1 holds, that we set C = |I| , and that Assumption 3.4 holds with ξ 1 , ξ 2 ≤ 1 /K . Denote σ as the upper bound σ max ( A ♮ m ) ≤ σ, ∀ m . In addition, assume that ∀L ⊂ I c , |L| = N -2 |I c | , rank ( F ♮ (: , L )) = K . Then, for some α > 0 , and S > S 0 where S 0 as the smallest integer such that κ ( e ♮ i ) ≥ ϵ g ( S 0 ) , ∀ i ∈ I , the following result holds with probability greater than 1 -2 /S -K/T α :

where η 2 = O ( βMT α / √ S ( √ M log S +( ∥ X ∥ R F ) 0 . 25 )) , Π a permutation matrix, X is defined as before, and T = N -|I| . In addition, we have exact outlier detection, i.e., ̂ I = I .

The proof is in Appendix D. We should mention that we set C = |I| for notation simplicity. With a notation-wise slightly more complicated definition of L , the same proof holds under C ≥ |I| , which leads to T = N -C and I ⊆ ̂ I . That is, over-estimated C still enables identifying I with the price of discarding some nominal samples. While the criterion (9) offers the desired identifiability, it requires the presence of class-specialist annotators and the anchor data points (c.f. Assumption 3.4), which are considered relatively restrictive [19, 31]. In [19], the CE loss was combined with a simplex volume minimization-based regularization to establish identifiability of f ♮ under more relaxed conditions, namely, the sufficiently scattered condition (SSC) from the NMF literature [53]. We show that this is also viable in the presence of outliers:

Theorem 3.6 ( Enhanced Identifiability). Let ( { ̂ A m } , { ̂ e ( m ) n } , ̂ f ) be any optimal solution of (9) with ̂ W ≜ [ ̂ A ⊤ 1 , . . . , ̂ A ⊤ M ] ⊤ admitting the minimum volume of conv { ̂ W } (i.e., the simplex spanned by its columns) over all possible optimal solutions. Assume that we set C = |I| and that F ♮ (: , I c ) satisfies the SSC. Under Assumptions 3.2 and the same conditions used in Lemma 3.1, the following result holds with probability greater than 1 -δ , when S grows to infinity:

In addition, we have ∥ ̂ A m -A ♮ m Π ∥ F = 0 , ∀ m and ̂ I = I .

The proof is relegated to Appendix E, which also holds for C ≥ |I| with slight modifications as discussed before. Theorem 3.6 shows that the target classifier can be accurately estimated in the presence of instance-dependent outliers, without needing anchor samples or class specialists.

Implementation. Our learning losses allows relatively easy implementation and optimization. We consider the following regularized criterion:

Note that we use L outlier = ∑ N n =1 ( ∑ M m =1 ∥ e ( m ) n ∥ 2 2 + ζ ) p 2 where ζ > 0 and 0 < p ≤ 1 to approximate the column-sparsity constraint. The ℓ 2 /ℓ p nonconvex mixed quasi-norm has proven to be a very effective approximator for column sparsity [54]. In addition, as the function is differentiable, it offers an optimization-friendly surrogate.

For the minimum-volume loss, we use L vol = -log det( FF ⊤ ) , where F = [ f ( x 1 ) , . . . , f ( x N )] . Note that the term encourages maximizing the volume of conv { F } , which in turn minimizes the volume of conv { W } under the model ˜ G = WF . Therefore, the volume of conv { W } can be minimized via using either log det( W ⊤ W ) or -log det( FF ⊤ ) as the regularizer. The reason why we choose the latter is because minimizing log det( W ⊤ W ) is ill-defined-the term encourages a rank-deficient W instead of a small-volume full-rank W . The remedy in the literature is to use log det( W ⊤ W + γ I ) with γ > 0 [54] or adding structural constraints (e.g., diagonal dominance) to W [19], but these require more parameter tuning and more prior knowledge.

The constraints on A m 's can be handled by adding softmax to the columns. The function class F can be a designated deep neural network model (e.g., ResNet), where the outputs are also constrained by softmax . The constraint on e ( m ) n can be handled by parameterizing each e m n = ˜ e ( m ) n -( 1 ⊤ ˜ e ( m ) n ) 1 /K , where ˜ e ( m ) n is trainable weight vector. More details of the implementation are provided in Appendix G.1. We name our approach as Crowdsourcing-based Outlier-robust criterion and INstancedependence-aware deep neural Network learning , abbreviated as COINNet .

4 Related Works

Our development is related to works in transition matrix-based [14-19,21,23,27-29,33,47,55,56] and sample selection-based [28, 33, 35, 37, 39] noisy label learning, confusion identification in crowdsourcing [14,18,57], end-to-end crowdsourcing [20,30,31,49] (with some paying particular attention to incomplete annotations [58, 59]), and identifiability of NMF [53, 60]-see detailed discussions of the related work in Appendix B.

A particular connection to highlight is the usage of multiple annotators as a key enabler of our identifiability guarantees. This result (implicitly) aligns with some recently proposed approaches that advocate for the use of multiple labels in learning under label noise [45, 49]. Specifically, the work by [45] established the identifiability of instance-independent label noise using the consensus of at least three similar data items with the same true label. Similarly, the work by [49] leveraged Kruskal's identifiability-based arguments to recommend using at least three noisy labels to establish the identifiability of the instance-dependent noise transition matrix. These works, particularly [49], were more concerned with the theoretical limitations other than realizable methods. They did not connect the existence of repeated noisy labels to crowdsourcing, but used clusterability properties of datasets to support their arguments. In this work, we advocate diversity in crowd wisdom -labels from multiple annotators-based on an intuitive geometric characteristic of the model, which leads to clearly realizable implementations.

5 Experiments

Baselines. The proposed method is compared with a number of existing baselines in noisy label learning. We choose several end-to-end crowdsourcing methods, namely, GeoCrowdNet(F)&(W) [31], TraceReg [30], CrowdLayer [20], and Max-MIG [61]. We also employ four different instancedependent noisy learning approach, namely, MEIDTM [36], PTD [27], and BLTM [33]-which all use a single annotator. In addition, we present the result of VolMinNet [19] that uses the instanceindependent confusion model and the volume minimization regularization. We also use two noiserobust loss function-based approaches, namely GCE [10] and Reweight [21]. For baselines that were developed for single annotator cases, we train them using the labels after majority voting. Note that all the confusion matrix-based noise correction methods (including ours) inherently have a permutation ambiguity (cf. the Π term in Theorems 3.5 and 3.6). Hence, we report the highest classification accuracy attained among all possible permutation matrices for every method. In practice, Π can be removed by additional annotator inspection on several anchor samples, but we skipped these steps to keep the evaluation simple.

5.1 Experiments with Machine Annotations

Dataset. We consider the CIFAR-10 [62] and the STL-10 datasets [63]-see Appendix G for details.

Experiment Setup. To generate multiple noisy labels, we use M = 3 machine annotators for each dataset. In order to simulate a wide range of annotation behaviors, we employ different classification and clustering methods such as k -nearest neighbors (kNN), logistic regression, and convolutional neural networks. Each machine annotator is trained by randomly choosing a subset of the training data. We control the labeling accuracies of these machine annotators by varying the size of the dataset and the number of epochs during their training phase. This results in annotators with different labeling qualities, with their average individual noise rates around 20% ( good ), 40% ( medium ), and 70% ( low ). This way, we set up the following three cases: (i) High Noise: Three machine annotators are with low, medium, and good quality, respectively. (i) Medium Noise: Two machine annotators are with medium quality and one machine annotator with good quality. (i) Low Noise: One machine annotators is with medium quality and two machine annotators with good quality. More details are in Appendix G.2.

Neural Network and Optimizer Settings. We use ResNet-34 and ResNet-9 architectures [64] as the backbone to run all methods on CIFAR-10 and STL-10 datasets, respectively. For our proposed approach COINNet , we fix ζ = 10 -10 , p = 0 . 4 , and µ 1 = µ 2 = 0 . 01 . Adam [65] is used as the optimizer with weight decay of 10 -4 , learning rate of 0.01, and batch size of 512.

More details and ablation studies with the hyperparameters are provided in Appendix G.

Results. Table 1 presents the average testing accuracy of the methods. One can see that our approach COINNet consistently outperforms all baselines in different scenarios under test. Notably, the performance gap between COINNet and the second-best method is more significant in the high noise regime. For example, in the CIFAR-10 high noise scenario, COINNet shows around 4% improvement over the second-best performing baseline GeoCrowdNet(F) . Another key observation is that the instance-dependent modeling based baselines such as MEIDTM , PTD , and BLTM , exhibit much degraded performance in high-noise scenarios, possibly due to their multi-stage procedures that accumulate errors easily under such circumstances. In addition, our approach outperforms the instance-independent confusion-based baselines, such as VolMinNet , Reweight and GeoCrowdNet(W) , by nontrivial margins, showing that modeling instance-dependent outliers is indeed beneficial.

5.2 Experiments Using Real Annotations

In this section, we use three datasets to evaluate the algorithms.

CIFAR-10N. The first dataset that we use is the CIFAR-10N dataset [66]. The data has N = 60 , 000 samples from K = 10 classes and the samples were labeled by M = 3 anonymous real annotators. The error rates of the 3 annotators are 17.23%, 18.12%, and 17.64%, respectively.

LabelMe. We also test the algorithms over the LabelMe dataset [67,68]. The dataset has N = 2 , 688 samples from K = 8 classes, and M = 59 anonymous real annotators were involved in labeling the data. The average error rate is 25.95%.

ImageNet-15N. In addition to existing datasets, we also acquire noisy annotations by asking AMT workers to annotate some images from ImageNet. We select K = 15 classes and submit randomly chosen images to AMT for labeling. Eventually we collect annotations for N = 2 , 514 images from M = 100 anonymous real annotators, which serve as our training set. The average error rate of the annotators is 42.68%. The validation and testing sets have 1,462 and 13,157 images, respectively. We release the code and our acquired noisy annotations at https://github.com/ductri/COINNet .

Settings. For CIFAR-10N, we employ the ResNet-34 architecture to serve as f . For the LabelMe and ImageNet-15N datasets, we employ similar settings as in [20]. Specifically, as the training sets are small, the pretrained VGG-16 [69] and CLIP [70] models are used to first extract image embeddings for the experiments on LabelMe and ImageNet-15N, respectively. The embeddings are then fed to f , which is a fully connected neural network with one hidden layer and 128 hidden ReLU units. The same encoders and architecture choices are employed for the all methods under test. For our approach COINNet , we set µ 1 = µ 2 = 0 . 1 for LabelMe, and µ 1 = µ 2 = 0 . 01 for CIFAR-10N and ImageNet-15N.

Results. Table 2 presents the average testing accuracy of different methods on the three datasets. The proposed approach, COINNet , shows promising results in all cases, clearly outperforming the baselines by a noticeable edge. This is consistent with the machine annotator experiments.

Fig. 2 demonstrates the outlier identification results using the CIFAR-10N dataset. Here, we define the outlier indicator as s n = ∑ M m =1 ∥ ̂ e ( m ) n ∥ 2 , ∀ n , where ̂ e ( m ) n are the learned instance-dependent perturbations in the model (5). The histogram plot of these scores in Fig. 2 clearly shows that our method does output two types of samples, i.e., nominal samples and outliers, based on these perturbations. In the figure, the images on the middle and the right are examples from the low and high outlier indicator value regimes, respectively. The images with high s n values show more instance-dependent confusion characteristics (such as background noise and blurring) compared to

Sn=0.005

Sn=0.005

Sn=0.004

Sn =0.005

Sn =0.004

Sn = 0.176

Sn = 0.176

Sn = 0.177

Sn = 0.176

Sn =0.177

Sn=0.005

Sn=0.005

Sn =0.005

Sn=0.005

Sn=0.005

Sn=0.139

Sn =0.178

Sn=0.176

Sn=0.177 Sn=0.176

s

n

= 0

.

24

true label: penguin

s

n

= 0

.

26

true label: container ship

s

n

= 0

.

36

true label: airliner

s

n

= 0

.

35

true label: container ship

s

n

= 0

.

33

true label: dog

s

n

= 0

.

34

true label: tiger cat

s

n

= 1

.

55

true label: freight car

s

n

= 1

.

77

true label: trailer truck

s

n

= 1

.

71

true label: soccer ball

s

n

= 1

.

59

true label: penguin

s

n

= 1

.

57

true label: orange

s

n

= 1

.

81

true label: lemon

K&NA

those in the middle. Overall, these results indicate the effectiveness of our outlier-based model in real-world settings.

Fig. 3 shows some examples selected from highs n and lows n regimes from the results output by COINNet in the ImageNet-15N experiment. Similar as before, the images in the first row that have lower s n scores are visually much easier to recognize. The images in the second row that have about 5 times higher s n scores are apparently more visually confusing.

More experiments can be found in Appendix H.

6 Conclusion

In this work, we considered the noisy label learning problem under a confusion matrix-based model. We developed theory and algorithms in the presence of instance-dependent outliers. Our study revealed that relying solely on single-annotator labels is insufficient for effective outlier detection under the model of interest. We further demonstrated that a crowdsourcing approach, leveraging multiple annotators and a sparsity-constrained loss function, can successfully detect outliers and identify the desired, label noise-free learning system under reasonable conditions. Our analyses and design feature a one-stage differentiable training loss that is optimization-friendly. Empirical results underscore the plausibility of our model and the effectiveness of our proposed method, showing noticeable performance improvements over existing baselines.

Limitations. Our work explores a noise transition matrix-based model and treats instance-dependent noisy labels as outliers. The key assumption is that the outliers do not occur overwhelmingly. The assumption is useful, but also could be debatable. In principle, all labels could be generated in an instance-dependent manner. More sophisticated models are needed to deal with this more general case. In addition, our treatment is outlier structure-agnostic. The upshot is that this treatment can also help exclude the negative impacts of other types of outliers. However, the downside is that the structural prior knowledge of the outliers is not fully exploited. If the outlier-generating function could be learned, it can help detect 'difficult samples' before sending for annotation, which would potentially save resources and reduce annotation errors. We leave this direction for future work.

Acknowledgment

The work was supported in part by the National Science Foundation (NSF) under Project IIS-2007836.

References

Devansh Arpit, Stanisław Jastrzundefinedbski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of International Conference on Machine Learning , page 233-242, 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In Proceedings of International Conference on Learning Representations , 2016.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In Proceedings of International Conference on Machine Learning , volume 80, pages 2304-2313, 2018.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Proceedings of International Conference on Machine Learning , volume 80, pages 4334-4343, 2018.

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In Proceedings of International Conference on Machine Learning , pages 7164-7173, 2019.

Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In Advances in Neural Information Processing Systems , volume 30, 2017.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems , page 8536-8546, 2018.

Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin-Yau Kwok. Searching to exploit memorization effect in learning with noisy labels. In Proceedings of International Conference on Machine Learning , volume 119, pages 10789-10798, 2020.

Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in Neural Information Processing Systems , volume 26, 2013.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems , volume 31, 2018.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In IEEE/CVF International Conference on Computer Vision (ICCV) , pages 322-330, 2019.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Curriculum loss: Robust learning and generalization against label corruption. In Proceedings of International Conference on Learning Representations , 2019.

Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L DMI : A novel information-theoretic loss function for training deep nets robust to label noise. In Advances in Neural Information Processing Systems , volume 32, 2019.

Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics , pages 20-28, 1979.

Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping, 2015.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1944-1952, 2017.

Jacob Whitehill, Ting fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems , volume 22, pages 2035-2043. 2009.

Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I. Jordan. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. Journal of Machine Learning Research , 17(102):1-44, 2016.

Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In Proceedings of International Conference on Machine Learning , pages 6403-6413, 2021.

Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), 2018.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38:447-461, 2016.

Tri Nguyen, Shahana Ibrahim, and Xiao Fu. Deep clustering with incomplete noisy pairwise annotations: A geometric regularization approach. In International Conference on Machine Learning , pages 25980-26007. PMLR, 2023.

Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 6226-6236, 2020.

Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 12501-12512, 2021.

Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. In Advances in Neural Information Processing Systems , volume 31, 2018.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32, 2019.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instancedependent label noise. In Advances in Neural Information Processing Systems , volume 33, pages 7597-7610, 2020.

Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In Proceedings of International Conference on Machine Learning , volume 119, pages 1789-1799, 2020.

David Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems , volume 24, pages 1953-1961, 2011a.

Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C. Alexander, and Nathan Silberman. Learning from noisy labels by regularized estimation of annotator confusion. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1123611245, 2019.

Shahana Ibrahim, Tri Nguyen, and Xiao Fu. Deep learning from crowdsourced labels: Coupled cross-entropy minimization, identifiability, and regularization. In Proceedings of International Conference on Learning Representations , 2023.

Kristen Grauman and Bastian Leibe. Visual Object Recognition . Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2011.

Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent Bayes-label transition matrix using a deep neural network. In Proceedings of International Conference on Machine Learning , 2021.

Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instancedependent label-noise learning under a structural causal model, 2021.

Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instancedependent label noise, 2021.

D. Cheng, T. Liu, Y. Ning, N. Wang, B. Han, G. Niu, X. Gao, and M. Sugiyama. Instancedependent label-noise learning with manifold-regularized transition matrix estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1660916618, jun 2022.

Shikun Li, Xiaobo Xia, Jiankang Deng, Shiming Ge, and Tongliang Liu. Transferring annotator- and instance-dependent transition matrix for learning from crowds. arXiv preprint arXiv:2306.03116 , 2023.

Qizhou Wang, Bo Han, Tongliang Liu, Gang Niu, Jian Yang, and Chen Gong. Tackling instancedependent label noise via a universal probabilistic model. Proceedings of the AAAI Conference on Artificial Intelligence , 35(11):10183-10191, May 2021.

Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. In Proceedings of International Conference on Learning Representations , 2021.

Zhimeng Jiang, Kaixiong Zhou, Zirui Liu, Li Li, Rui Chen, Soo-Hyun Choi, and Xia Hu. An information fusion approach to learning with instance-dependent label noise. In Proceedings of International Conference on Learning Representations , 2022.

Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In Proceedings of International Conference on Learning Representations , 2021.

Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Cicarrelli, Frederik Barkhof, and Daniel Alexander. Disentangling human error from ground truth in segmentation of medical images. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15750-15762. Curran Associates, Inc., 2020.

Hui Guo, Boyu Wang, and Grace Yi. Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model. In Advances in Neural Information Processing Systems , 2023.

Qizhou Wang, Jiangchao Yao, Chen Gong, Tongliang Liu, Mingming Gong, Hongxia Yang, and Bo Han. Learning with group noise. Proceedings of the AAAI Conference on Artificial Intelligence , 35:10192-10200, 05 2021.

Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 12912-12923, 2021.

Xiao Fu, Wing-Kin Ma, José Bioucas-Dias, and Tsung-Han Chan. Semiblind hyperspectral unmixing in the presence of spectral library mismatches. arXiv preprint arXiv:1507.01661 , 2015.

Qiang Liu, Jian Peng, and Alexander T Ihler. Variational inference for crowdsourcing. In Advances in neural information processing systems , volume 25, pages 692-700, 2012.

Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In Proceedings of the European conference on computer vision (ECCV) , pages 68-83, 2018.

Yang Liu, Hao Cheng, and Kun Zhang. Identifiability of label noise transition matrix. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 21475-21496, 2023.

Zhendong Chu, Jing Ma, and Hongning Wang. Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems , volume 30, 2017.

Shahana Ibrahim, Xiao Fu, Nikos Kargas, and Kejun Huang. Crowdsourcing via pairwise co-occurrences: Identifiability and algorithms. In Advances in Neural Information Processing Systems , volume 32, pages 7847-7857, 2019.

K. Huang, N. Sidiropoulos, and A. Swami. Non-negative matrix factorization revisited: Uniqueness and algorithm for symmetric decomposition. IEEE Trans. Signal Process. , 62(1):211-224, 2014.

Xiao Fu, Kejun Huang, Bo Yang, Wing-Kin Ma, and Nicholas D Sidiropoulos. Robust volume minimization-based matrix factorization for remote sensing and document clustering. IEEE Trans. Signal Process. , 64(23):6254-6268, 2016.

Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y Ng. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 254-263, 2008.

Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems , pages 2424-2432, 2010.

Panagiotis A Traganitis, Alba Pages-Zamora, and Georgios B Giannakis. Blind multiclass ensemble classification. IEEE Trans. Signal Process. , 66(18):4737-4752, 2018.

Hansong Zhang, Shikun Li, Dan Zeng, Chenggang Yan, and Shiming Ge. Coupled confusion correction: Learning from crowds with sparse annotations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 16732-16740, 2024.

Shikun Li, Tongliang Liu, Jiyong Tan, Dan Zeng, and Shiming Ge. Trustable co-label learning from multiple noisy annotators. IEEE Transactions on Multimedia , 25:1045-1057, 2021.

Nicolas Gillis. Nonnegative Matrix Factorization . Society for Industrial and Applied Mathematics, 2020.

Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. Max-Mig: An information theoretic approach for joint learning from crowds. In Proceedings of International Conference on Learning Representations , 2019.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of International Conference on Artificial Intelligence and Statistics , volume 15, pages 215-223, 2011.

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations, ICLR , 2015.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations , 2022.

Filipe Rodrigues, Mariana Lourenco, Bernardete Ribeiro, and Francisco C. Pereira. Learning supervised topic models for classification and regression from crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(12):2409-2422, 2017.

Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision , 77:157-173, 2007.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , pages 1-19, 2022.

Dana Angluin and Philip Laird. Learning from noisy examples. Mach. Learn. , 2(4):343-370, apr 1988.

Javed A. Aslam and Scott E. Decatur. On the sample complexity of noise-tolerant learning. Inf. Process. Lett. , 57(4):189-195, feb 1996.

Michael Kearns. Efficient noise-tolerant learning from statistical queries. J. ACM , 45(6):983-1006, nov 1998.

Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. In Shai Shalev-Shwartz and Ingo Steinwart, editors, Proceedings of the 26th Annual Conference on Learning Theory , volume 30, pages 489-511, 2013.

Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In Proceedings of International Conference on Machine Learning , volume 139, pages 825-836, 2021.

Mark A. Davenport, Yaniv Plan, Ewout van den Berg, and Mary Wootters. 1-bit matrix completion. Information and Inference: A Journal of the IMA , 3(3):189-223, 2014.

MSPinsker. The information stability of gaussian random variables and processes (in Russian). Dokl. Akad. Nauk SSSR , 133:28-30, 1960.

A.A. Fedotov, P. Harremoes, and F. Topsoe. Refinements of pinsker's inequality. IEEE Transactions on Information Theory , 49(6):1491-1498, 2003.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014.

Andreas Maurer. A vector-contraction inequality for rademacher complexities. ArXiv , abs/1605.00251, 2016.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices , page 210-268. Cambridge University Press, 2012.

G. Raskutti, Martin Wainwright, and B. Yu. Minimax rates of estimation for high-dimensional linear regression over ℓ q -balls. IEEE Transactions on Information Theory , 57:6976 - 6994, 11 2011.

X. Fu, W.-K. Ma, K. Huang, and N. D. Sidiropoulos. Blind separation of quasi-stationary sources: Exploiting convex geometry in covariance domain. IEEE Trans. Signal Process. , 63(9):2306-2320, May 2015.

Xiao Fu, Kejun Huang, and Nicholas D Sidiropoulos. On identifiability of nonnegative matrix factorization. IEEE Signal Process. Lett. , 25(3):328-332, 2018.

Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications , volume 11006, pages 369-386, 2019.

Supplementary Material of ' Noisy Label Learning with Instance-Dependent Outliers: Identifiability via Crowd Wisdom'

A Notation

The notations used in the paper are summarized in Table 3.

B Related Work

In this section, we briefly review some of the existing works that is relevant to our proposed method. Noisy label learning has been a prominent research focus within the machine learning community for many years-see a survey of these methods in [71]. Among the numerous approaches proposed, those based on label transition matrix-based models (cf. the model in (2)) have been proven effective and several variants of the model have been introduced; see, e.g., [14-19, 21, 23, 27-29, 33, 47, 52, 55, 56]. A broad classification of them is of instance-independent models and instance-dependent models.

Instance-independent Model-based Methods. One of the earliest instance-independent label noise models (i.e., assuming T ♮ ( x n ) = A ♮ , ∀ n in (2)) is the random classification model (RCN) [72-74], where true labels are flipped with a fixed noise rate independently at random. To extend this modeling to more general scenarios, class-conditioned model (CCN) was proposed [9], where the noise rates are assumed to be dependent on the true-class the data item belongs. CCN model has led to the

design of several statistically consistent classifiers, as the label transition matrix acts as a correction term for the loss associated with the noisy labels [21, 24, 26, 75]. To provably learn the CCN model, several notions are exploited, e.g., assuming the presence of anchor data items [21], exploiting the clusterability of data items [45], and minimizing the geometric volume of the transition matrix [19].

Instance-dependent Model-based Approaches. Instance-dependent label noise models are naturally more plausible. In these models, each data item has a label transition matrix dependent on the data's attributes (cf. Eq. (2)). Recently, a number of methods studied such models by using various simplified representations/parameterizations of the transition matrix T ♮ ( x n ) , e.g., imposing structural assumptions on label transition matrices [27,41,43], bounding noise rates [28], modeling the Bayes label transition matrix or posterior transition matrix instead of the true label transition matrix [33,37,40], leveraging additional forms of supervision such as confidence scores [76], and crafting specific regularization terms [36, 39, 42]. To be specific, the work in [27] assumed that the label transition matrix is a convex combination of several instance-independent transition matrices and the combination coefficients depend on the instances. The work in [28] assumed a known bounded noise rate and proposed an instance selection-based strategy. The method proposed in [39] also advocated an instance selection scheme, accompanied with a regularization term to combat instance-dependent label noise. In the work [36], the simplification relies on the assumption that many data items that lie close to each other in the feature space have similar label transition matrices. These interesting developments showed empirical progresses, but the theoretical understanding has been limited. Particularly, the aforementioned identifiability issue was not resolved. In addition, most approaches resort to multi-stage training strategies which may be practically inefficient due to potential error propagation. For example, the approach in [33] first performs instance selection, then estimates the (Bayes-label) transition matrices, and finally estimates the classifier, in a three-stage training scheme. Similar multi-stage approaches are employed in [27,28,36].

Our approach is also related to sample-selection-based approaches in noisy label learning, e.g., clean sample selection-based methods [28,33,35,37,39]. However, most of them treat sample selection as a pre-processing step based on strict prior assumptions (e.g., known noise rates) and then estimate the transition matrix using these selected samples. In contrast, our proposed method and identifiability analysis offer a more principled approach by integrating sample selection with the learning of the classifier and transition matrices in an end-to-end fashion.

C Proof of Lemma 3.1

LemmaC.1. Assume that the observed index pairs ( m,n ) ∈ S are sampled uniformly at random with replacement. Also assume f ♮ ∈ F , |I| ≤ C ≤ N/ 2 , and that each [ A m f ( x n ) + e ( m ) n ] k , ∀ A m ∈ A , ∀ e ( m ) n ∈ E , ∀ f ∈ F are lower bounded by (1 /β ) for a certain β > 0 . Then, with probability greater than 1 -δ , the following holds:

where R 2 S = K 2 log ( K ) + K ∥ X ∥ 2 F R F / S + CMK log √ NM and X = [ x n 1 , . . . , x n S ] are the annotated samples, in which ( m s , n s ) ∈ S for s = 1 , . . . , S .

The proof is essentially viewing the g ♮ ( m ) n estimation problem as a quantized matrix completion problem with missing blocks. The key ideas and steps all follow the proof of the end-to-end crowdsourcing work [31, Proposition 1] with the additional consideration of e ♮ ( m ) n . This addition is conceptually not hard to come up with, but the notations can be involved. Thus, we still derive the proof in detail.

Proof: Consider the cross entropy-based learning criterion given by

where S = |S| , and CE ( x , y ) = -∑ K k =1 1 [ y = k ] log( x ( k )) .

Let us denote the objective function in (11a) as follows:

where g ( m ) n is given by g ( m ) n = A m f ( x n ) + e ( m ) n , G is given by

and ̂ Y denotes the set of observed noisy labels, i.e., { ̂ y ( m ) n } ( m,n ) ∈S . Let { ̂ A m } , { ̂ e ( m ) n } and ̂ f denote optimal solutions of the learning criterion (11). Let us also define the following:

where

F ⊂ { f : R → R } is the neural network function class associated with the classifier function , n n n , D K

Let G ♮ represents the ground-truth such that

We aim to bound ∥ G ♮ -̂ G ∥ F . Towards this goal, we adopt the proof strategy based on the viewpoint of matrix completion as employed in [31,77].

Let us consider that we have S number of observations ω 1 , . . . , ω S such that ω s = ( m s , n s , ̂ y ( m s ) n s ) , where each ( m s , n s ) is sampled uniformly at random (with replacement) from [ M ] × [ N ] , i.e., the sampling probability for each ( m,n ) is given by π m,n = 1 NM . In addition, the noisy observations are generated as

Let D ω denote the joint probability distribution from where the observations ω s are sampled. Then we define the following:

We consider the following set of relations:

where the first inequality is obtained since ̂ G is the optimal solution as given by (13) and hence satisfies L S ( ̂ G ; ̂ Y ) ≤ L S ( G ♮ ; ̂ Y ) . The last inequality is from the given assumption that G ♮ ∈ G (This is satisfied since it is given that f ♮ ∈ F and ∥ E ♮ ∥ 0 ≤ C ).

Next, we consider the following set of relations:

where we applied Pinsker's inequality [78, 79] for the first inequality and the last inequality uses the fact that ∥ x ∥ 1 ≥ ∥ x ∥ 2 for any vector x ∈ R K having | [ x ] k | ≤ 1 , ∀ k ∈ [ K ] . Here D KL ( g 1 , g 2 ) denotes the Kullback-Leibler divergence between any g 1 , g 2 ∈ ∆ K , i.e., D KL ( g 1 , g 2 ) = ∑ K k =1 [ g 1 ] k log [ g 1 ] k [ g 2 ] k ] .

Combining (15) with (16), we get

Next, we proceed to characterize the R.H.S. of (17). To achieve this, we invoke the following theorem derived from Theorem 26.5 in [80]:

Theorem C.2. [80, Theorem 26.5] Assume that ∀ ( m,n, ̂ y ( m ) n ) and ∀ G ∈ G , we have | CE ( g ( m ) n , ̂ y ( m ) n ) | ≤ z max . Then for any G ∈ G , the following holds with probability greater than 1 -δ :

where ℓ ◦ G ◦ S denotes the set

and R ( X ) denotes the empirical Rademacher complexity of the set X .

To apply Theorem C.2, we need to characterize the terms z max and R ( ℓ ◦ G ◦ S ) . The term z max can be characterized as below:

Note that in the above relation (19), we utilize the given assumption that each [ g ] k ≥ 1 /β, ∀ g ∈ G . To characterize the Rademacher complexity R ( ℓ ◦ G ◦ S ) , we consider its definition as follows [80]:

where expectation is w.r.t. the independent Rademacher random variables σ s ∈ {-1 , 1 } . Since g ( m ) n are vectors, we utilize the following result to upper bound R ( ℓ ◦ G ◦ S ) :

Lemma C.3. [81] Assume that the function CE ( · , y ) , ∀ y has the Lipschitz constant L . Then

where σ sk is an independent (doubly indexed) Rademacher random variable and the expectations are taken w.r.t. the Rademacher random variables.

To apply Lemma C.3, let us define a vector z ≜ ( [ g ( m 1 ) n 1 ] 1 , [ g ( m 1 ) n 1 ] 2 , . . . , [ g ( m S ) n S ] K -1 , [ g ( m S ) n S ] K ) ∈ R SK and the set Z ◦ S ≜ { z = ( [ g ( m 1 ) n 1 ] 1 , [ g ( m 1 ) n 1 ] 2 , . . . , [ g ( m S ) n S ] K -1 , [ g ( m S ) n S ] K ) | G ∈ G} . Using these definitions, let us apply Lemma C.3 in (20) and get the following relation:

where β is an upper bound of the Lipschitz constant of the cross entropy loss function CE ( g , y ) = -∑ K k =1 1 [ y = k ] log[ g ] k when g ∈ ∆ K with [ x ] k > (1 /β ) . ∀ k .

Next, we will characterize R ( Z ◦ S ) using the covering number of the set Z ◦ S . To define the covering number, let us first consider the concept of ϵ -net covering of a set. The ϵ -net covering of a set X (denoted as X ϵ ) is defined as the finite subset of X (i.e., X ϵ ⊆ X ) such that for any x ∈ X , there exists an x ∈ X ϵ satisfying ∥ x -x ∥ 2 ≤ ϵ [82]. The smallest cardinality of the ϵ -nets of X is known as the covering number of X , which is denoted as N ( ϵ, X ) .

Let us consider a pair of vectors z S , z S ∈ Z ◦ S as below:

where A m s belongs to the ϵ -net covering of A , f belongs to the ϵ -net covering of F , and e ( m s ) n s is the appropriate sub-vector belonging to the ϵ -net covering of E 0 .

Let us also consider the following definitions:

Then, we have the following relation due to triangle inequality

Next, consider the second term on the R.H.S. of (22):

We also have the following relations for the first term on the R.H.S. of (22) :

In the above set of relations, the first inequality is by triangle inequality. The second inequality uses that fact that ( a + b ) 2 ≤ 2 a 2 +2 b 2 . The third inequality uses the fact that the Frobenius norm of A m ( x n ) 's are bounded by √ K and the ℓ 2 norm of f ( x n ) is bounded by 1 . Hence, combining (22)-(24), we get that in order to obtain an ε -net covering for the set Z ◦ S (i.e., ∥ z -z ∥ 2 ≤ ε ), we only need to show that there exists a ε 4 √ K -net covering for F ◦ S , an ε 2 √ S -net covering for E NM , ∀ m , and a ε 4 √ S -net covering for each A 's. Here we have

∥ E ∥ 0 = ∑ N n =1 1 { ∑ M m =1 ∥ e ( m ) n ∥ 2 > 0 } . Note that the full rank matrix A m ∈ R K × K can be represented as a K 2 -dimensional vector whose Euclidean norm is bounded by √ K . Hence, the cardinality of the ε 4 √ S -net covering for A m ∈ R K × K is at most ( 8 K √ SK ε ) K 2 [80]. We consider the covering number corresponding to the function class F ◦ S . Towards this, we invoke the following lemma:

Lemma C.4. [51, Theorem 3.3] Let fixed nonlinearities ( σ 1 , . . . , σ L ) and reference matrices ( M 1 , . . . , M L ) be given, where σ i is ρ i -Lipschitz and σ i (0) = 0 . Let spectral norm bounds ( s 1 , . . . , s L ) , and matrix (2,1)-norm bounds b 1 , . . . , b L be given. Let x 1 , . . . , x S be the S number of data items. Let h belongs to the following function class:

where C i ∈ R d i × d i -1 with d 0 = D and d L = K . Let H◦S denote the family of matrices obtained by evaluating the data items with all choices of network H :

Then for any ϵ > 0 , where e is the Euler's number.

Applying Lemma C.5, we get the cardinality of ε 2 √ S -net covering of of E NM as follows:

Note that in (25), we used the result that ∥ e NM ∥ 2 ≤ √ 2 CM ≤ 2 CM, ∀ e NM ∈ E NM .

Using the covering number results, the cardinality of the ε -net covering of set Z ◦ S is bounded by the following:

Now that we have characterized N ( ε, Z ◦ S ) , we invoke the Dudley entropy chaining technique lemma to obtain the Rademacher complexity R ( Z ◦ S ) [80]:

where X = [ x 1 , . . . , x S ] , R H = log(2 H 2 ) ( ∏ L j =1 s 2 j ρ 2 j )( ∑ i L ( b i /s i ) 2 / 3 ) 3 is called as the spectral-complexity upper bound of neural network function class H and H = max ℓ d ℓ .

Using Lemma C.4, we get the cardinality of the ε 8 √ SK -net covering for F ◦ N as below:

where X S = [ x 1 , . . . , x S ] ∈ R d × S and the parameter R F the spectral-complexity upper bound of F .

Next, we proceed to get the cardinality of the ε 2 √ S -net covering for E NM . Here, E NM is a set of sparse vectors with the number of non-zero elements upper-bounded by CMK , i.e., for any e NM ∈ E NM , ∥ e NM ∥ 0 ≤ CMK . To get the covering number of a set of sparse vectors, we invoke the following result:

Lemma C.5. [83] Let X = { z ∈ R N |∥ z ∥ 2 ≤ q, ∥ z ∥ 0 ≤ C } . Suppose that C ≤ N 2 . Then, there exists an ϵ -net covering denoted as X ϵ such that

Lemma C.6. Consider set A ⊆ R m , let c = min a max a ∈ A ∥ a -a ∥ . Then for any integer T > 0 ,

Since, applying Lemma C.6 on set Z ◦ S gives

Choosing T = log S/ (2 log 2) , we get the bound

Combining the upper bound of R ( Z ◦ S ) given by (27) with the upper bound of R ( ℓ ◦ G ◦ S ) as given by (21) and with the results in (18) and (19), we get that with probability greater than 1 -δ ,

where R S ( Z ◦ S ) is upper bounded by (27).

The above relation, combined with (16), implies that with probability greater than 1 -2 δ :

where R ( Z ◦ S ) is upper-bounded in (27).

D Proof of Theorem 3.5

̸

To obtain the results in Theorem 3.5, we first show that the criterion correctly distinguishes the instance-dependent and instance-independent samples, i.e., ̂ I = I where we define ̂ I ≜ { i ∈ [ N ] | ̂ e i = 0 } .

Recall ̂ A m 's, ̂ e ( m ) n 's and ̂ f are defined as in Theorem 3.5, and let ̂ e n = [( ̂ e (1) n ) ⊤ , . . . , ( ̂ e ( M ) n ) ⊤ ] ⊤ . We first show that ̂ I c ⊆ I c . From Lemma 3.1, the following holds:

Then the inequality also holds for any submatrix of the left hand side matrix. In particular, for i ∈ I and S > S 0 ,

̸

Therefore, ̂ e i = 0 and that concludes ̂ I c ⊆ I . On the other hand, since ̂ e ( m ) n 's minimizes ∑ N n =1 1 { ∑ M m =1 ∥ ̂ e ( m ) n ∥ 2 > 0 } , it holds that I c ⊆ ̂ I c . This implies that I c = ̂ I c must hold.

With I c = ̂ I c , we have the following relation from (30):

To proceed, we invoke a result from the work [31] considering Assumption 3.4 satisfied, and the set I c is uniformly chosen from [ N ] , with probability at least 1 -2 S -2 K N α ,

E Proof of Theorem 3.6

The proof of Theorem 3.6 follows similar steps in finite case presented in Theorem 3.5, except that we have the exact recovery of ̂ G when S →∞ . We repeat the derivation here for completeness.

We first show that the criterion correctly distinguishes the instance-dependent and instanceindependent labels.

Let ( ̂ A m 's, ̂ e ( m ) n 's and ̂ f ) be an optimal solutions as state in Theorem E. From Lemma (3.1), the following holds as the number of observations grows S infinitely large:

Consider the following relation:

Here, each column of the matrix G ♮ is given by

Similarly, we can also define

From (32), we get that

We will use contradiction-based argument towards this. Let us start by considering the case where there exists i ⋆ ∈ I c as well as i ⋆ ∈ ̂ I c holds. Then, combining (34) and (35) we get

We can rewrite the relation (36) as

Since any submatrix formed by N -2 |I c | columns of F ♮ = [ f ( x 1 ) , . . . , f ( x N )] has rank K and that rank ( W ♮ ) = K , we get that there exists a nonsingular matrix Q ∈ R K × K such that

Next, we consider the case where there exists i ⋆ ∈ I and i ⋆ ∈ ̂ I c . Then, again from (34), (35) we get that

However, by applying the relation (37) in (38), we get

where q = Q ̂ f ( x i ⋆ ) -f ♮ ( x i ⋆ ) is a nonzero vector since it is assumed that e ♮ i ⋆ = 0 . The above relation implies that e ♮ i ⋆ ∈ range( W ♮ ) must hold which is a contradiction to the assumption that e ♮ i ⋆ / ∈ range( W ♮ ) when i ⋆ ∈ I . Hence, the relation (38) is feasible only if e ♮ i ⋆ = 0 . It means that if there exists i ⋆ ∈ ̂ I c , we should have i ⋆ ∈ I c , leading to the conclusion that ̂ I c ⊆ I c .

On the other hand, if ̂ I c ⊂ I c holds, then

Since ̂ e ( m ) n 's are the optimal solution of Problem (9) with minimal ∑ N n =1 1 { ∑ M m =1 ∥ ̂ e ( m ) n ∥ 2 > 0 } , the following should hold

̸

which is a contradiction. This implies that I = ̂ I must hold.

Since I = ̂ I holds, we have the following relation from (32):

where we utilized the following notations:

Next, we connect the det( W ⊤ W ) term to our optimal solutions. This part is similar to the classical proofs in [84, 85], with proper modifications. Consider the following result extracted from the proof of [84, Theorem 1]:

Lemma E.1. Suppose a matrix Y ∈ R K × J satisfies Y ≥ 0 , 1 ⊤ Y = 1 ⊤ , rank ( Y ) = K , and SSC. Then, for any ̂ Y = QY satisfying ̂ Y ≥ 0 , 1 ⊤ ̂ Y = 1 ⊤ , the following holds:

The equality holds only if Q is a permutation matrix.

Since ̂ W is the optimal solution of the criterion given by Theorem 3.6, the following holds:

We also have W ♮ and F ♮ (: , I c ) are of rank K . Hence, from (39), we get that ̂ W and ̂ F (: , I c ) satisfies ̂ W = W ♮ Q -1 , ̂ F (: , I c ) = QF ♮ (: , I c ) , for a certain invertible matrix Q . Let us assume that Q is not a permutation matrix, Then, we have

where the last inequality is from Lemma E.1 using the SSC condition on F ♮ (: , I c ) . Note that the result is a contradiction from (40). Hence, Q must be a permutation matrix. This implies that the optimal solution ̂ W and ̂ F (: , I c ) satisfies the following:

F Proof of Fact 2.1

For rank ( A ♮ ) = K , when N →∞ , the objective in (6) seeks the optimal solutions A ⋆ , f ⋆ and e ⋆ n such that

This can be seen from Lemma 3.1 with M = 1 . One solution that satisfies the equality (42) and the sparsity constraint in (6) is A ⋆ = I , f ⋆ ( x ) = A ♮ f ♮ ( x n ) + e ♮ n , and e ⋆ n = 0 , which can always hold when f is a universal function approximator.

G More Details on Experiment Settings and Datasets

G.1 Implementation

We train with batch size of 512, number of epochs 200, Adam optimizer with learning rate of 0 . 01 and learning rate scheduler OneCycleLR [86]. We initialize A m using an identity matrix going through softmax and e m = 0 for all m ∈ [ M ] . We also apply standard data augmentation including random cropping, random flipping, when training on CIFAR-10, CIFAR-10N, and STL-10. The experiment results are averaged over 3-5 random trials. All runs have been conducted using either Nvidia A40 or Nvidia DGX H100 GPU. Each run of COINNet on CIFAR-10 takes about 60 minutes using Nvidia A40, and 30 minutes using Nvidia DGX H100 GPU.

G.2 Machine Annotations

For machine annotator experiments, we consider the CIFAR-10 [62] and the STL-10 datasets [63]. The CIFAR-10 dataset consists of 60 , 000 labeled color images of animals, vehicles and so on, each having a size of 32 × 32 and belonging K = 10 different classes. The images are split into training and testing sets with size 50,000 and 10,000, respectively. The STL-10 dataset consists of 13,000 labeled images from 10 different classes, similar to CIFAR-10. Of these, 5,000 images are designated for training and 8,000 for testing.

Table 4 shows the details regarding the individual label noise rates and training methods in our experiments with machine annotators.

G.3 Real Annotations

For real annotation experiments, we use 2 popular public datasets: CIFAR-10N [66] and LabelMe [67, 68], and construct the ImageNet-15N dataset with a higher number of annotators. All three datasets are labeled by anonymous annotators from the AMT platform. The CIFAR-10N is a 'noisy" version of the popular CIFAR-10 dataset, containing 50,000 images for training, and a separate

1

2

3

4

M

0.0

0.2

0.4

0.6

0.8

1.0

score

0.335

0.676

0.752

0.801

0.900

0.921

0.927

0.944

Outlier Detection Rate

1

2

3

4

M

0.0

0.2

0.4

0.6

0.8

1.0

score

0.202

0.359

0.549

0.636

0.870

0.898

0.918

0.932

Outlier Detection Rate

10,000 images for testing. We randomly split the training set into 47,500 and 2,500 to use as train and validation set for all methods. Every image in CIFAR-10N has 3 labels.

The LabelMe dataset comprises 2,688 images from eight distinct classes: highway, inside city, tall building, street, forest, coast, mountain, and open country. Out of the available images, 1,000 have been annotated by 59 AMT workers. In total, approximately 2,547 image annotations were collected, with labeling accuracy ranging from 0% to 100% and an average accuracy of 69.20%. To enhance the training dataset, standard augmentation techniques such as rescaling, cropping, and horizontal flips were applied, resulting in a training set of 10,000 images annotated by the same 59 workers-see more details in [67]. The validation set comprises 500 images, while the remaining 1,188 images are reserved for testing.

We select 15 classes from the ImageNet dataset with the intention of including easy-to-confuse classes. These classes are: dog, leopard, sports_car, tiger_cat, airship, aircraft_carrier, trailer_truck, orange, penguin, lemon, soccer_ball, airliner, freight_car, container_ship, passenger_car. We collect annotations for N = 2 , 514 images from M = 100 anonymous real annotators. These serve as our training set. The average error rate of the annotators is 42.68%. The validation and testing sets have 1,462 and 13,157 images, respectively. We release the noisy annotations at https: //github.com/ductri/COINNet .

H More Experiments

H.1 Experiments with Synthetic Annotators

Noisy Label Generation. Here, we consider labels corrupted by synthetically generated annotator confusions. To generate the confusion matrices, we utilize the following strategy. We control the instance-independent label noise rate using a parameter τ ∈ (0 , 1) . By employing this parameter, we construct the ground-truth confusion matrix A ♮ , where the diagonal entries of A ♮ are set as [ A ♮ ] k,k = 1 -τ, ∀ k , and the off-diagonal entries are set as [ A ♮ ] k,j = τ K -1 , ∀ k = j . First, we randomly select 1 -η fraction of the data samples and generate their labels using the instanceindependent confusion matrix A ♮ . For the remaining η percent of the samples, we adopt an instancedependent noise generation process, by following the strategy outlined in [27]. Specifically, for such 'outliers", the label confusions vary depending on the features of such instances.

̸

We generate synthetic noisy labels for CIFAR-10 dataset with M = 3 . CIFAR-10 contains 50,000 images of size 32 × 32 as training set, and 10,000 images as test set. We keep the test set untouched while randomly splitting the train set into 2 parts (47,500; 2,500) to serve as training and validation set. We use the same neural network architecture and the optimization settings as used in machine annotator case. The results are presented in Table 5. One can observe that our approach consistently perform well in all scenarios.

To demonstrate the effectiveness of the advocated crowdsourcing approach in outlier detection, we vary the number of annotators M and present the average outlier detection rate and the testing accuracy, where

We observe that increasing the number of annotators M shows improvement over both outlier detection and the final accuracy score as shown in Fig. 4.

H.2 Ablation Study

Effect of regularization terms. Tables 6, 7 and 8 show the effectiveness of having both the proposed regularization terms (i.e., sparsity and volume).

Effect of missing annotations. Table 9 and Table 10 show results under different levels of missing annotations under various parameters settings. Table 11 and Table 12 present results where only a single label is available for each image. In all four cases, the proposed method demonstrates superior robustness against the negative effect of missing annotations relative to MaxMIG and GeoCrowdNet(F) , two competitive baselines.

Effect of initialization. In Table 13, we test the performance of our approach using different initialization strategies. In particular, we conduct the machine annotator experiment on CIFAR-10 using the following initialization strategies: the confusion matrices are (i) initialized using identity matrices, and (ii) initialized by the GeoCrowdNet(F) method. We also include the performance of GeoCrowdNet(F) for reference. We observe a slight improvement (around 0.1-0.4%) in accuracy when using initialization from GeoCrowdNet(F) with the cost of training GeoCrowdNet(F) for 10 epochs.

0.005

0.005

0.005

0.005

0.005

0.005

0.005

0.004

0.005

0.004

0.005

0.005

0.004

0.005

0.004

0.004

0.004

0.005

0.004

0.005

0.004

0.004

0.005

0.004

0.005

0.005

0.005

0.004

0.005

0.005

0.005

0.005

0.004

0.005

0.005

0.005

0.004

0.005

0.005

0.005

0.005

0.004

0.005

0.005

0.005

0.005

0.005

0.004

0.005

0.005

H.3 Outlier Detection in CIFAR-10N Dataset

Following the result in Fig. 2, we show more examples of nominal data items and outlier data items as separated by our approach in Fig. 5 and Fig. 6, respectively. Clearly, more instance-varying confusing characteristics are present in the latter case.

0.176

0.176

0.175

0.175

0.171

0.167

0.172

0.174

0.175

0.176

0.167

0.172

0.174

0.175

0.176

0.166

0.177

0.176

0.175

0.172

0.166

0.177

0.176

0.175

0.172

0.171

0.176

0.180

0.178

0.167

0.171

0.176

0.180

0.178

0.167

0.172

0.174

0.175

0.172

0.167

0.172

0.174

0.175

0.172

0.167

0.173

0.177

0.174

0.176

0.176

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims have been discussed and supported by our analysis in Section 2 and 3, particularly by Fact 2.1, Theorem 3.5 and Theorem 3.6 with empirical evidence in Section 5

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations at the end of Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have presented all assumptions we have used in Section 3. All proof are presented in the Appendix, particularly in Appendix D, Appendix E, Appendix F.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have described in details experiment settings as well as the implementation techniques in Section 3, Section 5 and Appendix G. We also include our code in this submission as supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have included code, including instruction how to run in this submission. It is as almost the same code that we use to to produce all experiment results, except removing all identity related information. All data we used are publicly accessible.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe these in Appendix G

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are repeated at least 3 times. All numbers are accompanied with standard deviation, as described in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have included these information in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have made to our best effort to remove all identity-related information in both the main paper supplementary material.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work tackles the noisy labels problem which is ubiquitous, especially in the big data era, and hence bring positive impact to society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not see any potential harm in publishing data and trained models of the proposed method.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all datasets used in Section 5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There is no new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: There is no human subjects or crowdsourcing experiments conducted in the work. All datasets are from published sources.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: As mentioned above, our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper addresses the problem of learning with noisy labels by considering a model where instance-dependent confusion matrices occur occasionally across the samples, and the rest of data share a common nominal confusion matrix. The paper claims two main contributions: (1) showing that a single confusion matrix is insufficient to identify outliers and proposing a crowdsourcing strategy with a column sparsity constraint to overcome this, and (2) presenting an end-to-end, one-stage learning loss that is differentiable and easily optimized. These contributions are validated through experiments that demonstrate improved testing accuracy.

STRENGTHS:
● The problem setting, considering the instance-dependent noisy samples as outliers, is realistic and really interesting.
● The paper posits that existing methods relying on sparsity priors are insufficient for outlier detection and proposes a novel crowdsourcing strategy as a solution with theoretical grounding and generalization guarantees.

WEAKNESSES:
Lack of large-scale datasets in experiments

QUESTIONS:
Can the authors discuss any limitations their method might have in scenarios of highly imbalanced noise?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-05

comments: SUMMARY:
The paper investigates the challenge of instance-dependent noisy labels, which are modeled using an instance-dependent confusion matrix reflecting annotator errors. Traditional approaches assume a consistent confusion matrix across instances, which simplifies the problem but is unrealistic. This study models the instance-dependent confusion matrices as outliers. The authors propose using a crowdsourcing strategy with multiple annotators and a specialized loss function to effectively detect outliers and identify the target classifier. The method is supported by extensive theoretical results. Experimental results also confirm the efficacy of the proposed method.

STRENGTHS:
1. The work aims at an important and the proposed solution is solid.
2. The presentation is clear and the proposed method is supported by thorough theoretical results.
3. Experiments on real-world label noise such as CIFAR-N validate the effectiveness of the proposed method.

WEAKNESSES:
1. Theorem 3.6 requires $S\rightarrow \infty$. However, we know if we have sufficient annotators, the aggregated noise rate could be 0, making the result trivial. It is interesting to show the result with finite $S$. 
2. The organization part of the proposed method could be polished.

QUESTIONS:
1. Does this method heavily rely on multiple annotators? If there is only one label for each example, is there any approximation method to generate extra labels to simulate multiple annotators?
2. Would existing estimators for noisy labels, such as [R1], provide a good initialization for the variable $\mathbf A_m$ in the proposed loss function?

[R1] Unmasking and improving data credibility: A study with datasets for training harmless language models. ICLR 2024.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends further estimation of transition matrix in label noise learning. In previous studies, label noise is often assumed to be class-dependent. Hence, one can use the noisy label data and apply loss correction to learn a good classifier. The paper extends from such a modelling approach by considering most of samples having class-dependent label noise and some samples having class- and instance-dependent noise. Those with instance-dependent label noise are then considered as outlier in the newly proposed modelling approach. Despite such a modelling, it still results in a non-identifiable solution. To overcome that, the paper incorporates multiple labels from multiple annotators, so that one can solve for the nominal transition matrix and the classifier of interest. The paper also provides theoretical results to guarantee that the estimation holds under certain confidence level. Empirical results show that models trained on the new loss function performs competitively with prior methods.\n\nSTRENGTHS:\n- The paper thoroughly presents the motivation and the formulation of their modelling approach. The paper also connects to previous studies when formulating the noisy label learning with instance-dependent noise as outliers. In general, the paper is well-written and easy to follow.\n- The main contribution of the paper is to extend further the modelling considering only class-dependent transition matrix by adding *perturbation* induced by instance-dependent label noise samples. The paper also analyses the theoretical guarantee when estimating the classifier of interest in terms of PAC-style formulae.\n\nWEAKNESSES:\nThe major weakness of the paper may lie at the assumption that majority of samples would share the same nominal transition matrix, while a few would have their own transition matices. However, this does not affect too much to the contribution of the paper.\n\nAnother weakness is the usage of multiple annotators, while the conventional noisy label learning does not consider any additional labels. Of course, some studies (e.g., [43]) show that using a single noisy label would result in non-identifiability, unless additional constraints are exploited. Nevertheless, multiple annotator setting would be closer to current practice. The downside is that there are not many multi-rater learning datasets for benchmarking.\n\n### Minors\n- Line 76: typo \"traget\" -> target\n- The abbreviation \"NMF\" at line 100 is not defined.\n- Line 111: $E(x_{n})$ is a matrix. Thus, please make it clear when specifying $E(x_{n}) > 0$.\n\nQUESTIONS:\nI do enjoy reading the paper. It is coherent and easy to follow. I do not have any questions.\n\nLIMITATIONS:\nAs mentioned in the weakness section above and in the conclusion of the paper, the paper only considers a few samples with instance-dependent label noise, while the remaining samples are class-dependent label noise, to reduce the complexity of the modelling and analysis. Such an assumption may not always hold in all settings.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the identifiability problem of instance-dependent label noise with multiple annotators. To achieve the identifiability, this work first claimed a fact that only a proportion of all instances may have a labeling difficulty that significantly deviates from the general population. Then, it connected the problem to the uniqueness of non-negative matrix factorization with mild assumptions. Inspired by the identifiability result, this work proposed a end-to-end one-stage method to learn from crowds via identifying instance-dependent label noise. Experiments on multiple datasets with machine annotations and human annotators showed the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This work provided some interesting identifiability results for learning from crowds, which may inspired further work.\n2. The writing is excellent, making it easy to understand.\n3. The proposed method is end-to-end and one-stage, which is nice for application.\n4. The case study in Figure 2 clearly showed the rationality of the results.\n\nWEAKNESSES:\n1. As this work claimed, the number of annotators is important to identify the instance-dependent label noise, is there some experimental results verifying this analysis？\n\n2.  For human annotations,  the annotations are usually sparse. Does the annotation sparsity level influence the identifiability and the performance of the proposed method? I suggest the authors do some experiments as existing works to clarify it [1-3].\n\n3. The meaning of some terms, like \"outliers\", \"the model of interest\", \"neural systems\", is not clear when just reading the abstract. Besides, since the sparsity prior of the outliers is a basis assumption for this work, it should be referred to in the abstract.\n\n[1] Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model. NeurIPS 2023\n\n[2] Coupled confusion correction: learning from crowds with sparse annotations. AAAI 2024\n\n[3] Trustable co-label learning from multiple noisy annotators. TMM 2023\n\nQUESTIONS:\n1. The meaning of some terms, like \"outliers\", \"the model of interest\", \"neural systems\", is not clear when just reading the abstract.\n2. It seems that there is a symbol that is not displayed in the restrictive condition of Eq.(6).\n3. What does $e^\\star_n$ mean in Line 129?\n4. What does \"over-canceling” mean in Line 211?\n5. I suggest placing the learning-from-crowds methods after the learning-from-single-annotator methods in Table 1 and 2.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}