Title: Label Correction of Crowdsourced Noisy Annotations with an Instance-Dependent Noise Transition Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: Noisy Label

Instance-Dependent Transition Matrix

Label Correction

Crowdsourcing
Abstract: The predictive ability of supervised learning algorithms hinges on the quality of annotated examples, whose labels often come from multiple crowdsourced annotators with diverse expertise. To aggregate noisy crowdsourced annotations, many existing methods employ an annotator-specific instance-independent noise transition matrix  to characterize the labeling skills of each annotator. Learning an instance-dependent noise transition model, however, is challenging and remains relatively less explored. To address this problem, in this paper, we formulate the noise transition model in a Bayesian framework and subsequently design a new label correction algorithm. Specifically, we approximate the instance-dependent noise transition matrices using a Bayesian network with a hierarchical spike and slab prior. To theoretically characterize the distance between the noise transition model and the true instance-dependent noise transition matrix, we provide a posterior-concentration theorem that ensures the posterior consistency in terms of the Hellinger distance. We further formulate the label correction process as a hypothesis testing problem and propose a novel algorithm to infer the true label from the noisy annotations based on the pairwise likelihood ratio test. Moreover, we establish an information-theoretic bound on the Bayes error for the proposed method. We validate the effectiveness of our approach through experiments on benchmark and real-world datasets.

Label Correction of Crowdsourced Noisy Annotations with an Instance-Dependent Noise Transition Model

Abstract

The predictive ability of supervised learning algorithms hinges on the quality of annotated examples, whose labels often come from multiple crowdsourced annotators with diverse expertise. To aggregate noisy crowdsourced annotations, many existing methods employ an annotator-specific instance-independent noise transition matrix to characterize the labeling skills of each annotator. Learning an instance-dependent noise transition model, however, is challenging and remains relatively less explored. To address this problem, in this paper, we formulate the noise transition model in a Bayesian framework and subsequently design a new label correction algorithm. Specifically, we approximate the instance-dependent noise transition matrices using a Bayesian network with a hierarchical spike and slab prior. To theoretically characterize the distance between the noise transition model and the true instance-dependent noise transition matrix, we provide a posterior-concentration theorem that ensures the posterior consistency in terms of the Hellinger distance. We further formulate the label correction process as a hypothesis testing problem and propose a novel algorithm to infer the true label from the noisy annotations based on the pairwise likelihood ratio test. Moreover, we establish an information-theoretic bound on the Bayes error for the proposed method. We validate the effectiveness of our approach through experiments on benchmark and real-world datasets.

1 Introduction

Deep neural networks (DNNs) have achieved remarkable performance in various tasks [1, 2], and they have proven to be useful in handling sizable labeled data. Acquiring large accurately annotated datasets, however, is usually expensive and time consuming. To enhance the efficiency of annotation, in many applications, crowdsourcing [3] is employed as an alternative way for data labeling, where the labels are provided by multiple annotators with varying and imperfect labeling skills, and thus, the collected labels suffer from unavoidable noise. As deep models have a strong memorization power, using these noisy labels as the ground truth deteriorates the performance of DNNs [4, 5], and most importantly, yields erroneous learning results. Further, potentially substantial disagreement among

the annotators for each instance presents extra challenges in the application of traditional supervised learning algorithms. Hence, in the crowdsourcing scenario, to effectively train DNNs on noisy labeled datasets, a fundamental question is how to aggregate the noisy crowdsourced annotations and infer the latent true labels [6].

One naive approach to aggregate the crowdsourced labels is simply by computing the majority vote, which can be ineffective when the number of annotators is not large enough or the labeling task is difficult [7, 8]. Recent research has developed more powerful techniques for inferring the ground truth labels [7, 9-11], among which the annotator-specific noise transition matrix , aka annotator confusion , plays an important role by modeling the labeling process for each individual annotator. To estimate the transition matrix, available research [8, 11-13] usually makes the instance-independent assumption that for annotator r , given the true label y , the corruption process is independent of the input x , i.e., P (˜ y ( r ) = l | y = k, x ) = P (˜ y ( r ) = l | y = k ) , where x denotes the random variable for instance/feature, ˜ y ( r ) represents the noisy label given by annotator r , and y is the underlying ground truth label. This assumption, however, is often violated in applications. Instance-dependent annotation noise is more realistic and appropriate for real-world datasets, as suggested by the example that factors such as the quality of ultrasound images and the domain expertise of human annotators can greatly influence the actual diagnostic process in medical analysis [14, 15]. For annotator r , the transition matrix τ ( r ) ( x ) is a matrix-valued function, with the ( k, l ) element defined as τ ( r ) kl ( x ) = P (˜ y ( r ) = l | y = k, x ) . Unfortunately, the case of instance-dependent annotation noise remains challenging and less explored. Most existing works considering instance-dependent noise are designed for the single annotator case [16-18]. For the case with multiple annotators, existing methods investigate the human annotation process and use different models to estimate instance-dependent noise matrices. Approaches in [3, 19-22] use traditional classification models such as logistic regression, while others [23-25] cater to large datasets and deep models. Methods in [3, 19-22] and [24, 25] are heuristic in nature and lack theoretical guarantees in estimating instance-dependent noise matrices. [23] makes some theoretical progress in justifying the use of the trace regularisation, and extends the work of [8] which establishes the theory only for settings with an instance-independent noise matrix. The theory in [23] is constrained to individual samples rather than the population setting. Importantly, the theoretical characterization of the distance of the noise model and the true annotator confusion remains absent from the literature.

In this paper, we address this notable problem by framing it within the Bayesian paradigm. We formulate the instance-dependent annotator-specific label transition matrix , and further propose a novel algorithm to infer the underlying ground truth by aggregating the noisy annotations . To model the noise transition matrix, we invoke the Bayesian generalized linear mixed effects model (GLMM), which can be learned by deploying anchor points within the deep learning framework [12, 26, 27]. To facilitate the fact that the number of anchor points learned from the noisy training data is relatively small compared to the sample size, we employ a hierarchical spike and slab prior on the network parameters. This approach offers an interpretable mechanism for variable selection and allows us to establish the theoretical result within the deep learning setup. Our study reveals that the proposed noise transition model is close to the underlying true transition matrix with respect to the Hellinger distance in the Bayesian framework. Such a result is established for independently, nonidentically distributed (i.n.i.d.) observations, substantially extending the existing sparse Bayesian theories within the deep learning paradigm. Further, we develop a label correction method using the pairwise likelihood ratio test to aggregate and infer the ground truth from the noisy crowdsourced annotations. This development is carried out by formulating the label correction process as a hypothesis testing problem and utilizing the proposed Bayesian model in place of the unknown transition matrix in the pairwise likelihood ratio test (LRT). More importantly, with the posterior consistency result, we also derive information-theoretic bounds on the Bayes error for the proposed algorithm even without access to the underlying true noise transition matrix.

This research brings forth several noteworthy advancements: (1) We formulate the annotator-specific noise transition matrix in the Bayesian framework (Section 3.1). This method offers a practical and flexible framework to address real-world problems with noisy annotators. (2) We theoretically characterize the closeness of the proposed model and the underlying annotator confusions with respect to the Hellinger distance. (Section 3.2). (3) We develop a novel label correction algorithm by aggregating the noisy annotations using the pairwise likelihood ratio test, and identify informationtheoretic bounds on the Bayes error (Section 3.3). The effectiveness of the proposed algorithm is

confirmed by the application to both synthetic and real-world noisy datasets (Section 5). Code is available at https://github.com/hguo1728/BayesianIDNT.

2 Problem Setup

Objective and Data. Consider a classification task with a feature space X ⊂ R p and a label space Y = [ K ] , where p is the dimension of the features, K is the number of classes, and [ k ] represents { 1 , ..., k } for any positive integer k . Our goal is to develop a classifier h : X ↦→ Y , which can accurately predict the true label for a test instance. However, in applications, the true label y ∈ Y is often not observed for each input vector x ∈ X . Instead, we receive a set of noisy crowdsourced labels ˜ y = { ˜ y (1) , .., ˜ y ( R ) } from R distinct annotators, where ˜ y ( r ) ∈ Y represents the label given by the r th annotator for r ∈ [ R ] . Thus, a noisy dataset D of size N is defined as D = { x i , ˜ y (1) i , .., ˜ y ( R ) i } N i =1 , where for each instance x i , the true label y i is unobserved. Under this setting, we aim to learn a reliable classifier h by utilizing the noisy crowdsourced dataset D .

In practice, on commercial crowdsourcing platforms, large-scale labels can often be collected from independent human annotators. We thereby make a common assumption that the R annotators independently label the instances [7, 8]. The conditional probability of the R noisy labels, given an instance, can then be formulated as

where for k ∈ Y , P (y = k | x ) , called the base model [28], denotes the conditional probability of the latent true label y given x , which can be modeled by the output of a DNN parameterized by a parameter vector, say ϑ ; and P (˜ y ( r ) | y = k, x ) is the noise transition model for the r th annotator [8], satisfying ∑ K l =1 P (˜ y ( r ) = l | y = k, x ) = 1 for any x ∈ X and k ∈ [ K ] . For ease of theoretical presentation, we assume the accessibility to all the annotations from the R workers for now, and consider more general situations in the experimental part in Section 5; extensions to accommodating the case where each instance is only annotated by a subset of annotators are straightforward.

̸

Notation. In this paper, sets are denoted by calligraphic upper case letters, and vectors and matrices are denoted by bold lower and upper case letters, respectively. For a vector v , v j denote its j th element, and v ⊤ denotes its transpose. For v = ( v 1 , ..., v d ) ⊤ , we denote ∥ v ∥ q = ( ∑ d j =1 | v j | q ) 1 /q for q > 0 , ∥ v ∥ ∞ = max j | v j | , and ∥ v ∥ 0 = ∑ d j =1 1 ( v j = 0) , with 1 ( · ) denoting the indicator function. The L 2 norm of v is also denoted by ∥ v ∥ for simplicity. For a matrix V , we use V i,j to represent its ( i, j ) element. Let (Ω , G , µ ) denote the measure space under consideration, where Ω is a set, G is the σ -field of subsets of Ω , and µ is the associated measure. For a measurable function f : Ω → R d , we write ∥ f ∥ q ≜ ∥ f ∥ L q (Ω) when there is no ambiguity of the domain, where ∥ f ∥ L q (Ω) = ( ∫ Ω ∑ d j =1 | f j ( x ) | q dµ ) 1 /q for q > 0 . For two sequences, { a n } and { b n } , we write a n ⪯ b n if there exists a positive constant C such that a n ≤ Cb n for large enough n , and we write a n ≍ b n if a n ⪯ b n and b n ⪯ a n .

3 Main Results

3.1 Instance-dependent transition matrix with multiple annotators

Annotator-specific instance-dependent noise transition model. Given an instance x , the conditional probability mass function of noisy annotations can be characterized by R instance-dependent matrices of dimension K × K , termed transition matrices or annotator confusions [8, 13], with the k th row of the r th matrix denoted ( P (˜ y ( r ) = 1 | y = k, x ) , . . . , P (˜ y ( r ) = K | y = k, x ) ) . Thus, the distribution of noisy annotation depends on the instance in different ways due to the differences in the annotator r and the underlying true label y , which can be characterized by a Bayesian generalized linear mixed effects model (GLMM) [29, 30] in the deep learning framework.

Specifically, conditioned on the true label y = k and the feature vector x , we treat the noisy label ˜ y ( r ) from annotator r as a random variable generated from the distribution:

where s ( k,r ) = ( s ( k,r ) 1 , ..., s ( k,r ) K ) ⊤ ∈ S K -1 with S K -1 = { ( s 1 , ..., s K ) ⊤ ∈ R K : s j ≥ 0 for j ∈ [ K ] and ∑ K j =1 s j = 1 } representing the ( K -1) -dimensional simplex, and Cat ( s ( k,r ) ) represents a categorical distribution specified by the parameter vector s ( k,r ) . We extend existing works on mixed effects neural networks (MNN) [31, 32] by employing two nonlinear transformations ψ 1 and ψ 2 to incorporate different effects in the instance-dependent noise transition model, and set

where A ( r ) 0 = ( α ( r ) 10 , ..., α ( r ) K 0 ) ⊤ and B ( k ) 0 = ( β ( k ) 10 , ..., β ( k ) K 0 ) ⊤ are the regression weights; ψ 1 ( x ) and ψ 2 ( x ) can be modeled by some suitable networks; and G is a function mapping R K to S K -1 , which, in practice, is chosen to be the softmax function in the final layer. Utilizing two different network components ψ 1 and ψ 2 enables us to flexibly reflect possibly different effects of the annotator expertise ( r ) and the ground truth ( k ) in the annotation process, which can be interpreted as the input in mixed effects models.

Approximating the transition matrices. The proposed instance-dependent noise transition model can be learned by leveraging anchor points [12, 26, 27], or instances that are similar to anchor points learned from noisy training data [33]. An instance x is defined to be an anchor point of class k if it belongs to the k th class almost surely, that is, P (y = k | x ) = 1 , and hence, P (˜ y ( r ) | x ) = P (˜ y ( r ) | y = k, x ) . For k ∈ [ K ] , let D 0 ,k be the set of anchor points of the k th class and the associated noisy annotations, i.e., D 0 ,k = {{ x i , ˜ y i } : P (y i = k | x i ) = 1 } . Define D 0 = D 0 , 1 ∪ D 0 , 2 ∪ . . . ∪ D 0 ,K , and let n denote the subsample size of the learned anchor points, i.e., the cardinality of D 0 . Paired variables { x i , ˜ y i } in D 0 are independent, but not necessarily identically distributed (i.n.i.d). We write the input dimension p as p n from now on to emphasize that its dependence on n is allowed.

In applications, overfitting can occur when the subsample size n of the learned anchor points is relatively small compared to the main sample size N . To address this issue, we propose to learn ψ j ( x ) with a sparse Bayesian DNN, denoted ψ j ( x ; θ ( j ) ) , where θ ( j ) represents the vector of all involved parameters in the network with j = 1 , 2 . Furthermore, invoking the sparse Bayesian setting allows us to rigorously characterize the distance between the proposed model and the underlying true transition matrices, as presented in Theorem 1 of Section 3.2.

3.2 Bayesian analysis and posterior consistency result

Prior specification. To implement sparse Bayesian analysis, we utilize the spike and slab prior [34] on the network parameters, offering an interpretable mechanism for variable selection. The spike and slab model is formulated by constructing a prior hierarchy of the involved parameters and selects nonzero coefficients according to the posterior inclusion probability. Marginally, these priors are mutually independent and have a mixture distribution consisting of a flat distribution (slab) and a distribution concentrated at zero (spike). Parameters with a small posterior mean will be set to zero to achieve sparsity.

Specifically, for network ψ j ( x ; θ ( j ) ) , we write θ ( j ) as θ ( j ) = ( θ ( j ) 1 , ..., θ ( j ) J j ) ⊤ with J j denoting the length of θ ( j ) for j = 1 , 2 . For k ∈ [ J j ] , we treat θ ( j ) k as a random variable generated from the following prior hierarchy:

where γ ( j ) k ∈ { 0 , 1 } indicates whether or not θ ( j ) k is nonzero, c nj is specified as a very small positive number, σ 2 nj and c nj σ 2 nj are the parameters related to the variances of distributions π 1 ( · ) and π 0 ( · ) , respectively, and λ nj ∈ (0 , 1) determines the ratio of the mixture distribution. As c nj → 0 ,

π 0 ( θ ( j ) k ; c nj σ 2 nj ) becomes the degenerate distribution at zero. The marginal distribution of θ ( j ) k is then determined by

which is presented as π ( j ) ( · ) for short; and this is taken as the prior distribution of θ ( j ) k .

To further incorporate the effects of the true label information and the randomness from different annotators in (3), we place the following probabilistic structure on the generic weights for A ( r ) 0 and B ( k ) 0 in (3), A ( r ) = ( α ( r ) 1 , ..., α ( r ) K ) and B ( k ) = ( β ( k ) 1 , ..., β ( k ) K ) :

for j, k ∈ [ K ] and r ∈ [ R ] , where Σ ( r ) α and Σ ( k ) β are nonnegative definite matrices. We use π ( r ) A ( · ) and π ( k ) B ( · ) to denote the prior distribution of A ( r ) and B ( k ) in (6). Here the regression weights A ( r ) and B ( k ) can be seen as fully connected layers on top of ψ 1 ( x ; θ (1) ) and ψ 2 ( x ; θ (2) ) , respectively. The conditions on the aforementioned priors are given in Appendix A.3.

Prior and posterior probability measure. Let θ = ( θ (1) ⊤ , θ (2) ⊤ , vec( B (1) ) ⊤ , . . . , vec( B ( K ) ) ⊤ , ⊤

vec( A (1) ) ⊤ , . . . , vec( A ( R ) ) ⊤ ) ∈ Θ stand for the vector of all involved parameters in the noise transition model, with Θ denoting the parameter space. We use θ 0 to represent the true value of θ , which is an interior point of Θ . The foregoing specification of the prior distribution places a prior probability measure, denoted Π( · ) , on θ . With the data D 0 , the posterior probability measure Π( ·|D 0 ) is given by

where G is the σ -field on Θ , and p n θ is the joint probability density or mass function for the observations in D 0 under θ . Let P n θ ( · ) denote the probability measure associated with p n θ ( · ) , and write P n 0 ( · ) ≜ P n θ 0 ( · ) . Hence, the data D 0 is generated from P n 0 ( · ) in our setup.

Let f denote the unknown density of x . For { x , ˜ y } ∈ D 0 ,k , let f ( k,r ) 0 and f ( k ) 0 respectively represent the underlying true distributions for ˜ y ( r ) and ˜ y , given { y = k, x } , determined by (2) and (3); and let f ( k,r ) θ and f ( k ) θ denote the corresponding distributions characterized by the model indexed by θ . We let p θ ,i ≜ f ( k i ) θ f denote the probability density or mass function of the i th component in D 0 under θ , with k i ∈ [ K ] denoting the class that the instance belongs to almost surely. Then, the joint probability density or mass function p n θ is calculated as p n θ ≜ ∏ n i =1 p θ ,i . The following theorem describes the closeness of the proposed noise transition model and the true annotator confusions with respect to the Hellinger distance within the Bayesian framework.

Theorem 1. Suppose Conditions A.1-A.4 in Appendix A.2 and B.1-B.3 in Appendix A.3 are satisfied. Let d ( · , · ) and d n ( · , · ) denote the Hellinger distance given in Definition 1 and the semimetric defined in (16) in Appendix B.1, respectively. Then there exists a sequence of constants { ϵ 2 n } ∞ n =1 with ϵ 2 n = O ( ϖ n 1 + ϖ n 2 + ζ n ) and log(1 /ϵ 2 n ) ≺ nϵ 2 n , satisfying 0 < ϵ 2 n < 1 , ϵ n → 0 and nϵ 2 n →∞ as n →∞ , such that 2 ,

in P n 0 probability for every M n →∞ , where { ϖ jn } is a sequence of nonnegative numbers converging to 0 as n → ∞ for j = 1 , 2 as given in (14), and { ζ n } ∞ n =1 is a sequence given in Appendix A.3

depending on the structures of ψ 1 ( ·| θ (1) ) and ψ 2 ( ·| θ (1) ) with ζ n → 0 as n → ∞ . If we further assume that |D 0 ,k | / |D 0 | > ς 1 for some positive constant ς 1 , with | · | representing the cardinality of a set, then for any k ∈ [ K ] and r ∈ [ R ]

Intuitively, Theorem 1 reveals that the sparse noise transition model is close to the underlying true transition matrix with respect to the Hellinger distance under mild conditions. Notably, our posterior consistency result extends the existing theories in sparse Bayesian learning [35-37] to the setup of i.n.i.d observations. Moreover, this result on the convergence rate of the posterior measure allows us to infer the underlying true label from the noisy annotations with a theoretical guarantee on the bounds of the Bayes error, which will be discussed in detail in the following section.

3.3 Pairwise likelihood ratio test for label correction

The asymptotic result (9) in Theorem 1 indicates that for each annotator, the underlying true instancedependent transition matrix can be accurately modeled under the Bayesian framework. This enables us to aggregate and infer the ground truth label from noisy crowdsourced annotations.

A novel label correction algorithm. To highlight the idea, we first assume that the noise transition matrix P (˜ y ( r ) | y = k, x ) , or f k,r 0 ( · ) , is known. To simplify the notation, for each x i in the noisy dataset D , denote τ ( r ) i,kl ≜ τ ( r ) kl ( x i ) ≜ P (˜ y ( r ) = l | y = k, x i ) for i ∈ [ N ] and k, l ∈ [ K ] . We assign class prior [38] ℏ i = ( ℏ i, 1 , ..., ℏ i,K ) ⊤ for the ground truth label for the i th task, where the ℏ i,k for k ∈ [ K ] are nonnegative weights satisfying ∑ K k =1 ℏ i,k = 1 . For each instance, with the class prior and the noise transition matrices, the label correction process can be formulated as a hypothesis testing problem, where different hypotheses are generated from different choices of the true label values. Specifically, selecting the label for the instance x i from { g, g ′ } , with 1 ≤ g < g ′ ≤ K , is equivalent to choosing from the two competitors P (˜ y | y = g, x i ) and P (˜ y | y = g ′ , x i ) . We thereby consider the following hypothesis testing problem: H g : ˜ y i |{ y i , x i } ∼ P (˜ y | y = g, x i ) versus H g ′ : ˜ y i |{ y i , x i } ∼ P (˜ y | y = g ′ , x i ) . By the Neyman-Pearson Lemma [39], the Bayes testing error is minimized by the likelihood ratio test, and the decision region for hypothesis H g is given by

Building from the abovementioned reformulation of the label correction process, we now propose an algorithm to infer the underlying ground truth by aggregating noisy crowdsourced annotations with the help of the annotator confusions. Formally, we propose the following label correction method by setting the estimated label of x i to be y i ≜ g if

̸

where Ω ≥ 1 is a pre-specified threshold.

̸

Information-theoretic bounds on the Bayes error. To theoretically justify the effectiveness of the proposed label correction method (10), we derive information-theoretic bounds on the Bayes error, given the instances. Let D = { x i , y i } ¯ n i =1 denote the collection of instances with estimated labels, where ¯ n represents the size of D . We write y = { y i } ¯ n i =1 and the corresponding true label is denoted y = { y i } ¯ n i =1 . A loss measured by the accuracy of the estimated labels is given by L ( y , y ) = 1 ¯ n ∑ ¯ n i =1 1 (y i = y i ) . Let P ( ·| y ; τ ) denote the joint probability distribution of { ˜ y i } , given y and τ , and let E ( ·| y ; τ ) denote the associated expectation operator, where τ ≜ { τ i } ¯ n i =1 ≜ { τ ( r ) i :

r ∈ [ R ] } ¯ n i =1 represents the collection of the corresponding transition matrices τ ( r ) i having τ ( r ) i,kl as its ( k, l ) element. Then, the Bayes risk is defined as [40]

or ℜ Bayes for short, where ℏ ( y ) is the joint prior probability of y calculated from ℏ ≜ { ℏ i } ¯ n i =1 . The following theorem identifies bounds for the Bayes risk.

Theorem 2. Let D KL ( τ ( r ) i,g ∗ ∥ τ ( r ) i,g ′ ∗ ) denote the Kullback-Leibler (KL) divergence for discrete distributions τ ( r ) i,g ∗ and τ ( r ) i,g ′ ∗ , where, for i ∈ [ n ] , r ∈ [ R ] , and g, g ′ ∈ [ K ] , τ ( r ) i,g ∗ and τ ( r ) i,g ′ ∗ stand for the g th and g ′ th rows of τ ( r ) i , respectively. For ℏ = { ℏ i } n i =1 and τ = { τ i } n i =1 , define

̸

For i ∈ ¯ n and g ∈ [ K ] , let I ( g ) Ω ( ℏ i , τ i ) = min g ′ = g C ( i ) gg ′ . Then the Bayes error defined in (11) is bounded as follows:

Remark 1 . Theorem 2 establishes information-theoretic bounds on the Bayes error ℜ Bayes for arbitrary priors ℏ i with i ∈ [ n ] , which theoretically quantifies the combined impact of the prior knowledge and annotators' expertise on label accuracy using algorithm (10). The lower bound is proved in light of the concept of f -informativity [40-42], and is stronger than the commonly-used Bayes lower bound based on Fano's inequality [43, 44]. The proof of the upper bound considers the inference procedure of y and applies Markov's inequality. The details are given in Appendix B.

Remark 2 . The quantity C ( i ) gg ′ in Theorem 2 reflects how the identified upper bound of the Bayes error may be influenced by the prior ℏ i and the ability of the R annotators to distinguish between labels g and g ′ for instance i . If we set ℏ i to be the uniform prior and let Ω = 1 , C ( i ) gg ′ will degenerates to the average of the Chernoff information between { τ ( r ) i,g ∗ } R r =1 and { τ ( r ) i,g ′ ∗ } R r =1 [45, 46], which is a statistical divergence measuring the deviation between two probability distributions.

Result under the sparse Bayesian model. The label correction method (10) is not immediately applicable if we have no access to the underlying true annotator confusions, which is usually the case in real-world applications. To get around the issue induced from the unknownness of the true noise transition probability f ( k,r ) 0 , we consider the model f ( k,r ) θ given in Theorem 1 and write the corresponding transition matrices as τ ( r ) i,kl ≜ τ ( r ) kl ( x i ) ≜ f ( k,r ) θ (˜ y | x i ) ∣ ∣ ˜ y = l for k, l ∈ [ K ] and r ∈ [ R ] .

We then replace the τ ( r ) i,gl in (10) with τ ( r ) i,gl to determine corrected labels. With a slight abuse of notation, we still use D = { x i , y i } ¯ n i =1 to denote the set of instances with estimated labels but let ℜ Bayes denote the resulting Bayes error. We let D x = { x i } n i =1 represent the set of the considered instances. Combining Theorems 1 and 2 yields the following corollary.

Corollary 3. Suppose that the conditions in Theorem 1 are satisfied, and further assume that f ( x i ) > ς 2 for x i ∈ D x and min r ∈ [ R ] ,i ∈ [ n ] ,k,l ∈ [ K ] τ ( r ) i,kl ≥ ς 3 , where f ( · ) is the probability density

function of x , and ς 2 and ς 3 are some positive constants. Then, for any ϵ > 0 ,

in P n 0 probability as n →∞ , where I ( g ) Ω ( ℏ i , τ i ) is given in Theorem 2 for i ∈ [¯ n ] and g ∈ [ K ] .

4 Algorithm

Learning the noise transition model. In the warm-up stage, we train the base models on noisy training data and obtain the set of anchor points D 0 [26]. With D 0 , we first obtain the maximum a posteriori (MAP) estimate of network parameters of the transition model θ by maximizing the log posterior distribution of θ , with the constant term omitted,

where p θ ,i is the probability mass function of the i th component in D 0 given before Theorem 1, and π ( θ ) is the probability density function of θ relative to the prior probability measure Π( · ) . Given the MAP estimate ̂ θ , according to the prior hierarchy (4), the posterior inclusion probability of the k th parameter in the network ψ j ( · ; θ ( j ) ) is calculated as

for k ∈ [ J j ] with j = 1 , 2 . If the posterior inclusion probability is smaller than a pre-specified threshold, chosen to be 0.5 in our experiments, the associated parameter is zet to be zero. We then fine tune the sparse network and obtain the noise transition model.

Training the classifiers with corrected labels. With the noise transition model trained, we then train the base models with the label correction algorithm proposed in Section 3.3. Specifically, we train two base classifiers to reciprocally provide class priors for each instance in the label correction process. In the t th epoch, for instance x i , if y i satisfies the condition (10) for the pre-specified threshold Ω t , we put { x i , y i } in D t . The base models are then updated on the collected dataset D t . The complete pseudocode of our algorithm is included in Appendix C.

5 Experiments

Datasets. We assess the effectiveness of our method on three image datasets with synthetic annotations, MNIST [47], CIFAR-10 [48], and CIFAR-100 [48], and two datasets with human annotations, CIFAR-10N [49] and LabelMe [50, 51]. Detailed information can be found in Appendix C. For all the datasets except LabelMe, we leave out 10% of the training data as a noisy validation set.

Noise generation. For the three datasets, MNIST, CIFAR10, and CIFAR100, we consider three groups of annotators with varying expertise, with an average labeling error rate of about 20%, 35% and 50%, respectively. We abbreviate these three groups as IDN-LOW, IDN-MID, and IDN-HIGH, which represent instance-dependent annotators with low, middle ('mid" for short), and high labeling error rates, respectively. To generate noisy annotations, we independently simulate R = 5 annotators for each group according to Algorihtm 2 in [12], with IDNτ denoting that the noise rate is upper bounded by τ for each annotator. For each instance, we then randomly choose one of the annotations given by the R annotators, which is designed to evaluate the methods under incomplete annotator labeling. We manually corrupt the three datasets according to the following three groups of annotators:

(I) IDN-LOW. IDN-10%, IDN-10%, IDN-20%, IDN-20%, IDN-30%;

(II) IDN-MID. IDN-30%, IDN-30%, IDN-40%, IDN-40%, IDN-50%;

(III) IDN-HIGH. IDN-50%, IDN-50%, IDN-60%, IDN-60%, IDN-70%.

Experiment setup. The network structure for the MNIST dataset is chosen to be Lenet-5 [52]. We choose ResNet-18 [2] for CIFAR-10 and CIFAR-10N, and ResNet-34 architecture [2] for CIFAR-100 dataset. As in [53], we employ the pretrained VGG-16 network followed by a fully connected layer and a softmax output layer for the LabelMe dataset, using 50% dropout. More implementation details can be found in Appendix C.

Competing methods. We compare the proposed method with the following state-of-art methods: (1) CE (Clean), which trains the network with the standard cross entopy loss on the clean datasets; (2) CE (MV), which trains the network using the labels from majority voting; (3) CE (EM) [9]; (4) DoctorNet [54]; (5) GCE [55]; (6) Co-teaching [56]; (6) Co-teaching+ [57]; (7) BLTM [17]; (8) MBEM[11]; (9) CrowdLayer [53]; (10) TraceReg [8]; (11) Max-MIG [7]; (12) CoNAL [58]; (13) GeoCrowdNet (F) [13]; and (14) GeoCrowdNet (W) [13]. Among these methods, GCE, Co-teaching, Co-teaching+, and BLTM are strong baselines dealing with single noisy label issues, and we adapt them to the multiple annotations setting by utilizing the majority vote labels for loss computation.

Ablation study. In Figure 1, We plot the average estimation error for the noise transition matrices to demonstrate the effectiveness of the proposed method in modeling the instance-dependent annotator confusions. For instance x i with clean class label y i in the validation set, we analyse the y i th row rather than the whole noise transition matrix as in previous studies [16, 12]. Specifically, let ̂ τ ( r ) ( x i ) and τ ( r ) ( x i ) represent the estimated and the true noise transition matrix for annotator r , respectively. The estimation error on instance x i is defined as err ( r ) i = max l ∈ K | ̂ τ ( r ) ( x i ) y i ,l -τ ( r ) ( x i ) y i ,l | , where ̂ τ ( r ) ( x i ) y i ,l and τ ( r ) ( x i ) y i ,l stand for the (y i , l ) element in the corresponding matrices. The average estimation error for annotator r is then calculated as 1 n v ∑ n v i =1 err ( r ) i with n v denoting the size of the validation set. For each annotator, we compare the average estimation error of the proposed method with six baselines, CrowdLayer [53], TraceReg [8], GeoCrowdNet (F) [13], GeoCrowdNet (W) [13], MBEM [11], and BLTM [17], on the CIFAR10 dataset. In most of the cases, the proposed method leads to smaller estimation error especially when the noise rate is high, which shows the efficacy of the proposed sparse Bayesian model.

1

2

3

4

5

Annotator

0.10

0.15

0.20

0.25

0.30

0.35

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(a) IDN-LOW

1

2

3

4

5

Annotator

0.20

0.25

0.30

0.35

0.40

0.45

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(b) IDN-MID

1

2

3

4

5

Annotator

0.30

0.35

0.40

0.45

0.50

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(c) IDN-HIGH

5

10

30

50

100

Number of Annotators

72.5

75.0

77.5

80.0

82.5

85.0

87.5

CE(Clean)

CE(MV)

CE(EM)

DoctorNet

GCE

Co-teaching

Co-teaching+

BLTM

MBEM

CrowdLayer

TraceReg

CoNAL

GeoCrowdNet (F)

GeoCrowdNet (W)

Ours

(a) IDN-LOW

5

10

30

50

100

Number of Annotators

65

70

75

80

85

CE(Clean)

CE(MV)

CE(EM)

DoctorNet

GCE

Co-teaching

Co-teaching+

BLTM

MBEM

CrowdLayer

TraceReg

CoNAL

GeoCrowdNet (F)

GeoCrowdNet (W)

Ours

(b) IDN-MID

5

10

30

50

100

Number of Annotators

45

50

55

60

65

70

75

80

85

CE(Clean)

CE(MV)

CE(EM)

DoctorNet

GCE

Co-teaching

Co-teaching+

BLTM

MBEM

CrowdLayer

TraceReg

CoNAL

GeoCrowdNet (F)

GeoCrowdNet (W)

Ours

(c) IDN-HIGH

Classification accuracy. Table 1 presents the average test accuracy of 5 random trials on the datasets of CIFAR-10, CIFAR-100, CIFAR-10N and LabelMe, together with the standard errors of the test accuracies of the random trials, expressed after the plus/minus sign ± , where the two highest accuraries are bold faced; standard errors of the accuracies are calculated based on repeating those experiments 5 times, each with a different random seed. All the results demonstrate the superior performance of the proposed method on both synthetic and real-world noisy datasets. Moreover, to investigate the influence of the sparsity of annotations, we conduct more experiments with the number of annotators varying from 5 to 100, and each instance only has one label. Figure 2 shows the average accuracy with various numbers of annotators, which further exhibit the advantages of the proposed method under different settings. Additional experimental results, including the test accuracy on MNIST, the average estimation error on MNIST and CIFAR100, and the accuracy of the corrected labels using algorithm (10), are deferred to Appendix C to save space.

6 Conclusion

In this paper, we address the challenge of training classifiers using noisy crowdsourced labels, a common issue in various applications. We formulate the annotator-specific instance-dependent noise transition matrix within the Bayesian framework, and theoretically characterize the closeness of the proposed model and the true annotator confusions with respect to the Hellinger distance. Our result is established for the setup of i.n.i.d. observations, which substantially broadens the application scope of our method. Building on the convergence rate of the posterior measure, we propose a novel algorithm to aggregate noisy annotations and infer the ground truth label based using pairwise LRT. Additionally, we provide information-theoretic bounds on the Bayes error of the proposed algorithm. Empirical evidence demonstrates the effectiveness of our algorithm on both synthetic and real-world noisy datasets.

Limitations and Extensions

Our work can be further extended in different directions. It is interesting to generalize the setup here to the hierarchical classification setup. Instance-dependent transition matrices can be further refined with varying structures imposed and are learned with manifold regularization.

References

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine Learning , 95:291-327, 2014.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning , 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

James Surowiecki. The Wisdom of Crowds . Anchor, 2005.

Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. Max-mig: an information theoretic approach for joint learning from crowds. arXiv preprint arXiv:1905.13436 , 2019.

Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silberman. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C , 28(1):20-28, 1979.

Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems , 22, 2009.

Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. arXiv preprint arXiv:1712.04577 , 2017.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems , 32, 2019.

Shahana Ibrahim, Tri Nguyen, and Xiao Fu. Deep learning from crowdsourced labels: Coupled crossentropy minimization, identifiability, and regularization. In The International Conference on Learning Representations , 2023.

Andrew B Rosenkrantz, Ruth P Lim, Mershad Haghighi, Molly B Somberg, James S Babb, and Samir S Taneja. Comparison of interreader reproducibility of the prostate imaging reporting and data system and likert scales for evaluation of multiparametric prostate mri. American Journal of Roentgenology , 201(4):W612-W618, 2013.

Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical Image Analysis , 65:101759, 2020.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems , 33:7597-7610, 2020.

Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instancedependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning , 2022.

Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Cicarrelli, Frederik Barkhof, and Daniel Alexander. Disentangling human error from ground truth in segmentation of medical images. Advances in Neural Information Processing Systems , 33:15750-15762, 2020.

Peter Welinder, Steve Branson, Pietro Perona, and Serge Belongie. The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems , 23, 2010.

Paul Ruvolo, Jacob Whitehill, and Javier R Movellan. Exploiting structure in crowdsourcing tasks via latent factor models. Citeseer , 2010.

Yan Yan, Rómer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer Dy. Modeling annotator expertise: Learning when everybody knows a bit of something. In Proceedings of the International Conference on Artificial Intelligence and Statistics , 2010.

Wei Bi, Liwei Wang, James T Kwok, and Zhuowen Tu. Learning to predict from crowdsourced data. In The Conference on Uncertainty in Artificial Intelligence , 2014.

Joseph Jacob, Olga Ciccarelli, Frederik Barkhof, and Daniel C Alexander. Disentangling human error from the ground truth in segmentation of medical images. In Advances in Neural Information Processing Systems , 2021.

Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Marc Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane S Boning. Learning from multiple annotator noisy labels via sample-wise label fusion. In European Conference on Computer Vision , 2022.

Jingzheng Li, Hailong Sun, and Jiyi Li. Beyond confusion matrix: learning from multiple annotators with awareness of instance features. Machine Learning , 112(3):1053-1075, 2023.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38(3):447-461, 2015.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Neal S Grantham, Yawen Guan, Brian J Reich, Elizabeth T Borer, and Kevin Gross. Mimix: A bayesian mixed-effects model for microbiome data from designed experiments. Journal of the American Statistical Association , 115(530):599-609, 2020.

Wei Liu, Huazhen Lin, Shurong Zheng, and Jin Liu. Generalized factor model for ultra-high dimensional correlated variables with mixed types. Journal of the American Statistical Association , 118:1-17, 2021.

Yunyang Xiong, Hyunwoo J Kim, and Vikas Singh. Mixed effects neural networks (menets) with applications to gaze estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Giora Simchoni and Saharon Rosset. Using random effects to account for high-cardinality categorical features and repeated measures in deep neural networks. Advances in Neural Information Processing Systems , 34:25111-25122, 2021.

Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learning , 2020.

Hemant Ishwaran and J Sunil Rao. Spike and slab variable selection: frequentist and bayesian strategies. The Annals of Statistics , 33(2):730-773, 2005.

Yan Sun, Qifan Song, and Faming Liang. Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association , 117(540):1-15, 2021.

Yuexi Wang and Veronika Rocková. Uncertainty quantification for sparse deep learning. In International Conference on Artificial Intelligence and Statistics , 2020.

Yan Sun, Wenjun Xiong, and Faming Liang. Sparse deep learning: A new framework immune to local traps and miscalibration. Advances in Neural Information Processing Systems , 34:22301-22312, 2021.

Shantanu Jain, Martha White, and Predrag Radivojac. Estimating the class prior and posterior from noisy positives and unlabeled data. Advances in neural information processing systems , 29, 2016.

Jerzy Neyman and Egon Sharpe Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A , 231(694-706):289-337, 1933.

Xi Chen, Adityanand Guntuboyina, and Yuchen Zhang. On bayes risk lower bounds. The Journal of Machine Learning Research , 17(1):7687-7744, 2016.

Imre Csiszár. A class of measures of informativity of observation channels. Periodica Mathematica Hungarica , 2(1-4):191-213, 1972.

Anuran Makur and Lizhong Zheng. Comparison of contraction coefficients for f-divergences. Problems of Information Transmission , 56:103-156, 2020.

Hideaki Imamura, Issei Sato, and Masashi Sugiyama. Analysis of minimax error rate for crowdsourcing and its application to worker clustering model. In International Conference on Machine Learning , 2018.

Bin Yu. Festschrift for Lucien Le Cam , chapter 29. Springer, 1997.

MTCAJ Thomas and A Thomas Joy. Elements of information theory . Wiley-Interscience, 2006.

Chao Gao, Yu Lu, and Dengyong Zhou. Exact exponent in optimal rates for crowdsourcing. In International Conference on Machine Learning , pages 603-611. PMLR, 2016.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations , 2022.

Filipe Rodrigues, Mariana Lourenco, Bernardete Ribeiro, and Francisco C Pereira. Learning supervised topic models for classification and regression from crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(12):2409-2422, 2017.

Antonio Torralba, Bryan C Russell, and Jenny Yuen. Labelme: Online image annotation and applications. Proceedings of the IEEE , 98(8):1467-1484, 2010.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. Who said what: Modeling individual labelers improves classification. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems , 31, 2018.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems , 2018.

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning , pages 7164-7173. PMLR, 2019.

Zhendong Chu, Jing Ma, and Hongning Wang. Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Kolyan Ray, Botond Szabó, and Gabriel Clara. Spike and slab variational bayes for high dimensional logistic regression. Advances in Neural Information Processing Systems , 33:14423-14434, 2020.

Wenxin Jiang. Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities. The Annals of Statistics , 35(4):1487-1511, 2007.

Nicholas G Polson and Veronika Roˇ cková. Posterior concentration for sparse deep learning. Advances in Neural Information Processing Systems , 31, 2018.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems , 28, 2015.

Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine , 35(1):126-136, 2018.

Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning , 2019.

Philipp Petersen and Felix Voigtlaender. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks , 108:296-330, 2018.

Helmut Bolcskei, Philipp Grohs, Gitta Kutyniok, and Philipp Petersen. Optimal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science , 1(1):8-45, 2019.

Minerva Mukhopadhyay and David B Dunson. Targeted random projection for prediction from highdimensional features. Journal of the American Statistical Association , 115(532):1998-2010, 2020.

Michael R Sampford. Some inequalities on mill's ratio and related functions. The Annals of Mathematical Statistics , 24(1):130-132, 1953.

Rajarshi Guhaniyogi and David B Dunson. Bayesian compressed regression. Journal of the American Statistical Association , 110(512):1500-1514, 2015.

Abdulhamid A Alzaid and Maha A Omair. On the poisson difference distribution inference and applications. Bulletin of the Malaysian Mathematical Sciences Society. Second Series , 33(1):17-45, 2010.

C. M. Joshi and S. K. Bissu. Some inequalities of bessel and modified bessel functions. Journal of the Australian Mathematical Society , 50(2):333-342, 1991.

Martin J Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge University Press, 2019.

Subhashis Ghosal and Aad Van Der Vaart. Convergence rates of posterior distributions for noniid observations. The Annals of Statistics , 1:192-223, 2007.

Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological) , 28(1):131-142, 1966.

John Duchi. Information theory and statistics. Lecture Notes for Statistics , 311, 2019.

Michael Short. Improved inequalities for the poisson and binomial distribution and upper tail quantile functions. International Scholarly Research Notices , 2013.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Supplemental Materials

In the supplementary materials, we first summarize the regularity conditions on the underlying true model and prior distributions of the network parameters in Section A. The proofs of Theorem 1, Theorem 2, and Corollary 3 are provided in Sections B.2-B.4 with all the needed preliminaries presented in Section B.1. Implementation details of the proposed method and additional experiment results are exhibited in Section C, including the accuracy and number of selected labels using the proposed label correction algorithm (10), the test accuracy on MNIST, the hyperparameter analysis on CIFAR100, the classification accuracy on CIFAR100 with varying number of annotators, and the average estimation error on CIFAR10 and CIFAR100 with varying number of annotators.

A Regularity Conditions

A.1 Network structure

To incorporate the sparse high dimensional setting [59], we utilize sparse Bayesian DNNs to reconstruct ψ 1 ( x ) and ψ 2 ( x ) in (3) [37, 60]. Specifically, to approximate ψ j ( x ) with j = 1 , 2 , we consider a network with H nj -1 hidden layers and a width vector L ( j ) = ( L ( j ) 0 , L ( j ) 1 , ..., L ( j ) H nj ) ⊤ , where the width of the h th layer is denoted L ( j ) h for h = 0 , ..., H nj with L ( j ) 0 = p n for the input layer and L ( j ) H nj ≜ M ( j ) for the output layer. Then the DNN with network architecture { H nj , L ( j ) } is the nonlinear function of the form:

̸

where for h = 1 , ..., H nj , W ( j,h ) is a L ( j ) h × L ( j ) h -1 weight matrix, b ( j,h ) ∈ R L ( j ) h is the bias of layer h , σ ( · ) is a nonlinear activation function, and θ ( j ) represents the J nj × 1 vector formed from stacking { W ( j,h ) , b ( j,h ) } H nj h =1 from bottom to the top, with J nj ≜ ∑ H nj h =1 ( L ( j ) h -1 × L ( j ) h + L ( j ) h ) . We treat weights and biases equally without distinguishing them in θ ( j ) , and write θ ( j ) as θ ( j ) = ( θ ( j ) 1 , ..., θ ( j ) J nj ) ⊤ . Let γ ( j ) = ( γ ( j ) 1 , ..., γ ( j ) J nj ) ⊤ denote the indicator vector, with γ ( j ) k = 1 ( θ ( j ) k = 0) for k = 1 , ..., J nj . For ease of presentation, we use ψ j ( x ; θ ( j ) ) and ψ j ( x ; θ ( j ) , γ ( j ) ) exchangeably to represent the model for ψ j ( x ) , and let F n = F ( H n 1 , H n 2 , L (1) , L (2) , C 1 , C 2 , ϵ 1 ) denote the space of all sparse networks that satisfy Condition A.3 in Appendix A.2 and are constrained by positive constants C 1 , C 2 and ϵ 1 .

To determine the parameters of the sparse DNNs ψ j ( x ; θ ( j ) , γ ( j ) ) that best approximate ψ j ( x ) for j = 1 , 2 , we define

where for j = 1 , 2 , ϖ nj is an n -dependent positive constant satisfying ϖ nj → 0 as n →∞ 3 We call θ (1) ∗ , γ (1) ∗ , θ (2) ∗ , and γ (2) ∗ the true parameters of θ (1) , γ (1) , θ (2) , and γ (2) , respectively.

A.2 Conditions for the sparse GLM

A.1 The input vector is standardized so that x ∈ Ω ⊂ [ -1 , 1] p n , where Ω is the support of x , and the probability density f ( · ) of x satisfies that sup x ∈ Ω | f ( x ) | ≤ C 0 for some positive constant C 0 .

A.2 The activation function σ ( · ) is 1-Lipschitz.

A.3 The true sparse DNN model satisfies the following conditions.

A.3.1 For j = 1 , 2 , let r nj = ∥ γ ( j ) ∥ 1 denote the connectivity of γ ( j ) , let L nj = max 1 ≤ h ≤ H nj L ( j ) h denote the maximum layer width, and let s nj represent the input dimension of γ ( j ) . Let ζ n = { ( r n 1 + r n 2 )( H n 1 + H n 2 ) log n + ( r n 1 + r n 2 )(log L n 1 +log L n 2 )+( s n 1 + s n 2 ) log p n } /n . The true sparse DNN model satisfies that ζ n ≤ C 1 n -ϵ 1 for some constants C 1 > 0 and 0 < ϵ 1 < 1 .

A.3.2 For j = 1 , 2 , ∥ θ ( j ) ∥ ∞ ≤ E nj , where positive constant E nj ≤ n C 2 C 2 > 0 .

for some constant

A.4 Write the function G ( · ) in (3) as ( G 1 ( · ) , ..., G K ( · )) ⊤ . For x ∈ Ω , r ∈ [ R ] and k ∈ [ K ] , write ω ( k,r ) = A ( r ) T ψ 1 ( x ; θ (1) ) + B ( k ) T ψ 2 ( x ; θ (2) ) .

A.4.1 For k ∈ [ K ] and r ∈ [ R ] , in the neighbourhood of θ 0 ,

for some constants C 3 > 0 and ϵ 2 ∈ (0 , 1) , where the latter requirement can be achieved if G j ( ω k,r 0 ) > ς for some positive constant ς > 0 .

A.4.2 For r ∈ [ R ] and j, k ∈ [ K ] , ∥ α ( r ) j 0 ∥ ∞ ≤ F 1 and ∥ β ( k ) j 0 ∥ ∞ ≤ F 2 , where F 1 and F 2 are positive constants.

A.3 Conditions for the prior

B.1 For j = 1 , 2 , assume each element of θ has independent continuous prior distribution, denoted π ( j ) θ ( · ) . Thus, its minimum value on the interval [ -E nj -1 , E nj +1] exits, and let π ( j ) θ denote it. For a sequence of positive constants I ( j ) n with log I ( j ) n = O (log n ) , let δ nj and δ ′ nj be two sequences of constants satisfying that δ nj < 1 /nJ nj ( c 0 I ( j ) n ) H nj ( n/H nj ) H nj and δ ′ nj < 1 /nJ nj ( c 0 E nj ) H nj ( r nj /H nj ) H nj for some constant c 0 > 1 , respectively. Assume that:

( j )

for some constant S ( j ) 0 > 2 ;

B.2 For k ∈ [ K ] , let B ( k ) n and B ( k ) n denote the largest and the smallest eigenvalues of Σ ( k ) β , respectively, and for r ∈ [ R ] , let A ( r ) n and A ( r ) n denote the largest and the smallest eigenvalues of Σ ( r ) α , respectively. Assume that for large enough n ,

B.2.1 A ( r ) n ≤ S (1) 1 M (1) q 1 and A ( r ) n ≥ S (1) 1 { log M (1) } -1 for some positive constants S (1) 1 , S (1) 2 and q 1 .

B.2.2 B ( k ) n ≤ S (2) 1 M (2) q 1 and B ( k ) n ≥ S (2) 2 { log M (2) } -1 for some positive constants S (2) 1 , S (2) 2 and q 2 ;

A.4 Remark

Assumption A.1 specifies that the hypothesis set we consider is a class of DNNs, which is a common setting in the literature [61, 35].

Assumption A.2 ensures that the underlying noise transition probability can be approximated by a sparse model. Existing works [62-64] empirically show that large DNNs often contain a large number of redundant parameters and propose methods for compressing neural networks without affecting performance. Moreover, theoretical works [65, 66] in approximation theory provide theories that guarantee uniform approximation rates for a broad family of function classes. Similar assumptions can be found in [35, 67].

Assumption A.3 specifies the constraints on the prior distribution we use. As in our experiments in Section 5, we may employ the spike-and-slab prior λ n N (0 , σ 2 1 n ) + (1 -λ n ) N (0 , σ 2 0 n ) for each element of the parameter vector θ ( j ) of the sparse Bayesian DNN ψ j (x; θ ( j ) ) , with j = 1 , 2 ; we take the normal prior N (0 , σ 2 n ) for each element of the regression weights. It can be verified that Condition B.1 in A.3 is satisfied if the values of λ n , σ 1 n , and σ 0 n are properly chosen [35]. In particular, the value of λ n is related to the sparsity of the model and we require it to satisfy that λ n = O (1 /J nj [ n H n 1 + H n 2 ( L n 1 + L n 2 ) p n )] c ) for some positive constant c and j = 1 , 2 , which should be chosen by considering the network structure and the number of data points, n . Moreover, by using techniques such as Mill's ratio [68], Condition B.2 in A.3 is satisfied if c 1 < σ n < c 2 for some positive constants c 1 and c 2 , which are related to S ( j ) 1 , S ( j ) 2 , q j and M ( j ) for j = 1 , 2 in Condition B.2 [60, 67]. Similar assumptions can be found in [60, 35, 67].

B Proofs

In this section, we present the proofs of Theorem 1, Theorem 2, and Corollary 3. Specifically, we provide all the need preliminaries in Section B.1, where the definitions and results in Sections B.1.1-B.1.4 will be used in the proof of Theorem 1, and the information-theoretical definitions and lemmas provided in Section B.1.5 will be utilized in the proof of Theorem 2 and Corollary 3.

B.1 Preliminaries

B.1.1 Definitions of some discrepancy measures

Definition 1 ([69]) . Let f and f 0 denote two conditional probability density/mass functions of ˜ y , given x . Let ν 1 ( d x ) denote the probability measure for x associated with the density f ( x ) and let ν 2 ( d ˜ y ) = ⊗ R r =1 ν 2 ,r ( d ˜ y ( r ) ) be a dominating measure for f and f 0 , and hence, a dominating measure of ( x , ˜ y ) is taken as the product ν 1 ( d x ) ν 2 ( d ˜ y ) .

(i) The Hellinger distance between f and f 0 is defined as

(ii) For any t > 0 , define

(iii) The Kullback-Leibler divergence between f and f 0 is defined as

(iv) For q > 1 , define

For q = 2 , the index of V 2 ( · , · ) is omitted and the discrepancy measure is denoted V ( · , · ) .

[60] shows that (1) d ( f, f 0 ) ≤ √ d 0 ( f, f 0 ) ; (2) d t ( f, f 0 ) decreases to d 0 ( f, f 0 ) as t decreases to 0.

B.1.2 Mathematical details about the regression weights

Adapting the proof in [69], we prove the following proposition.

Proposition 1. Assume that β ∼ N ( 0 , D β ) , where D β is a positive definite matrix. Then, for any given dim ( β ) × 1 vector of functions ϕ ( x ; θ ) of x and θ , and for any constant ∆ > 0 ,

where B and B are the largest and the smallest eigenvalues of D β , respectively.

Proof. The proof is established in two steps.

Step 1. We first prove that

where X ∼ Pois ( ∆ 1 2 ) and Y ∼ Pois ( λ 2 ) , with ∆ 1 = ∆ 2 ϕ ( x ; θ ) T D β ϕ ( x ; θ ) and λ = { ϕ ( x ; θ ) T β 0 } 2 ϕ ( x ; θ ) T D β ϕ ( x ; θ ) , and X and Y are independent.

By the definition of the noncentral chi-squared distribution, it can be easily seen that T ≜ { ϕ ( x ; θ ) T β -ϕ ( x ; θ ) T β 0 } 2 ϕ ( x ; θ ) T D β ϕ ( x ; θ ) is distributed according to the noncentral chi-squared distribution χ 2 1 ( λ ) . Thus, by utilizing the cumulative distribution function (CDF) of χ 2 1 ( λ ) , we obtain that

where Q ( · ; 1 + 2 j ) is the CDF of χ 2 1+2 j , the central chi-squared distribution with 1 + 2 j degrees of freedom.

Noting that Q (∆ 1 ; 1 + 2 j ) > Q (∆ 1 ; 2 + 2 j ) , by the result that Z j 1 + ... + Z j,j +1 ∼ χ 2 2+2 j if Z j 1 , ..., Z j,j +1 i . i . d . ∼ χ 2 2 d ≡ exp( 1 2 ) , we obtain that Q (∆ 1 ; 1 + 2 j ) > P { Z j 1 + ... + Z j,j +1 < ∆ 1 } . According to the relationship between the Poisson counting process and exponential variables, we have that the counting process with Z j 1 , ..., Z j,j +1 as inter-arrival times is the Poisson process with rate 1 2 . Let N (∆ 1 ) denote the total number of occurrences or events that have happened up to time ∆ 1 . Then, N (∆ 1 ) follows the Poisson distribution with parameter ∆ 1 2 , i.e., N (∆ 1 ) d ≡ X , and { Z j 1 + ... + Z j,j +1 < ∆ 1 } = { N (∆ 1 ) > j +1 } . Consequently, we have that

where we use the distribution assumptions and the independence assumption for X and Y . Thus, (15) is proved.

Step 2. Since X ∼ Pois ( ∆ 1 2 ) and Y ∼ Pois ( λ 2 ) , and they are independent, we have that X -Y follows the Poisson difference distribution (aka the Skellam distribution): for k = 0 , ± 1 , ± 2 , ... ,

where I ν ( z ) = ∑ ∞ m =0 ( z/ 2) ν +2 m m !Γ( ν + m +1) is the modified Bessel function of the first kind [70]. Using the fact that I ν ( z ) > z ν 2 ν Γ( ν +1) for z > 0 [71] and plugging in ∆ 1 and λ , we obtain that

where the last inequality follows from the fact that for a symmetric matrix D , sup ∥ u ∥ =1 u T Du = λ max ( D ) and inf ∥ u ∥ =1 u T Du = λ min ( D ) , where λ max ( D ) and λ min ( D ) represent the largest and the smallest eigenvalues of D , respectively. This completes the proof.

B.1.3 Mathematical details about the sparse Bayesian DNNs

Consider a DNN with network architecture ( H n , L ) , where H n -1 is the number of hidden layers, and L = ( L 0 , L 1 , ..., L H n ) T is the width vector with L 0 = p n for the input layer and L H n = M for the output layer. For the corresponding indicator vector γ and for h = 1 , ..., H n , let r h denote the number of nonzero connections to the h th hidden layer which includes the bias for the h th hidden layer and the weights between the ( h -1) th and the h th layer, such that ∑ r h = | γ | 1 . = r n . For a parameter vector θ , let O h,j ( θ , x ) represent the output value of the j th node of the h th hidden layer for j = 1 , ..., L h .

Lemma 1 ([37], Lemma S1) . Consider a sparse DNN with parameter vector θ = ( θ 1 , ..., θ q ) ⊤ with dimension q , the corresponding indicator vector γ and the network architecture as mentioned above. Suppose that Conditions A.1-A.2 in Section A.2 are satisfied and | θ | ∞ ≤ E n for a positive constant E n . Then, for 1 ≤ h ≤ H n , the summation of the outputs of the h th layer is upper bounded by

Lemma 2 ([37], Lemma S2) . Consider a sparse DNN ϕ ( θ , x ) with indicator vector γ satisfying the conditions in Lemma 1, and a DNN ϕ ( ¯ θ , x ) with ¯ θ ∈ A , where A is defined as

for given δ 1 > 0 and δ 2 > 0 . Then

B.1.4 A useful lemma

Assume P n is a sequence of sets of probability densities and P c n denotes the complement of P n for each n . Let ϵ n be a sequence of positive numbers. An ϵ n -cover of P n with respect to (w.r.t.)

distance d is a set { f 1 , ..., f k } ⊂ P n such that for each f ∈ P n , there exists j ∈ { 1 , ..., k } such that d ( f, f j ) ≤ ϵ n . The ϵ n -covering number is the cardinality of the smallest ϵ n -cover [72]. Let N ( ϵ n , P n , d ) denote the ϵ n -covering number of P n w.r.t. the distance d .

Consider a vector of independently (not necessarily identically) distributed observations D n , where the i th component is generated from distribution P θ ,i with the density p θ ,i relative to a σ -finite measure ν i on ( X i , A i ) for i ∈ [ n ] , and θ is the vector of parameters in the parameter space Θ . We define P n θ to be the product measure ⊗ n i =1 P θ ,i on the corresponding product measurable space ⊗ n i =1 ( X i , A i ) . Assume that D n is generated from the true distribution P n θ 0 . We define the square of the semimetric d n as in [73]:

which can be seen as the average of the squares of the Hellinger distances. For ϵ > 0 , we define the ϵ -neighborhood around θ 0 :

where K i ( θ , θ 0 ) = K ( P θ ,i , P θ 0 ,i ) and V i ( θ , θ 0 ) = V ( P θ ,i , P θ 0 ,i ) , defined in Definition 1.

Let Π( · ) denote the prior probability measure on θ , and let Π( ·|D n ) represent the associated posterior measure given the data D n . For ease of exposition, we put Π(Θ ∗ ) for Π( { θ ∈ Θ ∗ } ) for any Θ ∗ ⊂ Θ . The following lemma is modified from Theorem 4 in [73] and will be used in the proof of Theorem 1.

Lemma 3. Suppose that for a sequence of sets Θ n ⊂ Θ and for a sequence of positive numbers { ϵ n } ∞ n =1 such that ϵ n → 0 as n →∞ and nϵ 2 n is bounded away from zero, the following conditions hold for large enough n :

(a) sup ϵ>ϵ n log N ( ϵ/ 36 , { θ ∈ Θ n : d n ( θ , θ 0 ) < ϵ } , d n ) ≤ nϵ 2 n ;

(b) Π(Θ \ Θ n ) = o (exp {-( r +2) nϵ 2 n } ) ;

(c) Π( G ∗ n ( θ 0 , ϵ n )) ≥ exp( -rnϵ 2 n )

for some constant r > 0 . Then for any M n > 0 P n θ 0 Π( θ : d n ( θ , θ 0 ) ≥ M n ϵ n |D n ) → 0 as M n →∞ .

B.1.5 Useful information-theoretical definitions and lemmas

We give some useful information-theoretical definitions and lemmas in this subsection, which will be used in the proof of Theorem 2 and Corollary 3.

Lemma 4 (Log sum inequality, [45]) . For positive numbers, a 1 , .., a n and b 1 , ..., b n ,

with equality if and only if a i b i equals for i = 1 , ..., n .

Definition 2 ([74, 75]) . Let P and Q be probability distributions on the set X , and let f : R + -→ R be a convex function satisfying f (1) = 0 . Without loss of generality, assume that P and Q are absolutely continuous with respect to the base measure µ . The f -divergence between P and Q is then defined as

where p and q are the densities of P and Q with respect to the measure µ , respectively, and f ′ ( ∞ ) represents lim x →∞ f ( x ) /x .

Example 1. By taking different f functions, we provide some popular examples of f -divergences.

Kullback-Leibler (KL) divergence: taking f ( t ) = t log t gives D f ( P ∥ Q ) ≜ D KL ( P ∥ Q ) = ∫ p log( p/q ) dµ , which is also denoted d 0 ( p, q ) in Definition 1.

The total variation distance: taking f ( t ) = 1 2 | t -1 | yields D f ( P ∥ Q ) ≜ ∥ P -Q ∥ TV = 1 2 ∫ ∣ ∣ p q -1 ∣ ∣ qdµ = sup A ⊂X | P ( A ) -Q ( A ) | , which is also denoted d 0 ( p, q ) in Definition 1.

The Hellinger distance: taking f ( t ) = ( √ t -1) 2 = t -2 √ t + 1 leads to the squared Hellinger distance D f ( P ∥ Q ) ≜ H 2 ( P ∥ Q ) = ∫ ( √ p - √ q ) 2 dµ , which is also denoted d 2 ( p, q ) in Definition 1.

The χ 2 -divergence: taking f ( t ) = 1 2 ( t -1) 2 produces the χ 2 -divergence D f ( P ∥ Q ) ≜ χ 2 ( P ∥ Q ) = 1 2 ∫ ( p q -1) 2 dµ .

Lemma 5 ([75]) . For the quantities defined in Example 1, the following relationships hold:

(i) For the Hellinger distance,

(ii) Pinsker's inequality: for any distributions P and Q ,

Definition 3 ([41, 40]) . Let P = { P θ : θ ∈ Θ } be a family of probability measures on a space X , indexed by θ ∈ Θ , and let ω be a probability measure on Θ . For each f satisfying the conditions in Definition 2, the f -informativity, I f ( ω, P ) , is defined as

where the infimum is taken over all possible probability measures Q on X . In particular, when f ( t ) = t log t , the f -informativity is equal to the mutual information and is denoted by I ( ω, P ) .

For each f satisfying the conditions in Definition 2, let ϕ f : [0 , 1] 2 → R be the function defined as follows: for a, b ∈ [0 , 1] 2 , ϕ f ( a, b ) is the f -divergence between the two probability measures P and Q on { 0 , 1 } given by P { 1 } = a and Q { 1 } = b . Then, ϕ f ( a, b ) has the following expression:

The following lemma implies monotonicity and convexity properties of ϕ f by taking it as a univariate function with one argument of ϕ f fixed at a given value.

Lemma 6 ([40]) . For each f satisfying the conditions in Definition 2, consider ϕ f defined in (18). Then

(a). for every fixed b > 0 , the map g ( a ) : a ↦→ ϕ f ( a, b ) is non-increasing for a ∈ [0 , b ] and g ( a ) is convex and continuous in a ;

(b). for every fixed a < 1 , the map h ( b ) : b ↦→ ϕ f ( a, b ) is non-decreasing for b ∈ [ a, 1] .

Lemma 7 ([40]) . Let P = { P θ : θ ∈ Θ } be a family of probability measures on a space X and let ω be a probability measure on Θ . Suppose that the loss function L is zero-one valued. Define the Bayes error as ℜ Bayes ( ω ) = inf d ∫ Θ E L ( θ, d ( X )) ω ( dθ ) with d : X → Θ denoting a mapping from the sample space to the parameter space. For any f satisfying the conditions in Definition 2, we have that

where ϕ f is given by (18), and ℜ 0 is defined as ℜ 0 ≜ inf a ∈ Θ ∫ Θ L ( θ, a ) ω ( dθ ) .

B.2 Proof of Theorem 1

We now establish the proof of Theorem 1 by checking the three conditions in Lemma 3 established for the sparse Bayesian deep learning framework. We first provide a useful result on the discrepancy measures for generalized linear models in Section B.2.1, and then verify the three conditions in Sections B.2.2-B.2.4 following the proof techniques in [37, 67].

B.2.1 Discrepancy measures for generalized linear models

Paired variables in the set of anchor points D 0 = D 0 , 1 ∪ . . . ∪ D 0 ,K in Section 3.1 can be seen as independently (not necessarily identically) distributed, and we let D n denote D 0 in the following derivations to emphasize its dependence on the sample size n . For { x , ˜ y } ∈ D 0 ,k with k ∈ [ K ] , we write the r th element of ˜ y , ˜ y ( r ) , in the 1-ofK fashion, i.e., only the j th element is equal to 1 while others are all 0 if ˜ y ( r ) is the j th class, and then, the conditional probability density/mass function of ˜ y ( r ) induced by θ is given by

for r ∈ [ R ] . We denote the joint conditional probability density/mass function of ˜ y as f ( k ) θ (˜ y ) , given by

and in contrast, we denote the underlying true conditional probability density/mass functions for ˜ y ( r ) and ˜ y as f ( k,r ) 0 and f ( k ) 0 , respectively.

Result 1. If Conditions A.4 in Section A.2 are satisfied, then, for any k ∈ [ K ] ,

for some positive constants C K and C V in the neighbourhood of θ 0 4 .

Proof. By Definition (1) (iii) and (20), we have that

According to (19) and by taking the Taylor's expansion of log ( f k,r θ ) at ω ( k,r ) 0 , we obtain that

where ¯ ω = (¯ ω 1 , ..., ¯ ω K ) ⊤ is between ω ( k,r ) and ω ( k,r ) 0 and

Since ˜ y ( r ) is discretely distributed, according to (19) and Condition A.4.1, we further obtain that

Thus, the proof of inequality (21) is completed.

Similarly, we write V ( f ( k ) θ , f ( k ) 0 ) as:

̸

where the second term can be upper bounded by

̸

and for the first term, we have

according to Condition A.4.1. If G j ( ω ( k,r ) 0 ) > ∥ ω ( k,r ) ) -ω ( k,r ) 0 ) ∥ 1 , similar to the proof for (21), we obtain that

where ¯ ω is between ω ( k,r ) 0 and ω ( k,r ) . Thus, the inequality (22) holds in the neighbourhood of θ 0 .

B.2.2 Verification of Condition (c) in Lemma 3

Proof. For the sequence { ϵ n } ∞ n =1 in Lemma 3, by (21) and (22) in Result 1 and (17), we obtain that

where ∆ n = U 1 ϵ 2 n for some positive constant U 1 .

For simplicity of presentation, we now omit k and r in f ( k,r ) 0 , f ( k,r ) θ , ω ( k,r ) 0 , ω ( k,r ) , B ( k ) and A ( r ) in this proof here. Note that

. Corresponding to each term above, we consider the following six terms:

Corresponding to B ( k ) for (6), we write B = ( β 1 , ..., β K ) and B 0 = ( β 10 , ..., β K 0 ) . Then for (I), we have that

where the last equality holds since all columns of B have independent and identical prior.

Taking ∆ = ∆ n 6 K = U 1 ϵ 2 n 6 K , by Proposition 1, we have that for any given nonzero ψ 2 ( x ; θ (2) ) , M 1 > 0 , and M 2 > 0 ,

for large enough n , where the second inequality follows from Condition A.4.2 in Section A.2 and Condition B.2.2 in Section A.3, and the third inequality holds since ∆ = ∆ n 6 K = U 1 ϵ 2 n 6 K by definition and ϵ n → 0 as n → ∞ . The last inequality holds if 4 log(1 /ϵ 2 n ) ≤ M 2 nϵ 2 n for large enough n and log ∥ ψ 2 ( x ; θ (2) ) ∥ ≺ M 2 nϵ 2 n , where, according to Lemma 1, the latter holds if H n 2 ( log E n 2 +log r n 2 H n 2 ) ≺ M 2 nϵ 2 n , which can be guaranteed by Condition A.3.1 in Section A.2. Since the result above holds for any given ψ 2 ( x ; θ (2) ) , by summarizing the discussion above, we obtain that

Now we consider the second term (II) in (23), which can be written as

where the last inequality follows from Condition A.4.2 in Section A.2.

Consider the set

where η n and η ′ n will be specified later. Let r h denote the number of nonzero connections to the h th hidden layer which includes the bias for the h th hidden layer and the weights between the ( h -1) th and the h th layer, such that ∑ r h = ∥ γ (2) ∗ ∥ 1 ≜ r n 2 . Then, for any x satisfying ∥ x ∥ ∞ ≤ 1 ,

where the first inequality follows from Lemma 2, and the second and the third inequalities hold if we take η n = U 3 ϵ 2 n / { H n 2 ( U 2 E n 2 ) H n 2 ( r n 2 /H n 2 ) H n 2 } and η ′ n = U 3 ϵ 2 n / { J n 2 ( U 2 E n 2 ) H n 2 ( r n 2 /H n 2 ) H n 2 } for some constants U 2 > 1 and U 3 > 0 . By taking a small enough U 3 , we can obtain that E x ∥ ψ 2 ( x ; θ (2) ) -ψ 2 ( x ; θ (2) ∗ ) ∥ 1 ≤ ∆ n 6 KF 1 for any θ (2) ∈ S (2) defined in (24). Then, it suffices to prove that Π( S (2) ) = Π (2) ( S (2) ) ≥ exp( -M 3 nϵ 2 n ) for some positive constant M 3 , where Π (2) ( · ) is the prior measure of θ (2) as given in Section A.3.

As defined in (5) in Section 3.2, each element of θ (2) has an independent continuous prior, denoted π (2) θ ( · ) , and we use π (2) θ to denote the minimal density value of π (2) θ ( · ) on the interval [ -E n 2 -1 , E n 2 +1] . Then, we obtain that

and thus, according to Conditions B.1.1 and B.1.2 in Section A.3, we have that

where the last asymptotic equality follows from Condition A.3 in Section A.2 and the fact that log(1 /ϵ n ) = O (log n ) . Thus, (II) ≥ Π( S (2) ) ≥ exp( -M 3 nϵ 2 n ) holds if r n 2 H n 2 log n + r n 2 log L n 2 ≤ U 4 nϵ 2 n for some sufficiently small positive constant U 4 .

Now we consider the third term in (23):

where the second inequality follows from Condition A.4.2 in Section A.2 and the last inequality follows from the definition of the true model given in (14). Thus, we can take ϵ 2 n = O ( ϖ n 2 ) so that (III) ≈ 1 for large enough n .

Similar discussion can be applied to (I) ′ , (II) ′ and (III) ′ in (23) and we can obtain that

where M ′ 1 , M ′ 2 and M ′ 3 are constants, and S (1) can be similarly defined as in (24). Since the discussion above holds for any k ∈ [ K ] and r ∈ [ R ] , by choosing proper M 1 , M 2 , M 3 , M ′ 1 , M ′ 2 and M ′ 3 , Condition (c) of Lemma 3 can be verified.

B.2.3 Verification of Condition (a) in Lemma 3

Proof. Condition (a) in Lemma 3 can be verified if we can find a parameter set Θ n ⊂ Θ such that log N ( ϵ n / 36 , P n , d n ) ≤ nϵ 2 , where P n represents the set of all densities that can be represented by model (2) in Section 3.1 with parameters in Θ n . Consider the following parameter space:

where

Here, | γ (1) θ | in and | γ (2) θ | in denote the input dimensions of the sparse network structures γ (1) θ and γ (1) θ , respectively, and I (1) n , I (2) n , δ ′ n 1 , δ ′ n 2 , k n 1 ( ≤ n/r n 1 ) , k n 2 ( ≤ n/r n 2 ) , k ′ n 1 ( ≤ n/s n 1 ) , k ′ n 2 ( ≤ n/s n 2 ) , b n and a n are positive constants whose values will be described later.

The parameter space Θ n can be covered by a set of L ∞ balls of the form ( ϱ j -ρ n , ϱ j + ρ n ) W n j =1 , where for the j th coordinate, ϱ j is the center which lies inside Θ n , ρ n is the radius, and W n = J n 1 + J n 2 + K 2 M (1) + RKM (2) is the number of all involved parameters. It can be verified that the number of balls needed to cover the parameter space is upper bounded by

where for l = 1 , 2 , χ ( l ) ( j ) denotes the number of all valid networks with exactly j connections and no more than k ′ nl s nl inputs.

Then we consider the number of d n -balls needed to cover the set of all densities, P n . Consider two parameters in Θ n : θ u ≜ ( θ (1) ⊤ u , θ (2) ⊤ u , B (1) ⊤ u , . . . , B ( K ) ⊤ u , A (1) ⊤ u , . . . , A ( R ) ⊤ u ) ⊤ and θ v ≜ ( θ (1) ⊤ v , θ (2) ⊤ v , B (1) ⊤ v , . . . , B ( K ) ⊤ v , A (1) ⊤ v , . . . , A ( R ) ⊤ v ) ⊤ , which satisfy that

(i) there exists a network structure γ (1) such that | γ (1) | ≤ k n 1 r n 1 , | γ (1) | in ≤ k ′ n 1 s n 1 , | θ (1) u,j -θ (1) v,j | ≤ ρ n for j ∈ γ (1) , θ (1) u,j ≤ δ ′ n 1 and θ (1) v,j ≤ δ ′ n 1 for j / ∈ γ (1) ;

(ii) there exists a network structure γ (2) such that | γ (2) | ≤ k n 2 r n 2 , | γ (2) | in ≤ k ′ n 2 s n 2 , | θ (2) u,j -θ (2) v,j | ≤ ρ n for j ∈ γ (2) , θ (2) u,j ≤ δ ′ n 2 and θ (2) v,j ≤ δ ′ n 2 for j / ∈ γ (2) ;

(iii) | β ( k ) ul,j -β ( k ) vi,j | ≤ ρ n for k, l ∈ [ K ] and j ∈ [ M (1) ] ;

where s n 1 and s n 2 are defined in Condition A.3.1 in Appendix A.2, and ρ n is a positive constant whose values will be discussed later.

Let f ( k ) θ u and f ( k ) θ v denote the corresponding densities in P n for k ∈ [ K ] . By (21) in Result 1 and the fact that d ( f 1 , f 2 ) ≤ √ K ( f 1 , f 2 ) in Appendix B.1.1, we obtain that d 2 n ( θ u , θ v ) ≤ max k ∈ [ K ] ,r ∈ [ R ] U 5 E x ∥ ω ( k,r ) u -ω ( k,r ) v ∥ 1 for some positive constant U 5 , where ω ( k,r ) u = B ( k ) ⊤ u ψ 2 ( x ; θ (2) u )+ A ( r ) ⊤ u ψ 1 ( x ; θ (1) u ) and ω ( k,r ) v = B ( k ) ⊤ v ψ 2 ( x ; θ (2) v )+ A ( r ) ⊤ v ψ 1 ( x ; θ (1) v )

for k ∈ [ K ] and r ∈ [ R ] . For ease of presentation, we omit k and r in B ( k ) u , B ( k ) v , A ( r ) u , A ( r ) v , ω ( k,r ) u and ω ( k,r ) v , and further obtain that

where the preceding four boundness assumptions for θ u and θ v , together with the definition of A n and B n , are used.

Similar to the proof in Appendix B.2.2, by using Lemmas 1 and 2, we can further obtain that d 2 n ( θ u , θ v ) ≤ ( ϵ n / 36) 2 if we choose ρ n , δ ′ n 1 and δ ′ n 2 in the preceding derivations as follows:

for some constants U 8 , 1 , U 8 , 2 > 0 and U 8 , 1 , U 8 , 2 > 1 .

Thus, the log covering number of P n , log N ( ϵ n / 36 , P n , d n ) , can be upper bounded by log K n = log K n 1 +log K n 2 +log K n 3 +log K n 4 , with K nj given in (25) for j = 1 , 2 , 3 , 4 .

Considering the fact that log χ (1) ( j ) ≤ log ( p n k ′ n 1 s n 1 )( k ′ n 1 s n 1 + H n 1 L n 1 j ) ≤ k ′ n 1 s n 1 log( p n ) + j log( k ′ n 1 s n 1 + H n 1 L 2 n 1 ) ≤ k ′ n 1 s n 1 log( p n ) + k n 1 r n 1 log { H n 1 ( k ′ n 1 s n 1 + L n 1 ) 2 } , and by choosing log I (1) n = O (log n ) , log I (2) n = O (log n ) , a n ≍ √ nϵ 2 n and b n ≍ √ nϵ 2 n , we have that log K n 1

where the last equality holds since k n 1 r n 1 ≤ n , k n 2 r n 2 ≤ n , k ′ n 1 s n 1 ≤ n and k ′ n 1 s n 1 ≤ n . By choosing k ′ n 1 and k n 1 such that k n 1 r n 1 ( H n 1 log n + H n 2 log n +log L n 1 +log L n 2 ) ≍ k ′ n 1 s n 1 log( p n ) ≍ nϵ 2 n , we have nϵ 2 n = O { r n 1 H n 1 log n + r n 1 H n 2 log n + r n 1 log L n 1 + r n 1 log L n 2 + s n 1 log( p n ) } . By applying similar discussion on log K n 2 , log K n 3 and log K n 4 , we further obtain that log N ( P n , ϵ n ) ≤ log K n ≤ nϵ 2 n , where the sequence

B.2.4 Verification of Condition (b) in Lemma 3

Proof. Since Π(Θ c n ) ≤ Π( S c n 1 ) + Π( S c n 2 ) + Π( B c n ) + Π( A c n ) , we first examine Π( B c n ) and Π( A c n ) . For any b n > 0 , by the prior assumption given in (6), we have that

where π ( k ) ( · ) denotes the measure induced by the prior distribution of B ( k ) as defined in Section 3.2.

B Applying the bounds on the Mills ratio [76] for the standard normal distribution: 1 x -1 x 3 < 1 -Φ( x ) φ ( x ) < 1 x -1 x 3 + 3 x 5 for -∞ < x < ∞ , where Φ( · ) and φ ( · ) denote the CDF and probability density function (PDF) of the standard normal distribution, respectively, we further obtain that, by taking b n = max k ∈ [ K ] √ 5 B ( k ) n nϵ 2 n ,

where A j,j represents the ( j, j ) element of matrix A , the first step is due to the Mills ratio bounds, the second step comes from the definition of the PDF of the standard normal distribution, the third step is due to the fact that the sum of the eigenvalues of a matrix equals the trace of the matrix and the positivity of ( Σ ( k ) β ) j,j , and the last step comes from the choice of b n . Therefore, it follows that Π( B c n ) < K 2 M (2) exp( -5 2 nϵ 2 n ) for large enough n .

Similarly, we can obtain that Π( A c n ) < K 2 M (1) exp( -5 2 nϵ 2 n ) for large enough n by taking a n = max r ∈ [ R ] √ 5 A ( r ) n nϵ 2 n .

We then consider Π( S c n 1 ) , which can be upper bounded by

where ν n = 1 -π (1) θ ([ -δ ′ n 1 , δ ′ n 1 ]) ≤ 1 J n 1 exp { ( H n 1 + H n 2 ) log n +log L n 1 +log L n 2 +log p n } according to Condition B.1.2 in Section A.3. For the first term in (26), we have that J n 1 π (1) θ ( | θ (1) 1 | > I (1) n | ) ≺ exp {-(2 + ϵ ( k ) 3 ) nϵ 2 n } following Condition B.1.3 in Section A.3, where ϵ ( k ) 3 > 0 is a

constant. The second term can be upper bounded by

where the last inequality is due to the choice of k n 1 in Appendix B.2.3, and S (1) 0 > 2 is a constant given in Condition B.1.2 in Section A.3.

Similarly, for the third term in (26) we have that

Thus, Condition (b) in Lemma 3 is verified.

B.3 Proof of Theorem 2

We now prove Theorem 2. The proof of the upper bound adapts the techniques in [46], and the proof of the lower bound employs a general mechanism for lower bounding the Bayes risk [40], which is tighter than the Fano's inequality [44].

Proof. Upper Bound. The Bayes risk can be written as

̸

̸

̸

We first consider the situation g = 1 :

̸

For each g ′ ≥ 2 , considering the inference procedure of y i , we have that

̸

where the inequality in the third line follows from Markov's inequality. Denote C ( i ) gg ′ = -min 0 ≤ λ ≤ 1 1 R { -λ log Ω ℏ i,g ℏ i,g ′ + ∑ R r =1 log { ∑ K l =1 ( τ ( r ) i,gl ) 1 -λ ( τ ( r ) i,g ′ l ) λ }} and I ( g ) Ω ( ℏ i , τ i ) = min g ′ = g C ( i ) gg ′ , we further obtain that

̸

Thus, we can obtain the upper bound of the Bayes risk:

Lower Bound. Using Markov's inequality, we obtain that

̸

̸

where the second step is due to Markov's inequality. Using Lemma 7, we can obtain that

where P is the set of distributions of { ˜ y i } ¯ n i =1 induced by y , and for 0-1 loss, ℜ 0 has the expression in [40]: ℜ 0 = 1 -sup a ∈ [ K ] ¯ n ℏ ( B ( a )) with B ( a ) = { y ∈ [ K ] ¯ n : 1 ( y = a ) = 0 } . Then, it can be easily verified that ℜ 0 = 1 -∏ ¯ n i =1 max k ∈ [ K ] ℏ i,k .

̸

Let g ( t ) = ϕ f ( t, ℜ 0 ) , which is, by Lemma 6, non-increasing for t ∈ [0 , ℜ 0 ] , convex, and continuous in t . Using the convexity of g ( t ) , we obtain that for every t ∈ (0 , ℜ 0 ]

where ϕ ′ f ( t -, ℜ 0 ) denotes the left derivative of x ↦→ ϕ f ( x, ℜ 0 ) at x = t . Then we can obtain that

where the inequalities come from the fact that I ( ℏ , P ) ≥ ϕ f ( ℜ , ℜ 0 ) by Lemma 7 and that ϕ ′ f ( t -, ℜ 0 ) ≤ 0 due to the non-increasing function g ( t ) over t ∈ [0 , ℜ 0 ] . By taking f ( t ) = t log t and t = ℜ 0 1+ ℜ 0 , we obtain that

where I ( ℏ , P ) is the mutual information of ℏ and P .

Let Y and Z denote two independent random variables such that Y ∼ ℏ and Z ∼ P , and let D KL ( ·∥· ) denote the KL divergence of the associated distributions. For ease of notation, we use P ( Y ) , P ( Z ) , P ( Y, Z ) and P ( Z | Y ) to denote the ℏ , P , the joint distribution of Y and Z , and the conditional distribution of Z , given Y . Then I ( ℏ , P ) can be evaluated as follows [43]:

Using the log sum inequality in Lemma 4, we further obtain that

Combining the discussion above, we obtain that

B.4 Proof of Corollary 3

Proof. For k ∈ [ K ] , r ∈ [ R ] , and consider the following set:

where θ , ϵ n and M n satisfy the conditions in Theorem 1, ω ( k,r ) and ω ( k,r ) 0 depend on x , and D x represents the set of observations of the instances in D . Then, according to Theorem 1 and using the union bound, we have that

in P n 0 probability as n → ∞ , where ∥ · ∥ TV denotes the total variation distance of the associated distributions as defined in Example 1, and U 9 is a positive constant. Here, the second inequality is due to the definition of the total variation distance of discrete distributions and the fact that f ( x ) > ς 2 for x ∈ D x , and the third inequality follows the (i) in Lemma 5. Thus, Π( S|D 0 ) -→ 1 in P n 0 probability as n →∞ , where S can be written as

̸

We now consider the Bayes risk ℜ Bayes, which, according to the proof of Theorem 2, can be written as ℜ Bayes = inf y { 1 ¯ n ∑ ¯ n i =1 ∑ K g =1 ℏ i,g P (y i = g | y i = g ; τ ) } . We first consider g = 1 :

̸

If { τ ( r ) k ∗ ( · ) } k ∈ [ K ] ,r ∈ [ R ] ∈ S , using Taylor's expansion, we have that

for some constant κ > 0 and large enough n . For any ϵ > 0 , we have that κM n ϵ n < ϵ by taking M n = ϵ/ (2 κM n ϵ n ) . Thus, we further obtain that

̸

for { τ ( r ) k ∗ ( · ) } k ∈ [ K ] ,r ∈ [ R ] ∈ S , where in the second step, we use Markov's inequality and the definition of I (1) Ω ( ℏ i , τ i ) . Similar results also hold for g = 2 , ..., K . Hence, according to Theorem 1, we have that

where Π( S|D 0 ) -→ 1 in P n 0 probability as n →∞ . Thus, the proof is completed.

C Implementation details and additional experimental results

C.1 Implementation details

Dataset description. We assess the effectiveness of our method on three image datasets with synthetic annotations, MNIST [47], CIFAR-10 [48], and CIFAR-100 [48], and two datasets with human annotations, CIFAR-10N [49] and LabelMe [50, 51]. MNIST consists of 28 × 28 grayscale images with 10 classes, containing 60,000 training images and 10,000 test images. CIFAR10 has 10 classes of 32 × 32 × 3 color images, with 50,000 images used for training and 10,000 for testing. CIFAR100 also consists of 50,000 training images and 10,000 test images, whose size is 32 × 32 × 3 , but with 100 fine-grained classes. We further consider two additional datasets with human annotations, CIFAR-10N [49] and LabelMe [50, 51]. For each instance in CIFAR10, CIFAR-10N provides three independent human annotated noisy labels, with the aggregation of three noisy labels by majority voting being 9 . 03% . LabelMe is an image classification dataset consists of 10,000 training images, 500 validation images, and 1,188 test images. For images in the training set, LabelMe has noisy and incomplete labels provided by a total of R = 59 annotators, with each image being labeled by an average of 2.547 annotations. For all the datasets except LabelMe, we leave out 10% of the training data as a noisy validation set.

Experiment setup. The network structure for the MNIST dataset is chosen to be Lenet-5 [52]. We choose ResNet-18 [2] for CIFAR-10 and CIFAR-10N, and ResNet-34 architecture [2] for CIFAR-100. As in [53], we employ the pretrained VGG-16 network, followed by a fully connected layer and a softmax output layer for the LabelMe dataset, using 50% dropout. We take the batch size to be 128 for all the datasets. For MNIST, we use the SGD optimizer with momentum 0.9, weight decay 5 × 10 -4 , and an initial learning rate of 10 -2 . The learning rate is divided by 10 at the 40th epoch, and we set 80 epochs in total, in which the first 20 epochs are used to warm up the model on the noisy dataset and to determine anchor points. For CIFAR10, CIFAR100, CIFAR10N and LabelMe, the Adam optimizer [77] is utilized with the weight decay 5 × 10 -4 . For CIFAR10, CIFAR100 and CIFAR10N, the initial learning rate is set to be 10 -3 , and the network is trained for 120, 150, and 120 epochs for CIFAR10, CIFAR100 and CIFAR10N, repectively, with the first 30 epochs used as the warm-up stage. The model is trained on LabelMe for 100 epochs, with an initial learning rate 5 × 10 -4 and first 20 epochs as the warm-up stage.

Baselines. We compare the proposed method with the following state-of-art methods: (1) CE (Clean), which trains the network with the standard cross entopy loss on the clean datasets; (2) CE (MV), which trains the network using the labels from majority voting; (3) CE (EM) [9], which obtains the aggregated labels utilizing the EM algorithm; (4) DoctorNet [54], which models the annotators individually and then learns averaging weights for combining them; (5) GCE [55], which generalizes

the mean absolute error and the cross entopy loss to combat errors in training labels; (6) Co-teaching [56], which trains two networks and cross-trains on instances with small loss values; (6) Co-teaching+ [57], which bridges the 'Update by Disagreement' strategy with the Co-teaching method; (7) BLTM [17], which directly models the transition matrix from Bayes optimal labels to noisy labels and learns a classifier to predict Bayes optimal labels; (8) MBEM [11], which alternates in rounds between estimating annotator quality from disagreement with the current model and updating the model by optimizing the a loss function that accounts for the current estimate of worker quality; (9) CrowdLayer [53], which concatenates the classifier with multiple annotator-specific layers and simultaneously learns the parameters; (10) TraceReg [8], which uses a loss function similar to CrowdLayer [53], but adds a regularization to establish identifiability of the confusion matrices and the classifier; (11) Max-MIG [7], which jointly aggregates the noisy crowdsourced labels and trains the classifier; (12) CoNAL [58], which decomposes the annotation noise into common and individual confusions; (13) GeoCrowdNet (F) [13]; and (14) GeoCrowdNet (W) [13], which are two regularized variants of the coupled cross-entropy minimization to enhance the identifiability of the confusion matrices. Among these methods, GCE and Co-teaching are strong baselines dealing with single noisy label issue, and we adapt them to the multiple annotations setting by utilizing the majority vote labels for loss computation.

Implementation details. We first warm up the base models on the noisy dataset with majority vote labels, obtain the set of anchor points D 0 , and train the sparse Bayesian model on D 0 by maximizing the log posterior distribution of network parameters and excluding non-informative parameters with low posterior inclusion probability. With the noise transition model trained, we then iteratively implement the label correction algorithm (10) and update the base models, where linearly increase the threshold Ω t in the training process. Specifically, let ER denoted the estimated average noise rate, and let r 0 and r 1 represent two prespecified constants with r 1 ≥ r 0 > 0 , which determine the magnitude of Ω t at the beginning and the end of the training process. In the t th epoch, we set the threshold Ω t to be Ω t = (1 -ER ) · ( r 0 + t · r 1 -r 0 T ) in the experiment. The implementation procedure of the proposed method is summarized in Algorithm 1.

Algorithm 1: Annotator-Specific Instance-Dependent Label Noise Learning via Sparse Bayesian Network and Pairwise LRT

C.2 Additional experimental results.

Accuracy and number of selected labels. To validate the effectiveness of the proposed label correction method, we plot the accuracy and the number of corrected labels using criterion (10) in Figures 3 and 4 for all the considered datasets, where the solid line is the accuracy (left y-axis) and the dashed line is the number (right y-axis) of corrected labels.

45000

46000

47000

48000

49000

50000

51000

52000

53000

Number of selected labels

10

20

30

40

50

60

Epoch

0.988

0.990

0.992

0.994

0.996

0.998

1.000

Accuracy of selected labels

IDN-HIGH: Accuracy

IDN-HIGH: Accuracy

IDN-LOW: Accuracy

IDN-HIGH: Number

IDN-MID: Number

IDN-LOW: Number

(a) MNIST

30000

31000

32000

33000

34000

35000

36000

37000

38000

39000

Number of selected labels

20

40

60

80

Epoch

0.88

0.90

0.92

0.94

0.96

0.98

Accuracy of selected labels

IDN-HIGH: Accuracy

IDN-MID: Accuracy

IDN-LOW: Accuracy

IDN-HIGH: Number

IDN-MID: Number

IDN-LOW: Number

(b) CIFAR10

30000

32000

34000

36000

38000

40000

42000

44000

Number of selected labels

20

40

60

80

100

120

Epoch

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

Accuracy of selected labels

IDN-HIGH: Accuracy

IDN-MID: Accuracy

IDN-LOW: Accuracy

IDN-HIGH: Number

IDN-MID: Number

IDN-LOW: Number

(c) CIFAR100

40000

40500

41000

41500

42000

42500

43000

Number of selected labels

20

40

60

80

Epoch

0.950

0.955

0.960

0.965

0.970

0.975

0.980

0.985

0.990

Accuracy of selected labels

IDN-HIGH: Accuracy

IDN-HIGH: Number

(a) CIFAR10N

8400

8600

8800

9000

9200

9400

9600

Number of selected labels

10

20

30

40

50

60

70

80

Epoch

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

Accuracy of selected labels

IDN-HIGH: Accuracy

IDN-HIGH: Number

(b) LabelMe

Classification accuracy on MNIST. The test accuracy on CIFAR10, CIFAR100, CIFAR10, and LabelMe are provided in Section 5. Here we present the average test accuracy on the MNIST dataset in Table 2, where the two highest accuraries are bold faced. Clearly, that the proposed method achieves the best performance.

Hyperparameter analysis on the CIFAR100 dataset. We conduct sensitivity analyses about hyperparameters r 0 and r 1 on the CIFAR100 dataset, where we choose r 0 from { 3 , 5 , 10 , 15 , 20 } and r 1 from { 200 , 250 , 300 } . As discussed in the implementation details in Section C.1, r 0 and r 1 determine the magnitude of the threshold Ω t at the beginning and the end of the training process, respectively. With higher values of r 0 and r 1 , the accuracy of the corrected labels using algorithm (10) will increase accordingly, but the number of corrected labels will decrease. As shown in Table 3, with different choices of r 0 and r 1 , the proposed method consistently outperforms all the compared methods.

Classification accuracy on CIFAR100 with varying number of annotators. Figure 5 shows the average accuracy on CIFAR100 with the number of annotators varying from 5 to 100, which further demonstrate the superiority of the proposed method under different settings.

5

10

30

50

100

Number of Annotators

20

30

40

50

60

CE(Clean)

CE(MV)

CE(EM)

DoctorNet

GCE

Co-teaching

Co-teaching+

BLTM

MBEM

CrowdLayer

TraceReg

CoNAL

GeoCrowdNet (F)

GeoCrowdNet (W)

Ours

(a) IDN-LOW

5

10

30

50

100

Number of Annotators

20

30

40

50

60

CE(Clean)

CE(MV)

CE(EM)

DoctorNet

GCE

Co-teaching

Co-teaching+

BLTM

MBEM

CrowdLayer

TraceReg

CoNAL

GeoCrowdNet (F)

GeoCrowdNet (W)

Ours

(b) IDN-MID

5

10

30

50

100

Number of Annotators

20

30

40

50

60

CE(Clean)

CE(MV)

CE(EM)

DoctorNet

GCE

Co-teaching

Co-teaching+

BLTM

MBEM

CrowdLayer

TraceReg

CoNAL

GeoCrowdNet (F)

GeoCrowdNet (W)

Ours

(c) IDN-HIGH

Average estimation error. For synthetic noisy datasets CIFAR10 and CIFAR100, we compare the average estimation error of the proposed method with six competing methods, CrowdLayer [53], TraceReg [8], GeoCrowdNet (F) [13], GeoCrowdNet (W) [13], MBEM [11], and BLTM [17]. The definition of the average estimation error and the results on CIFAR10 with 5 annotators are given in the Ablation study of Section 5. In Figures 6 and 7, we respectively present the average estimation error on the validation set of CIFAR10 and CIFAR100 with varying numbers of annotators, where the results of CrowdLayer, TraceReg, GeoCrowdNet (F) and GeoCrowdNet (W) overlap in some subfigures of Figure 7. The proposed method outperforms all the compared methods with lower average estimation error for each annotator in most of the cases, further demonstrating the effectiveness of the proposed sparse Bayesian model.

1

2

3

4

5

Annotator

0.10

0.15

0.20

0.25

0.30

0.35

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(a) IDN-LOW

1

2

3

4

5

Annotator

0.20

0.25

0.30

0.35

0.40

0.45

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(b) IDN-MID

1

2

3

4

5

Annotator

0.30

0.35

0.40

0.45

0.50

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(c) IDN-HIGH

1

2

3

4

5

6

7

8

9

10

Annotator

0.10

0.15

0.20

0.25

0.30

0.35

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(d) IDN-LOW

1

2

3

4

5

6

7

8

9

10

Annotator

0.25

0.30

0.35

0.40

0.45

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(e) IDN-MID

1

2

3

4

5

6

7

8

9

10

Annotator

0.300

0.325

0.350

0.375

0.400

0.425

0.450

0.475

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(f) IDN-HIGH

3

6

9

12

15

18

21

24

27

30

Annotator

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(g) IDN-LOW

3

6

9

12

15

18

21

24

27

30

Annotator

0.25

0.30

0.35

0.40

0.45

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(h) IDN-MID

3

6

9

12

15

18

21

24

27

30

Annotator

0.325

0.350

0.375

0.400

0.425

0.450

0.475

0.500

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(i) IDN-HIGH

5

10

15

20

25

30

35

40

45

50

Annotator

0.1

0.2

0.3

0.4

0.5

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(j) IDN-LOW

5

10

15

20

25

30

35

40

45

50

Annotator

0.25

0.30

0.35

0.40

0.45

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(k) IDN-MID

5

10

15

20

25

30

35

40

45

50

Annotator

0.325

0.350

0.375

0.400

0.425

0.450

0.475

0.500

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(l) IDN-HIGH

10

20

30

40

50

60

70

80

90

100

Annotator

0.1

0.2

0.3

0.4

0.5

0.6

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(m) IDN-LOW

10

20

30

40

50

60

70

80

90

100

Annotator

0.25

0.30

0.35

0.40

0.45

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(n) IDN-MID

10

20

30

40

50

60

70

80

90

100

Annotator

0.325

0.350

0.375

0.400

0.425

0.450

0.475

0.500

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(o) IDN-HIGH

1

2

3

4

5

Annotator

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(a) IDN-LOW

1

2

3

4

5

Annotator

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(b) IDN-MID

1

2

3

4

5

Annotator

0.44

0.46

0.48

0.50

0.52

0.54

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(c) IDN-HIGH

1

2

3

4

5

6

7

8

9

10

Annotator

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(d) IDN-LOW

1

2

3

4

5

6

7

8

9

10

Annotator

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(e) IDN-MID

1

2

3

4

5

6

7

8

9

10

Annotator

0.44

0.46

0.48

0.50

0.52

0.54

0.56

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(f) IDN-HIGH

3

6

9

12

15

18

21

24

27

30

Annotator

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(g) IDN-LOW

3

6

9

12

15

18

21

24

27

30

Annotator

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(h) IDN-MID

3

6

9

12

15

18

21

24

27

30

Annotator

0.46

0.48

0.50

0.52

0.54

0.56

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(i) IDN-HIGH

5

10

15

20

25

30

35

40

45

50

Annotator

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(j) IDN-LOW

5

10

15

20

25

30

35

40

45

50

Annotator

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(k) IDN-MID

5

10

15

20

25

30

35

40

45

50

Annotator

0.46

0.48

0.50

0.52

0.54

0.56

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(l) IDN-HIGH

10

20

30

40

50

60

70

80

90

100

Annotator

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(m) IDN-LOW

10

20

30

40

50

60

70

80

90

100

Annotator

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(n) IDN-MID

10

20

30

40

50

60

70

80

90

100

Annotator

0.48

0.50

0.52

0.54

0.56

0.58

0.60

0.62

Estimation Error

CrowdLayer

TraceReg

GeoCrowdNet (F)

GeoCrowdNet (W)

MBEM

BLTM

Ours

(o) IDN-HIGH

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors formulate the noise transition model in a Bayesian framework and design a new label correction algorithm. The authors further formulate the label correction process as a hypothesis testing problem and propose a novel algorithm to infer the true label from the noisy annotations based on the pairwise likelihood ratio test (LRT). The experimental results on benchmark and real-world datasets validate the effectiveness of the proposed approach.\n\nSTRENGTHS:\n1. This paper provides a posterior-concentration theorem, which guarantees the posterior consistency of the noise transition model in terms of the Hellinger distance.\n\n2. This paper formulates the label correction process as a hypothesis testing problem and propose a novel algorithm to infer the true label from the noisy annotations based on the pairwise likelihood ratio test (LRT).\n\n3. This paper presents extensive experiments on benchmark and real-world datasets to validate the effectiveness of the proposed approach.\n\nWEAKNESSES:\nThe performance of the proposed approach outperforms other algorithms on three image datasets with synthetic annotations, but does not have significant advantages on two datasets with human annotations, CIFAR-10N and LabelMe. The authors do not provide a reasonable explanation for these results. In addition, some of the author's statements in the abstract need to be considered, such as “ Learning an instance-dependent noise transition model, however, is challenging and remains less explored.”.  To my knowledge, there are many algorithms proposed to solve this problem.\n\nQUESTIONS:\n1. From Table 1, the performance of the proposed approach outperforms other algorithms on three image datasets with synthetic annotations, but does not have significant advantages on two datasets with human annotations, CIFAR-10N and LabelMe. The authors have less analytical discussion of the experimental results. The authors should give a fuller discussion of the experimental results and add other datasets with human annotations to verify the validity of the proposed approach.\n\n2. The authors claim that “Learning an instance-dependent noise transition model, however, is challenging and remains less explored.”. But to my knowledge, there are many algorithms proposed to solve this problem. The authors should notice t and cite these papers.\n\n3. The proof section of the text is relatively substantial, but the author's explanation of the motivation of the article is slightly lacking.\n\nLIMITATIONS:\n1. From Table 1, the performance of the proposed approach outperforms other algorithms on three image datasets with synthetic annotations, but does not have significant advantages on two datasets with human annotations, CIFAR-10N and LabelMe. The authors have less analytical discussion of the experimental results. The authors should give a fuller discussion of the experimental results and add other datasets with human annotations to verify the validity of the proposed approach.\n\n2. The authors claim that “Learning an instance-dependent noise transition model, however, is challenging and remains less explored.”. But to my knowledge, there are many algorithms proposed to solve this problem. The authors should notice t and cite these papers.\n\n3. The proof section of the text is relatively substantial, but the author's explanation of the motivation of the article is slightly lacking.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method for modelling input-dependent label noise for classification in the presence of multiple annotators. The generative process of annotation is modelled as a hierarchical Bayesian model in which the observed labels are noisy versions of the latent true labels that are corrupted through a noise transition model that is dependent on the input as well as the annotator. The authors propose a method for inferring the true labels based on the approximated noise model  and demonstrate benefits on classification benchmarks with both synthetic and real annotation noise.\n\nSTRENGTHS:\n- The idea of modelling labeling noise as a function of both input and annotator is much more realistic than the majority of related works which drop the input dependence or ignore the differences between annotators. \n- They propose a practical algorithm with demonstrable efficacy on classification benchmarks with both synthetic  & real label noise. \n- The authors provide a sound theoretical justification for the above algorithm; 1) the concentration theorem shows the \"tightness\" of the approximation of the noise transition model; 2) formulating inference of true labels as a statistical test gives an information-theoretic bound on the error.\n\nWEAKNESSES:\n-  The empirical analysis lacks insights into which components of the annotation model are actually important e.g., how sensitive is the model to a different specification of the prior distribution.\n-  There seems to be no comparison with other works that employ instance-specific noise transition models --- e.g., [18] seems to propose a much simpler model that simultaneously learn the instance-specific confusion matrix per annotator and the true label distribution.\n\nQUESTIONS:\n- how does the algorithm scale with the number of annotators?\n- how does the accuracy of the method depend on the \"sparsity\" of labels (e.g., if we had only one label per image, would it still be possible to model the input-dependent noise)?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with the noisy annotation problem with crowdsourcing data. Instead of traditional transition matrices, the paper proposed to learn an instance-dependent transition model that uses a Bayesian network with guaranteed consistent posterior to approximate the noise in labels. The paper then proposes to infer the true label from the noisy annotations and improve the performance based on the corrected labels.\n\nSTRENGTHS:\n1. The paper is clearly motivated and deals with a potentially meaningful problem for supervised learning with noisy annotations.\n2. The paper provides a detailed theoretical analysis of the proposed Bayesian model and an information-theoretic bound on the Bayes error.\n3. The experiments on MNIST and CIFAR datasets show that the proposed method outperforms the baselines most of the time.\n\nWEAKNESSES:\n1. The assumption that an instance-dependent noise model can be learned with limited training data is quite strong. It seems that the proposed method is still associated with preset thresholds like existing methods, which is particularly difficult to set in a noisy problem with multiple annotators.\n2. The conditions for Theorem 1 are not listed in the main paper. I think some remarks are necessary for people to have a fair understanding of the proposed theorems.\n3. The proposed method does not always outperform baselines, and the advantage is not significant. In some cases, the biggest difference appears in the IDN-MID case, which is not thoroughly investigated or explained.\n\nQUESTIONS:\n1. Can there be a more convincing claim that an instance-dependent noise model can be learned with limited training data?\n2. Is the main theoretical result asymptotic? Can there be a clearer message that indicates the dependency on the number of data samples?\n3. Can there be more analysis on the reasons for the change of performances with the number of annotators on some datasets?\n\nLIMITATIONS:\nAlthough the paper does mention limitations and extensions, the impact of modeling annotators and instance-dependent noise model is not adequately addressed. The noisy problem is often associated with biases or fairness concerns, so some careful consideration needs to be made.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of estimating annotator-specific instance-dependent transition matrices. The authors propose a solution based on a family of Bayesian estimators to approximate these matrices, supported by theoretical foundations. Through rigorous mathematical proofs, the authors demonstrate that their estimator can effectively approximate transition matrices under claimed mild conditions. Additionally, they introduce a novel label correction method with bounded Bayes error. To evaluate the effectiveness of their proposed method, the authors conduct experiments using three commonly used benchmarks and synthetic annotations. Overall, this paper has potential significant insights that could benefit the community. The experiment evaluation is not very convincing, but I'm tending to accept it at this stage.\n\nSTRENGTHS:\nThere are several main theoritical contributions in this paper:\n\n1. Authors proposed a sparse Bayesian approach to learn the annotator-specific instance-dependent transition matrices to model the relationship between label noise and crowdsourced annotators.\n\n2. The proposed estimator is claimed to approxmiate true transition processs with guarantee (minimizes Hellinger distance).\n\n3. Based on the derived theorem, the authors also provided a novel correction algorithm named that leverages likelihood ratio test, an upper-bound of the Bayes error for the proposed algorithm is provided.\n\nWith the approxmiation guarantee and error-bound on the correction algorithm, the proposed method is therefore considered to have significant theoritical implications and considerable novelties, provided that the proof are rigourous and correct, which is under further checking.\n\nWEAKNESSES:\n1. The section numbers provided in the appendix do not correspond to the references in the main paper, where are assumptions A.1-A.4 in A.1? \n\n2. Compared baselines are not known for their advantages in modeling instance-dependent label noise, methods such as PartT [1], BLTM [2] should be compared instead. Also, SOTA methods based on geometrical properties such as [3,4] should also be compared. \n\n3. One of the main issues is that the use of sparse Bayesian approach is not strongly motivated, it remains unclear to me why using sparse Bayesian estimators is better or non-trivial compared with existing works.\n\n[1] Xia, Xiaobo, et al. \"Part-dependent label noise: Towards instance-dependent label noise.\" Advances in Neural Information Processing Systems 33 (2020): 7597-7610.\n\n[2] Yang, Shuo, et al. \"Estimating instance-dependent bayes-label transition matrix using a deep neural network.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Li, Xuefeng, et al. \"Provably end-to-end label-noise learning without anchor points.\" International conference on machine learning. PMLR, 2021.\n\n[4] Yong, L. I. N., et al. \"A Holistic View of Label Noise Transition Matrix in Deep Learning and Beyond.\" The Eleventh International Conference on Learning Representations. 2022.\n\nQUESTIONS:\n1. Why are the results on CE trained with clean labels so low? This is inconsistent with prior works.\n\n2. How are the sythentic annotations generated?\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for label correction of crowdsourced data via an instance-dependent noise transition model. It parameterizes the instance-dependent noise transition matrices by a Bayesian network. A theoretical analysis that guarantees the posterior consistency of noise transition matrices is provided.   A new label correction way based on the instance-dependent noise transition matrices is also provided. Experiments show the proposed method can achieve better performance than some existing methods.\n\nSTRENGTHS:\n1. Estimating the instance-dependent noise transition matrices for crowdsourcing is an important topic.\n2. This paper provides some theoretical analyses to understand and guarantee some useful properties of the proposed method.\n3. Experiments on synthetic and real crowdsourced datasets verified the effectiveness of the proposed methods.\n\nWEAKNESSES:\n1.  General instance-dependent noise matrix is not new for crowdsourcing. For example, several prior methods [1-5] have studied it in the early period. Recently, this topic receives more attention [6-8], due to the increasing demand for large-scale well-annotated datasets. I think the paper should include these background and existing works, and discuss the difference with them.\n\n2.  The conditions to guarantee the posterior consistency of the noise transition model seems very complex. How can we know such conditions are met in real scenes? Besides, since the anchor point assumption is strong in label-noise learning [9-11], the requirement of the proposed method that a set of anchor points can be used for training the noise transition model seems much more difficult.\n\n3.  The experiments are insufficient in terms of the following aspects: \n- The number of annotators is usually large, while the experiments on synthetic data only assume 5 annotators, which is not a typical setting. \n- Also due to the small number of annotators, the annotations are not sparse for each annotator as usual case.  \n- I suggest the authors compare the proposed method with some baselines that model instance-dependent noise matrices [6-8], and more truth inference baselines [12-14] to show its advantage more clearly.  \n- Since the experimental datasets are all image classification datasets, it would be better to do some experiments on the dataset from other modalities, e.g. Music dataset [15].\n\n4.  Lack of ablation discussion on label correction way. How to use noise transition for inferring clean labels has been explored in previous works [16,13], I think it is necessary to discuss and compare the proposed label correction way with the existing truth inference methods with noise transition matrix to when to use such label correction way.\n\n[1] The multidimensional wisdom of crowds. NeurIPS 2010\n\n[2] Exploiting structure in crowdsourcing tasks via latent factor models. Citeseer. 2010\n\n[3] Modeling annotator expertise: Learning when everybody knows a bit of something. AISTATS 2010\n\n[4] Learning from multiple annotators with varying expertise. MLJ 2014\n\n[5] Learning to Predict from Crowdsourced Data. UAI 2014\n\n[6] Disentangling human error from the ground truth in segmentation of medical images. NeurIPS 2020\n\n[7] Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusion. ECCV 2022\n\n[8] Beyond confusion matrix: learning from multiple annotators with awareness of instance features. MLJ 2023\n\n[9] Are Anchor Points Really Indispensable in Label-Noise Learning? NeurIPS 2019\n\n[10] Provably End-to-end Label-noise Learning without Anchor Points. ICML 2021\n\n[11] Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning. NeurIPS 2022\n\n[12] Bayesian classifier combination. AISTATS 2012\n\n[13] Learning from noisy singly-labeled data. ICLR 2018\n\n[14]  Exploiting worker correlation for label aggregation in crowdsourcing. ICML 2019\n\n[15] Gaussian process classification and active learning with multiple annotators. In ICML 2014\n\n[16] Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society, 1979\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe authors have discussed the limitations at the end of the main body, while I think the demand of a set of anchor points is difficult to meet in real-world scenes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}