SlowGuess/NeurIPS_2023_OCR huggingface dataset:
in  $\mathbb{P}_0^n$  probability for every  $M_n\to \infty$ , where  $\{\varpi_{jn}\}$  is a sequence of nonnegative numbers converging to 0 as  $n\to \infty$  for  $j = 1,2$  as given in (14), and  $\{\zeta_n\}_{n = 1}^{\infty}$  is a sequence given in Appendix A.3
2The anchor point assumption can be relaxed and Theorem 1 can be extended to a more general setting. For any given  $\delta \in (0,1)$ , we define the  $\delta$ -pseudo anchor point for class  $k$  as  $\mathbb{P}(y = k|\mathbf{x}) \geq 1 - \delta$  and denote  $\bar{\mathcal{D}}_{\delta}$  as the set of all  $\delta$ -pseudo anchor points accordingly. Then, the following result holds:  $\Pi \{\theta \in \Theta : d_n(\theta, \theta_0) > M_n \epsilon_n + C \delta |\bar{\mathcal{D}}_{\delta}\} \to 0$  in  $\mathbb{P}_0^n$  probability for every  $M_n \to \infty$ , where  $C$  is a positive constant. From the modified theory, as  $\delta$  approaches 0 slowly, the Hellinger distance of the transition model and the true transition probability converges to zero at a slow rate. In other words, the transition model will still converge even if the collection of a set of anchor points is not guaranteed, albeit at a slow rate.
depending on the structures of  $\psi_{1}(\cdot |\pmb{\theta}^{(1)})$  and  $\psi_{2}(\cdot |\pmb{\theta}^{(1)})$  with  $\zeta_n\to 0$  as  $n\to \infty$ . If we further assume that  $|\overline{\mathcal{D}}_{0,k}| / |\overline{\mathcal{D}}_0| > \varsigma_1$  for some positive constant  $\varsigma_{1}$ , with  $|\cdot |$  representing the cardinality of a set, then for any  $k\in [K]$  and  $r\in [R]$
$$
\Pi \left\{\boldsymbol {\theta} \in \Theta : d \left(f _ {\boldsymbol {\theta}} ^ {(k, r)}, f _ {0} ^ {(k, r)}\right) > M _ {n} \epsilon_ {n} | \overline {{\mathcal {D}}} _ {0} \right\}\rightarrow 0, \tag {9}
$$
in  $\mathbb{P}_0^n$  probability for any  $M_{n}\to \infty$
Intuitively, Theorem 1 reveals that the sparse noise transition model is close to the underlying true transition matrix with respect to the Hellinger distance under mild conditions. Notably, our posterior consistency result extends the existing theories in sparse Bayesian learning [35-37] to the setup of i.n.i.d observations. Moreover, this result on the convergence rate of the posterior measure allows us to infer the underlying true label from the noisy annotations with a theoretical guarantee on the bounds of the Bayes error, which will be discussed in detail in the following section.
3.3 Pairwise likelihood ratio test for label correction
The asymptotic result (9) in Theorem 1 indicates that for each annotator, the underlying true instance-dependent transition matrix can be accurately modeled under the Bayesian framework. This enables us to aggregate and infer the ground truth label from noisy crowdsourced annotations.
A novel label correction algorithm. To highlight the idea, we first assume that the noise transition matrix  $\mathbb{P}(\tilde{\mathbf{y}}^{(r)}|\mathbf{y} = k,\mathbf{x})$ , or  $f_0^{k,r}(\cdot)$ , is known. To simplify the notation, for each  $\mathbf{x}_i$  in the noisy dataset  $\mathcal{D}$ , denote  $\tau_{i,kl}^{(r)}\triangleq \tau_{kl}^{(r)}(\mathbf{x}_i)\triangleq \mathbb{P}(\tilde{\mathbf{y}}^{(r)} = l|\mathbf{y} = k,\mathbf{x}_i)$  for  $i\in [N]$  and  $k,l\in [K]$ . We assign class prior [38]  $\pmb{\hbar}_{i} = (\hbar_{i,1},\dots,\hbar_{i,K})^{\top}$  for the ground truth label for the  $i$ th task, where the  $\hbar_{i,k}$  for  $k\in [K]$  are nonnegative weights satisfying  $\sum_{k = 1}^{K}\hbar_{i,k} = 1$ . For each instance, with the class prior and the noise transition matrices, the label correction process can be formulated as a hypothesis testing problem, where different hypotheses are generated from different choices of the true label values. Specifically, selecting the label for the instance  $\mathbf{x}_i$  from  $\{g,g'\}$ , with  $1\leq g < g' \leq K$ , is equivalent to choosing from the two competitors  $\mathbb{P}(\tilde{\mathbf{y}} |\mathbf{y} = g,\mathbf{x}_i)$  and  $\mathbb{P}(\tilde{\mathbf{y}} |\mathbf{y} = g',\mathbf{x}_i)$ . We thereby consider the following hypothesis testing problem:  $H_{g}:\tilde{\mathbf{y}}_{i}|\{\mathbf{y}_{i},\mathbf{x}_{i}\} \sim \mathbb{P}(\tilde{\mathbf{y}} |\mathbf{y} = g,\mathbf{x}_{i})$  versus  $H_{g'}:\tilde{\mathbf{y}}_i|\{\mathbf{y}_i,\mathbf{x}_i\} \sim \mathbb{P}(\tilde{\mathbf{y}} |\mathbf{y} = g',\mathbf{x}_i)$ . By the Neyman-Pearson Lemma [39], the Bayes testing error is minimized by the likelihood ratio test, and the decision region for hypothesis  $H_{g}$  is given by
$$
\left\{\tilde {\mathbf {y}}: \frac {\hbar_ {i , g} \mathbb {P} (\tilde {\mathbf {y}} | \mathbf {y} = g , \mathbf {x} _ {i})}{\hbar_ {i , g ^ {\prime}} \mathbb {P} (\tilde {\mathbf {y}} | \mathbf {y} = g ^ {\prime} , \mathbf {x} _ {i})} = \frac {\hbar_ {i , g} \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left\{\tau_ {i , g l} ^ {(r)} \right\} ^ {\mathbb {I} (\tilde {y} ^ {(r)} = l)}}{\hbar_ {i , g ^ {\prime}} \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left\{\tau_ {i , g ^ {\prime} l} ^ {(r)} \right\} ^ {\mathbb {I} (\tilde {y} ^ {(r)} = l)}} > 1 \right\}.
$$
Building from the abovementioned reformulation of the label correction process, we now propose an algorithm to infer the underlying ground truth by aggregating noisy crowdsourced annotations with the help of the annotator confusions. Formally, we propose the following label correction method by setting the estimated label of  $\mathbf{x}_i$  to be  $\overline{\mathbf{y}}_i \triangleq g$  if
$$
\frac {\hbar_ {i , g} \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left\{\tau_ {i , g l} ^ {(r)} \right\} ^ {\mathbf {1} \left(\tilde {y} _ {i} ^ {(r)} = l\right)}}{\hbar_ {i , g ^ {\prime}} \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left\{\tau_ {i , g ^ {\prime} l} ^ {(r)} \right\} ^ {\mathbf {1} \left(\tilde {y} _ {i} ^ {(r)} = l\right)}} > \Omega \text {f o r a n y} g ^ {\prime} \neq g, \tag {10}
$$
where  $\Omega \geq 1$  is a pre-specified threshold.
Information-theoretic bounds on the Bayes error. To theoretically justify the effectiveness of the proposed label correction method (10), we derive information-theoretic bounds on the Bayes error, given the instances. Let  $\overline{\mathcal{D}} = \{\mathbf{x}_i,\overline{\mathbf{y}}_i\}_{i = 1}^{\bar{n}}$  denote the collection of instances with estimated labels, where  $\bar{n}$  represents the size of  $\overline{\mathcal{D}}$ . We write  $\overline{\mathbf{y}} = \{\overline{\mathbf{y}}_i\}_{i = 1}^{\bar{n}}$  and the corresponding true label is denoted  $\mathbf{y} = \{\mathbf{y}_i\}_{i = 1}^{\bar{n}}$ . A loss measured by the accuracy of the estimated labels is given by  $\mathcal{L}(\overline{\mathbf{y}},\mathbf{y}) = \frac{1}{\bar{n}}\sum_{i = 1}^{\bar{n}}\mathbf{1}(\overline{\mathbf{y}}_i\neq \mathbf{y}_i)$ . Let  $\mathbb{P}(\cdot |\mathbf{y};\pmb {\tau})$  denote the joint probability distribution of  $\{\tilde{\mathbf{y}}_i\}$ , given  $\mathbf{y}$  and  $\pmb{\tau}$ , and let  $\mathbb{E}(\cdot |\mathbf{y};\pmb{\tau})$  denote the associated expectation operator, where  $\pmb{\tau}\triangleq \{\pmb{\tau}_i\}_{i = 1}^{\bar{n}}\triangleq \{\pmb{\tau}_i^{(r)}:$
$r \in [R]\}_{i=1}^{\bar{n}}$  represents the collection of the corresponding transition matrices  $\pmb{\tau}_i^{(r)}$  having  $\tau_{i,kl}^{(r)}$  as its  $(k,l)$  element. Then, the Bayes risk is defined as [40]
$$
\mathfrak {R} _ {\text {B a y e s}} (\boldsymbol {\hbar}, \mathcal {L}) = \inf  _ {\overline {{\mathbf {y}}}} \left[ \sum_ {\mathbf {y} \in [ K ] ^ {\bar {n}}} \hbar (\mathbf {y}) \mathbb {E} \{\mathcal {L} (\overline {{\mathbf {y}}}, \mathbf {y}) | \mathbf {y}; \tau \} \right], \tag {11}
$$
or  $\Re_{\mathrm{Bayes}}$  for short, where  $\hbar (\mathbf{y})$  is the joint prior probability of  $\mathbf{y}$  calculated from  $\pmb{\hbar} \triangleq \{\pmb{\hbar}_i\}_{i=1}^n$ . The following theorem identifies bounds for the Bayes risk.
Theorem 2. Let  $D_{KL}\left(\pmb{\tau}_{i,g*}^{(r)}\|\pmb{\tau}_{i,g'}^{(r)}\right)$  denote the Kullback-Leibler (KL) divergence for discrete distributions  $\pmb{\tau}_{i,g*}^{(r)}$  and  $\pmb{\tau}_{i,g'}^{(r)}$ , where, for  $i \in [\overline{n}]$ ,  $r \in [R]$ , and  $g, g' \in [K]$ ,  $\pmb{\tau}_{i,g*}^{(r)}$  and  $\pmb{\tau}_{i,g'}^{(r)}$  stand for the  $g$ th and  $g'$ th rows of  $\pmb{\tau}_{i}^{(r)}$ , respectively. For  $\pmb{\hbar} = \{\pmb{\hbar}_i\}_{i=1}^{\overline{n}}$  and  $\pmb{\tau} = \{\pmb{\tau}_i\}_{i=1}^{\overline{n}}$ , define
$$
\overline {{D}} _ {K L} (\boldsymbol {\hbar}, \boldsymbol {\tau}) = \frac {1}{\bar {n}} \sum_ {i = 1} ^ {\bar {n}} \sum_ {r = 1} ^ {R} \sum_ {g = 1} ^ {K} \sum_ {g ^ {\prime} = 1} ^ {K} \hbar_ {i, g} \hbar_ {i, g ^ {\prime}} D _ {K L} \left(\boldsymbol {\tau} _ {i, g *} ^ {(r)} \| \boldsymbol {\tau} _ {i, g ^ {\prime} *} ^ {(r)}\right) a n d
$$
$$
C _ {g g ^ {\prime}} ^ {(i)} = - \min _ {0 \leq \lambda \leq 1} \frac {1}{R} \left[ - \lambda \log \left(\frac {\Omega \hbar_ {i , g}}{\hbar_ {i , g ^ {\prime}}}\right) + \sum_ {r = 1} ^ {R} \log \left\{\sum_ {l = 1} ^ {K} \left(\tau_ {i, g l} ^ {(r)}\right) ^ {1 - \lambda} \left(\tau_ {i, g ^ {\prime} l} ^ {(r)}\right) ^ {\lambda} \right\} \right].
$$
For  $i \in \bar{n}$  and  $g \in [K]$ , let  $I_{\Omega}^{(g)}(\pmb{\hbar}_i, \pmb{\tau}_i) = \min_{g' \neq g} C_{gg'}^{(i)}$ . Then the Bayes error defined in (11) is bounded as follows:
$$
\begin{array}{l} \frac {1}{\bar {n}} \left[ 1 - \frac {\overline {{D}} _ {K L} (\boldsymbol {\hbar} , \boldsymbol {\tau}) + \frac {1}{\bar {n}} \log (2 - \prod_ {i = 1} ^ {\bar {n}} \max  _ {k \in [ K ]} \hbar_ {i , k})}{\left\{\sum_ {i = 1} ^ {\bar {n}} \log \left(\max  _ {k \in [ K ]} \hbar_ {i , k}\right) \right\} / \bar {n}} \right] \\ \leq \Re_ {B a y e s} (\boldsymbol {\hbar}, \mathcal {L}) \leq \frac {K - 1}{\bar {n}} \sum_ {i = 1} ^ {\bar {n}} \sum_ {g = 1} ^ {K} \hbar_ {i, g} \exp \left\{- R I _ {\Omega} ^ {(g)} (\boldsymbol {\hbar} _ {i}, \boldsymbol {\tau} _ {i}) \right\}. \\ \end{array}
$$
Remark 1. Theorem 2 establishes information-theoretic bounds on the Bayes error  $\Re_{\mathrm{Bayes}}$  for arbitrary priors  $\pmb{\hbar}_i$  with  $i\in [\overline{n} ]$ , which theoretically quantifies the combined impact of the prior knowledge and annotators' expertise on label accuracy using algorithm (10). The lower bound is proved in light of the concept of  $f$ -informativity [40-42], and is stronger than the commonly-used Bayes lower bound based on Fano's inequality [43, 44]. The proof of the upper bound considers the inference procedure of  $\overline{y}$  and applies Markov's inequality. The details are given in Appendix B.
Remark 2. The quantity  $C_{gg'}^{(i)}$  in Theorem 2 reflects how the identified upper bound of the Bayes error may be influenced by the prior  $\pmb{\hbar}_i$  and the ability of the  $R$  annotators to distinguish between labels  $g$  and  $g'$  for instance  $i$ . If we set  $\pmb{\hbar}_i$  to be the uniform prior and let  $\Omega = 1$ ,  $C_{gg'}^{(i)}$  will degenerate to the average of the Chernoff information between  $\{\pmb{\tau}_{i,g*}^{(r)}\}_{r=1}^R$  and  $\{\pmb{\tau}_{i,g'*}^{(r)}\}_{r=1}^R$  [45, 46], which is a statistical divergence measuring the deviation between two probability distributions.
Result under the sparse Bayesian model. The label correction method (10) is not immediately applicable if we have no access to the underlying true annotator confusions, which is usually the case in real-world applications. To get around the issue induced from the unknowness of the true noise transition probability  $f_0^{(k,r)}$ , we consider the model  $f_{\theta}^{(k,r)}$  given in Theorem 1 and write the corresponding transition matrices as  $\overline{\tau}_{i,kl}^{(r)} \triangleq \overline{\tau}_{kl}^{(r)}(\mathbf{x}_i) \triangleq f_{\theta}^{(k,r)}(\tilde{y}|\mathbf{x}_i)\big|_{\tilde{y} = l}$  for  $k,l \in [K]$  and  $r \in [R]$ . We then replace the  $\tau_{i,gl}^{(r)}$  in (10) with  $\overline{\tau}_{i,gl}^{(r)}$  to determine corrected labels. With a slight abuse of notation, we still use  $\overline{\mathcal{D}} = \{\mathbf{x}_i,\overline{\mathbf{y}}_i\}_{i=1}^n$  to denote the set of instances with estimated labels but let  $\overline{\mathfrak{R}}_{\mathrm{Bayes}}$  denote the resulting Bayes error. We let  $\overline{\mathcal{D}}_{\mathbf{x}} = \{\mathbf{x}_i\}_{i=1}^n$  represent the set of the considered instances. Combining Theorems 1 and 2 yields the following corollary.
Corollary 3. Suppose that the conditions in Theorem 1 are satisfied, and further assume that  $f(\mathbf{x}_i) > \varsigma_2$  for  $\mathbf{x}_i \in \overline{\mathcal{D}}_{\mathbf{x}}$  and  $\min_{r \in [R], i \in [\overline{n}], k, l \in [K]} \overline{\tau}_{i,kl}^{(r)} \geq \varsigma_3$ , where  $f(\cdot)$  is the probability density
function of  $\mathbf{x}$ , and  $\varsigma_{2}$  and  $\varsigma_{3}$  are some positive constants. Then, for any  $\epsilon > 0$ ,
$$
\Pi \left[ \boldsymbol {\theta}: \overline {{\mathfrak {R}}} _ {B a y e s} \leq \frac {K - 1}{\bar {n}} \sum_ {i = 1} ^ {\bar {n}} \sum_ {g = 1} ^ {K} \hbar_ {i, g} \exp \left\{- R I _ {\Omega} ^ {(g)} (\pmb {\hbar} _ {i}, \pmb {\tau} _ {i}) + \epsilon \right\} \middle | \overline {{\mathcal {D}}} _ {0} \right] \longrightarrow 1
$$
in  $\mathbb{P}_0^n$  probability as  $n\to \infty$ , where  $I_{\Omega}^{(g)}(\pmb{\hbar}_i,\pmb{\tau}_i)$  is given in Theorem 2 for  $i\in [\bar{n} ]$  and  $g\in [K]$ .
Learning the noise transition model. In the warm-up stage, we train the base models on noisy training data and obtain the set of anchor points  $\overline{\mathcal{D}}_0$  [26]. With  $\overline{\mathcal{D}}_0$ , we first obtain the maximum a posteriori (MAP) estimate of network parameters of the transition model  $\theta$  by maximizing the log posterior distribution of  $\theta$ , with the constant term omitted,
$$
\widehat {\boldsymbol {\theta}} = \arg \max  _ {\boldsymbol {\theta}} \left\{\sum_ {i = 1} ^ {n} \log p _ {\boldsymbol {\theta}, i} + \log \pi (\boldsymbol {\theta}) \right\}, \tag {12}
$$
where  $p_{\theta ,i}$  is the probability mass function of the  $i$ th component in  $\overline{\mathcal{D}}_0$  given before Theorem 1, and  $\pi (\pmb {\theta})$  is the probability density function of  $\pmb{\theta}$  relative to the prior probability measure  $\Pi (\cdot)$ . Given the MAP estimate  $\widehat{\pmb{\theta}}$ , according to the prior hierarchy (4), the posterior inclusion probability of the  $k$ th parameter in the network  $\psi_j(\cdot ;\pmb{\theta}^{(j)})$  is calculated as
$$
\mathbb {P} \left(\gamma_ {k} ^ {(j)} = 1 \mid \widehat {\boldsymbol {\theta}}\right) = \frac {\lambda_ {n j} \pi_ {1} \left(\widehat {\theta} _ {k} ^ {(j)} ; \sigma_ {n j} ^ {2}\right)}{\lambda_ {n j} \pi_ {1} \left(\widehat {\theta} _ {k} ^ {(j)} ; \sigma_ {n j} ^ {2}\right) + \left(1 - \lambda_ {n j}\right) \pi_ {0} \left(\widehat {\theta} _ {k} ^ {(j)} ; c _ {n j} \sigma_ {n j} ^ {2}\right)} \tag {13}
$$
for  $k \in [J_j]$  with  $j = 1,2$ . If the posterior inclusion probability is smaller than a pre-specified threshold, chosen to be 0.5 in our experiments, the associated parameter is zet to be zero. We then fine tune the sparse network and obtain the noise transition model.
Training the classifiers with corrected labels. With the noise transition model trained, we then train the base models with the label correction algorithm proposed in Section 3.3. Specifically, we train two base classifiers to reciprocally provide class priors for each instance in the label correction process. In the  $t$ th epoch, for instance  $\mathbf{x}_i$ , if  $\overline{\mathbf{y}}_i$  satisfies the condition (10) for the pre-specified threshold  $\Omega_t$ , we put  $\{\mathbf{x}_i, \overline{\mathbf{y}}_i\}$  in  $\overline{D}_t$ . The base models are then updated on the collected dataset  $\overline{D}_t$ . The complete pseudocode of our algorithm is included in Appendix C.
Datasets. We assess the effectiveness of our method on three image datasets with synthetic annotations, MNIST [47], CIFAR-10 [48], and CIFAR-100 [48], and two datasets with human annotations, CIFAR-10N [49] and LabelMe [50, 51]. Detailed information can be found in Appendix C. For all the datasets except LabelMe, we leave out  $10\%$  of the training data as a noisy validation set.
Noise generation. For the three datasets, MNIST, CIFAR10, and CIFAR100, we consider three groups of annotators with varying expertise, with an average labeling error rate of about  $20\%$ ,  $35\%$  and  $50\%$ , respectively. We abbreviate these three groups as IDN-LOW, IDN-MID, and IDN-HIGH, which represent instance-dependent annotators with low, middle ("mid" for short), and high labeling error rates, respectively. To generate noisy annotations, we independently simulate  $R = 5$  annotators for each group according to Algorithm 2 in [12], with IDN- $\tau$  denoting that the noise rate is upper bounded by  $\tau$  for each annotator. For each instance, we then randomly choose one of the annotations given by the  $R$  annotators, which is designed to evaluate the methods under incomplete annotator labeling. We manually corrupt the three datasets according to the following three groups of annotators:
(I) IDN-LOW.  $IDN - 10\%$ $IDN - 10\%$ $IDN - 20\%$ $IDN - 20\%$ $IDN - 30\%$
(I) IDN-MID.  ${IDN} - {30}\% ,{IDN} - {30}\% ,{IDN} - {40}\% ,{IDN} - {40}\% ,{IDN} - {50}\%$  ;
(III) IDN-HIGH.  ${IDN} - {50}\% ,{IDN} - {50}\% ,{IDN} - {60}\% ,{IDN} - {60}\% ,{IDN} - {70}\%$  .
Experiment setup. The network structure for the MNIST dataset is chosen to be Lenet-5 [52]. We choose ResNet-18 [2] for CIFAR-10 and CIFAR-10N, and ResNet-34 architecture [2] for CIFAR-100 dataset. As in [53], we employ the pretrained VGG-16 network followed by a fully connected layer and a softmax output layer for the LabelMe dataset, using  $50\%$  dropout. More implementation details can be found in Appendix C.
Competing methods. We compare the proposed method with the following state-of-art methods: (1) CE (Clean), which trains the network with the standard cross entropy loss on the clean datasets; (2) CE (MV), which trains the network using the labels from majority voting; (3) CE (EM) [9]; (4) DoctorNet [54]; (5) GCE [55]; (6) Co-teaching [56]; (7) BLTM [17]; (8) MBEM [11]; (9) CrowdLayer [53]; (10) TraceReg [8]; (11) Max-MIG [7]; (12) CoNAL [58]; (13) GeoCrowdNet (F) [13]; and (14) GeoCrowdNet (W) [13]. Among these methods, GCE, Co-teaching, Co-teaching+, and BLTM are strong baselines dealing with single noisy label issues, and we adapt them to the multiple annotations setting by utilizing the majority vote labels for loss computation.
Ablation study. In Figure 1, We plot the average estimation error for the noise transition matrices to demonstrate the effectiveness of the proposed method in modeling the instance-dependent annotator confusions. For instance  $\mathbf{x}_i$  with clean class label  $y_i$  in the validation set, we analyse the  $y_i$ th row rather than the whole noise transition matrix as in previous studies [16, 12]. Specifically, let  $\widehat{\tau}^{(r)}(\mathbf{x}_i)$  and  $\tau^{(r)}(\mathbf{x}_i)$  represent the estimated and the true noise transition matrix for annotator  $r$ , respectively. The estimation error on instance  $\mathbf{x}_i$  is defined as  $err_i^{(r)} = \max_{l \in K} |\widehat{\tau}^{(r)}(\mathbf{x}_i)_{y_i,l} - \tau^{(r)}(\mathbf{x}_i)_{y_i,l}|$ , where  $\widehat{\tau}^{(r)}(\mathbf{x}_i)_{y_i,l}$  and  $\tau^{(r)}(\mathbf{x}_i)_{y_i,l}$  stand for the  $(y_i,l)$  element in the corresponding matrices. The average estimation error for annotator  $r$  is then calculated as  $\frac{1}{n_v} \sum_{i=1}^{n_v} err_i^{(r)}$  with  $n_v$  denoting the size of the validation set. For each annotator, we compare the average estimation error of the proposed method with six baselines, CrowdLayer [53], TraceReg [8], GeoCrowdNet (F) [13], GeoCrowdNet (W) [13], MBEM [11], and BLTM [17], on the CIFAR10 dataset. In most of the cases, the proposed method leads to smaller estimation error especially when the noise rate is high, which shows the efficacy of the proposed sparse Bayesian model.
Classification accuracy. Table 1 presents the average test accuracy of 5 random trials on the datasets of CIFAR-10, CIFAR-100, CIFAR-10N and LabelMe, together with the standard errors of the test accuracies of the random trials, expressed after the plus/minus sign  $\pm$ , where the two highest accuracies are bold faced; standard errors of the accuracies are calculated based on repeating those experiments 5 times, each with a different random seed. All the results demonstrate the superior performance of the proposed method on both synthetic and real-world noisy datasets. Moreover, to investigate the influence of the sparsity of annotations, we conduct more experiments with the number of annotators varying from 5 to 100, and each instance only has one label. Figure 2 shows the average accuracy with various numbers of annotators, which further exhibit the advantages of the proposed method under different settings. Additional experimental results, including the test accuracy on MNIST, the average estimation error on MNIST and CIFAR100, and the accuracy of the corrected labels using algorithm (10), are deferred to Appendix C to save space.
In this paper, we address the challenge of training classifiers using noisy crowdsourced labels, a common issue in various applications. We formulate the annotator-specific instance-dependent noise transition matrix within the Bayesian framework, and theoretically characterize the closeness of the proposed model and the true annotator confusions with respect to the Hellinger distance. Our result is established for the setup of i.n.i.d. observations, which substantially broadens the application scope of our method. Building on the convergence rate of the posterior measure, we propose a novel algorithm to aggregate noisy annotations and infer the ground truth label based using pairwise LRT. Additionally, we provide information-theoretic bounds on the Bayes error of the proposed algorithm. Empirical evidence demonstrates the effectiveness of our algorithm on both synthetic and real-world noisy datasets.
Limitations and Extensions
Our work can be further extended in different directions. It is interesting to generalize the setup here to the hierarchical classification setup. Instance-dependent transition matrices can be further refined with varying structures imposed and are learned with manifold regularization.
Yi is the Canada Research Chair in Data Science (Tier 1). Her research was supported by the Canada Research Chairs Program and the Natural Sciences and Engineering Research Council of Canada (NSERC).