SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hui Guo
hguo288@uwo.ca
Boyu Wang *
bwang@csd.uwo.ca
Grace Y. Yi*
Department of Statistical and Actuarial Sciences
gyi5@uwo.ca
The predictive ability of supervised learning algorithms hinges on the quality of annotated examples, whose labels often come from multiple crowdsourced annotators with diverse expertise. To aggregate noisy crowdsourced annotations, many existing methods employ an annotator-specific instance-independent noise transition matrix to characterize the labeling skills of each annotator. Learning an instance-dependent noise transition model, however, is challenging and remains relatively less explored. To address this problem, in this paper, we formulate the noise transition model in a Bayesian framework and subsequently design a new label correction algorithm. Specifically, we approximate the instance-dependent noise transition matrices using a Bayesian network with a hierarchical spike and slab prior. To theoretically characterize the distance between the noise transition model and the true instance-dependent noise transition matrix, we provide a posterior-concentration theorem that ensures the posterior consistency in terms of the Hellinger distance. We further formulate the label correction process as a hypothesis testing problem and propose a novel algorithm to infer the true label from the noisy annotations based on the pairwise likelihood ratio test. Moreover, we establish an information-theoretic bound on the Bayes error for the proposed method. We validate the effectiveness of our approach through experiments on benchmark and real-world datasets.
Deep neural networks (DNNs) have achieved remarkable performance in various tasks [1, 2], and they have proven to be useful in handling sizable labeled data. Acquiring large accurately annotated datasets, however, is usually expensive and time consuming. To enhance the efficiency of annotation, in many applications, crowdsourcing [3] is employed as an alternative way for data labeling, where the labels are provided by multiple annotators with varying and imperfect labeling skills, and thus, the collected labels suffer from unavoidable noise. As deep models have a strong memorization power, using these noisy labels as the ground truth deteriorates the performance of DNNs [4, 5], and most importantly, yields erroneous learning results. Further, potentially substantial disagreement among
the annotators for each instance presents extra challenges in the application of traditional supervised learning algorithms. Hence, in the crowdsourcing scenario, to effectively train DNNs on noisy labeled datasets, a fundamental question is how to aggregate the noisy crowdsourced annotations and infer the latent true labels [6].
One naive approach to aggregate the crowdsourced labels is simply by computing the majority vote, which can be ineffective when the number of annotators is not large enough or the labeling task is difficult [7, 8]. Recent research has developed more powerful techniques for inferring the ground truth labels [7, 9-11], among which the annotator-specific noise transition matrix, aka annotator confusion, plays an important role by modeling the labeling process for each individual annotator. To estimate the transition matrix, available research [8, 11-13] usually makes the instance-independent assumption that for annotator  $r$ , given the true label  $y$ , the corruption process is independent of the input  $x$ , i.e.,  $\mathbb{P}(\tilde{y}^{(r)} = l|y = k,\mathbf{x}) = \mathbb{P}(\tilde{y}^{(r)} = l|y = k)$ , where  $x$  denotes the random variable for instance/feature,  $\tilde{y}^{(r)}$  represents the noisy label given by annotator  $r$ , and  $y$  is the underlying ground truth label. This assumption, however, is often violated in applications. Instance-dependent annotation noise is more realistic and appropriate for real-world datasets, as suggested by the example that factors such as the quality of ultrasound images and the domain expertise of human annotators can greatly influence the actual diagnostic process in medical analysis [14, 15]. For annotator  $r$ , the transition matrix  $\tau^{(r)}(\mathbf{x})$  is a matrix-valued function, with the  $(k,l)$  element defined as  $\tau_{kl}^{(r)}(\mathbf{x}) = \mathbb{P}(\tilde{y}^{(r)} = l|y = k,\mathbf{x})$ . Unfortunately, the case of instance-dependent annotation noise remains challenging and less explored. Most existing works considering instance-dependent noise are designed for the single annotator case [16-18]. For the case with multiple annotators, existing methods investigate the human annotation process and use different models to estimate instance-dependent noise matrices. Approaches in [3, 19-22] use traditional classification models such as logistic regression, while others [23-25] cater to large datasets and deep models. Methods in [3, 19-22] and [24, 25] are heuristic in nature and lack theoretical guarantees in estimating instance-dependent noise matrices. [23] makes some theoretical progress in justifying the use of the trace regularisation, and extends the work of [8] which establishes the theory only for settings with an instance-independent noise matrix. The theory in [23] is constrained to individual samples rather than the population setting. Importantly, the theoretical characterization of the distance of the noise model and the true annotator confusion remains absent from the literature.
In this paper, we address this notable problem by framing it within the Bayesian paradigm. We formulate the instance-dependent annotator-specific label transition matrix, and further propose a novel algorithm to infer the underlying ground truth by aggregating the noisy annotations. To model the noise transition matrix, we invoke the Bayesian generalized linear mixed effects model (GLMM), which can be learned by deploying anchor points within the deep learning framework [12, 26, 27]. To facilitate the fact that the number of anchor points learned from the noisy training data is relatively small compared to the sample size, we employ a hierarchical spike and slab prior on the network parameters. This approach offers an interpretable mechanism for variable selection and allows us to establish the theoretical result within the deep learning setup. Our study reveals that the proposed noise transition model is close to the underlying true transition matrix with respect to the Hellinger distance in the Bayesian framework. Such a result is established for independently, nonidentically distributed (i.n.i.d.) observations, substantially extending the existing sparse Bayesian theories within the deep learning paradigm. Further, we develop a label correction method using the pairwise likelihood ratio test to aggregate and infer the ground truth from the noisy crowdsourced annotations. This development is carried out by formulating the label correction process as a hypothesis testing problem and utilizing the proposed Bayesian model in place of the unknown transition matrix in the pairwise likelihood ratio test (LRT). More importantly, with the posterior consistency result, we also derive information-theoretic bounds on the Bayes error for the proposed algorithm even without access to the underlying true noise transition matrix.
This research brings forth several noteworthy advancements: (1) We formulate the annotator-specific noise transition matrix in the Bayesian framework (Section 3.1). This method offers a practical and flexible framework to address real-world problems with noisy annotators. (2) We theoretically characterize the closeness of the proposed model and the underlying annotator confusions with respect to the Hellinger distance. (Section 3.2). (3) We develop a novel label correction algorithm by aggregating the noisy annotations using the pairwise likelihood ratio test, and identify information-theoretic bounds on the Bayes error (Section 3.3). The effectiveness of the proposed algorithm is
confirmed by the application to both synthetic and real-world noisy datasets (Section 5). Code is available at https://github.com/hguo1728/BayesianIDNT.
Objective and Data. Consider a classification task with a feature space  $\mathcal{X} \subset \mathbb{R}^p$  and a label space  $\mathcal{Y} = [K]$ , where  $p$  is the dimension of the features,  $K$  is the number of classes, and  $[k]$  represents  $\{1,\dots,k\}$  for any positive integer  $k$ . Our goal is to develop a classifier  $h: \mathcal{X} \mapsto \mathcal{Y}$ , which can accurately predict the true label for a test instance. However, in applications, the true label  $\mathbf{y} \in \mathcal{Y}$  is often not observed for each input vector  $\mathbf{x} \in \mathcal{X}$ . Instead, we receive a set of noisy crowdsourced labels  $\tilde{\mathbf{y}} = \{\tilde{\mathbf{y}}^{(1)},\dots,\tilde{\mathbf{y}}^{(R)}\}$  from  $R$  distinct annotators, where  $\tilde{\mathbf{y}}^{(r)} \in \mathcal{Y}$  represents the label given by the  $r$ th annotator for  $r \in [R]$ . Thus, a noisy dataset  $\mathcal{D}$  of size  $N$  is defined as  $\mathcal{D} = \{\mathbf{x}_i, \tilde{\mathbf{y}}_i^{(1)},\dots,\tilde{\mathbf{y}}_i^{(R)}\}_{i=1}^N$ , where for each instance  $\mathbf{x}_i$ , the true label  $\mathbf{y}_i$  is unobserved. Under this setting, we aim to learn a reliable classifier  $h$  by utilizing the noisy crowdsourced dataset  $\mathcal{D}$ .
In practice, on commercial crowdsourcing platforms, large-scale labels can often be collected from independent human annotators. We thereby make a common assumption that the  $R$  annotators independently label the instances [7, 8]. The conditional probability of the  $R$  noisy labels, given an instance, can then be formulated as
$$
\mathbb {P} \left(\tilde {\mathcal {Y}} ^ {(1)},.., \tilde {\mathcal {Y}} ^ {(R)} | \mathbf {x}\right) = \prod_ {r = 1} ^ {R} \mathbb {P} \left(\tilde {\mathcal {Y}} ^ {(r)} | \mathbf {x}\right) = \prod_ {r = 1} ^ {R} \sum_ {k \in \mathcal {Y}} \left\{\mathbb {P} \left(\tilde {\mathcal {Y}} ^ {(r)} | \mathrm {y} = k, \mathbf {x}\right) P (\mathrm {y} = k | \mathbf {x}) \right\}, \tag {1}
$$
where for  $k \in \mathcal{V}$ ,  $\mathbb{P}(\mathrm{y} = k|\mathbf{x})$ , called the base model [28], denotes the conditional probability of the latent true label  $\mathrm{y}$  given  $\mathbf{x}$ , which can be modeled by the output of a DNN parameterized by a parameter vector, say  $\vartheta$ ; and  $\mathbb{P}(\tilde{\mathrm{y}}^{(r)}|\mathrm{y} = k,\mathbf{x})$  is the noise transition model for the  $r$ th annotator [8], satisfying  $\sum_{l=1}^{K}\mathbb{P}(\tilde{\mathrm{y}}^{(r)} = l|\mathrm{y} = k,\mathbf{x}) = 1$  for any  $\mathbf{x} \in \mathcal{X}$  and  $k \in [K]$ . For ease of theoretical presentation, we assume the accessibility to all the annotations from the  $R$  workers for now, and consider more general situations in the experimental part in Section 5; extensions to accommodating the case where each instance is only annotated by a subset of annotators are straightforward.
Notation. In this paper, sets are denoted by calligraphic upper case letters, and vectors and matrices are denoted by bold lower and upper case letters, respectively. For a vector  $\pmb{v}$ ,  $v_{j}$  denote its  $j$ th element, and  $\pmb{v}^{\top}$  denotes its transpose. For  $\pmb{v} = (v_{1},\dots,v_{d})^{\top}$ , we denote  $\| \pmb{v}\|_q = (\sum_{j=1}^d |v_j|^q)^{1/q}$  for  $q > 0$ ,  $\| \pmb{v}\|_{\infty} = \max_j |v_j|$ , and  $\| \pmb{v}\|_0 = \sum_{j=1}^d \mathbf{1}(v_j \neq 0)$ , with  $\mathbf{1}(\cdot)$  denoting the indicator function. The  $L_2$  norm of  $\pmb{v}$  is also denoted by  $\| \pmb{v}\|$  for simplicity. For a matrix  $\pmb{V}$ , we use  $V_{i,j}$  to represent its  $(i,j)$  element. Let  $(\Omega, \mathcal{G}, \mu)$  denote the measure space under consideration, where  $\Omega$  is a set,  $\mathcal{G}$  is the  $\sigma$ -field of subsets of  $\Omega$ , and  $\mu$  is the associated measure. For a measurable function  $f: \Omega \to \mathbb{R}^d$ , we write  $\| f\|_q \triangleq \| f\|_{L^q(\Omega)}$  when there is no ambiguity of the domain, where  $\| f\|_{L^q(\Omega)} = \left( \int_{\Omega} \sum_{j=1}^d |f_j(x)|^q d\mu \right)^{1/q}$  for  $q > 0$ . For two sequences,  $\{a_n\}$  and  $\{b_n\}$ , we write  $a_n \preceq b_n$  if there exists a positive constant  $C$  such that  $a_n \leq C b_n$  for large enough  $n$ , and we write  $a_n \preceq b_n$  if  $a_n \preceq b_n$  and  $b_n \preceq a_n$ .
3.1 Instance-dependent transition matrix with multiple annotators
Annotator-specific instance-dependent noise transition model. Given an instance  $\mathbf{x}$ , the conditional probability mass function of noisy annotations can be characterized by  $R$  instance-dependent matrices of dimension  $K \times K$ , termed transition matrices or annotator confusions [8, 13], with the  $k$ th row of the  $r$ th matrix denoted  $\left(\mathbb{P}(\tilde{\mathbf{y}}^{(r)} = 1 | \mathbf{y} = k, \mathbf{x}), \ldots, \mathbb{P}(\tilde{\mathbf{y}}^{(r)} = K | \mathbf{y} = k, \mathbf{x})\right)$ . Thus, the distribution of noisy annotation depends on the instance in different ways due to the differences in the annotator  $r$  and the underlying true label  $\mathbf{y}$ , which can be characterized by a Bayesian generalized linear mixed effects model (GLMM) [29, 30] in the deep learning framework.
Specifically, conditioned on the true label  $y = k$  and the feature vector  $\mathbf{x}$ , we treat the noisy label  $\tilde{y}^{(r)}$  from annotator  $r$  as a random variable generated from the distribution:
$$
\tilde {\mathbf {y}} ^ {(r)} \mid \left\{\mathrm {y} = k, \mathbf {x} \right\} \sim \operatorname {C a t} \left(\boldsymbol {s} ^ {(k, r)}\right), \tag {2}
$$
where  $\pmb{s}^{(k,r)} = (s_1^{(k,r)},\dots,s_K^{(k,r)})^\top \in S^{K - 1}$  with  $S^{K - 1} = \{(s_1,\dots,s_K)^\top \in \mathbb{R}^K:s_j\geq 0$  for  $j\in [K]$  and  $\sum_{j = 1}^{K}s_{j} = 1\}$  representing the  $(K - 1)$ -dimensional simplex, and  $\mathrm{Cat}(s^{(k,r)})$  represents a categorical distribution specified by the parameter vector  $\pmb{s}^{(k,r)}$ . We extend existing works on mixed effects neural networks (MNN) [31, 32] by employing two nonlinear transformations  $\psi_{1}$  and  $\psi_{2}$  to incorporate different effects in the instance-dependent noise transition model, and set
$$
\boldsymbol {s} ^ {(k, r)} = G \left(\boldsymbol {\omega} _ {0} ^ {(k, r)}\right) \text {w i t h} \boldsymbol {\omega} _ {0} ^ {(k, r)} = \mathbf {A} _ {0} ^ {(r)} \psi_ {1} (\mathbf {x}) + \mathbf {B} _ {0} ^ {(k)} \psi_ {2} (\mathbf {x}), \tag {3}
$$
where  $\mathbf{A}_0^{(r)} = (\pmb{\alpha}_{10}^{(r)},\dots,\pmb{\alpha}_{K0}^{(r)})^\top$  and  $\mathbf{B}_0^{(k)} = (\beta_{10}^{(k)},\dots,\beta_{K0}^{(k)})^\top$  are the regression weights;  $\psi_1(\mathbf{x})$  and  $\psi_2(\mathbf{x})$  can be modeled by some suitable networks; and  $G$  is a function mapping  $\mathbb{R}^K$  to  $S^{K - 1}$  which, in practice, is chosen to be the softmax function in the final layer. Utilizing two different network components  $\psi_{1}$  and  $\psi_{2}$  enables us to flexibly reflect possibly different effects of the annotator expertise  $(r)$  and the ground truth  $(k)$  in the annotation process, which can be interpreted as the input in mixed effects models.
Approximating the transition matrices. The proposed instance-dependent noise transition model can be learned by leveraging anchor points [12, 26, 27], or instances that are similar to anchor points learned from noisy training data [33]. An instance  $\mathbf{x}$  is defined to be an anchor point of class  $k$  if it belongs to the  $k$ th class almost surely, that is,  $\mathbb{P}(\mathrm{y} = k|\mathbf{x}) = 1$ , and hence,  $\mathbb{P}(\tilde{\mathbf{y}}^{(r)}|\mathbf{x}) = \mathbb{P}(\tilde{\mathbf{y}}^{(r)}|\mathbf{y} = k,\mathbf{x})$ . For  $k \in [K]$ , let  $\overline{\mathcal{D}}_{0,k}$  be the set of anchor points of the  $k$ th class and the associated noisy annotations, i.e.,  $\overline{\mathcal{D}}_{0,k} = \{\{\mathbf{x}_i,\tilde{\mathbf{y}}_i\} : \mathbb{P}(\mathrm{y}_i = k|\mathbf{x}_i) = 1\}$ . Define  $\overline{\mathcal{D}}_0 = \overline{\mathcal{D}}_{0,1} \cup \overline{\mathcal{D}}_{0,2} \cup \ldots \cup \overline{\mathcal{D}}_{0,K}$  and let  $n$  denote the subsample size of the learned anchor points, i.e., the cardinality of  $\overline{\mathcal{D}}_0$ . Paired variables  $\{\mathbf{x}_i,\tilde{\mathbf{y}}_i\}$  in  $\overline{\mathcal{D}}_0$  are independent, but not necessarily identically distributed (i.n.i.d). We write the input dimension  $p$  as  $p_n$  from now on to emphasize that its dependence on  $n$  is allowed.
In applications, overfitting can occur when the subsample size  $n$  of the learned anchor points is relatively small compared to the main sample size  $N$ . To address this issue, we propose to learn  $\psi_j(\mathbf{x})$  with a sparse Bayesian DNN, denoted  $\psi_j(\mathbf{x};\pmb{\theta}^{(j)})$ , where  $\pmb{\theta}^{(j)}$  represents the vector of all involved parameters in the network with  $j = 1,2$ . Furthermore, invoking the sparse Bayesian setting allows us to rigorously characterize the distance between the proposed model and the underlying true transition matrices, as presented in Theorem 1 of Section 3.2.
3.2 Bayesian analysis and posterior consistency result
Prior specification. To implement sparse Bayesian analysis, we utilize the spike and slab prior [34] on the network parameters, offering an interpretable mechanism for variable selection. The spike and slab model is formulated by constructing a prior hierarchy of the involved parameters and selects nonzero coefficients according to the posterior inclusion probability. Marginally, these priors are mutually independent and have a mixture distribution consisting of a flat distribution (slab) and a distribution concentrated at zero (spike). Parameters with a small posterior mean will be set to zero to achieve sparsity.
Specifically, for network  $\psi_{j}(\mathbf{x};\pmb{\theta}^{(j)})$ , we write  $\pmb{\theta}^{(j)}$  as  $\pmb{\theta}^{(j)} = (\theta_1^{(j)},\dots,\theta_{J_j}^{(j)})^\top$  with  $J_{j}$  denoting the length of  $\pmb{\theta}^{(j)}$  for  $j = 1,2$ . For  $k\in [J_j]$ , we treat  $\theta_k^{(j)}$  as a random variable generated from the following prior hierarchy:
$$
\gamma_ {k} ^ {(j)} \sim \operatorname {B e r n o u l l i} \left(\lambda_ {n j}\right), \tag {4a}
$$
$$
\left. \theta_ {k} ^ {(j)} \right| \gamma_ {k} ^ {(j)} \sim \gamma_ {k} ^ {(j)} \pi_ {1} \left(\theta_ {k} ^ {(j)}; \sigma_ {n j} ^ {2}\right) + (1 - \gamma_ {k} ^ {(j)}) \pi_ {0} \left(\theta_ {k} ^ {(j)}; c _ {n j} \sigma_ {n j} ^ {2}\right), \tag {4b}
$$
where  $\gamma_k^{(j)} \in \{0,1\}$  indicates whether or not  $\theta_k^{(j)}$  is nonzero,  $c_{nj}$  is specified as a very small positive number,  $\sigma_{nj}^2$  and  $c_{nj}\sigma_{nj}^2$  are the parameters related to the variances of distributions  $\pi_1(\cdot)$  and  $\pi_0(\cdot)$ , respectively, and  $\lambda_{nj} \in (0,1)$  determines the ratio of the mixture distribution. As  $c_{nj} \to 0$ ,
$\pi_0(\theta_k^{(j)};c_{nj}\sigma_{nj}^2)$  becomes the degenerate distribution at zero. The marginal distribution of  $\theta_k^{(j)}$  is then determined by
$$
\theta_ {k} ^ {(j)} \sim \lambda_ {n j} \pi_ {1} \left(\theta_ {k} ^ {(j)}; \sigma_ {n j} ^ {2}\right) + (1 - \lambda_ {n j}) \pi_ {0} \left(\theta_ {k} ^ {(j)}; c _ {n j} \sigma_ {n j} ^ {2}\right), \tag {5}
$$
which is presented as  $\pi^{(j)}(\cdot)$  for short; and this is taken as the prior distribution of  $\theta_k^{(j)}$ .
To further incorporate the effects of the true label information and the randomness from different annotators in (3), we place the following probabilistic structure on the generic weights for  $\mathbf{A}_0^{(r)}$  and  $\mathbf{B}_0^{(k)}$  in (3),  $\mathbf{A}^{(r)} = (\pmb{\alpha}_1^{(r)},\dots,\pmb{\alpha}_K^{(r)})$  and  $\mathbf{B}^{(k)} = (\beta_1^{(k)},\dots,\beta_K^{(k)})$ :
$$
\boldsymbol {\alpha} _ {j} ^ {(r)} \sim \mathcal {N} (\mathbf {0}, \boldsymbol {\Sigma} _ {\alpha} ^ {(r)}) \text {a n d} \beta_ {j} ^ {(k)} \sim \mathcal {N} (\mathbf {0}, \boldsymbol {\Sigma} _ {\beta} ^ {(k)}). \tag {6}
$$
for  $j, k \in [K]$  and  $r \in [R]$ , where  $\pmb{\Sigma}_{\alpha}^{(r)}$  and  $\pmb{\Sigma}_{\beta}^{(k)}$  are nonnegative definite matrices. We use  $\pi_A^{(r)}(\cdot)$  and  $\pi_B^{(k)}(\cdot)$  to denote the prior distribution of  $\mathbf{A}^{(r)}$  and  $\mathbf{B}^{(k)}$  in (6). Here the regression weights  $\mathbf{A}^{(r)}$  and  $\mathbf{B}^{(k)}$  can be seen as fully connected layers on top of  $\psi_1(\mathbf{x}; \pmb{\theta}^{(1)})$  and  $\psi_2(\mathbf{x}; \pmb{\theta}^{(2)})$ , respectively. The conditions on the aforementioned priors are given in Appendix A.3.
Prior and posterior probability measure. Let  $\pmb{\theta} = \left(\pmb{\theta}^{(1)\top},\pmb{\theta}^{(2)\top},\mathrm{vec}(\mathbf{B}^{(1)})^{\top},\dots,\mathrm{vec}(\mathbf{B}^{(K)})^{\top},\right.$ $\left.\mathrm{vec}(\mathbf{A}^{(1)})^{\top},\dots,\mathrm{vec}(\mathbf{A}^{(R)})^{\top}\right)^{\top}\in \Theta$  stand for the vector of all involved parameters in the noise transition model, with  $\Theta$  denoting the parameter space. We use  $\pmb{\theta}_0$  to represent the true value of  $\pmb{\theta}$ , which is an interior point of  $\Theta$ . The foregoing specification of the prior distribution places a prior probability measure, denoted  $\Pi (\cdot)$ , on  $\pmb{\theta}$ . With the data  $\overline{\mathcal{D}}_0$ , the posterior probability measure  $\Pi (\cdot |\overline{\mathcal{D}}_0)$  is given by
$$
\Pi (G | \overline {{\mathcal {D}}} _ {0}) = \frac {\int_ {G} p _ {\boldsymbol {\theta}} ^ {n} (\overline {{\mathcal {D}}} _ {0}) d \Pi (\boldsymbol {\theta})}{\int_ {\Theta} p _ {\boldsymbol {\theta}} ^ {n} (\overline {{\mathcal {D}}} _ {0}) d \Pi (\boldsymbol {\theta})} \text {f o r a n y} G \in \mathcal {G}, \tag {7}
$$
where  $\mathcal{G}$  is the  $\sigma$ -field on  $\Theta$ , and  $p_{\theta}^{n}$  is the joint probability density or mass function for the observations in  $\overline{\mathcal{D}}_0$  under  $\theta$ . Let  $\mathbb{P}_{\theta}^{n}(\cdot)$  denote the probability measure associated with  $p_{\theta}^{n}(\cdot)$ , and write  $\mathbb{P}_0^n (\cdot)\triangleq \mathbb{P}_{\theta_0}^n (\cdot)$ . Hence, the data  $\overline{\mathcal{D}}_0$  is generated from  $\mathbb{P}_0^n (\cdot)$  in our setup.
Let  $f$  denote the unknown density of  $\mathbf{x}$ . For  $\{\mathbf{x},\tilde{\mathbf{y}}\} \in \overline{\mathcal{D}}_{0,k}$ , let  $f_0^{(k,r)}$  and  $f_0^{(k)}$  respectively represent the underlying true distributions for  $\tilde{y}^{(r)}$  and  $\tilde{\mathbf{y}}$ , given  $\{\mathrm{y} = k,\mathbf{x}\}$ , determined by (2) and (3); and let  $f_{\theta}^{(k,r)}$  and  $f_{\theta}^{(k)}$  denote the corresponding distributions characterized by the model indexed by  $\pmb{\theta}$ . We let  $p_{\theta ,i}\triangleq f_{\theta}^{(k_i)}f$  denote the probability density or mass function of the  $i$ th component in  $\overline{\mathcal{D}}_0$  under  $\pmb{\theta}$ , with  $k_{i}\in [K]$  denoting the class that the instance belongs to almost surely. Then, the joint probability density or mass function  $p_{\theta}^{n}$  is calculated as  $p_{\theta}^{n}\triangleq \prod_{i = 1}^{n}p_{\theta ,i}$ . The following theorem describes the closeness of the proposed noise transition model and the true annotator confusions with respect to the Hellinger distance within the Bayesian framework.
Theorem 1. Suppose Conditions A.1-A.4 in Appendix A.2 and B.1-B.3 in Appendix A.3 are satisfied. Let  $d(\cdot, \cdot)$  and  $d_n(\cdot, \cdot)$  denote the Hellinger distance given in Definition 1 and the semimetric defined in (16) in Appendix B.1, respectively. Then there exists a sequence of constants  $\{\epsilon_n^2\}_{n=1}^\infty$  with  $\epsilon_n^2 = O(\varpi_{n1} + \varpi_{n2} + \zeta_n)$  and  $\log(1/\epsilon_n^2) \prec n\epsilon_n^2$ , satisfying  $0 < \epsilon_n^2 < 1$ ,  $\epsilon_n \to 0$  and  $n\epsilon_n^2 \to \infty$  as  $n \to \infty$ , such that
$$
\Pi \left\{\boldsymbol {\theta} \in \Theta : d _ {n} (\boldsymbol {\theta}, \boldsymbol {\theta} _ {0}) > M _ {n} \epsilon_ {n} | \overline {{\mathcal {D}}} _ {0} \right\}\rightarrow 0 \tag {8}
$$