SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} \left(\bar {y} _ {i} \neq 1 \mid y _ {i} = 1; \boldsymbol {\tau}\right) \\ \leq \sum_ {g ^ {\prime} = 2} ^ {K} \mathbb {P} (\bar {y} _ {i} = g ^ {\prime} | y _ {i} = 1; \boldsymbol {\tau}) \\ \leq \sum_ {g ^ {\prime} = 2} ^ {K} \mathbb {P} \left\{\left(\frac {\hbar_ {i , g ^ {\prime}}}{\Omega \hbar_ {i , 1}}\right) \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left(\frac {\overline {{\tau}} _ {i , g ^ {\prime} l} ^ {(r)}}{\overline {{\tau}} _ {i , 1 l} ^ {(r)}}\right) ^ {\mathbf {1} \left(\mathrm {y} _ {i} ^ {(r)} = l\right)} > 1 \Bigg | \mathrm {y} _ {i} = 1; \boldsymbol {\tau} \right\} \\ = \sum_ {g ^ {\prime} = 2} ^ {K} \mathbb {P} \left\{\left(\frac {\hbar_ {i , g ^ {\prime}}}{\Omega \hbar_ {i , 1}}\right) \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left(\frac {\tau_ {i , g ^ {\prime} l} ^ {(r)}}{\tau_ {i , 1 l} ^ {(r)}}\right) ^ {\mathbf {1} (\mathbf {y} _ {i} ^ {(r)} = l)} \cdot \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left(\frac {\tau_ {i , 1 l} ^ {(r)}}{\overline {{\tau}} _ {i , 1 l} ^ {(r)}} \cdot \frac {\overline {{\tau}} _ {i , g ^ {\prime} l} ^ {(r)}}{\tau_ {i , g ^ {\prime} l} ^ {(r)}}\right) ^ {\mathbf {1} (\mathbf {y} _ {i} ^ {(r)} = l)} > 1 \Bigg | \mathrm {y} _ {i} = 1; \boldsymbol {\tau} \right\}. \\ \end{array}
$$
If  $\{\overline{\tau}_{k*}^{(r)}(\cdot)\}_{k\in [K],r\in [R]}\in S$  using Taylor's expansion, we have that
$$
\begin{array}{l} \log \left\{\prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left(\frac {\tau_ {i , 1 l} ^ {(r)}}{\overline {{\tau}} _ {i , 1 l} ^ {(r)}} \cdot \frac {\overline {{\tau}} _ {i , g ^ {\prime} l} ^ {(r)}}{\tau_ {i , g ^ {\prime} l}}\right) ^ {\mathbf {1} (\mathrm {y} _ {i} ^ {(r)} = l)} \right\} \\ = \sum_ {r = 1} ^ {R} \sum_ {k = 1} ^ {K} \left(\log \frac {\overline {{\tau}} _ {i , g ^ {\prime} l} ^ {(r)}}{\tau_ {i , g ^ {\prime} l} ^ {(r)}} - \log \frac {\overline {{\tau}} _ {i , 1 l} ^ {(r)}}{\tau_ {i , 1 l} ^ {(r)}}\right) \mathbf {1} \left(\mathrm {y} _ {i} ^ {(r)} = l\right) \leq \kappa M _ {n} \epsilon_ {n}, \\ \end{array}
$$
for some constant  $\kappa > 0$  and large enough  $n$ . For any  $\epsilon > 0$ , we have that  $\kappa M_n \epsilon_n < \epsilon$  by taking  $M_n = \epsilon / (2\kappa M_n \epsilon_n)$ . Thus, we further obtain that
$$
\begin{array}{l} \mathbb {P} \left(\bar {y} _ {i} \neq 1 | \mathrm {y} _ {i} = 1; \boldsymbol {\tau}\right) \leq \sum_ {g ^ {\prime} = 2} ^ {K} \mathbb {P} \left(\exp (\epsilon) \left(\frac {\hbar_ {i , g ^ {\prime}}}{\Omega \hbar_ {i , 1}}\right) \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left(\frac {\tau_ {i , g ^ {\prime} l} ^ {(r)}}{\tau_ {i , 1 l} ^ {(r)}}\right) ^ {\mathbf {1} \left(\mathrm {y} _ {i} ^ {(r)} = l\right)} > 1 \Bigg | \mathrm {y} _ {i} = 1; \boldsymbol {\tau}\right) \\ \leq (K - 1) \exp \left\{- R I _ {\Omega} ^ {(1)} (\pmb {\hbar} _ {i}, \pmb {\tau} _ {i}) + \epsilon \right\} \\ \end{array}
$$
for  $\{\overline{\tau}_{k*}^{(r)}(\cdot)\}_{k\in [K],r\in [R]}\in \mathcal{S}$ , where in the second step, we use Markov's inequality and the definition of  $I_{\Omega}^{(1)}(\pmb {\hbar}_i,\pmb {\tau}_i)$ . Similar results also hold for  $g = 2,\ldots ,K$ . Hence, according to Theorem 1, we have that
$$
\Pi \left\{\overline {{\boldsymbol {\tau}}}:\mathfrak {R} _ {\mathrm {B a y e s}}\leq \frac {K - 1}{\bar {n}} \sum_ {i = 1} ^ {\bar {n}}\sum_ {g = 1} ^ {K}\hbar_ {i, g}e ^ {- R I _ {\Omega} ^ {(g)} (\pmb {\hbar} _ {i},\pmb {\tau} _ {i}) + \epsilon}\bigg| \overline {{\mathcal {D}}} _ {0} \right\} \geq \Pi (\mathcal {S} | \overline {{\mathcal {D}}} _ {0}),
$$
where  $\Pi (\mathcal{S}|\overline{\mathcal{D}}_0)\longrightarrow 1$  in  $\mathbb{P}_0^n$  probability as  $n\to \infty$ . Thus, the proof is completed.
C Implementation details and additional experimental results
Dataset description. We assess the effectiveness of our method on three image datasets with synthetic annotations, MNIST [47], CIFAR-10 [48], and CIFAR-100 [48], and two datasets with human annotations, CIFAR-10N [49] and LabelMe [50, 51]. MNIST consists of  $28 \times 28$  grayscale images with 10 classes, containing 60,000 training images and 10,000 test images. CIFAR10 has 10 classes of  $32 \times 32 \times 3$  color images, with 50,000 images used for training and 10,000 for testing. CIFAR100 also consists of 50,000 training images and 10,000 test images, whose size is  $32 \times 32 \times 3$ , but with 100 fine-grained classes. We further consider two additional datasets with human annotations, CIFAR-10N [49] and LabelMe [50, 51]. For each instance in CIFAR10, CIFAR-10N provides three independent human annotated noisy labels, with the aggregation of three noisy labels by majority voting being  $9.03\%$ . LabelMe is an image classification dataset consists of 10,000 training images, 500 validation images, and 1,188 test images. For images in the training set, LabelMe has noisy and incomplete labels provided by a total of  $R = 59$  annotators, with each image being labeled by an average of 2.547 annotations. For all the datasets except LabelMe, we leave out  $10\%$  of the training data as a noisy validation set.
Experiment setup. The network structure for the MNIST dataset is chosen to be Lenet-5 [52]. We choose ResNet-18 [2] for CIFAR-10 and CIFAR-10N, and ResNet-34 architecture [2] for CIFAR-100. As in [53], we employ the pretrained VGG-16 network, followed by a fully connected layer and a softmax output layer for the LabelMe dataset, using  $50\%$  dropout. We take the batch size to be 128 for all the datasets. For MNIST, we use the SGD optimizer with momentum 0.9, weight decay  $5 \times 10^{-4}$ , and an initial learning rate of  $10^{-2}$ . The learning rate is divided by 10 at the 40th epoch, and we set 80 epochs in total, in which the first 20 epochs are used to warm up the model on the noisy dataset and to determine anchor points. For CIFAR10, CIFAR100, CIFAR10N and LabelMe, the Adam optimizer [77] is utilized with the weight decay  $5 \times 10^{-4}$ . For CIFAR10, CIFAR100 and CIFAR10N, the initial learning rate is set to be  $10^{-3}$ , and the network is trained for 120, 150, and 120 epochs for CIFAR10, CIFAR100 and CIFAR10N, respectively, with the first 30 epochs used as the warm-up stage. The model is trained on LabelMe for 100 epochs, with an initial learning rate  $5 \times 10^{-4}$  and first 20 epochs as the warm-up stage.
Baselines. We compare the proposed method with the following state-of-art methods: (1) CE (Clean), which trains the network with the standard cross entropy loss on the clean datasets; (2) CE (MV), which trains the network using the labels from majority voting; (3) CE (EM) [9], which obtains the aggregated labels utilizing the EM algorithm; (4) DoctorNet [54], which models the annotators individually and then learns averaging weights for combining them; (5) GCE [55], which generalizes
the mean absolute error and the cross entropy loss to combat errors in training labels; (6) Co-teaching [56], which trains two networks and cross-trains on instances with small loss values; (6) Co-teaching+ [57], which bridges the "Update by Disagreement" strategy with the Co-teaching method; (7) BLTM [17], which directly models the transition matrix from Bayes optimal labels to noisy labels and learns a classifier to predict Bayes optimal labels; (8) MBEM [11], which alternates in rounds between estimating annotator quality from disagreement with the current model and updating the model by optimizing the a loss function that accounts for the current estimate of worker quality; (9) CrowdLayer [53], which concatenates the classifier with multiple annotator-specific layers and simultaneously learns the parameters; (10) TraceReg [8], which uses a loss function similar to CrowdLayer [53], but adds a regularization to establish identifiability of the confusion matrices and the classifier; (11) Max-MIG [7], which jointly aggregates the noisy crowdsourced labels and trains the classifier; (12) CoNAL [58], which decomposes the annotation noise into common and individual confusions; (13) GeoCrowdNet (F) [13]; and (14) GeoCrowdNet (W) [13], which are two regularized variants of the coupled cross-entropy minimization to enhance the identifiability of the confusion matrices. Among these methods, GCE and Co-teaching are strong baselines dealing with single noisy label issue, and we adapt them to the multiple annotations setting by utilizing the majority vote labels for loss computation.
Implementation details. We first warm up the base models on the noisy dataset with majority vote labels, obtain the set of anchor points  $\overline{\mathcal{D}}_0$ , and train the sparse Bayesian model on  $\overline{\mathcal{D}}_0$  by maximizing the log posterior distribution of network parameters and excluding non-informative parameters with low posterior inclusion probability. With the noise transition model trained, we then iteratively implement the label correction algorithm (10) and update the base models, where linearly increase the threshold  $\Omega_t$  in the training process. Specifically, let  $ER$  denote the estimated average noise rate, and let  $r_0$  and  $r_1$  represent two prespecified constants with  $r_1 \geq r_0 > 0$ , which determine the magnitude of  $\Omega_t$  at the beginning and the end of the training process. In the  $t$ th epoch, we set the threshold  $\Omega_t$  to be  $\Omega_t = (1 - ER) \cdot (r_0 + t \cdot \frac{r_1 - r_0}{T})$  in the experiment. The implementation procedure of the proposed method is summarized in Algorithm 1.
C.2 Additional experimental results.
Accuracy and number of selected labels. To validate the effectiveness of the proposed label correction method, we plot the accuracy and the number of corrected labels using criterion (10) in Figures 3 and 4 for all the considered datasets, where the solid line is the accuracy (left y-axis) and the dashed line is the number (right y-axis) of corrected labels.
Classification accuracy on MNIST. The test accuracy on CIFAR10, CIFAR100, CIFAR10, and LabelMe are provided in Section 5. Here we present the average test accuracy on the MNIST dataset in Table 2, where the two highest accuracies are bold faced. Clearly, that the proposed method achieves the best performance.
Hyperparameter analysis on the CIFAR100 dataset. We conduct sensitivity analyses about hyperparameters  $r_0$  and  $r_1$  on the CIFAR100 dataset, where we choose  $r_0$  from \{3, 5, 10, 15, 20\} and  $r_1$  from \{200, 250, 300\}. As discussed in the implementation details in Section C.1,  $r_0$  and  $r_1$  determine the magnitude of the threshold  $\Omega_t$  at the beginning and the end of the training process, respectively. With higher values of  $r_0$  and  $r_1$ , the accuracy of the corrected labels using algorithm (10) will increase accordingly, but the number of corrected labels will decrease. As shown in Table 3, with different choices of  $r_0$  and  $r_1$ , the proposed method consistently outperforms all the compared methods.
Classification accuracy on CIFAR100 with varying number of annotators. Figure 5 shows the average accuracy on CIFAR100 with the number of annotators varying from 5 to 100, which further demonstrate the superiority of the proposed method under different settings.
Average estimation error. For synthetic noisy datasets CIFAR10 and CIFAR100, we compare the average estimation error of the proposed method with six competing methods, CrowdLayer [53], TraceReg [8], GeoCrowdNet (F) [13], GeoCrowdNet (W) [13], MBEM [11], and BLTM [17]. The definition of the average estimation error and the results on CIFAR10 with 5 annotators are given in the Ablation study of Section 5. In Figures 6 and 7, we respectively present the average estimation error on the validation set of CIFAR10 and CIFAR100 with varying numbers of annotators, where the results of CrowdLayer, TraceReg, GeoCrowdNet (F) and GeoCrowdNet (W) overlap in some subfigures of Figure 7. The proposed method outperforms all the compared methods with lower average estimation error for each annotator in most of the cases, further demonstrating the effectiveness of the proposed sparse Bayesian model.