SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {V a r} \left(\psi_ {\text {g e n i e}} (\widetilde {\mathcal {D}} _ {\mathrm {A}})\right) = \operatorname {V a r} \left(\psi_ {\text {s o f t}} (\mathcal {D}) - \frac {1}{n} \sum_ {i = 1} ^ {n} Z _ {c _ {i}}\right) \\ \stackrel {\mathrm {(a)}} {=} \operatorname {V a r} \left(\psi_ {\text {s o f t}} (\mathcal {D})\right) + \frac {1}{n} \operatorname {V a r} (Z) \\ \stackrel {\text {(b)}} {=} \frac {\operatorname {V a r} \left(Y _ {M : M}\right)}{n} + \frac {1}{n} \operatorname {V a r} (Z), \tag {42} \\ \end{array}
$$
where (a) follows from the independence between  $\mathbf{Y}$  and  $\mathbf{Z}$ , and the fact that  $Z_{j}$ 's (with  $j\in [M]$ ) are i.i.d.; and (b) follows by Proposition 1.
Finally, to show the asymptotic normality, we observe that from (41), we have that
$$
\begin{array}{l} \psi_ {\mathrm {g e n i e}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) = \frac {1}{n} \sum_ {i = 1} ^ {n} (1 - (\boldsymbol {y} _ {i}) _ {c _ {i}} - (\boldsymbol {z} _ {i}) _ {c _ {i}}) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - \max  _ {j \in [ M ]} \left(\boldsymbol {y} _ {i}\right) _ {j} - \left(\boldsymbol {z} _ {i}\right) _ {c _ {i}}\right) \\ = \frac {d}{n} \sum_ {i = 1} ^ {n} \left(1 - \max  _ {j \in [ M ]} \left(\boldsymbol {y} _ {i}\right) _ {j} - \left(\boldsymbol {z} _ {i}\right) _ {1}\right), \tag {43} \\ \end{array}
$$
where the last equality follows since  $(z_{i})_{k} \stackrel{d}{=} (z_{i})_{\ell}$  for all  $(k, \ell) \in [M]^{2}$ . Since (43) is the sample mean of  $n$  realizations from  $1 - Y_{M:M} - Z$ , the central limit theorem leads to
$$
\begin{array}{l} \sqrt {n} (\psi_ {\mathrm {g e n i e}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) - P _ {e}) \stackrel {d} {\to} \mathcal {N} (0, \mathrm {V a r} (1 - Y _ {M: M} - Z)) \\ \stackrel {d} {=} \mathcal {N} (0, \operatorname {V a r} \left(Y _ {M: M}\right) + \operatorname {V a r} (Z)), \tag {44} \\ \end{array}
$$
where the last equality is due to the independence between  $Y_{M:M}$  and  $Z$ . This concludes the proof of Proposition 2.
C Details on existing BER estimators / Additional experiments
C.1 Details on existing BER estimators
C.1.1 k-NN BER bounds [16]
Let  $P_{e}^{\mathrm{NN}}$  be the error rate of the 1-nearest neighbor (NN) classifier with  $n$  samples. Then, for binary classification, it follows that for  $n \to \infty$  [16],
$$
\frac {1}{2} \left(1 - \sqrt {1 - 2 P _ {e} ^ {\mathrm {N N}}}\right) \leq P _ {e} \leq P _ {e} ^ {\mathrm {N N}}. \tag {45}
$$
The above bounds were proved by showing the convergence of the conditional posterior probability of the nearest neighbor to the true conditional posterior probability. For details, we refer an interested reader to [16].
To generalize (45) to  $M$ -classification problems, it was shown that for  $n \to \infty$  [16],
$$
\frac {M - 1}{M} \left(1 - \sqrt {1 - \frac {M}{M - 1} P _ {e} ^ {\mathrm {N N}}}\right) \leq P _ {e} \leq P _ {e} ^ {\mathrm {N N}}. \tag {46}
$$
In our experiments, we chose  $k = 1$  as this value provides the tightest bounds among larger values of  $k$  [16, 72], and we plotted the upper and lower bounds defined in (46). In particular, in order to evaluate  $P_{e}^{\mathrm{NN}}$  on given data samples, we generated true one-hot labels for all data samples from their soft labels by taking the index having the maximum value in the soft labels.
C.1.2 Generalized Henze-Penrose (GHP) divergence BER bounds [75]
Consider a parameter  $p \in (0,1)$  and two distributions  $f_0$  and  $f_1$ . Then, the Henze-Penrose (HP) divergence between  $f_0$  and  $f_1$  is defined as,
$$
D _ {p} \left(f _ {0}, f _ {1}\right) = \frac {1}{4 p (1 - p)} \left(\int \frac {\left(p f _ {0} (\boldsymbol {x}) - (1 - p) f _ {1} (\boldsymbol {x})\right) ^ {2}}{p f _ {0} (\boldsymbol {x}) + (1 - p) f _ {1} (\boldsymbol {x})} \mathrm {d} \boldsymbol {x} - (2 p - 1) ^ {2}\right). \tag {47}
$$
An appealing feature of  $D_{p}$  is that it can be estimated directly from the data using the generalized Friedman-Rafsky (FR) statistic [28, 75], which uses the Euclidean minimal spanning tree.
Then, for  $f_0$  and  $f_1$  with prior probabilities  $p$  and  $1 - p$ , the BER is bounded as [6],
$$
\frac {1}{2} - \frac {1}{2} \sqrt {u _ {p} \left(f _ {0} , f _ {1}\right)} \leq P _ {e} \leq \frac {1}{2} - \frac {1}{2} u _ {p} \left(f _ {0}, f _ {1}\right), \tag {48}
$$
$$
u _ {p} \left(f _ {0}, f _ {1}\right) = 4 p (1 - p) D _ {p} \left(f _ {0}, f _ {1}\right) + (2 p - 1) ^ {2}, \tag {49}
$$
and  $D_{p}(f_{0},f_{1})$  can be estimated based on the minimal spanning tree [6].
To generalize the HP bounds in (48), consider an  $M$ -class classification problem with  $p_1, \ldots, p_M$  as class prior probabilities and class conditional probability densities given by  $f_k(\boldsymbol{x}) \coloneqq f(\boldsymbol{x} | c = k)$ ,  $k = 1, \ldots, M$ . Let  $f^{(M)}(\boldsymbol{x}) \coloneqq \sum_{k=1}^{M} p_k f_k(\boldsymbol{x})$ . Then, the generalized Henze-Penrose (GHP)-integral is defined as,
$$
\mathsf {G H P} ^ {(M)} \left(f _ {i}, f _ {j}\right) = \int_ {\mathcal {S}} \frac {f _ {i} (\boldsymbol {x}) f _ {j} (\boldsymbol {x})}{f ^ {(M)} (\boldsymbol {x})} \mathrm {d} \boldsymbol {x}, \tag {50}
$$
where  $S$  is the support of  $f^{(M)}(\pmb{x})$ . Let  $\delta_{i,j}^{(M)} := \int \frac{p_i p_j f_i(\pmb{x}) f_j(\pmb{x})}{f^{(M)}(\pmb{x})} \, \mathrm{d}\pmb{x}$ . Then, it was shown in [75] that the BER upper bound is
$$
P _ {e} \leq 2 \sum_ {i = 1} ^ {M - 1} \sum_ {j = i + 1} ^ {M} \delta_ {i, j} ^ {(M)}, \tag {51}
$$
and the BER lower bound is
$$
P _ {e} \geq \frac {M - 1}{M} \left(1 - \left(1 - \frac {2 M}{M - 1} \sum_ {i = 1} ^ {M - 1} \sum_ {j = i + 1} ^ {M} \delta_ {i, j} ^ {(M)}\right) ^ {\frac {1}{2}}\right). \tag {52}
$$
In our experiments, we plotted the GHP BER upper and lower bounds defined in (51) and (52). Similar to the  $k$ -NN BER bounds, we generated true one-hot labels for all data samples before evaluating the GHP divergence  $D_{p}$ . Computing the above bounds requires to evaluate  $\delta_{i,j}^{(M)}$  and this can be done by using the generalized FR statistics (for more details, see [75, Section IV.]).
C.2 Additional experiments
Figure 5: Misclassification error rates of several classifiers on CIFAR-10 and CIFAR-10H datasets.
In Figure 5, we compare the estimated BER with the misclassification error rate of a few popular neural network models on the CIFAR-10 and CIFAR-10H datasets. We chose ResNet [40], VGG [78],
(b) Noisy soft labels.
Figure 6: Comparison of  $\mathsf{MoB}_K(\psi_{\mathsf{C}})$  with the GHP bounds and the  $k$ -NN bounds with  $k = 1$ . The Gaussian samples are generated according to  $\mathcal{N}(c, \sigma^2 I_2)$ , where  $c \in \{(1,1), (-1,1), (-1, -1), (1, -1)\}$  is equally likely. We consider  $\mathbf{Z} \sim \mathcal{N}\left(\mathbf{0}, \frac{1}{5} I_4\right)$ .  $\mathsf{MoB}_K(\psi_{\mathsf{C}})$  uses  $K = \lfloor \sqrt{n} \rfloor$ , the Euclidean distance for  $\mathsf{d}$ , and  $r = 1/5$ . We iterate the experiments 10 times and plot the average of them.
DenseNet [43], and Vision Transformer (ViT) [24] as classifiers to compare with and we trained these models on the CIFAR-10 dataset. For the error rate evaluated on the CIFAR-10H dataset, we followed the same experiment setup as in [46], under which we generated the ground-truth labels for testset images from the soft labels (i.e., the probabilities for each class being associated with the input image) and evaluated the testset error rate. We iterated this procedure 20 times and plotted the average of the error rates. Figure 5 shows that the error rates on CIFAR-10 from the models are worse than the estimated BER, with the exception of ViT. As noted in [46], the reasons for having a lower error rate for ViT than the estimated BER are mainly due to 1) testset over-fitting and 2) various labeling difficulties on the dataset CIFAR-10H that is used for estimating the BER. After compensating for such difficulties on CIFAR-10H, all models' error rates (evaluated on CIFAR-10H) are worse than the estimated BER; this suggests that the estimated BER may be a good estimation since the error rate of any classifier is larger than the BER.
Figure 6 shows the comparison of  $\mathrm{MoB}_K(\psi_C)$  with the GHP and  $k$ -NN bounds on the BER. For each class  $c \in \{(1,1), (-1,1), (-1, -1), (1, -1)\}$ , we generated 500 Gaussian samples  $\mathcal{N}(c, \sigma^2 I_2)$  to have a total of 2,000 samples. The label noise was generated according to  $\mathbf{Z} \sim \mathcal{N}(\mathbf{0}, \frac{1}{5} I_4)$ . The case of no outliers is considered in this experiment. We observe that our BER estimator outperforms the other estimators over  $\sigma^2 \in [0.1, 2]$  in both cases of noiseless soft labels and noisy soft labels.
We also made performance comparisons varying different parameters (i.e.,  $\mu, \sigma^2, r$ ) in Figures 7, 8, 9, and 10. Similar to the experiment setting for Figure 6, we generated a total of 2,000 Gaussian samples with different parameters that are specified in the caption of each figure. The effect of  $\mu$  and  $\sigma^2$  on the BER estimation is shown in Figures 7 and 8 under different choices of the radius  $r$ . We also provide performance comparisons of our estimator with the other estimators for different values of the noise power  $\sigma_{\mathbf{Z}}^2$  in Figures 9 and 10. Choosing an appropriate value for  $r$  is critical in the estimation of the BER as shown in Figure 10. An appropriate value for  $r$  can be empirically determined by using the elbow method [84], which is commonly used for choosing the number of clusters in  $k$ -means clustering algorithms. Using Figure 10, we selected  $r = 0.2$  (elbow point) for the main experiment in Figure 2. Overall, from these figures, we observe that our estimator outperforms the GHP and the 1-NN classifier BER bounds in various settings.
<sup>11</sup>We fine-tuned the official ViT-16B model (pre-trained onImagenet21k) on CIFAR-10 dataset; for this we used the implementation in [48]. Moreover, we trained the other models using the implementation in [70].
(a)  $r = 0.3$
(b)  $r = 0.4$
Figure 7: Comparison of  $\mathsf{MoB}_K(\psi_{\mathbb{C}})$  with the GHP bounds and the  $k$ -NN bounds with  $k = 1$ . The Gaussian samples are generated according to  $\mathcal{N}(\pmb {c},I_2)$ , where  $\pmb {c}\in \{(\mu ,\mu),(-\mu ,\mu),(-\mu , - \mu),(\mu , - \mu)\}$  with equal probability. We consider  $\mathbf{Z}\sim \mathcal{N}\big(0,\frac{1}{5} I_4\big)$ .  $\mathsf{MoB}_K(\psi_{\mathbb{C}})$  uses  $K = \lfloor \sqrt{n}\rfloor$ , the Euclidean distance for  $\mathsf{d}$ , and  $r\in \{0.3,0.4\}$ . We iterate the experiments 10 times and plot the average of them.
(a)  $r = 0.3$
Figure 8: Comparison of  $\mathrm{MoB}_K(\psi_{\mathbb{C}})$  with the GHP bounds and the  $k$ -NN bounds with  $k = 1$ . The Gaussian samples are generated according to  $\mathcal{N}(c, \sigma^2 I_2)$ , where  $c \in \{(1,1), (-1,1), (-1, -1), (1, -1)\}$  with equal probability. We consider  $\mathbf{Z} \sim \mathcal{N}\left(\mathbf{0}, \frac{1}{5} I_4\right)$ .  $\mathrm{MoB}_K(\psi_{\mathbb{C}})$  uses  $K = \lfloor \sqrt{n} \rfloor$ , the Euclidean distance for  $\mathsf{d}$ , and  $r \in \{0.3, 0.4\}$ . We iterate the experiments 10 times and plot the average of them. As expected, a larger value of  $r$  in Figure 8b yields better performance than a smaller value of  $r$  in Figure 8a.
(b)  $r = 0.4$
(a)  $\sigma_{\mathbf{Z}}^{2} = 0.3$
(b)  $\sigma_{\mathbf{Z}}^2 = 0.4$
Figure 9: Comparison of  $\mathrm{MoB}_K(\psi_{\mathbb{C}})$  with the GHP bounds and the  $k$ -NN bounds with  $k = 1$ . The Gaussian samples are generated according to  $\mathcal{N}(c, I_2)$ , where  $c \in \{(\mu, \mu), (-\mu, \mu), (-\mu, -\mu), (\mu, -\mu)\}$  with equal probability. We consider  $\mathbf{Z} \sim \mathcal{N}\left(\mathbf{0}, \sigma_{\mathbf{Z}}^2 I_4\right)$ .  $\mathrm{MoB}_K(\psi_{\mathbb{C}})$  uses  $K = \lfloor \sqrt{n} \rfloor$ , the Euclidean distance for  $d$ , and  $r = 0.2$ . We iterate the experiments 10 times and plot the average of them. In both figures, the performance of our BER estimator at the extreme values of  $\mu$  is worse than the performance of the other estimators; this is due to an inappropriate choice of  $r$ .
(a)  $\sigma_{\mathbf{Z}}^2 = 0.3$
Figure 10: Comparison of  $\mathsf{MoB}_K(\psi_C)$  with the GHP bounds and the  $k$ -NN bounds with  $k = 1$  with respect to the cluster radius  $r$ . The Gaussian samples are generated according to  $\mathcal{N}(c, I_2)$ , where  $c \in \{(1,1), (-1,1), (-1, -1), (1, -1)\}$ . We consider  $\mathbf{Z} \sim \mathcal{N}\left(\mathbf{0}, \sigma_{\mathbf{Z}}^2 I_4\right)$ .  $\mathsf{MoB}_K(\psi_C)$  uses  $K = \lfloor \sqrt{n} \rfloor$  and the Euclidean distance for  $d$ . We iterate the experiments 10 times and plot the average of them.
(b)  $\sigma_{\mathbf{Z}}^2 = 0.4$