SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Direct estimation of the BER. As an alternative approach to using bounds on the BER, some recent research directly estimates the BER [13, 46, 67, 83]. In particular, the authors in [67] expressed the BER as a function of the  $f$ -divergence and proposed a density ratio estimator to evaluate such an  $f$ -divergence that yields the BER. Under various label assumptions, a direct BER estimator was proposed in [46]; this estimator is unbiased and consistent for soft labels, and it has desirable properties. Another direct estimator for the BER was proposed in [13]. In particular, this estimator uses the relationship between the BER and the miss-classified samples.
Extension to multi-class classification. The extension of the BER in binary classification to multi-class classification has been extensively investigated both in terms of generalizing existing bounds and deriving new approaches. For example, the Chernoff bound [34], the Jensen-Shannon divergence [55], the Fano's inequality [15], the general mean distance [8], the  $L_{2}$  norm of posterior probabilities [19], and the generalized Henze-Penrose divergence [75] can be used to derive bounds on the BER in multi-class classification problems. Non-parametric methods [31, 58], such as the  $k$ -NN classifier, can also bound the BER in multi-class classification settings. Bounds based on the error probability of a  $k$ -NN classifier in the asymptotic regime were studied in [16], and some bounds in the non-asymptotic regime can be found in [10, 32].
10Interested readers are referred to [2, 11, 56, 72]. In particular, comparisons of a variety of upper bounds on the BER were studied in [56, 72], and an arbitrary tight upper bound was proposed in [2].
B Proofs and auxiliary results
Proof. By taking the expectation of  $\psi_{\mathrm{soft}}$  in (4) with respect to  $P_{\mathbf{X},\mathbf{Y}}$ , we obtain
$$
\begin{array}{l} \mathbb {E} [ \psi_ {\mathrm {s o f t}} (\mathcal {D}) ] = \mathbb {E} \left[ \frac {1}{n} \sum_ {(\pmb {x}, \pmb {y}) \in \mathcal {D}} \left(1 - \max _ {j \in [ M ]} y _ {j}\right) \right] \\ = \frac {1}{n} \sum_ {(\boldsymbol {x}, \boldsymbol {y}) \in \mathcal {D}} \mathbb {E} \left[ 1 - \max  _ {j \in [ M ]} \Pr (C = j | \mathbf {X}) \right] \\ = 1 - \mathbb {E} \left[ \max _ {j \in [ M ]} \operatorname * {P r} (C = j | \mathbf {X}) \right] \\ = P _ {e}, \tag {16} \\ \end{array}
$$
where the second equality follows from (3), and the last equality is due to (2). This shows that  $\psi_{\mathrm{soft}}$  in (4) is an unbiased estimator of  $P_{e}$ .
Next, let  $U_{k} \stackrel{i.i.d.}{\sim} Y_{M:M}$  for all  $k \in [n]$ , and note that  $U_{k} \in \left[\frac{1}{M}, 1\right] := \mathcal{U}$ . Define a function  $f: \mathcal{U}^{n} \to \left[0, 1 - \frac{1}{M}\right]$  as  $f(U_{1}, \ldots, U_{n}) := \frac{1}{n} \sum_{k=1}^{n} (1 - U_{k})$ , which is equivalent to  $\psi_{\mathrm{soft}}$  in (4). Then, for any  $u_{k} \in \mathcal{U}$  and for all  $k \in [n]$ , we have that
$$
\begin{array}{l} \sup  _ {v \in \mathcal {U}} | f (u _ {1}, \dots , u _ {i - 1}, u _ {i}, u _ {i + 1}, \dots , u _ {n}) - f (u _ {1}, \dots , u _ {i - 1}, v, u _ {i + 1}, \dots , u _ {n}) | \\ = \sup  _ {v \in \mathcal {U}} \left| \frac {1}{n} \left(\left(1 - u _ {i}\right) - (1 - v)\right) \right| \\ \leq \frac {1}{n} \left(1 - \frac {1}{M}\right) := c _ {i}. \tag {17} \\ \end{array}
$$
By using McDiarmid's inequality [23], together with (16) and (17), we arrive at
$$
\begin{array}{l} \Pr \left(\left| \psi_ {\text {s o f t}} - P _ {e} \right| \geq \varepsilon\right) = \Pr \left(\left| f \left(U _ {1}, \dots , U _ {n}\right) - \mathbb {E} [ f \left(U _ {1}, \dots , U _ {n}\right) ] \right| \geq \varepsilon\right) \\ \leq 2 \exp \left(- \frac {2 n \varepsilon^ {2}}{\left(1 - \frac {1}{M}\right) ^ {2}}\right). \tag {18} \\ \end{array}
$$
By taking  $n\to \infty$  , we obtain
$$
\lim  _ {n \rightarrow \infty} \Pr \left(\left| \psi_ {\text {s o f t}} - P _ {e} \right| \geq \varepsilon\right) \leq \lim  _ {n \rightarrow \infty} 2 \exp \left(- \frac {2 n \varepsilon^ {2}}{\left(1 - \frac {1}{M}\right) ^ {2}}\right) = 0, \tag {19}
$$
which shows that  $\psi_{\mathrm{soft}}$  in (4) is a consistent estimator of  $P_{e}$ .
By setting  $\varepsilon = \sqrt{\frac{\left(1 - \frac{1}{M}\right)^2}{2n}}\ln \frac{2}{\delta}$  in (18) where  $\delta \in (0,1)$ , we obtain
$$
\Pr \left(| \psi_ {\text {s o f t}} - P _ {e} | <   \sqrt {\frac {\left(1 - \frac {1}{M}\right) ^ {2}}{2 n} \ln \frac {2}{\delta}}\right) > 1 - \delta . \tag {20}
$$
This shows the convergence rate.
The estimator  $\psi_{\mathrm{soft}}$  in (4) can be viewed as the sample mean of  $1 - \max_{j\in [M]}y_{j}$  in  $\mathcal{D}$ . Since  $\max_{j\in [M]}y_j\sim Y_{M:M}$ , from the central limit theorem, as  $n\to \infty$ , we obtain that
$$
\begin{array}{l} \sqrt {n} \left(\psi_ {\text {s o f t}} - P _ {e}\right)\rightarrow \mathcal {N} (0, \operatorname {V a r} \left(1 - Y _ {M: M}\right)) \\ = \mathcal {N} (0, \operatorname {V a r} \left(Y _ {M: M}\right)). \tag {21} \\ \end{array}
$$
This concludes the proof of Theorem 1.
Proof. We have that  $Y_{M:M} \sim \max_{j \in [M]} Y_j$ . Then, by letting  $U_k \stackrel{i.i.d.}{\sim} Y_{M:M}$  for all  $k \in [n]$ , we obtain
$$
\begin{array}{l} \operatorname {V a r} \left(\psi_ {\text {s o f t}}\right) = \operatorname {V a r} \left(\frac {1}{n} \sum_ {k = 1} ^ {n} \left(1 - U _ {k}\right)\right) \\ = \frac {1}{n} \operatorname {V a r} \left(1 - U _ {k}\right) \\ = \frac {1}{n} \operatorname {V a r} \left(U _ {k}\right) \\ = \frac {1}{n} \operatorname {V a r} \left(Y _ {M: M}\right). \tag {22} \\ \end{array}
$$
Moreover, by using the Bhatia-Davis inequality [7], we obtain  $\operatorname{Var}(Y_{M:M}) = \operatorname{Var}(1 - Y_{M:M}) \leq (1 - \frac{1}{M} - P_e)P_e$  since  $\mathbb{E}[\psi_{\mathrm{soft}}] = P_e$ , which gives
$$
\operatorname {V a r} \left(\psi_ {\text {s o f t}}\right) \leq \frac {\left(1 - \frac {1}{M}\right) P _ {e} - P _ {e} ^ {2}}{n}. \tag {23}
$$
Finally, we can maximize the term  $\left(1 - \frac{1}{M} - P_e\right) P_e$  over  $0 \leq P_e \leq 1 - \frac{1}{M}$ . The maximum is indeed  $\frac{\left(1 - \frac{1}{M}\right)^2}{4}$  attained by  $P_e = \frac{1}{2}\left(1 - \frac{1}{M}\right)$ . Hence,
$$
\begin{array}{l} \operatorname {V a r} \left(\psi_ {\text {s o f t}}\right) \leq \frac {\left(1 - \frac {1}{M}\right) P _ {e} - P _ {e} ^ {2}}{n} \\ \leq \frac {\left(1 - \frac {1}{M}\right) ^ {2}}{4 n}, \tag {24} \\ \end{array}
$$
which concludes the proof of Proposition 1.
Proof. We start by noting that, by viewing  $\psi_{\mathrm{soft}}$  in (4) as a sample mean of  $(1 - Y_{M:M})$ , the estimator  $\mathrm{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  is a median-of-means estimator. It has been shown in [62] that, provided that  $\mathbb{E}[|1 - Y_{M:M} - P_e|^3 ] < \infty$ , a median-of-means estimator satisfies
$$
\left| \operatorname {M o B} _ {K} \left(\psi_ {\text {s o f t}}, \mathcal {D}\right) - P _ {e} \right| \leq 3 \left(\operatorname {V a r} \left(Y _ {M: M}\right)\right) ^ {\frac {1}{2}} \left(\frac {\mathbb {E} \left[ \left| 1 - Y _ {M : M} - P _ {e} \right| ^ {3} \right]}{\operatorname {V a r} \left(Y _ {M : M}\right) ^ {\frac {3}{2}}} \frac {K}{n - K} + \sqrt {\frac {s}{n - K}}\right), \tag {25}
$$
with probability at least  $1 - 4e^{-2s}$ , for all  $s \lesssim K$ . From [54, eq. (3.8)], since  $\left(1 - Y_{M:M}\right) \in \left[0,1 - \frac{1}{M}\right]$  we have that
$$
\begin{array}{l} \mathbb {E} \left[ | 1 - Y _ {M: M} - P _ {e} | ^ {3} \right] \leq P _ {e} \left(1 - \frac {1}{M} - P _ {e}\right) \left(1 - \frac {1}{M} - 2 P _ {e}\right) \\ \leq \frac {\left(1 - \frac {1}{M}\right) ^ {3}}{6 \sqrt {3}}, \tag {26} \\ \end{array}
$$
where the last inequality follows by maximizing  $P_{e}\left(1 - \frac{1}{M} -P_{e}\right)\left(1 - \frac{1}{M} -2P_{e}\right)$  with respect to  $0\leq P_{e}\leq 1 - \frac{1}{M}$ . Substituting (26) into (25) leads to
$$
\begin{array}{l} \left| \operatorname {M o B} _ {K} \left(\psi_ {\text {s o f t}}, \mathcal {D}\right) - P _ {e} \right| \leq 3 \left(\operatorname {V a r} \left(Y _ {M: M}\right)\right) ^ {\frac {1}{2}} \left(\frac {\left(1 - \frac {1}{M}\right) ^ {3}}{6 \sqrt {3} \left(\operatorname {V a r} \left(Y _ {M : M}\right)\right) ^ {\frac {3}{2}}} \frac {K}{n - K} + \sqrt {\frac {s}{n - K}}\right) \\ = \left(\frac {\left(1 - \frac {1}{M}\right) ^ {3}}{2 \sqrt {3} \operatorname {V a r} \left(Y _ {M : M}\right)} \frac {K}{\sqrt {n - K}} + 3 \sqrt {s \operatorname {V a r} \left(Y _ {M : M}\right)}\right) \sqrt {\frac {1}{n - K}}. \tag {27} \\ \end{array}
$$
Under an appropriate choice of  $n$  and  $K$  (e.g.,  $K = o(n)$ ), the upper bound in (27) converges to 0, which proves the consistency of  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$ .
For the breakdown point of  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$ , we start by observing that, if we have  $\left\lfloor \frac{K + 1}{2}\right\rfloor$  outliers, then at most  $\left\lfloor \frac{K + 1}{2}\right\rfloor$  base estimators  $\psi_{\mathrm{soft}}(\mathcal{D}_k)$ 's in (6) can be affected by these outliers, i.e., they can be bad. In this case,  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  gives a bad estimate since more than half of the base estimators  $\psi_{\mathrm{soft}}(\mathcal{D}_k)$ 's are bad. However, if the number of outliers is smaller than  $\left\lfloor \frac{K + 1}{2}\right\rfloor$ , e.g.,  $\left\lfloor \frac{K + 1}{2}\right\rfloor - 1$ , then the median of  $\psi_{\mathrm{soft}}(\mathcal{D}_k)$ ,  $k \in [K]$  is not a bad estimate since the number of outliers is smaller than half of the base estimators. Thus, from Definition 3 with  $\tau = \left\lfloor \frac{K + 1}{2}\right\rfloor$  and  $\kappa + \tau = n$ , we obtain  $B(\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})) = \left\lfloor \frac{K + 1}{2}\right\rfloor \frac{1}{n}$ .
The proof of the asymptotic normality of  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  follows directly from [62, Theorem 4]. In particular, if  $K\to \infty$  and  $K = o(\sqrt{n})$  as  $n\to \infty$ , it holds that
$$
\sqrt {n} \left(\mathsf {M o B} _ {K} \left(\psi_ {\text {s o f t}}, \mathcal {D}\right) - P _ {e}\right) \xrightarrow {d} \mathcal {N} \left(0, \frac {\pi}{2} \operatorname {V a r} \left(Y _ {M: M}\right)\right). \tag {28}
$$
This concludes the proof of Theorem 2.
B.4 Random permutation noise
One of the most common type of label noises is a noise that randomly permutes/shuffles the data labels [27], i.e., we have  $(\boldsymbol{x},\tilde{\boldsymbol{y}})\in \widetilde{\mathcal{D}}_{\mathrm{P}}$  where  $\widetilde{\mathcal{D}}_{\mathrm{P}}$  denotes the dataset with randomly permuted labels. This noise model can be categorized into two types [27], namely instance-independent and instance-dependent. The former perturbs the label  $\pmb{y}$  with a random permutation according to some probability distribution (the extreme noise case, namely unlabeled data or data without correspondence, where the label is permuted with probability one using a random permutation matrix, has also been studied extensively [36, 71, 73, 79, 87, 95]), which is independent of the input features. For example, a corruption by an instance-independent noise to  $\boldsymbol{y} = [0.1,0.7,0.2]^{\top}$  may result in  $\tilde{\boldsymbol{y}} = [0.1,0.2,0.7]^{\top}$ . The other type of noise, the instance-dependent noise, flips/permutes the labels depending on the instance. For example, the label "car" is more likely to be flipped to "truck" rather than "cat".
We note that  $\psi_{\mathrm{soft}}$  in (4) only depends on the maximum value of  $\pmb{y}$ , which does not change after that a random permutation is applied on  $\pmb{y}$ . This property ensures that  $\psi_{\mathrm{soft}}$  is still an effective estimator of  $P_{e}$  even when the labels are randomly permuted, as stated in the next theorem.
Theorem 4. Let  $\widetilde{\mathcal{D}}_{\mathrm{P}} = \{(x_i,\tilde{y}_i)\}_{i = 1}^n$  be a dataset that consists of noisy soft labels  $\tilde{\pmb{y}}_i = P_i\pmb {y}_i$  where  $P_{i},i\in [M]$  is any random permutation matrix of size  $M\times M$ . Then,  $\psi_{\mathrm{soft}}(\widetilde{\mathcal{D}}_{\mathrm{P}})$  satisfies all of the properties in Theorem 1 and Proposition 1.
Proof. Since  $\max_{j\in [M]}y_j$  is permutation-invariant with respect to  $\pmb{y}$ , it follows that
$$
\begin{array}{l} \psi_ {\text {s o f t}} \left(\left\{\left(\boldsymbol {x} _ {i}, \tilde {\boldsymbol {y}} _ {i}\right) \right\} _ {i = 1} ^ {n}\right) = \psi_ {\text {s o f t}} \left(\left\{\left(\boldsymbol {x} _ {i}, P _ {i} \boldsymbol {y} _ {i}\right) \right\} _ {i = 1} ^ {n}\right) \\ = \psi_ {\text {s o f t}} \left(\left\{\left(\boldsymbol {x} _ {i}, \boldsymbol {y} _ {i}\right) \right\} _ {i = 1} ^ {n}\right), \tag {29} \\ \end{array}
$$
where the first equality follows since  $\tilde{\pmb{y}}_i = P_i\pmb{y}_i$ , and the second equality is due to the permutation-invariance property of the max function. The results in Theorem 1 and Proposition 1 then prove Theorem 4.
Proof. Without loss of generality, we let  $\mathcal{X} = \{\pmb{u}_1, \pmb{u}_2, \dots, \pmb{u}_K\}$ , where  $K = |\mathcal{X}|$  is the cardinality of  $\mathcal{X}$ , and  $\pmb{u}_k$  is the unique feature vector (or tensor). We denote by  $\pmb{v}_k \in \mathcal{Y}$  the soft label corresponding to  $\pmb{u}_k$ , i.e.,  $\pmb{v}_k = [\operatorname*{Pr}(C = 1 | \mathbf{X} = \pmb{u}_k), \dots, \operatorname*{Pr}(C = M | \mathbf{X} = \pmb{u}_k)]^\top$ . With these definitions, we observe that due to the law of large numbers, we have
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} \boldsymbol {s} (\boldsymbol {u} _ {k}) = \mathbb {E} [ \widetilde {\mathbf {Y}} | \mathbf {X} = \boldsymbol {u} _ {k} ] \\ \stackrel {\mathrm {(a)}} {=} \mathbb {E} [ \mathbb {E} [ \widetilde {\mathbf {Y}} | \mathbf {Y} ] | \mathbf {X} = \boldsymbol {u} _ {k} ] \\ \stackrel {(b)} {=} \mathbb {E} [ \mathbf {Y} | \mathbf {X} = \boldsymbol {u} _ {k} ] \\ = \boldsymbol {v} _ {k}, \tag {30} \\ \end{array}
$$
where (a) follows from the law of total expectation and (b) is due to the fact that  $\mathbb{E}[\widetilde{\mathbf{Y}} |\mathbf{Y}] = \mathbf{Y}$  since the noise is zero mean. Now, since for any  $i\in [n]$  there exists a  $k\in [K]$  such that  $\pmb {x}_i = \pmb {u}_k$  , we can write that
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} \mathrm {i d} x \left(\boldsymbol {x} _ {i}\right) = \lim  _ {n \rightarrow \infty} \mathrm {i d} x \left(\boldsymbol {u} _ {k}\right) \\ \stackrel {\mathrm {(a)}} {=} \arg \max  _ {j \in [ M ]} \left\{\left(\boldsymbol {v} _ {k}\right) _ {j} \right\} \\ = \arg \max  _ {j \in [ M ]} \left\{\Pr (C = j | \mathbf {X} = \boldsymbol {u} _ {k}) \right\} \\ := \widehat {\operatorname {i d} \times} \left(\boldsymbol {u} _ {k}\right), \tag {31} \\ \end{array}
$$
where the equality in (a) follows from (30). Then, we can write  $\psi_{\mathrm{DN}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$  in (12) as
$$
\psi_ {\mathrm {D N}} \left(\widetilde {\mathcal {D}} _ {\mathrm {A}}\right) = \sum_ {k = 1} ^ {K} \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} \left\{\boldsymbol {x} _ {i} = \boldsymbol {u} _ {k} \right\} \left(1 - \left(\tilde {\boldsymbol {y}} _ {i}\right) _ {\mathrm {i d} \times \left(\boldsymbol {x} _ {i}\right)}\right). \tag {32}
$$
When  $n\to \infty$  , it follows that
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} \psi_ {\mathrm {D N}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) \stackrel {(\mathrm {a})} {=} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {X} = \boldsymbol {u} _ {k} \right\}\left(1 - \widetilde {\mathbf {Y}} _ {\widehat {\operatorname {i d x}} (\boldsymbol {u} _ {k})}\right)\right] \\ \stackrel {(b)} {=} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {X} = \boldsymbol {u} _ {k} \right\} \left(1 - \mathbb {E} [ \widetilde {\mathbf {Y}} | \mathbf {Y} ] _ {\widehat {\mathrm {i d} \times} (\boldsymbol {u} _ {k})}\right) \right] \\ \stackrel {\mathrm {(c)}} {=} \sum_ {k = 1} ^ {K} \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {X} = \boldsymbol {u} _ {k} \right\} \left(1 - \mathbf {Y} _ {\widehat {\operatorname {i d} \times} (\boldsymbol {u} _ {k})}\right) \right] \\ = \sum_ {k = 1} ^ {K} \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {X} = \boldsymbol {u} _ {k} \right\} \left(1 - \max  _ {j \in [ M ]} \Pr (C = j | \mathbf {X} = \boldsymbol {u} _ {k})\right) \right] \\ = \sum_ {k = 1} ^ {K} \mathbb {E} \left[ 1 - \max  _ {j \in [ M ]} \Pr (C = j | \mathbf {X} = \boldsymbol {u} _ {k}) \right| \mathbf {X} = \boldsymbol {u} _ {k} \Bigg ] \Pr (\mathbf {X} = \boldsymbol {u} _ {k}) \\ = \mathbb{E}\left[1 - \max_{j\in [M]}\Pr (C = j|\mathbf{X})\right] \\ = P _ {e}, \tag {33} \\ \end{array}
$$
where the labeled equalities follow from: (a) the law of large numbers, the assumption that  $(\pmb{x}_i,\tilde{\pmb{y}}_i)\sim P_{\mathbf{X},\widetilde{\mathbf{Y}}}$ , and using (31); (b) using the law of total expectation; and (c) the fact that  $\mathbb{E}[\widetilde{\mathbf{Y}} |\mathbf{Y}] = \mathbf{Y}$  since the noise is zero mean. This proves the consistency of  $\psi_{\mathrm{DN}}$ .
The asymptotic unbiasedness directly follows from the consistency together with the assumption of bounded support for the noise. Specifically, since  $Z_{j} \in [a,b]$  for all  $j \in [M]$ , we have that
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} \mathbb {E} \left[ \psi_ {\mathrm {D N}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) \right] = \mathbb {E} \left[ \lim  _ {n \rightarrow \infty} \psi_ {\mathrm {D N}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) \right] \\ = \mathbb {E} [ P _ {e} ] \\ = P _ {e}, \tag {34} \\ \end{array}
$$
where the first equality follows by the dominated convergence theorem that can be applied since  $|\psi_{\mathrm{DN}}(\widetilde{\mathcal{D}}_{\mathrm{A}})| \leq \max \{|a|, |1 - \frac{1}{M} + b|\}$ , where  $a$  and  $b$  are the minimum and the maximum value in the support of the noise, respectively, and they are both finite. This shows the asymptotic unbiasedness of  $\psi_{\mathrm{DN}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$ .
We are left to prove the denoising consistency. The denoised label  $s(\boldsymbol{x}_i)$  in (11) is an unbiased estimator of  $\boldsymbol{y}_i$  since
$$
\begin{array}{l} \mathbb {E} [ \boldsymbol {s} (\mathbf {X} _ {i}) ] \stackrel {\mathrm {(a)}} {=} \mathbb {E} \left[ \frac {\sum_ {j = 1} ^ {n} \mathbb {1} \{\mathbf {X} _ {i} = \mathbf {X} _ {j} \} \widetilde {\mathbf {Y}} _ {j}}{\sum_ {j = 1} ^ {n} \mathbb {1} \{\mathbf {X} _ {i} = \mathbf {X} _ {j} \}} \right] \\ \stackrel {\mathrm {(b)}} {=} \mathbb {E} \left[ \mathbb {E} \left[ \frac {\sum_ {j = 1} ^ {n} \mathbb {1} \left\{\mathbf {X} _ {i} = \mathbf {X} _ {j} \right\} \widetilde {\mathbf {Y}} _ {j}}{\sum_ {j = 1} ^ {n} \mathbb {1} \left\{\mathbf {X} _ {i} = \mathbf {X} _ {j} \right\}} \mid N, \mathbf {X} _ {i} \right] \right] \\ \stackrel {(c)} {=} \mathbb {E} \left[ \frac {1}{N} \sum_ {k = 1} ^ {N} \mathbb {E} \left[ \widetilde {\mathbf {Y}} _ {k} \mid N, \mathbf {X} _ {i} \right] \right] \\ \stackrel {\mathrm {(d)}} {=} \mathbb {E} \left[ \frac {1}{N} \sum_ {k = 1} ^ {N} \mathbb {E} \left[ \mathbf {Y} _ {i} \mid \mathbf {X} _ {i} \right] \right] \\ = \mathbb {E} \left[ \mathbf {Y} _ {i} \mid \mathbf {X} _ {i} \right], \tag {35} \\ \end{array}
$$
where the labeled equalities follow from: (a) letting  $\mathbf{X}_j$ ,  $\mathbf{Y}_j$ , and  $\widetilde{\mathbf{Y}}_j$  for all  $j \in [n]$  be independent copies of  $\mathbf{X}$ ,  $\mathbf{Y}$ , and  $\widetilde{\mathbf{Y}}$ , respectively; (b) introducing a random variable  $N \in [n]$  that counts the number of  $\mathbf{X}_j$  such that  $\mathbf{X}_j = \mathbf{X}_i$  and using the law of total expectation; (c) re-indexing  $\widetilde{\mathbf{Y}}_k$ ,  $k \in [N]$  for the  $N$  pairs  $(\mathbf{X}_j, \widetilde{\mathbf{Y}}_k)$  such that  $\mathbf{X}_i = \mathbf{X}_j$ ; and (d) the fact that the noise is zero mean.
Note that  $s(\pmb{x}_i)$  is the average of  $n_{\pmb{x}_i}$  noisy labels  $\tilde{\pmb{y}}$ 's corresponding to  $\pmb{x}_i$ . By Hoeffding's inequality, we then obtain
$$
\Pr \left(| \boldsymbol {s} (\boldsymbol {x} _ {i}) - \boldsymbol {y} _ {i} | \geq \varepsilon\right) \leq 2 \exp \left(- \frac {2 n _ {\boldsymbol {x} _ {i}} \varepsilon^ {2}}{\left(1 - \frac {1}{M} - a + b\right) ^ {2}}\right). \tag {36}
$$
Setting  $\varepsilon = \sqrt{\frac{\left(1 - \frac{1}{M} - a + b\right)^2}{2n_{\boldsymbol{x}_i}}\ln\frac{2}{\delta}}$  leads to
$$
\Pr \left(\left| \boldsymbol {s} \left(\boldsymbol {x} _ {i}\right) - \boldsymbol {y} _ {i} \right| <   \sqrt {\frac {\left(1 - \frac {1}{M} - a + b\right) ^ {2}}{2 n _ {\boldsymbol {x} _ {i}}} \ln \frac {2}{\delta}}\right) > 1 - \delta . \tag {37}
$$
B.6 Properties of  $\psi_{\mathrm{genie}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$
The following proposition defines  $\psi_{\mathrm{genie}}$  and provides some properties of it.
Proposition 2. Let  $\widetilde{\mathcal{D}}_{\mathrm{A}} = \{(\pmb{x}_i,\tilde{\pmb{y}}_i)\}_{i=1}^n$  be a dataset that consists of noisy soft labels  $\tilde{\pmb{y}}_i = \pmb{y}_i + \pmb{z}_i$  with  $z_i \stackrel{i.i.d.}{\sim} P_{\mathbf{Z}}$  such that  $\mathbb{E}[\mathbf{Z}] = \mathbf{0}$ . Let  $c_i = \arg \max_{j \in [M]} (\pmb{y}_i)_j$ . Then, the following estimator
$$
\psi_ {\text {g e n i e}} \left(\widetilde {\mathcal {D}} _ {\mathrm {A}}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - \left(\tilde {\boldsymbol {y}} _ {i}\right) _ {c _ {i}}\right) \tag {38}
$$
satisfies the following properties:
1. (Unbiasedness): It is an unbiased estimator of the BER;
2. (Consistency): It is a consistent estimator of the BER;
3. (Variance):  $Var\left(\psi_{\mathrm{genie}}(\widetilde{\mathcal{D}}_{\mathrm{A}})\right) = \frac{Var(Y_{M:M}) + Var(Z)}{n}$ ;
4. (Asymptotic Normality):  $\sqrt{n} (\psi_{\mathrm{genie}}(\widetilde{\mathcal{D}}_{\mathrm{A}}) - P_{e}) \xrightarrow{d} \mathcal{N}(0, \mathrm{Var}(Y_{M:M}) + \mathrm{Var}(Z))$  as  $n \to \infty$ .
Proof. Define the random variable  $C = \arg \max_{j\in [M]}Y_{j}$  that indicates the index of  $\mathbf{Y}$  having the largest value, i.e.,  $Y_{C} = \max_{j\in [M]}Y_{j}$ . We have that
$$
\begin{array}{l} \mathbb {E} [ \psi_ {\mathrm {g e n i e}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) ] = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ 1 - \widetilde {\mathbf {Y}} _ {C} ] \\ \stackrel {(a)} {=} \mathbb {E} [ \mathbb {E} [ 1 - \tilde {\mathbf {Y}} _ {C} | \mathbf {Y} ] ] \\ \stackrel {(\mathrm {b})} {=} \mathbb {E} [ 1 - \mathbf {Y} _ {C} ] \\ = \mathbb {E} \left[ 1 - \max  _ {j \in [ M ]} Y _ {j} \right] \\ = P _ {e}, \tag {39} \\ \end{array}
$$
where the labeled equalities follow from: (a) the law of total expectation; and (b) the fact that  $\mathbb{E}[\widetilde{\mathbf{Y}} |\mathbf{Y}] = \mathbf{Y}$  since  $\mathbb{E}[\mathbf{Z}] = \mathbf{0}$ . This shows the unbiasedness of  $\psi_{\mathrm{genie}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$ .
Now, we show the consistency of  $\psi_{\mathrm{genie}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$ . Due to the law of large numbers, as  $n\to \infty$ , we obtain that
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} \psi_ {\text {g e n i e}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) = \lim  _ {n \rightarrow \infty} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - \left(\tilde {\boldsymbol {y}} _ {i}\right) _ {c _ {i}}\right) \\ = \mathbb {E} [ 1 - \widetilde {\mathbf {Y}} _ {C} ] \\ = \mathbb {E} [ \mathbb {E} [ 1 - \bar {\mathbf {Y}} _ {C} | \mathbf {Y} ] ] \\ = \mathbb {E} [ 1 - \mathbf {Y} _ {C} ] \\ = P _ {e}, \tag {40} \\ \end{array}
$$
which demonstrates the consistency of  $\psi_{\mathrm{genie}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$ .
We now compute the variance of  $\psi_{\mathrm{genie}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$ . We have that
$$
\begin{array}{l} \psi_ {\text {g e n i e}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) = \frac {1}{n} \sum_ {i = 1} ^ {n} (1 - (\tilde {\boldsymbol {y}} _ {i}) _ {c _ {i}}) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - \left(\boldsymbol {y} _ {i}\right) _ {c _ {i}} - \left(\boldsymbol {z} _ {i}\right) _ {c _ {i}}\right) \\ = \psi_ {\text {s o f t}} (\mathcal {D}) - \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\boldsymbol {z} _ {i}\right) _ {c _ {i}}. \tag {41} \\ \end{array}
$$