SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which follow from (2) and (3). The bounds in (10) demonstrate that with labels corrupted by additive noise,  $\psi_{\mathrm{soft}}$  is (in general) an inconsistent estimator of  $P_e$ . By following similar steps, it can be easily proved that  $\psi_{\mathrm{soft}}$  is also a biased estimator of  $P_e$ . In particular, the bounds in (10) show that  $\psi_{\mathrm{soft}}$  has a bias which is bounded by the expected value of the smallest (lower bound) and of the largest (upper bound) order statistics of the noise. In what follows, we provide two examples of distributions for which bounds on these expected values have been computed.
Example 1. Let  $\mathbf{Z} \stackrel{i.i.d.}{\sim}$  Uniform  $(-a, a)$ . Then,  $\mathbb{E}[Z_{i:M}] = \frac{2ai}{M + 1} - a$ , which as  $n \to \infty$  implies that  $|P_e - \psi_{\mathrm{soft}}(\widetilde{\mathcal{D}}_A)| \leq a \frac{M - 1}{M + 1}$ .
Example 2. Let  $\mathbf{Z} \stackrel{i.i.d.}{\sim} \gamma^2$ -sub-Gaussian. Then, we have that  $\mathbb{E}[Z_{M:M}] = -\mathbb{E}[Z_{1:M}]$  and  $\mathbb{E}[Z_{M:M}] \leq \sqrt{2\gamma^2 \ln M}$  [9] which as  $n \to \infty$  implies that  $|P_e - \psi_{\mathrm{soft}}(\widetilde{\mathcal{D}}_A)| \leq \sqrt{2\gamma^2 \ln M}$ .
Our analysis above shows that  $\psi_{\mathrm{soft}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$  is (in general) an inconsistent and biased estimator of  $P_{e}$ . Because of this, we next propose a new BER estimator, which (different from  $\psi_{\mathrm{soft}}$ ) leverages the features  $\{\pmb{x}_i\}_{i=1}^n$  to denoise  $\tilde{\pmb{y}}_i$ . We refer to this estimator as  $\psi_{\mathrm{DN}}$ . Our main intuition behind proposing  $\psi_{\mathrm{DN}}$  stems from the fact that data samples having the same feature  $\pmb{x}$  should be labeled with the same (or at least similar) label. This can be attained by noting that, even if labels are noisy, it is possible to minimize the effect of a zero-mean noise by averaging the noisy labels associated with the same feature. We next formally define our denoising BER estimator  $\psi_{\mathrm{DN}}: \mathbb{R}^{M} \to [0,1]$ .
Definition 5 (Denoise estimator). For a noisy dataset  $\widetilde{\mathcal{D}}_{\mathrm{A}} = \{(x_i,\tilde{y}_i)\}_{i = 1}^n$ , let the denoised label  $s(x_{i})$  for all  $i\in [n]$ , be defined as
$$
\boldsymbol {s} \left(\boldsymbol {x} _ {i}\right) = \frac {\sum_ {j = 1} ^ {n} \mathbb {1} \left\{\boldsymbol {x} _ {j} = \boldsymbol {x} _ {i} \right\} \tilde {\boldsymbol {y}} _ {j}}{\sum_ {j = 1} ^ {n} \mathbb {1} \left\{\boldsymbol {x} _ {j} = \boldsymbol {x} _ {i} \right\}}, \tag {11}
$$
and let  $\mathrm{id}\times (\pmb {x}_i) = \arg \max_{j\in [M]}\{(\pmb {s}(\pmb {x}_i))_j\}$ . Then, the denoise BER estimator is defined as
$$
\psi_ {\mathrm {D N}} \left(\widetilde {\mathcal {D}} _ {\mathrm {A}}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - \left(\tilde {\boldsymbol {y}} _ {i}\right) _ {\mathrm {i d x} \left(\boldsymbol {x} _ {i}\right)}\right). \tag {12}
$$
The next theorem (proof in Appendix B.5) provides some important properties of  $\psi_{\mathrm{DN}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$ .
Theorem 3. Let  $\widetilde{\mathcal{D}}_{\mathrm{A}} = \{(x_i,\tilde{y}_i)\}_{i = 1}^n$  be a dataset that consists of noisy soft labels  $\tilde{\pmb{y}}_i = \pmb {y}_i + \pmb {z}_i$  where  $z_{i}\stackrel {i,i,d}{\sim}P_{\mathbf{Z}}$  is the zero mean noise. Then,  $\psi_{\mathrm{DN}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$  is a consistent estimator of  $P_{e}$
Moreover, if the noise has bounded support, that is  $\operatorname{Pr}(\mathbf{Z} \in [a, b]^M) = 1$ , then
1. (Asymptotical unbiasedness):  $\psi_{\mathrm{DN}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$  is an asymptotically unbiased estimator of  $P_{e}$ ;
2. (Denoising Consistency): For any  $\delta \in (0,1)$ , it holds that  $|\pmb{s}(\pmb{x}) - \pmb{y}| < \sqrt{\frac{(1 - \frac{1}{M} - a + b)^2}{2n_x}\ln\frac{2}{\delta}}$  with probability at least  $1 - \delta$ , where  $n_{\pmb{x}} = \sum_{j=1}^{n} \mathbb{1}\{\pmb{x}_j = \pmb{x}\}$  and  $\pmb{s}(\pmb{x})$  is the denoised label defined in (11).
The results in Theorem 3 broaden the applicability of an effective BER estimator to a larger class of datasets, e.g., to scenarios where the dataset includes multiple data samples representing the same feature. For example,  $\psi_{\mathrm{DN}}$  works well on a noisy dataset where each data sample has multiple labels. Moreover, as we will see in Section 4.2,  $\psi_{\mathrm{DN}}$  is also useful for one-hot labels. We also expect that  $\psi_{\mathrm{DN}}$  will work well with privatized datasets where noise was added to enhance privacy [26].
Remark 1. Our estimator  $\psi_{\mathrm{DN}}$  only requires the dataset  $\widetilde{\mathcal{D}}_{\mathrm{A}}$ . This is a more practical approach than the one in [46] which instead also requires the true one-hot label. The estimator in [46], which is referred to as a genie estimator  $\psi_{\mathrm{genie}}$ , can be obtained by replacing  $s(x)$  in (11) with the true one-hot label. In Appendix B.6, we provide properties of  $\psi_{\mathrm{genie}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$  that generalize the results in [46] for any  $M \geq 2$ . As expected,  $\psi_{\mathrm{genie}}$  is a consistent and unbiased estimator of  $P_e$ .
<sup>5</sup>We say that a random variable  $Z$  is  $\gamma^2$ -sub-Gaussian if  $\mathbb{E}[e^{\lambda (Z - \mathbb{E}[Z])}]\leq e^{\frac{\lambda^2\gamma^2}{2}}$ ,  $\forall \lambda \in \mathbb{R}$ .
4.2 One-hot labels
A dataset consisting of one-hot labels, denoted as  $\overline{D} = \{(x_i,\overline{y}_i)\}_{i = 1}^n$ , contains little information about the class posterior probability. We assume that the one-hot label is constructed from the corresponding soft label as follows,
$$
\overline {{\mathbf {Y}}} = \mathbf {e} _ {i} \text {w i t h p r o b a b i l i t y} Y _ {i}, \tag {13}
$$
where: (i)  $\overline{\mathbf{Y}}$  is the one-hot random vector; (ii)  $\mathbf{Y}$  is the soft label random vector in (3); and (iii)  $\mathbf{e}_i\in \mathbb{R}^M$  is the standard basis vector with a one in the  $i$ -th position and zero in all the other entries.
We can think of the one-hot label construction in (13) as a noisy label. Specifically, the noise  $\mathbf{Z} \in \{\mathbf{e}_i - \mathbf{y}\}_{i=1}^M$  with the probability mass function  $p_{\mathbf{Z}|\mathbf{Y}}(\mathbf{z}|\mathbf{y}) = \sum_{j=1}^{M} y_j \mathbb{1}\{\mathbf{z} = \mathbf{e}_j - \mathbf{y}\}$  satisfies  $\overline{\mathbf{Y}} = \mathbf{Y} + \mathbf{Z}$ . It is not difficult to see that  $\mathbb{E}[\mathbf{Z}] = \mathbb{E}[\mathbb{E}[\mathbf{Z}|\mathbf{Y}]] = \mathbf{0}$  and hence, our denoise estimator  $\psi_{\mathrm{DN}}(\widetilde{\mathcal{D}}_{\mathrm{A}})$  in (12), or its robust version  $\mathrm{MoB}_K(\psi_{\mathrm{DN}}, \widetilde{\mathcal{D}}_{\mathrm{A}})$ , can estimate the BER also for the case of one-hot labels. However, we also note that with the above construction, the labels might be too noisy to estimate the BER in practice. This suggests that a more refined denoising method than the one in Definition 5 for one-hot labels might be needed for a more effective BER estimation.
Motivated by the above discussion, we next propose a denoising method for one-hot labels, which leverages neighbor samples to mitigate the noise. Our choice of such a denoising method mainly stems from the fact that one would expect that neighboring samples should have similar posterior probabilities. In particular, for each  $\boldsymbol{x}_i$ ,  $i \in [n]$ , we consider all of its neighbors (features that are at most at a distance  $r$  from  $\boldsymbol{x}_i$ ) and we average the corresponding noisy labels in a spirit similar to (11). We next formally define our denoising BER estimator, which we refer to as  $\psi_{\mathrm{C}}$ .
Definition 6 (Cluster denoise estimator). Given  $\mathsf{d}:\mathcal{X}^2\to \mathbb{R}_+$  and  $r\in \mathbb{R}_+$ , define  $\text{Cluster}_i\coloneqq \{(x,y)\in \widetilde{\mathcal{D}}_{\mathrm{A}}:\mathsf{d}(x_i,x)\leq r\}$ . Then, the denoised label for  $\tilde{\pmb{y}}_i$ , for all  $i\in [n]$ , is defined as
$$
\hat {\boldsymbol {y}} _ {i} = \frac {\sum_ {(\boldsymbol {x} , \tilde {\boldsymbol {y}}) \in \operatorname {C l u s t e r} _ {i}} \tilde {\boldsymbol {y}}}{| \operatorname {C l u s t e r} _ {i} |}, \tag {14}
$$
and the cluster-based BER estimator is defined as
$$
\psi_ {\mathrm {C}} \left(\widetilde {\mathcal {D}} _ {\mathrm {A}}, \mathrm {d}, r\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - \max  _ {j \in [ M ]} \left\{\left(\hat {\boldsymbol {y}} _ {i}\right) _ {j} \right\}\right). \tag {15}
$$
Remark 2. Preprocessing the dataset that aggregates data samples of similar features can also be helpful in improving the BER estimate for noisy labels, and it can be paired with the cluster denoise estimator in Definition 6. A naive approach would consist of reducing the feature dimensionality and increasing the number of data samples inside clusters (14), which helps mitigate the effect of the noise. Some notable examples of data preprocessing are the classical principal component analysis (PCA) [77] and the representation learning [5] (e.g., variational auto encoder [52]).
Remark 3. Another viewpoint to the denoising method in (14) is the connection to the non-parametric estimation of the conditional expectation  $\mathbb{E}[\mathbf{Y}|\mathbf{X} = \boldsymbol{x}]$ , where  $\mathbf{Y}$  is the true soft label corresponding to  $\mathbf{X} = \boldsymbol{x}$ . In particular, estimating  $\mathbb{E}[\mathbf{Y}|\mathbf{X} = \boldsymbol{x}]$  is equivalent to denoising the noisy label associated with  $\boldsymbol{x}$  as in (14). Under such a viewpoint, the BER estimator in Definition 6 resembles the Nadaraya-Watson estimator with Parzen window kernel [63, 89].
We empirically validate our results using various datasets, namely: 1) a synthetic dataset with different noises, including one-hot labels; 2) two benchmark datasets CIFAR-10H [4] and Fashion-MNIST-H [46]; and 3) MovieLens [38], a real-world dataset for movie recommendations. We compare our estimators with two SOTA BER bounds [72], namely the generalized Henze-Penrose (GHP) BER bounds [75] and the  $k$ -Nearest Neighbor (NN) BER bounds [16]. Details on the  $k$ -NN bounds and GHP bounds, and additional results can be found in Appendix C.
<sup>6</sup>Several works have taken advantage of neighbor samples to estimate the BER or the probability divergence. We refer an interested reader to [67] and references therein.
Figure 1: 4-classes Gaussian data samples. The left figure shows the samples of each class, and the right figure displays the corresponding soft label information as the maximum value of the soft label.
(a) Soft labels.
(b) Noisy soft labels.
Figure 2: Comparison of  $\mathsf{MoB}_K(\psi_{\mathsf{C}})$  where  $\psi_{\mathsf{C}}$  is defined in (15) with the GHP bounds and the  $k$ -NN bounds with  $k = 1$ . We consider  $n = 2,000$  samples generated as in Figure 1 with noise  $\mathbf{Z} \sim \mathcal{N}\left(\mathbf{0}, \frac{1}{5} I_4\right)$ .  $\mathsf{MoB}_K(\psi_{\mathsf{C}})$  uses  $K = \lfloor \sqrt{n} \rfloor$  (with this choice, Theorem 2 ensures the asymptotic normality of  $\mathsf{MoB}_K$ ), the Euclidean distance for  $\mathsf{d}$ , and  $r = 1/5$ .
Synthetic dataset with soft labels and noisy labels. We consider a 4-class classification problem with equiprobable classes  $C \in \mathcal{C}_{\mu} \coloneqq \{(\mu, \mu), (-\mu, \mu), (-\mu, -\mu), (\mu, -\mu)\}$ , where  $\mu > 0$  is a parameter that controls the classification hardness. We generate the feature  $\mathbf{X} \in \mathbb{R}^2$  according to a 2-dimensional Gaussian distribution with mean  $\pmb{c}$  (i.e., a realization of  $C \in \mathcal{C}_{\mu}$ ) and covariance matrix  $I_2$ . The corresponding soft labels  $\pmb{y}_i$ 's are then obtained by the Bayes' theorem. In particular, since  $f_{\mathbf{X}|C} \sim \mathcal{N}(C, I_2)$  and  $P_C(\pmb{c}) = 1/4$  if  $\pmb{c} \in \mathcal{C}_{\mu}$  (and zero otherwise), according to (3) we have that  $(\pmb{y}_i)_k = \frac{f_{\mathbf{X}|C}(\pmb{x}_i|\pmb{c}) P_C(\pmb{c})}{\sum_{\alpha \in \mathcal{C}_{\mu}} f_{\mathbf{X}|C}(\pmb{x}_i|\alpha) P_C(\alpha)}$ . Figure 1 illustrates  $n = 1,000$  features (left figure), and the corresponding soft labels (right figure) for the synthetic dataset generated as explained above; in particular, for each feature  $\pmb{x}_i$ , we only reported the label  $(\pmb{y}_i)_{j^{\star}}$  where  $j^{\star} = \arg \max_{j \in [4]} (\pmb{y}_i)_j$  which is required for evaluating  $\psi_{\mathrm{soft}}$  in (4). As expected, a point near the decision boundary (lines for  $x_1 = 0$  or  $x_2 = 0$ ) has a smaller maximum value in its soft label.
The results in Figure 2 empirically demonstrate the effectiveness of our estimators, which consistently outperform the others. Moreover, our estimators estimate the exact value of the BER, while the others derive upper and lower bounds on the BER, which might not be tight.
Without loss of generality, we consider the following mapping:  $k = 1$  if  $c = (\mu, \mu)$ ;  $k = 2$  if  $c = (-\mu, \mu)$ ;  $k = 3$  if  $c = (-\mu, -\mu)$ ; and  $k = 4$  if  $c = (\mu, -\mu)$ .
Among the proposed estimators, in Figure 2 we only evaluated  $\mathsf{MoB}_K(\psi_C)$  since we observed that it performs well with respect to SOTA methods and hence, we omitted the other estimators.
(a)  $P_{e}$
(b)  $|\psi - P_e|$  and variance.
Figure 3: Comparison of  $\mathrm{MoB}_K(\psi_{\mathrm{C}})$  (evaluated with  $\psi_{\mathrm{C}}$  in (15)) with  $\psi_{\mathrm{soft}}$  on noisy soft labels. We consider  $\mathbf{Z} \sim \mathcal{N}(\mathbf{0}, 1/5I_4)$ . To model outliers, we added  $\tilde{Z} = 0.2 \cdot \operatorname{Ber}(1/2)$  with probability  $1/10$  to each entry of the soft labels. For different  $n$ , the parameters of  $\mathrm{MoB}_K(\psi_{\mathrm{C}})$  are chosen as  $K = \lfloor \sqrt{n} \rfloor$ ,  $d$  is the Euclidean distance, and  $r = 1/5$ . We iterate the experiment 50 times for each  $n$ .
We further conducted experiments to verify the effectiveness of our denoising and robustifying methods on data samples with label noise and outliers. Figure 3 shows the comparison of the denoising estimator  $\mathsf{MoB}_K(\psi_{\mathrm{C}})$  with the base estimator  $\psi_{\mathrm{soft}}$ . We observe that  $\psi_{\mathrm{soft}}$  suffers from the label noise, which leads to a bias in the estimation. However, our denoising estimator  $\mathsf{MoB}_K(\psi_{\mathrm{C}})$  suitably denoises the noisy labels and effectively estimates the BER. As shown in Figure 3b, in fact, the absolute error between the estimate and the BER as well as the variance decrease as  $n$  grows.
Synthetic dataset with one-hot labels. We analyze the performance of  $\mathsf{MoB}_K(\psi_{\mathrm{C}})$  in the same Gaussian setting described above, but with one-hot labels. Each soft label of the samples is mapped to a one-hot label according to the categorical distribution with probabilities equal to the soft labels (see (13)). Figure 4 shows that our denoising method is capable of suitably reducing the noise in one-hot labels; in fact, it correctly estimates the BER after around  $10^{4}$  samples when  $\mathsf{MoB}_K(\psi_{\mathrm{C}})$  converges to the BER. From Figure 4, we also observe that  $\psi_{\mathrm{soft}}$  performs poorly; this suggests that denoising methods are essential for the BER estimation task, hence worth further investigation.
Figure 4: Comparison of  $\mathrm{MoB}_K(\psi_{\mathbb{C}})$  with  $\psi_{\mathrm{soft}}$  for Gaussian samples with one-hot labels.
CIFAR-10H [4] and Fashion-MNIST-H [46].
The CIFAR-10H dataset is a variation of CIFAR-10 [53], constructed by labeling  $10^{4}$  images in the test dataset of CIFAR-10 by multiple labelers. The Fashion-MNIST-H dataset is populated by  $10^{4}$  images in the test dataset of Fashion-MNIST [93] in a similar manner. These two datasets have 10 classes, but we further categorized these into  $M \in \{2,3\}$  classes. In particular, for CIFAR-10H, we assigned  $C_1 = \{\text{airplane, automobile, ship, truck}\}$ ,  $C_2 = \{\text{bird, cat, deer, dog, frog, horse}\}$  when  $M = 2$ , and  $C_1 = \{\text{airplane, automobile, ship, truck}\}$ ,  $C_2 = \{\text{cat, dog, frog}\}$ ,  $C_3 = \{\text{bird, deer, horse}\}$  when  $M = 3$ . Similarly, for Fashion-MNIST-H, we assigned  $C_1 = \{\text{t-shirt/top, pullover, dress, coat, shirt}\}$ ,  $C_2 = \{\text{trouser, sandal, sneaker, bag, ankleboot}\}$  when  $M = 2$ , and  $C_1 = \{\text{t-shirt/top, pullover, coat, shirt}\}$ ,  $C_2 = \{\text{trouser, dress, bag}\}$ , and  $C_3 = \{\text{sandal, sneaker, ankleboot}\}$  when  $M = 3$ .
From Table 1, we observe that the estimates using  $\mathsf{MoB}_K(\psi_{\mathsf{DN}})$  with  $K = \lfloor \sqrt{n}\rfloor$  are slightly lower than  $\psi_{\mathsf{DN}}$ , which is due to the robustness of  $\mathsf{MoB}_K(\psi_{\mathsf{DN}})$  to outliers. We also highlight that the BER estimates can be leveraged to determine which task is more difficult. Intuitively, performing
In general, there might be outliers that lead to low values of the estimated BER. However, since the labels belong to  $[0,1]$  and the estimated BER values in our examples are very small, it is reasonable to assume that outliers with large values would be more impactful than outliers with small values.
Table 1: BER estimate on benchmark and real-world datasets.
# classes	CIFAR-10H	Fashion-MNIST-H	MovieLens
2	3	10	2	3	10	2	3
\( \psi_{\text{DN}} \)	0.0050	0.0177	0.0456	0.0348	0.0932	0.2825	0.3063	0.4031
\( MoB_K(\psi_{\text{DN}}) \)	0.0044	0.0168	0.0440	0.0347	0.0931	0.2816	0.3065	0.4035
the classification task over CIFAR-10H is much easier than performing it over Fashion-MNIST-H, and this is supported by the results in Table 1, i.e., the BERs associated with CIFAR-10H are smaller than those over Fashion-MNIST-H. It is worth noting that test dataset overfitting can happen when benchmark datasets are considered [3, 46, 51, 59, 91]. This can also be observed by the results in Table 1, where our BER estimates of the 10-class classifications are larger than the SOTA error rates of 0.005 on CIFAR-10 by [24] and of 0.0309 on Fashion-MNIST by [82]. We also suspect that, since the labels are assigned by humans, who might not be experts, the datasets might still have a considerable amount of label noise, which would lead to incorrect BER estimates. However, no estimators (including ours) can estimate the BER from a dataset that contains very noisy labels.
MovieLens [38]. This dataset consists of 25 million ratings to 62,000 movies by 162,000 users. Each rating ranges from 0.5 to 5 with step size 0.5. We first considered a movie classification task: a user either likes a movie or not. This is a complex binary classification task with the input feature being a movie (in general, a two-hour long video with audio) and the class being either 0 (dislike the movie) or 1 (like the movie). To ensure that a label belongs to  $[0,1]$ , we applied min-max normalization to each rating. Moreover, in order to have enough ratings for each movie when we applied the denoising method, we filtered out some data if the number of ratings was smaller than 100. Then,  $\psi_{\mathrm{DN}}$  and  $\mathrm{MoB}_K(\psi_{\mathrm{DN}})$  estimated the BER of such movie classification task, and they yielded a BER of around 0.3. We then categorized the ratings into  $M = 3$  classes, i.e., we assigned  $C_1 = \{0.5,1.0,1.5\}$ ,  $C_2 = \{2.0,\dots ,3.5\}$  and  $C_3 = \{4.0,4.5,5.0\}$ . On this task, the BER estimate is around 0.4. These BERs are quite large, implying that the movie classification is a hard task to perform. This may be justified by the fact that ratings of movies are subjective, and movie recommendations are indeed challenging without users' information (e.g., content-based [47] or item-based [74] recommendation systems use users' information).
In this paper, we investigated the challenge of estimating the BER for multi-class classifications. We proposed a BER estimator,  $\psi_{\mathrm{soft}}$ , which we proved to be unbiased, consistent, and asymptotically normal when applied to soft-labeled datasets. By leveraging the median-of-mean method, we also proposed a methodology to make any BER estimate robust. To ensure the applicability of the BER estimator in practical scenarios, we analyzed the challenges posed by noisy soft labels, including those with additive noise and one-hot labels. For such noisy labeled datasets, we developed denoising techniques that effectively mitigate the label noise by using the corresponding features. We showed that these denoising BER estimators are unbiased and consistent under mild noise assumptions. Our experimental results, drawn from synthetic and real-world datasets, validated our theoretical results.
Although in this work we assumed that the set of features is finite, several of our results (e.g., Theorem 1 and Theorem 2) can be easily shown to hold also for the infinite feature space. A research direction worth further investigation would consist of proving that the cluster denoise estimator in Definition 6 has similar appealing properties (see Theorem 3) as the denoise estimator in Definition 5 (which assumes that the set of the features is discrete). Always along these lines, a second interesting future work would be to analyze the rate of convergence of our BER estimator paired with the denoising method as a function of the characteristics (e.g., cardinality, distribution) of the feature space. Such an analysis would indeed provide insights into the difficulty of a classification task. Another relevant avenue for future research lies in assessing the optimal performance of multi-label learning, where a single data point can be associated with multiple classes. While adapting our framework to predict the minimum Hamming loss for multi-label learning might be relatively straightforward, estimating other metrics (e.g., the F1 score and the exact match ratio) might be a difficult task, potentially necessitating new methodologies.
It is well known that the BER of a binary classification problem can be achieved by a likelihood ratio test [66] and the BER of a multiclass classification problem can be attained by a Maximum a Posteriori (MAP) classifier [50]. The problem of estimating the BER using a dataset has been extensively investigated from the middle of the 20th century [16] to the present [13], leveraging different approaches, such as the  $k$ -nearest neighbors (NN) method, probabilistic divergence, and bounds on the BER. In the early stage, the main approach was to derive upper and lower bounds on the BER, while recent work mainly focuses on directly estimating the BER rather than deriving bounds on it.
BER bounds. Several bounds on the BER in binary classification tasks have been proposed using probability divergence and distance metrics. Notable approaches use the Mahalanobis distance [20, 86], the Bhattacharyya distance between two class conditional distributions [20, 49, 86], the Chernoff coefficient (a generalization of the Bhattacharyya distance) [14, 25, 29], the Henze-Penrose divergence [6], the Jensen-Shannon divergence [55], bounds for the  $\min\{a, b\}$  function [2, 39] (a generalization of the equivocation bound [41] and of the Bayesian bound [12]), the  $M_0$ -distance [85], the Fano's inequality [15], and the  $L_2$  norm of posterior probabilities [19]. Some of these bounds are tight, but they still require estimating the data distribution (e.g., class conditional distributions or posterior probabilities), which may not be practical, particularly for modern data (e.g., images and videos).<sup>10</sup> Bounds on the BER were also derived using non-parametric approaches [16, 21, 30, 33], such as the  $k$ -NN classifier. A remarkable advantage of a  $k$ -NN classifier for this problem is its distribution-free property to bound the BER (i.e., it works under no assumption on the data distribution). A convergence rate analysis for universal BER estimators was developed in [1, 22].