SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minoh Jeong  
Electrical and Computer Engineering  
University of Minnesota  
Minneapolis, MN 55455  
jeong316@umn.edu
Martina Cardone  
Electrical and Computer Engineering  
University of Minnesota  
Minneapolis, MN 55455  
mcardone@umn.edu
Alex Dytso  
Qualcomm Flarion Technology, Inc.  
Bridgewater, NJ 08807  
odytso2@gmail.com
Classification is a fundamental task in science and engineering on which machine learning methods have shown outstanding performances. However, it is challenging to determine whether such methods have achieved the Bayes error rate, that is, the lowest error rate attained by any classifier. This is mainly due to the fact that the Bayes error rate is not known in general and hence, effectively estimating it is paramount. Inspired by the work by Ishida et al. (2023), we propose an estimator for the Bayes error rate of supervised multi-class classification problems. We analyze several theoretical aspects of such estimator, including its consistency, unbiasedness, convergence rate, variance, and robustness. We also propose a denoising method that reduces the noise that potentially corrupts the data labels, and we improve the robustness of the proposed estimator to outliers by incorporating the median-of-means estimator. Our analysis demonstrates the consistency, asymptotic unbiasedness, convergence rate, and robustness of the proposed estimators. Finally, we validate the effectiveness of our theoretical results via experiments both on synthetic data under various noise settings and on real data.
Supervised classification problems are typical tasks in various fields of science and engineering, such as machine learning, statistical signal processing, estimation, and detection. In supervised classification, a dataset consisting of several input feature-label pairs is given. The goal of a supervised classification task is to design effective classifiers, by leveraging the given dataset, to suitably label future input features, or equivalently, to classify future input features into one of the classes.
As the dataset is the only available resource on the data distribution, the performance of a classifier is typically measured by its empirical misclassification rate on the test dataset (which is a subset of the given dataset). The best performance of an existing classifier, that is, the so-called state-of-the-art (SOTA) performance, tends to be the point of reference to measure the improvement of a new classifier. However, there are no guarantees that the SOTA performance is close to the theoretical minimum misclassification rate, namely the Bayes error rate (BER). Thus, comparing the empirical misclassification rate with the SOTA error rate provides only a relative improvement.
Having the knowledge of the BER is essential in theory and practice. The BER indeed provides a fundamental limit on the misclassification rate, which is critical for designing high-performing
classifiers. Moreover, one can leverage the BER to assess how good the SOTA performance is with respect to the theoretically optimal error rate. If the SOTA performance is nearly close to the BER, we can avoid wasting time and effort in designing a new classifier. Furthermore, the BER indicates the inherent hardness of a task and hence, it can be seen as a benchmark for comparing the hardness of different tasks [13, 46, 83]. Knowing the BER also brings the opportunity to detect whether test dataset overfitting occurs (which has sporadically happened [3, 51, 59, 91]); this overfitting can be detected since the BER provides the minimum misclassification rate and hence, no classifier will perform strictly better than it. We refer an interested reader to Appendix A for a thorough literature overview on methods to estimate and bound the BER.
In this paper, we investigate the problem of estimating the BER of an  $M$ -class classification task directly from a dataset, where  $M \geq 2$  is arbitrary. Our technique to estimate the BER is different from a plug-in approach that first estimates the distribution from which the data is drawn, and then evaluates the BER. Indeed, our BER estimators, which are proved to be unbiased, consistent and robust to label noise and outliers, do not require the estimation of the data probability density to perform an effective BER estimation. We start by assuming that the data labels are soft and real-valued, approximating the posterior probability of the class. We then relax this assumption on the data labels, and show the applicability of our estimators on one-hot labels and other noisy datasets.
Contributions. Our contribution is summarized as follows. Inspired by [46], in Section 3 we first propose a BER estimator for the case of soft data labels, and we show that it benefits from several appealing properties, e.g., it is consistent, unbiased and asymptotically normal. In Section 3, we also propose a methodology, inspired by the median-of-means estimator [65], to make any BER estimator robust. Then, in Section 4 we study the performance of the proposed estimators in scenarios where the soft labels are corrupted by two typical types of noise, i.e., the case of a noise that permutes/shuffles the labels, and the case of additive noise. For the first type of noise, our estimators have similar properties as in the noiseless case. However, for the additive noise case, our proposed estimators are not consistent. Because of this, we propose a denoising method that averages noisy labels associated with the same feature. The corresponding constructed estimator is shown to be consistent. In Section 4, we also showcase that the noisy soft label framework can be used to study the case of one-hot labels, and we provide a denoising method that encompasses the one proposed for the case of additive noise. In Section 5 we validate the effectiveness of the proposed estimators via experiments both on synthetic data under various noise settings (e.g., one-hot labels) and on real data using three different datasets, i.e., CIFAR-10H [4], Fashion-MNIST-H [46] and MovieLens [38]. Finally, in Section 6 we conclude the paper with some discussion on future research directions, which are worth further investigation.
Notation. Deterministic scalar quantities are denoted by lowercase letters, scalar random variables are denoted by uppercase letters, vectors are denoted by bold lowercase letters, and random vectors by bold uppercase letters (e.g.,  $x$ ,  $X$ ,  $\mathbf{x}$ ,  $\mathbf{X}$ ). We let  $x_{i}$  (resp.,  $(\pmb{x}_k)_i$  ) indicate the  $i$ -th value of  $\pmb{x}$  (resp.,  $\pmb{x}_k$ ). Calligraphic letters  $\mathcal{X}$  denote sets, and  $|\mathcal{X}|$  is the cardinality of  $\mathcal{X}$ .  $\mathbb{1}\{\mathcal{S}\}$  is the indicator function that yields 1 if  $\mathcal{S}$  is true and 0 otherwise.  $[M] := \{1,\dots ,M\}$ . For  $\mathbf{X} \in \mathbb{R}^{M}$ , we let  $X_{i:M}$  be the  $i$ -th order statistics [18] of  $\mathbf{X}$ , i.e., the  $i$ -th smallest value of  $\mathbf{X}$  with  $i \in [M]$ . Finally,  $I_{n}$  is the identity matrix of dimension  $n$ , and  $\stackrel{d}{\rightarrow}$  (resp.,  $\stackrel{d}{=}$ ) denotes convergence (resp., equality) in distribution.
We consider an  $M$ -class classification task in which an input feature  $\pmb{x}$  is classified into a class  $c \in \mathcal{C} \coloneqq [M]$ . Our goal is to estimate the minimum misclassification probability, that is the BER. In particular, we seek to estimate the BER based on a dataset  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$  that follows an unknown data distribution, i.e.,  $(x_i, y_i)^{i.i.d.} \sim P_{\mathbf{X}, \mathbf{Y}}$ , where  $\mathbf{X} \in \mathcal{X}$  and  $\mathbf{Y} \in \mathcal{Y} \subseteq [0,1]^M$ .
We assume that the label data  $\mathbf{y}$  contains the information about the class  $c$  of the input feature  $\mathbf{x}$ , implying that one can retrieve  $(\mathbf{x}, c)$  from  $(\mathbf{x}, \mathbf{y})$ . When a classifier  $\phi: \mathcal{X} \to \mathcal{C}$  is employed for a classification task, the corresponding misclassification probability is defined as  $\mathcal{E}(\phi) := \operatorname*{Pr}(\phi(\mathbf{X}) \neq C)$ , where  $C$  is the true class for  $\mathbf{X}$ . The minimum value of this misclassification probability is the so-called BER, denoted by  $P_e$ , which is formally defined next.
<sup>1</sup>We assume that  $\mathcal{X}$  is a finite set, but several of our results easily extend to the case when  $\mathcal{X}$  is an infinite set.
Definition 1 (Bayes error rate [50]). Consider an  $M$ -class classification problem, where an input feature  $\pmb{x} \in \mathcal{X}$  has to be classified into a class  $c \in \mathcal{C} := [M]$ . The BER is defined as
$$
P _ {e} = \inf  _ {\phi \in \Phi} \mathcal {E} (\phi) = \inf  _ {\phi \in \Phi} \mathbb {E} \left[ \mathbb {1} \left\{\phi (\mathbf {X}) \neq C \right\} \right], \tag {1}
$$
where  $\Phi$  is the set of all measurable functions  $\phi : \mathcal{X} \to \mathcal{C}$ , and the expectation is taken over  $P_{\mathbf{X}, C}$ .
The misclassification probability  $\mathcal{E}(\phi)$  depends on the quality of the classifier  $\phi$  and the BER is obtained by choosing an optimal classifier. In fact, an optimal classifier is theoretically equivalent to the Maximum a Posteriori (MAP) classifier [50], that is,  $\phi_{\mathrm{MAP}}(\pmb{x}) = \arg \max_{k\in \mathcal{C}}\operatorname*{Pr}(C = k|\mathbf{X} = \pmb{x})$ . Plugging the MAP classifier into (1), the BER can be written as
$$
P _ {e} = \mathbb {E} \left[ 1 - \max  _ {k \in \mathcal {C}} \Pr (C = k | \mathbf {X}) \right], \tag {2}
$$
where the expectation is taken with respect to  $P_{\mathbf{X}}$ . Note that  $P_{e}\in \left[0,1 - \frac{1}{M}\right]$
Our main objective is to effectively estimate the BER from the dataset  $\mathcal{D}$ . In the remaining of the paper, we let  $\psi : \mathcal{D} \to [0,1]$  denote the estimator of the BER.
3 BER estimation with soft labels
Soft labels have several favorable properties (e.g., they help to prevent an overfitting problem, they lead to a well-structured model, and they improve the prediction performance) that make them widely used in machine learning. For example, they are essential in label smoothing and knowledge distillation, which are widely applied methods to improve model performance [81, 97–101].
There exist several types of soft labels and here we assume that a label is soft if it represents the posterior probability. Specifically, we say that  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^n$  is a dataset with soft labels if  $x_{i}\in \mathcal{X}$  and  $y_{i}\in \mathcal{V}$  are such that
$$
\boldsymbol {y} _ {i} = \left[ \begin{array}{c} \Pr (C = 1 | \mathbf {X} = \boldsymbol {x} _ {i}) \\ \Pr (C = 2 | \mathbf {X} = \boldsymbol {x} _ {i}) \\ \vdots \\ \Pr (C = M | \mathbf {X} = \boldsymbol {x} _ {i}) \end{array} \right]. \tag {3}
$$
This is a standard and widely adopted assumption; it has indeed been argued [17, 35, 42, 61, 69, 101] that using soft labels to approximate posterior probabilities enhances model performance.
Definition 2 (Base BER estimator). The BER estimator  $\psi_{\mathrm{soft}}(\mathcal{D})$  is defined as
$$
\psi_ {\text {s o f t}} (\mathcal {D}) = \frac {1}{n} \sum_ {(\boldsymbol {x}, \boldsymbol {y}) \in \mathcal {D}} \left(1 - \max  _ {j \in [ M ]} y _ {j}\right). \tag {4}
$$
We will use the base estimator  $\psi_{\mathrm{soft}}$  in (4) as a building block for a robust BER estimation. We note that  $\psi_{\mathrm{soft}}$  in (4) with  $M = 2$  retrieves the estimator proposed in [46]. The next theorem (proof in Appendix B.1) provides three important properties of  $\psi_{\mathrm{soft}}$  in (4).
Theorem 1. Assume that  $\mathcal{D}$  contains soft labels as defined in (3). Then,  $\psi_{\mathrm{soft}}(\mathcal{D})$  satisfies the following properties:
1. (Unbiasedness):  $\mathbb{E}[\psi_{\mathrm{soft}}(\mathcal{D})] = P_e$ , that is,  $\psi_{\mathrm{soft}}(\mathcal{D})$  is an unbiased estimator of the BER;
2. (Consistency): For any  $\delta \in (0,1)$ , it holds that  $|\psi_{\mathrm{soft}} - P_e| < \sqrt{\frac{(1 - \frac{1}{M})^2}{2n}\ln\frac{2}{\delta}}$  with probability at least  $1 - \delta$ , that is,  $\psi_{\mathrm{soft}}(\mathcal{D})$  is a consistent estimator of the BER;
3. (Asymptotic Normality):  $\sqrt{n} (\psi_{\mathrm{soft}} - P_e)\stackrel {d}{\to}\mathcal{N}(0,Var(Y_{M:M}))$  as  $n\to \infty$
Theorem 1 shows that the BER can be effectively estimated directly from a dataset that contains soft labels as in (3). Moreover, it highlights that the convergence rate of  $\psi_{\mathrm{soft}}(\mathcal{D})$  is  $n^{-\frac{1}{2}}$ , which is indeed the optimal (parametric) convergence rate.
Another important aspect to assess the performance of  $\psi_{\mathrm{soft}}(\mathcal{D})$  is to measure how far it is from the BER  $P_e$ . Since  $\psi_{\mathrm{soft}}(\mathcal{D})$  is unbiased (from Theorem 1) this distance can be measured by computing the variance of  $\psi_{\mathrm{soft}}(\mathcal{D})$ , which is denoted as  $\mathrm{Var}(\psi_{\mathrm{soft}})$  and provided by the next proposition (proof in Appendix B.2).
Proposition 1. It holds that  $\text{Var}(\psi_{\text{soft}}) = \frac{1}{n}\text{Var}(Y_{M:M})$  and  $\text{Var}(\psi_{\text{soft}}) \leq \frac{\left(1 - \frac{1}{M}\right)P_e - P_e^2}{n} \leq \frac{\left(1 - \frac{1}{M}\right)^2}{4n}$ .
The exact computation of  $\mathrm{Var}(\psi_{\mathrm{soft}})$  in Proposition 1 requires the knowledge of the order statistics of  $\mathbf{Y}$  and hence, of the label distribution  $P_{\mathbf{Y}}$ . The upper bounds on  $\mathrm{Var}(\psi_{\mathrm{soft}})$  are instead distribution-independent. In particular, both upper bounds show that the rate of convergence of  $\mathrm{Var}(\psi_{\mathrm{soft}})$  is  $1 / n$ , which is in line with Theorem 1. Moreover, the first upper bound on  $\mathrm{Var}(\psi_{\mathrm{soft}})$  implies that  $\mathrm{Var}(\psi_{\mathrm{soft}}) \to 0$  when either  $P_e \to 0$  (i.e., also known as realizability assumption [76]) or  $P_e \to 1 - \frac{1}{M}$  (i.e., labels and features are independent). The first upper bound on  $\mathrm{Var}(\psi_{\mathrm{soft}})$  also allows to find an upper bound on  $P_e$  which becomes tight when  $P_e \to 1 - 1 / M$ .
3.1 Robustness of  $\psi_{\mathrm{soft}}$
We consider robustness to outliers, where an outlier is a data sample that is corrupted by high noise. We use the concept of breakdown point [44, 60] to measure the robustness of  $\psi_{\mathrm{soft}}$ . The breakdown point captures how robust an estimator is with respect to outliers. In the classical definition of breakdown point, the worst estimator (in terms of robustness) for a dataset  $\mathcal{D}$  with outliers is defined as  $\psi (\mathcal{D}) = \infty$ . However, in our setting  $\psi_{\mathrm{soft}}(\mathcal{D})\leq 1 - \frac{1}{M}$  since  $\psi_{\mathrm{soft}}(\mathcal{D})$  is an estimate of  $P_{e}$ . Because of this, we next adopt an alternative definition for the breakdown point which is commonly used for a bounded parameter space  $\Theta$  [45].
Definition 3 (Breakdown point). Consider an estimator  $\psi : \Omega^n \to \Theta$  of  $\theta \in \Theta \subseteq \mathbb{R}^L$ . Let  $\mathcal{D}^{(\kappa)} = \{D_1, \ldots, D_\kappa\}$ , where  $D_i \in \Omega$ ,  $\forall i \in [\kappa]$ , be a clean dataset without outliers, and let  $\widetilde{\mathcal{D}}^{(\tau)} = \{\widetilde{D}_1, \ldots, \widetilde{D}_{\tau}\}$  be a noisy dataset that is composed of  $\tau$  noisy data samples that can be arbitrarily replaced by outliers. Then, the breakdown point of  $\psi$  is defined as
$$
B (\psi) = \min  _ {\tau : 1 \leq \tau \leq \kappa} \left\{\frac {\tau}{\kappa + \tau}: \sup  \left\| \psi \left(\mathcal {D} ^ {(\kappa + \tau)}\right) - \psi \left(\mathcal {D} ^ {(\kappa)} \cup \widetilde {\mathcal {D}} ^ {(\tau)}\right) \right\| \geq \| \operatorname {r a d} (\Theta) \| \right\}, \tag {5}
$$
where the sup is taken over all  $\mathcal{D}^{(\kappa +\tau)},\mathcal{D}^{(\kappa)}$  and  $\widetilde{\mathcal{D}}^{(\tau)}$  such that  $\mathcal{D}^{(\kappa)}\subset \mathcal{D}^{(\kappa +\tau)}$ , and  $\operatorname {rad}(\Theta)$  is the vector consisting of the  $L$  radii of the largest  $L$ -dimensional ellipsoid in  $\Theta$ .<sup>3</sup>
$B(\psi)$  in (5) quantifies the value of the breakdown point for an estimator  $\psi : \Omega^{\tau + \kappa} \to \Theta$ . In particular, an estimator  $\psi$  "breaks down" if  $\psi(\mathcal{D}^{(\kappa + \tau)})$  changes of at least  $\|\mathrm{rad}(\Theta)\|$  when  $\tau$  clean data samples are replaced with  $\tau$  outliers. The breakdown point is defined as the minimum ratio between the number of outliers  $(\tau)$  and the total number of data samples  $(\kappa + \tau)$ , such that  $\tau$  outliers are sufficient to break the estimator  $\psi(\mathcal{D}^{(\kappa + \tau)})$ . From Definition 3, it is clear that the higher the value of the breakdown point, the more robust the estimator is. To measure the breakdown point of  $\psi_{\mathrm{soft}}$ , we start by noting that  $\psi_{\mathrm{soft}}$  is the sample mean of  $\{1 - \max_{j \in [M]} (\pmb{y}_i)_j\}_{i=1}^n$  (see (4)). Thus, using the notation as in Definition 3, we have that  $\Theta = [0, 1 - \frac{1}{M}]$ , which leads to  $\mathrm{rad}(\Theta) = \frac{1}{2} - \frac{1}{2M}$ . This implies that  $B(\psi_{\mathrm{soft}}) = \frac{1}{n}$  (e.g., when  $\mathcal{D}$  contains an outlier  $\pmb{y}$  such that  $\max_{j \in [M]} y_j = \infty$ ) and hence,  $\psi_{\mathrm{soft}}$  is not robust to outliers.
We next propose a methodology, inspired by the median-of-means estimator [65], to make any BER estimator (and hence, also  $\psi_{\mathrm{soft}}$ ) robust.
Definition 4 (Median-of-BERs (MoB) estimator). Consider any BER estimator  $\psi$  and a dataset  $\mathcal{D}$ . First, partition  $\mathcal{D}$  into  $K$  sub-datasets  $\mathcal{D}_k$ ,  $k \in [K]$  such that  $|\mathcal{D}| = n = cK$ , where  $c \in \mathbb{N}$  is the number of data samples in  $\mathcal{D}_k$ . Then, the Median-of-BERs (MoB) estimator is defined as
$$
\operatorname {M o B} _ {K} (\psi , \mathcal {D}) = \operatorname {m e d} \left(\left\{\psi \left(\mathcal {D} _ {k}\right): k \in [ K ] \right\}\right), \tag {6}
$$
where  $\operatorname{med}(\mathcal{S})$  is the sample median of  $\mathcal{S}$ .
2The sample mean can fail to estimate the true mean by a single outlier (e.g., a sample  $x$  with a value very different from the true mean). Thus, the sample mean has a breakdown point equal to  $1 / n$ , meaning that one outlier can break the estimate. Differently, the median is more robust since it has a breakdown point of  $1 / 2$ , i.e., half of the samples need to be outliers for breaking the estimate.
<sup>3</sup>Any  $L$ -dimensional ellipsoid (after rotation) can be defined as  $\frac{x_1^2}{a_1^2} + \frac{x_2^2}{a_2^2} + \ldots + \frac{x_L^2}{a_L^2} = 1$ . Then, the  $L$  radii are  $a_1, a_2, \ldots, a_L$ .
The next theorem (proof in Appendix B.3) provides three important properties of  $\mathrm{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$
Theorem 2. Assume that  $\mathcal{D}$  contains soft labels as defined in (3). Then, the MoB estimator  $\mathrm{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  satisfies the following properties:
1. (Consistency): For all  $t \lesssim K$ , with probability at least  $1 - 4e^{-2t}$ , it holds that
$$
\left| \operatorname {M o B} _ {K} \left(\psi_ {\text {s o f t}}, \mathcal {D}\right) - P _ {e} \right| \leq \left(\frac {\left(1 - \frac {1}{M}\right) ^ {3}}{2 \sqrt {3} \operatorname {V a r} \left(Y _ {M : M}\right)} \frac {K}{\sqrt {n - K}} + 3 \sqrt {t \operatorname {V a r} \left(Y _ {M : M}\right)}\right) \sqrt {\frac {1}{n - K}}, \tag {7}
$$
and hence,  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  is a consistent estimator of the BER;
2. (Breakdown): Its breakdown point is given by  $B(\mathsf{MoB}_K(\psi_{\mathrm{soft}}, \mathcal{D})) = \left\lfloor \frac{K + 1}{2} \right\rfloor \frac{1}{n}$ ;
3. (Asymptotic Normality): If  $K \to \infty$  and  $K = o(\sqrt{n})$  as  $n \to \infty$ , then  $\sqrt{n}(\mathsf{MoB}_K(\psi_{\mathrm{soft}}, \mathcal{D}) - P_e) \xrightarrow{d} \mathcal{N}\left(0, \frac{\pi}{2} \mathrm{Var}(Y_{M:M})\right)$ .
Theorem 2 shows that  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  has the same rate of convergence of  $n^{-\frac{1}{2}}$  as  $\psi_{\mathrm{soft}}$  (see Theorem 1). However,  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  has a higher breakdown point (and hence, is more robust) than  $\psi_{\mathrm{soft}}$ . Nonetheless, this robustness is attained at two expenses: (i)  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  is not an unbiased estimator of  $P_e$ ; and (ii) the variance of  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$  is larger than the one of  $\psi_{\mathrm{soft}}$  by a factor of  $\pi /2$  in the asymptotic regime.
Theorem 2 also points out to an interesting trade-off, with respect to  $K$ , between accuracy and robustness. For example, setting  $K = \sqrt{n}$  leads to a higher breakdown point (and hence, is more robust) than setting  $K = \ln n$ . However, the concentration bound in (7) (which measures the accuracy of the estimation) has a much larger value with  $K = \sqrt{n}$  than with  $K = \ln n$ .
4 BER estimation with noisy labels
The soft label data discussed in Section 3 might be challenging to obtain as data labels are often corrupted by noise or other perturbations. In such scenarios, data labels are unreliable [27, 64, 68] and several studies have been conducted on them [37, 57, 80, 88, 90, 92, 94, 96].
With the goal to broaden the applicability of  $\psi_{\mathrm{soft}}$  (or its robust version  $\mathsf{MoB}_K(\psi_{\mathrm{soft}},\mathcal{D})$ ), we here study their performance in scenarios where the soft labels are corrupted by various typical types of noise.4 We denote by  $\widetilde{\mathbf{Y}}\sim P_{\widetilde{\mathbf{Y}} |\mathbf{Y}}$  the noisy soft label, i.e., a noisy soft label  $\widetilde{\mathbf{Y}} = \widetilde{\boldsymbol{y}}_i$  is distributed according to the conditional probability distribution  $P_{\widetilde{\mathbf{Y}} |\mathbf{Y} = \boldsymbol {y}_i}$ , where  $\boldsymbol {y}_i$  is the true soft label.
4.1 Additive noise on the data labels
We here consider the case where the labels are corrupted by additive noise, i.e., we have  $(\pmb{x},\tilde{\pmb{y}})\in \widetilde{\mathcal{D}}_{\mathrm{A}}$  where  $\tilde{\pmb{y}} = \pmb {y} + \pmb{z}$ , where  $\pmb{z}$  is an  $M$ -dimensional random noise vector. We assume that  $\pmb{z}$  has i.i.d. components each with zero mean.
We start our analysis by showing that  $\psi_{\mathrm{soft}}$  in (4) applied on  $\widetilde{\mathcal{D}}_{\mathrm{A}}$  is neither a consistent nor an unbiased estimator of  $P_{e}$ . We, in fact, have that
$$
\psi_ {\text {s o f t}} \left(\widetilde {\mathcal {D}} _ {\mathrm {A}}\right) = 1 - \frac {1}{n} \sum_ {i = 1} ^ {n} \max  _ {j \in [ M ]} \left\{\left(\boldsymbol {y} _ {i}\right) _ {j} + \left(\boldsymbol {z} _ {i}\right) _ {j} \right\}, \tag {8}
$$
which, by the law of large numbers, converges to
$$
\lim  _ {n \rightarrow \infty} \psi_ {\text {s o f t}} (\widetilde {\mathcal {D}} _ {\mathrm {A}}) = 1 - \mathbb {E} \left[ \max  _ {j \in [ M ]} \left\{Y _ {j} + Z _ {j} \right\}\right]. \tag {9}
$$
Then, the inequalities  $\max_i x_i + \min_j y_j \leq \max_i \{x_i + y_i\} \leq \max_i x_i + \max_j y_j$  imply that
$$
\mathbb {E} \left[ Z _ {1: M} \right] \leq P _ {e} - \lim  _ {n \rightarrow \infty} \psi_ {\text {s o f t}} \left(\widetilde {\mathcal {D}} _ {\mathrm {A}}\right) \leq \mathbb {E} \left[ Z _ {M: M} \right], \tag {10}
$$
4We refer an interested reader to Appendix B.4 where a noise that randomly shuffles the labels is also studied.