Title: Demystifying the Optimal Performance of Multi-Class Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayes error

estimation

classification

minimum error probability
Abstract: Classification is a fundamental task in science and engineering on which machine learning methods have shown outstanding performances. However, it is challenging to determine whether such methods have achieved the Bayes error rate, that is, the lowest error rate attained by any classifier. This is mainly due to the fact that the Bayes error rate is not known in general and hence, effectively estimating it is paramount. Inspired by the work by Ishida et al. (2023), we propose an estimator for the Bayes error rate of supervised multi-class classification problems. We analyze several theoretical aspects of such estimator, including its consistency, unbiasedness, convergence rate, variance, and robustness. We also propose a denoising method that reduces the noise that potentially corrupts the data labels, and we improve the robustness of the proposed estimator to outliers by incorporating the median-of-means estimator. Our analysis demonstrates the consistency, asymptotic unbiasedness, convergence rate, and robustness of the proposed estimators. Finally, we validate the effectiveness of our theoretical results via experiments both on synthetic data under various noise settings and on real data.

Demystifying the Optimal Performance of Multi-Class Classification

Abstract

Classification is a fundamental task in science and engineering on which machine learning methods have shown outstanding performances. However, it is challenging to determine whether such methods have achieved the Bayes error rate, that is, the lowest error rate attained by any classifier. This is mainly due to the fact that the Bayes error rate is not known in general and hence, effectively estimating it is paramount. Inspired by the work by Ishida et al. (2023), we propose an estimator for the Bayes error rate of supervised multi-class classification problems. We analyze several theoretical aspects of such estimator, including its consistency, unbiasedness, convergence rate, variance, and robustness. We also propose a denoising method that reduces the noise that potentially corrupts the data labels, and we improve the robustness of the proposed estimator to outliers by incorporating the median-of-means estimator. Our analysis demonstrates the consistency, asymptotic unbiasedness, convergence rate, and robustness of the proposed estimators. Finally, we validate the effectiveness of our theoretical results via experiments both on synthetic data under various noise settings and on real data.

1 Introduction

Supervised classification problems are typical tasks in various fields of science and engineering, such as machine learning, statistical signal processing, estimation, and detection. In supervised classification, a dataset consisting of several input feature-label pairs is given. The goal of a supervised classification task is to design effective classifiers, by leveraging the given dataset, to suitably label future input features, or equivalently, to classify future input features into one of the classes.

As the dataset is the only available resource on the data distribution, the performance of a classifier is typically measured by its empirical misclassification rate on the test dataset (which is a subset of the given dataset). The best performance of an existing classifier, that is, the so-called state-of-theart (SOTA) performance, tends to be the point of reference to measure the improvement of a new classifier. However, there are no guarantees that the SOTA performance is close to the theoretical minimum misclassification rate, namely the Bayes error rate (BER). Thus, comparing the empirical misclassification rate with the SOTA error rate provides only a relative improvement.

Having the knowledge of the BER is essential in theory and practice. The BER indeed provides a fundamental limit on the misclassification rate, which is critical for designing high-performing

classifiers. Moreover, one can leverage the BER to assess how good the SOTA performance is with respect to the theoretically optimal error rate. If the SOTA performance is nearly close to the BER, we can avoid wasting time and effort in designing a new classifier. Furthermore, the BER indicates the inherent hardness of a task and hence, it can be seen as a benchmark for comparing the hardness of different tasks [13, 46, 83]. Knowing the BER also brings the opportunity to detect whether test dataset overfitting occurs (which has sporadically happened [3, 51, 59, 91]); this overfitting can be detected since the BER provides the minimum misclassification rate and hence, no classifier will perform strictly better than it. We refer an interested reader to Appendix A for a thorough literature overview on methods to estimate and bound the BER.

In this paper, we investigate the problem of estimating the BER of an M -class classification task directly from a dataset, where M ≥ 2 is arbitrary. Our technique to estimate the BER is different from a plug-in approach that first estimates the distribution from which the data is drawn, and then evaluates the BER. Indeed, our BER estimators, which are proved to be unbiased, consistent and robust to label noise and outliers, do not require the estimation of the data probability density to perform an effective BER estimation. We start by assuming that the data labels are soft and real-valued, approximating the posterior probability of the class. We then relax this assumption on the data labels, and show the applicability of our estimators on one-hot labels and other noisy datasets.

Contributions. Our contribution is summarized as follows. Inspired by [46], in Section 3 we first propose a BER estimator for the case of soft data labels, and we show that it benefits from several appealing properties, e.g., it is consistent, unbiased and asymptotically normal. In Section 3, we also propose a methodology, inspired by the median-of-means estimator [65], to make any BER estimator robust. Then, in Section 4 we study the performance of the proposed estimators in scenarios where the soft labels are corrupted by two typical types of noise, i.e., the case of a noise that permutes/shuffles the labels, and the case of additive noise. For the first type of noise, our estimators have similar properties as in the noiseless case. However, for the additive noise case, our proposed estimators are not consistent. Because of this, we propose a denoising method that averages noisy labels associated with the same feature. The corresponding constructed estimator is shown to be consistent. In Section 4, we also showcase that the noisy soft label framework can be used to study the case of one-hot labels, and we provide a denoising method that encompasses the one proposed for the case of additive noise. In Section 5 we validate the effectiveness of the proposed estimators via experiments both on synthetic data under various noise settings (e.g., one-hot labels) and on real data using three different datasets, i.e., CIFAR-10H [4], Fashion-MNIST-H [46] and MovieLens [38]. Finally, in Section 6 we conclude the paper with some discussion on future research directions, which are worth further investigation.

Notation. Deterministic scalar quantities are denoted by lowercase letters, scalar random variables are denoted by uppercase letters, vectors are denoted by bold lowercase letters, and random vectors by bold uppercase letters (e.g., x , X , x , X ). We let x i (resp., ( x k ) i ) indicate the i -th value of x (resp., x k ). Calligraphic letters X denote sets, and |X| is the cardinality of X . ✶ {S} is the indicator function that yields 1 if S is true and 0 otherwise. [ M ] := { 1 , . . . , M } . For X ∈ R M , we let X i : M be the i -th order statistics [18] of X , i.e., the i -th smallest value of X with i ∈ [ M ] . Finally, I n is the identity matrix of dimension n , and d → (resp., d = ) denotes convergence (resp., equality) in distribution.

2 Problem formulation

We consider an M -class classification task in which an input feature x is classified into a class c ∈ C := [ M ] . Our goal is to estimate the minimum misclassification probability, that is the BER. In particular, we seek to estimate the BER based on a dataset D = { ( x i , y i ) } n i =1 that follows an unknown data distribution, i.e., ( x i , y i ) i.i.d. ∼ P X , Y , where X ∈ X and Y ∈ Y ⊆ [0 , 1] M . 1

̸

We assume that the label data y contains the information about the class c of the input feature x , implying that one can retrieve ( x , c ) from ( x , y ) . When a classifier ϕ : X → C is employed for a classification task, the corresponding misclassification probability is defined as E ( ϕ ) := Pr( ϕ ( X ) = C ) , where C is the true class for X . The minimum value of this misclassification probability is the so-called BER, denoted by P e , which is formally defined next.

Definition 1 (Bayes error rate [50]) . Consider an M -class classification problem, where an input feature x ∈ X has to be classified into a class c ∈ C := [ M ] . The BER is defined as

̸

where Φ is the set of all measurable functions ϕ : X → C , and the expectation is taken over P X ,C .

The misclassification probability E ( ϕ ) depends on the quality of the classifier ϕ and the BER is obtained by choosing an optimal classifier. In fact, an optimal classifier is theoretically equivalent to the Maximum a Posteriori (MAP) classifier [50], that is, ϕ MAP ( x ) = arg max k ∈C Pr( C = k | X = x ) . Plugging the MAP classifier into (1), the BER can be written as

where the expectation is taken with respect to P X . Note that P e ∈ [ 0 , 1 -1 M ] .

Our main objective is to effectively estimate the BER from the dataset D . In the remaining of the paper, we let ψ : D → [0 , 1] denote the estimator of the BER.

3 BER estimation with soft labels

Soft labels have several favorable properties (e.g., they help to prevent an overfitting problem, they lead to a well-structured model, and they improve the prediction performance) that make them widely used in machine learning. For example, they are essential in label smoothing and knowledge distillation, which are widely applied methods to improve model performance [81, 97-101].

There exist several types of soft labels and here we assume that a label is soft if it represents the posterior probability. Specifically, we say that D = { ( x i , y i ) } n i =1 is a dataset with soft labels if x i ∈ X and y i ∈ Y are such that

This is a standard and widely adopted assumption; it has indeed been argued [17, 35, 42, 61, 69, 101] that using soft labels to approximate posterior probabilities enhances model performance.

Definition 2 (Base BER estimator) . The BER estimator ψ soft ( D ) is defined as

We will use the base estimator ψ soft in (4) as a building block for a robust BER estimation. We note that ψ soft in (4) with M =2 retrieves the estimator proposed in [46]. The next theorem (proof in Appendix B.1) provides three important properties of ψ soft in (4).

Theorem 1. Assume that D contains soft labels as defined in (3) . Then, ψ soft ( D ) satisfies the following properties:

(Unbiasedness): E [ ψ soft ( D )] = P e , that is, ψ soft ( D ) is an unbiased estimator of the BER;

(Consistency): For any δ ∈ (0 , 1) , it holds that | ψ soft -P e | < √ ( 1 -1 M ) 2 2 n ln 2 δ with probability at least 1 -δ , that is, ψ soft ( D ) is a consistent estimator of the BER;

(Asymptotic Normality) : √ n ( ψ soft -P e ) d →N (0 , Var ( Y M : M )) as n →∞ .

Theorem 1 shows that the BER can be effectively estimated directly from a dataset that contains soft labels as in (3). Moreover, it highlights that the convergence rate of ψ soft ( D ) is n -1 2 , which is indeed the optimal (parametric) convergence rate.

Another important aspect to assess the performance of ψ soft ( D ) is to measure how far it is from the BER P e . Since ψ soft ( D ) is unbiased (from Theorem 1) this distance can be measured by computing the variance of ψ soft ( D ) , which is denoted as Var ( ψ soft ) and provided by the next proposition (proof in Appendix B.2).

The exact computation of Var ( ψ soft ) in Proposition 1 requires the knowledge of the order statistics of Y and hence, of the label distribution P Y . The upper bounds on Var ( ψ soft ) are instead distributionindependent. In particular, both upper bounds show that the rate of convergence of Var ( ψ soft ) is 1 /n , which is in line with Theorem 1. Moreover, the first upper bound on Var ( ψ soft ) implies that Var ( ψ soft ) → 0 when either P e → 0 (i.e., also known as realizability assumption [76]) or P e → 1 -1 M (i.e., labels and features are independent). The first upper bound on Var ( ψ soft ) also allows to find an upper bound on P e which becomes tight when P e → 1 -1 /M .

3.1 Robustness of ψ soft

We consider robustness to outliers, where an outlier is a data sample that is corrupted by high noise. We use the concept of breakdown point [44, 60] to measure the robustness of ψ soft . The breakdown point captures how robust an estimator is with respect to outliers. 2 In the classical definition of breakdown point, the worst estimator (in terms of robustness) for a dataset D with outliers is defined as ψ ( D ) = ∞ . However, in our setting ψ soft ( D ) ≤ 1 -1 M since ψ soft ( D ) is an estimate of P e . Because of this, we next adopt an alternative definition for the breakdown point which is commonly used for a bounded parameter space Θ [45].

Definition 3 (Breakdown point) . Consider an estimator ψ : Ω n → Θ of θ ∈ Θ ⊆ R L . Let D ( κ ) = { D 1 , . . . , D κ } , where D i ∈ Ω , ∀ i ∈ [ κ ] , be a clean dataset without outliers, and let ˜ D ( τ ) = { ˜ D 1 , . . . , ˜ D τ } be a noisy dataset that is composed of τ noisy data samples that can be arbitrarily replaced by outliers. Then, the breakdown point of ψ is defined as

where the sup is taken over all D ( κ + τ ) , D ( κ ) and ˜ D ( τ ) such that D ( κ ) ⊂ D ( κ + τ ) , and rad (Θ) is the vector consisting of the L radii of the largest L -dimensional ellipsoid in Θ . 3

B ( ψ ) in (5) quantifies the value of the breakdown point for an estimator ψ : Ω τ + κ → Θ . In particular, an estimator ψ 'breaks down' if ψ ( D ( κ + τ ) ) changes of at least ∥ rad (Θ) ∥ when τ clean data samples are replaced with τ outliers. The breakdown point is defined as the minimum ratio between the number of outliers ( τ ) and the total number of data samples ( κ + τ ), such that τ outliers are sufficient to break the estimator ψ ( D ( κ + τ ) ) . From Definition 3, it is clear that the higher the value of the breakdown point, the more robust the estimator is. To measure the breakdown point of ψ soft , we start by noting that ψ soft is the sample mean of { 1 -max j ∈ [ M ] ( y i ) j } n i =1 (see (4)). Thus, using the notation as in Definition 3, we have that Θ = [ 0 , 1 -1 M ] , which leads to rad (Θ) = 1 2 -1 2 M . This implies that B ( ψ soft ) = 1 n (e.g., when D contains an outlier y such that max j ∈ [ M ] y j = ∞ ) and hence, ψ soft is not robust to outliers.

We next propose a methodology, inspired by the median-of-means estimator [65], to make any BER estimator (and hence, also ψ soft ) robust.

Definition 4 (Median-of-BERs (MoB) estimator) . Consider any BER estimator ψ and a dataset D . First, partition D into K sub-datasets D k , k ∈ [ K ] such that |D| = n = cK , where c ∈ N is the number of data samples in D k . Then, the Median-of-BERs (MoB) estimator is defined as

where med ( S ) is the sample median of S .

The next theorem (proof in Appendix B.3) provides three important properties of MoB K ( ψ soft , D ) .

Theorem 2. Assume that D contains soft labels as defined in (3) . Then, the MoB estimator MoB K ( ψ soft , D ) satisfies the following properties:

(Consistency): For all t ≲ K , with probability at least 1 -4 e -2 t , it holds that

and hence, MoB K ( ψ soft , D ) is a consistent estimator of the BER;

(Breakdown): Its breakdown point is given by B ( MoB K ( ψ soft , D )) = ⌊ K +1 2 ⌋ 1 n ;

(Asymptotic Normality): If K → ∞ and K = o ( √ n ) as n → ∞ , then √ n ( MoB K ( ψ soft , D ) -P e ) d →N ( 0 , π 2 Var ( Y M : M ) ) .

Theorem 2 shows that MoB K ( ψ soft , D ) has the same rate of convergence of n -1 2 as ψ soft (see Theorem 1). However, MoB K ( ψ soft , D ) has a higher breakdown point (and hence, is more robust) than ψ soft . Nonetheless, this robustness is attained at two expenses: (i) MoB K ( ψ soft , D ) is not an unbiased estimator of P e ; and (ii) the variance of MoB K ( ψ soft , D ) is larger than the one of ψ soft by a factor of π/ 2 in the asymptotic regime.

Theorem 2 also points out to an interesting trade-off, with respect to K , between accuracy and robustness. For example, setting K = √ n leads to a higher breakdown point (and hence, is more robust) than setting K = ln n . However, the concentration bound in (7) (which measures the accuracy of the estimation) has a much larger value with K = √ n than with K = ln n .

4 BER estimation with noisy labels

The soft label data discussed in Section 3 might be challenging to obtain as data labels are often corrupted by noise or other perturbations. In such scenarios, data labels are unreliable [27, 64, 68] and several studies have been conducted on them [37, 57, 80, 88, 90, 92, 94, 96].

With the goal to broaden the applicability of ψ soft (or its robust version MoB K ( ψ soft , D ) ), we here study their performance in scenarios where the soft labels are corrupted by various typical types of noise. 4 We denote by ˜ Y ∼ P ˜ Y | Y the noisy soft label, i.e., a noisy soft label ˜ Y = ˜ y i is distributed according to the conditional probability distribution P ˜ Y | Y = y i , where y i is the true soft label.

4.1 Additive noise on the data labels

We here consider the case where the labels are corrupted by additive noise, i.e., we have ( x , ˜ y ) ∈ ˜ D A where ˜ y = y + z , where z is an M -dimensional random noise vector. We assume that z has i.i.d. components each with zero mean.

Westart our analysis by showing that ψ soft in (4) applied on ˜ D A is neither a consistent nor an unbiased estimator of P e . We, in fact, have that

which, by the law of large numbers, converges to

Then, the inequalities max i x i +min j y j ≤ max i { x i + y i } ≤ max i x i +max j y j imply that

which follow from (2) and (3). The bounds in (10) demonstrate that with labels corrupted by additive noise, ψ soft is (in general) an inconsistent estimator of P e . By following similar steps, it can be easily proved that ψ soft is also a biased estimator of P e . In particular, the bounds in (10) show that ψ soft has a bias which is bounded by the expected value of the smallest (lower bound) and of the largest (upper bound) order statistics of the noise. In what follows, we provide two examples of distributions for which bounds on these expected values have been computed.

Example 1 . Let Z i.i.d. ∼ Uniform( -a, a ) . Then, E [ Z i : M ] = 2 ai M +1 -a , which as n →∞ implies that | P e -ψ soft ( ˜ D A ) | ≤ a M -1 M +1 .

Example 2 . Let Z i.i.d. ∼ γ 2 -sub-Gaussian. 5 Then, we have that E [ Z M : M ] = -E [ Z 1: M ] and E [ Z M : M ] ≤ √ 2 γ 2 ln M [9] which as n →∞ implies that | P e -ψ soft ( ˜ D A ) | ≤ √ 2 γ 2 ln M.

Our analysis above shows that ψ soft ( ˜ D A ) is (in general) an inconsistent and biased estimator of P e . Because of this, we next propose a new BER estimator, which (different from ψ soft ) leverages the features { x i } n i =1 to denoise ˜ y i . We refer to this estimator as ψ DN . Our main intuition behind proposing ψ DN stems from the fact that data samples having the same feature x should be labeled with the same (or at least similar) label. This can be attained by noting that, even if labels are noisy, it is possible to minimize the effect of a zero-mean noise by averaging the noisy labels associated with the same feature. We next formally define our denoising BER estimator ψ DN : R M → [0 , 1] .

Definition 5 (Denoise estimator) . For a noisy dataset ˜ D A = { ( x i , ˜ y i ) } n i =1 , let the denoised label s ( x i ) for all i ∈ [ n ] , be defined as

✶ and let idx ( x i ) = arg max j ∈ [ M ] { ( s ( x i )) j } . Then, the denoise BER estimator is defined as

The next theorem (proof in Appendix B.5) provides some important properties of ψ DN ( ˜ D A ) .

Theorem 3. Let ˜ D A = { ( x i , ˜ y i ) } n i =1 be a dataset that consists of noisy soft labels ˜ y i = y i + z i where z i i.i.d. ∼ P Z is the zero mean noise. Then, ψ DN ( ˜ D A ) is a consistent estimator of P e . Moreover, if the noise has bounded support, that is Pr( Z ∈ [ a, b ] M ) = 1 , then

(Asymptotical unbiasedness): ψ DN ( ˜ D A ) is an asymptotically unbiased estimator of P e ;

(Denoising Consistency): For any δ ∈ (0 , 1) , it holds that | s ( x ) -y | < √ ( 1 -1 M -a + b ) 2 2 n x ln 2 δ with probability at least 1 -δ , where n x = ∑ n j =1 ✶ { x j = x } and s ( x ) is the denoised label defined in (11) .

The results in Theorem 3 broadens the applicability of an effective BER estimator to a larger class of datasets, e.g., to scenarios where the dataset includes multiple data samples representing the same feature. For example, ψ DN works well on a noisy dataset where each data sample has multiple labels. Moreover, as we will see in Section 4.2, ψ DN is also useful for one-hot labels. We also expect that ψ DN will work well with privatized datasets where noise was added to enhance privacy [26].

Remark 1 . Our estimator ψ DN only requires the dataset ˜ D A . This is a more practical approach than the one in [46] which instead also requires the true one-hot label. The estimator in [46], which is referred to as a genie estimator ψ genie , can be obtained by replacing s ( x ) in (11) with the true one-hot label. In Appendix B.6, we provide properties of ψ genie ( ˜ D A ) that generalize the results in [46] for any M ≥ 2 . As expected, ψ genie is a consistent and unbiased estimator of P e .

4.2 One-hot labels

A dataset consisting of one-hot labels, denoted as D = { ( x i , y i ) } n i =1 , contains little information about the class posterior probability. We assume that the one-hot label is constructed from the corresponding soft label as follows,

where: (i) Y is the one-hot random vector; (ii) Y is the soft label random vector in (3); and (iii) e i ∈ R M is the standard basis vector with a one in the i -th position and zero in all the other entries.

We can think of the one-hot label construction in (13) as a noisy label. Specifically, the noise Z ∈ { e i -y } i M =1 with the probability mass function p Z | Y ( z | y ) = ∑ M j =1 y j ✶ { z = e j -y } satisfies Y = Y + Z . It is not difficult to see that E [ Z ] = E [ E [ Z | Y ]] = 0 and hence, our denoise estimator ψ DN ( ˜ D A ) in (12), or its robust version MoB K ( ψ DN , ˜ D A ) , can estimate the BER also for the case of one-hot labels. However, we also note that with the above construction, the labels might be too noisy to estimate the BER in practice. This suggests that a more refined denoising method than the one in Definition 5 for one-hot labels might be needed for a more effective BER estimation.

Motivated by the above discussion, we next propose a denoising method for one-hot labels, which leverages neighbor samples to mitigate the noise. 6 Our choice of such a denoising method mainly stems from the fact that one would expect that neighboring samples should have similar posterior probabilities. In particular, for each x i , i ∈ [ n ] , we consider all of its neighbors (features that are at most at a distance r from x i ) and we average the corresponding noisy labels in a spirit similar to (11). We next formally define our denoising BER estimator, which we refer to as ψ C .

Definition 6 (Cluster denoise estimator) . Given d : X 2 → R + and r ∈ R + , define Cluster i := { ( x , y ) ∈ ˜ D A : d ( x i , x ) ≤ r } . Then, the denoised label for ˜ y i , for all i ∈ [ n ] , is defined as

and the cluster-based BER estimator is defined as

Remark 2 . Preprocessing the dataset that aggregates data samples of similar features can also be helpful in improving the BER estimate for noisy labels, and it can be paired with the cluster denoise estimator in Definition 6. A naive approach would consist of reducing the feature dimensionality and increasing the number of data samples inside clusters (14), which helps mitigate the effect of the noise. Some notable examples of data preprocessing are the classical principal component analysis (PCA) [77] and the representation learning [5] (e.g., variational auto encoder [52]).

Remark 3 . Another viewpoint to the denoising method in (14) is the connection to the non-parametric estimation of the conditional expectation E [ Y | X = x ] , where Y is the true soft label corresponding to X = x . In particular, estimating E [ Y | X = x ] is equivalent to denoising the noisy label associated with x as in (14). Under such a viewpoint, the BER estimator in Definition 6 resembles the Nadaraya-Watson estimator with Parzen window kernel [63, 89].

5 Experiments

Weempirically validate our results using various datasets, namely: 1) a synthetic dataset with different noises, including one-hot labels; 2) two benchmark datasets CIFAR10 H [4] and Fashion-MNISTH [46]; and 3) MovieLens [38], a real-world dataset for movie recommendations. We compare our estimators with two SOTA BER bounds [72], namely the generalized Henze-Penrose (GHP) BER bounds [75] and the k -Nearest Neighbor (NN) BER bounds [16]. Details on the k -NN bounds and GHP bounds, and additional results can be found in Appendix C.

0.6

2

0.5

0.4

0.3

class

class

0.2

class2

class

2

class3

class

3

0.1

class 4

class 4

0

2

0

2

C1

C1

(a) Soft labels.

(b) Noisy soft labels.

BER

k-NN upper bound

GHP upper bound

MoBK(c)

GHP lower bound

K-NN lower bound

0.6

0.6

Q 0.4

Q 0.4

0.2

0.2

**+

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.10

0.10

2

0.05

0.05

000

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

μ

μ

Synthetic dataset with soft labels and noisy labels. We consider a 4 -class classification problem with equiprobable classes C ∈ C µ := { ( µ, µ ) , ( -µ, µ ) , ( -µ, -µ ) , ( µ, -µ ) } , where µ > 0 is a parameter that controls the classification hardness. We generate the feature X ∈ R 2 according to a 2 -dimensional Gaussian distribution with mean c (i.e., a realization of C ∈ C µ ) and covariance matrix I 2 . The corresponding soft labels y i 's are then obtained by the Bayes' theorem. In particular, since f X | C ∼ N ( C, I 2 ) and P C ( c ) = 1 / 4 if c ∈ C µ (and zero otherwise), according to (3) we have that ( y i ) k = f X | C ( x i | c ) P C ( c ) ∑ α ∈C µ f X | C ( x i | α ) P C ( α ) . 7 Figure 1 illustrates n = 1 , 000 features (left figure), and the corresponding soft labels (right figure) for the synthetic dataset generated as explained above; in particular, for each feature x i , we only reported the label ( y i ) j ⋆ where j ⋆ = arg max j ∈ [4] ( y i ) j which is required for evaluating ψ soft in (4). As expected, a point near the decision boundary (lines for x 1 = 0 or x 2 = 0 ) has a smaller maximum value in its soft label.

The results in Figure 2 empirically demonstrate the effectiveness of our estimators, which consistently outperform the others. 8 Moreover, our estimators estimate the exact value of the BER, while the others derive upper and lower bounds on the BER, which might not be tight.

(a)

P

e

.

(b)

|

ψ

-

P

e

|

and variance.

0.3

BER

0.2

t'soft

0.1

0.0

0.3

0.2

P°

0.1

喜

BER

MoB（c)

0.0

40

80

120160

200

240280320360

400

008

1600240032004000

Number of samples

0.3

soft

MoBK（c)

0.2

0.1

0.0

0.030

0.025

0.020

0.015

MoBk（c)

F010’0

200°0

0.000

0

500

1000

1500

2000

3000

3500

4000

Number of samples

We further conducted experiments to verify the effectiveness of our denoising and robustifying methods on data samples with label noise and outliers. Figure 3 shows the comparison of the denoising estimator MoB K ( ψ C ) with the base estimator ψ soft . We observe that ψ soft suffers from the label noise, which leads to a bias in the estimation. However, our denoising estimator MoB K ( ψ C ) suitably denoises the noisy labels and effectively estimates the BER. As shown in Figure 3b, in fact, the absolute error between the estimate and the BER as well as the variance decrease as n grows.

Synthetic dataset with one-hot labels. We analyze the performance of MoB K ( ψ C ) in the same Gaussian setting described above, but with one-hot labels. Each soft label of the samples is mapped to a one-hot label according to the categorical distribution with probabilities equal to the soft labels (see (13)). Figure 4 shows that our denoising method is capable of suitably reducing the noise in one-hot labels; in fact, it correctly estimates the BER after around 10 4 samples when MoB K ( ψ C ) converges to the BER. From Figure 4, we also observe that ψ soft performs poorly; this suggests that denoising methods are essential for the BER estimation task, hence worth further investigation.

CIFAR10 H [4] and Fashion-MNIST-H [46].

The CIFAR10 H dataset is a variation of CIFAR10 [53], constructed by labeling 10 4 images in the test dataset of CIFAR10 by multiple labelers. The Fashion-MNIST-H dataset is populated by 10 4 images in the test dataset of Fashion-MNIST [93] in a similar manner. These two datasets have 10 classes, but we further categorized these into M ∈ { 2 , 3 } classes. In particular, for CIFAR10 H, we assigned C 1 = { airplane , automobile , ship , truck } , C 2 = { bird , cat , deer , dog , frog , horse } when M = 2 , and C 1 = { airplane , automobile , ship , truck } , C 2 = { cat , dog , frog } , C 3 = { bird , deer , horse } when M = 3 . Similarly, for Fashion-MNIST-H, we assigned C 1 = { t -shirt / top , pullover , dress , coat , shirt } , C 2 = { trouser , sandal , sneaker , bag , ankleboot } when M = 2 , and C 1 = { t -shirt / top , pullover , coat , shirt } , C 2 = { trouser , dress , bag } , and C 3 = { sandal , sneaker , ankleboot } when M = 3 .

From Table 1, we observe that the estimates using MoB K ( ψ DN ) with K = ⌊ √ n ⌋ are slightly lower than ψ DN , which is due to the robustness of MoB K ( ψ DN ) to outliers. 9 We also highlight that the BER estimates can be leveraged to determine which task is more difficult. Intuitively, performing

0.30

0.25

0.20

BER

0.15

MoBk(c)

soft

0.10

0.05

0.00

40

80

120 160 200 240 280 320 360 400 800 1600 2400 3200 4000 6000 80001000012000

Numberofsamples

the classification task over CIFAR10 H is much easier than performing it over Fashion-MNIST-H, and this is supported by the results in Table 1, i.e., the BERs associated with CIFAR10 H are smaller than those over Fashion-MNIST-H. It is worth noting that test dataset overfitting can happen when benchmark datasets are considered [3, 46, 51, 59, 91]. This can also be observed by the results in Table 1, where our BER estimates of the 10 -class classifications are larger than the SOTA error rates of 0 . 005 on CIFAR10 by [24] and of 0 . 0309 on Fashion-MNIST by [82]. We also suspect that, since the labels are assigned by humans, who might not be experts, the datasets might still have a considerable amount of label noise, which would lead to incorrect BER estimates. However, no estimators (including ours) can estimate the BER from a dataset that contains very noisy labels.

MovieLens [38]. This dataset consists of 25 million ratings to 62 , 000 movies by 162 , 000 users. Each rating ranges from 0 . 5 to 5 with step size 0 . 5 . We first considered a movie classification task: a user either likes a movie or not. This is a complex binary classification task with the input feature being a movie (in general, a two-hour long video with audio) and the class being either 0 (dislike the movie) or 1 (like the movie). To ensure that a label belongs to [0 , 1] , we applied min-max normalization to each rating. Moreover, in order to have enough ratings for each movie when we applied the denoising method, we filtered out some data if the number of ratings was smaller than 100 . Then, ψ DN and MoB K ( ψ DN ) estimated the BER of such movie classification task, and they yielded a BER of around 0 . 3 . We then categorized the ratings into M = 3 classes, i.e., we assigned C 1 = { 0 . 5 , 1 . 0 , 1 . 5 } , C 2 = { 2 . 0 , . . . , 3 . 5 } and C 3 = { 4 . 0 , 4 . 5 , 5 . 0 } . On this task, the BER estimate is around 0 . 4 . These BERs are quite large, implying that the movie classification is a hard task to perform. This may be justified by the fact that ratings of movies are subjective, and movie recommendations are indeed challenging without users' information (e.g., content-based [47] or item-based [74] recommendation systems use users' information).

6 Conclusion

In this paper, we investigated the challenge of estimating the BER for multi-class classifications. We proposed a BER estimator, ψ soft , which we proved to be unbiased, consistent, and asymptotically normal when applied to soft-labeled datasets. By leveraging the median-of-mean method, we also proposed a methodology to make any BER estimate robust. To ensure the applicability of the BER estimator in practical scenarios, we analyzed the challenges posed by noisy soft labels, including those with additive noise and one-hot labels. For such noisy labeled datasets, we developed denoising techniques that effectively mitigate the label noise by using the corresponding features. We showed that these denoising BER estimators are unbiased and consistent under mild noise assumptions. Our experimental results, drawn from synthetic and real-world datasets, validated our theoretical results.

Although in this work we assumed that the set of features is finite, several of our results (e.g., Theorem 1 and Theorem 2) can be easily shown to hold also for the infinite feature space. A research direction worth further investigation would consist of proving that the cluster denoise estimator in Definition 6 has similar appealing properties (see Theorem 3) as the denoise estimator in Definition 5 (which assumes that the set of the features is discrete). Always along these lines, a second interesting future work would be to analyze the rate of convergence of our BER estimator paired with the denoising method as a function of the characteristics (e.g., cardinality, distribution) of the feature space. Such an analysis would indeed provide insights into the difficulty of a classification task. Another relevant avenue for future research lies in assessing the optimal performance of multilabel learning, where a single data point can be associated with multiple classes. While adapting our framework to predict the minimum Hamming loss for multi-label learning might be relatively straightforward, estimating other metrics (e.g., the F1 score and the exact match ratio) might be a difficult task, potentially necessitating new methodologies.

References

A. Antos, L. Devroye, and L. Gyorfi. Lower bounds for Bayes error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 21(7):643-645, 1999.

H. Avi-Itzhak and T. Diep. Arbitrarily tight upper and lower bounds on the Bayesian probability of error. IEEE Transactions on Pattern Analysis and Machine Intelligence , 18(1):89-91, 1996.

B. Barz and J. Denzler. Do we train on test data? purging CIFAR of near-duplicates. Journal of Imaging , 6(6):41, 2020.

R. M. Battleday, J. C. Peterson, and T. L. Griffiths. Capturing human categorization of natural images by combining deep networks and cognitive models. Nature Communications , 11(1):5418, 2020. doi: 10.1038/s41467-020-18946-z. URL https://doi.org/10.1038/s41467-020-18946-z .

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):1798-1828, 2013. doi: 10.1109/ TPAMI.2013.50.

V. Berisha, A. Wisler, A. O. Hero, and A. Spanias. Empirically estimable classification bounds based on a nonparametric divergence measure. IEEE Transactions on Signal Processing , 64(3):580-591, 2016. doi: 10.1109/TSP.2015.2477805.

R. Bhatia and C. Davis. A better bound on the variance. The American Mathematical Monthly , 107(4): 353-357, 2000.

D. E. Boekee and J. C. van der Lubbe. Some aspects of error bounds in feature selection. Pattern Recognition , 11(5-6):353-360, 1979.

S. Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, 2013.

L. Buturovi´ c. Improving k-nearest neighbor density and error estimates. Pattern Recognition , 26(4): 611-616, 1993.

C. Chen. Theoretical comparison of a class of feature selection criteria in pattern recognition. IEEE Transactions on Computers , C-20(9):1054-1056, 1971. doi: 10.1109/T-C.1971.223402.

C. H. Chen. On information and distance measures, error bounds, and feature selection. Information Sciences , 10(2):159-173, 1976.

Q. Chen, F. Cao, Y. Xing, and J. Liang. Evaluating classification model against Bayes error rate. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-16, 2023. doi: 10.1109/TPAMI. 2023.3240194.

H. Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics , pages 493-507, 1952.

T. M. Cover. Elements of Information Theory . John Wiley & Sons, 1999.

T. M. Cover and P. Hart. Nearest neighbor pattern classification. IEEE Transactions on Information Theory , 13(1):21-27, 1967. doi: 10.1109/TIT.1967.1053964.

T. Dao, G. M. Kamath, V. Syrgkanis, and L. Mackey. Knowledge distillation as semiparametric inference. In International Conference on Learning Representations , 2021. URL https://openreview.net/ forum?id=m4UCf24r0Y .

H. A. David and H. N. Nagaraja. Order Statistics . Wiley Online Library, 2004.

P. A. Devijver. On a new class of bounds on Bayes risk in multihypothesis pattern recognition. IEEE Transactions on Computers , C-23(1):70-80, 1974. doi: 10.1109/T-C.1974.223779.

P. A. Devijver and J. Kittler. Pattern Recognition . Prentice Hall, Old Tappan, NJ, July 1982.

L. Devroye. On the asymptotic probability of error in nonparametric discrimination. The Annals of Statistics , 9(6):1320-1327, 1981.

L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition , volume 31. Springer Science & Business Media, 2013.

J. L. Doob. Regularity properties of certain families of chance variables. Transactions of the American Mathematical Society , 47(3):455-486, 1940.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=YicbFdNTTy .

R. O. Duda, P. E. Hart, et al. Pattern Classification . John Wiley & Sons, 2006.

C. Dwork. Differential privacy. In Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 33 , pages 1-12. Springer, 2006.

B. Frenay and M. Verleysen. Classification in the presence of label noise: A survey. IEEE Transactions on Neural Networks and Learning Systems , 25(5):845-869, 2014. doi: 10.1109/TNNLS.2013.2292894.

J. H. Friedman and L. C. Rafsky. Multivariate generalizations of the wald-wolfowitz and smirnov two-sample tests. The Annals of Statistics , pages 697-717, 1979.

K. Fukunaga. Introduction to Statistical Pattern Recognition (2nd Ed.) . Academic Press Professional, Inc., USA, 1990. ISBN 0122698517.

K. Fukunaga and L. Hostetler. k-nearest-neighbor Bayes-risk estimation. IEEE Transactions on Information Theory , 21(3):285-293, 1975. doi: 10.1109/TIT.1975.1055373.

K. Fukunaga and D. M. Hummels. Bayes error estimation using Parzen and k-NN procedures. IEEE Transactions on Pattern Analysis and Machine Intelligence , (5):634-643, 1987.

K. Fukunaga and D. M. Hummels. Bias of nearest neighbor error estimates. IEEE Transactions on Pattern Analysis and Machine Intelligence , (1):103-112, 1987.

K. Fukunaga and D. Kessell. Nonparametric Bayes error estimation using unclassified samples. IEEE Transactions on Information Theory , 19(4):434-440, 1973. doi: 10.1109/TIT.1973.1055049.

F. Garber and A. Djouadi. Bounds on the Bayes classification error based on pairwise risk functions. IEEE Transactions on Pattern Analysis and Machine Intelligence , 10(2):281-288, 1988. doi: 10.1109/34.3891.

V. Grossmann, L. Schmarje, and R. Koch. Beyond hard labels: investigating data label distributions. arXiv preprint arXiv:2207.06224 , 2022.

L.-Z. Guo, Z.-Y. Zhang, Y. Jiang, Y.-F. Li, and Z.-H. Zhou. Safe deep semi-supervised learning for unseen-class unlabeled data. In International Conference on Machine Learning , pages 3897-3906. PMLR, 2020.

B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems , 31, 2018.

F. M. Harper and J. A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. , 5(4), dec 2015. ISSN 2160-6455. doi: 10.1145/2827872. URL https://doi.org/10.1145/ 2827872 .

W. Hashlamoun, P. Varshney, and V. Samarasooriya. A tight upper bound on the Bayesian probability of error. IEEE Transactions on Pattern Analysis and Machine Intelligence , 16(2):220-224, 1994. doi: 10.1109/34.273728.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

M. Hellman and J. Raviv. Probability of error, equivocation, and the Chernoff bound. IEEE Transactions on Information Theory , 16(4):368-372, 1970. doi: 10.1109/TIT.1970.1054466.

M. Huai, C. Miao, Y. Li, Q. Suo, L. Su, and A. Zhang. Metric learning from probabilistic labels. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1541-1550, 2018.

G. Huang, Z. Liu, L. V. D. Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2261-2269, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society. doi: 10.1109/CVPR.2017.243. URL https://doi.ieeecomputersociety.org/10.1109/CVPR.2017.243 .

P. J. Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics , 35(1):73 101, 1964. doi: 10.1214/aoms/1177703732. URL https://doi.org/10.1214/aoms/1177703732 .

P. J. Huber. Robust statistics. In International Encyclopedia of Statistical Science , pages 1248-1251. Springer, 2011.

T. Ishida, I. Yamane, N. Charoenphakdee, G. Niu, and M. Sugiyama. Is the performance of my deep network too good to be true? a direct approach to estimating the Bayes error in binary classification. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=FZdJQgy05rz .

U. Javed, K. Shaukat, I. A. Hameed, F. Iqbal, T. M. Alam, and S. Luo. A review of content-based and context-based recommendation systems. International Journal of Emerging Technologies in Learning (iJET) , 16(3):274-306, 2021.

E. Jeon. ViT-PyTorch, 2020. https://github.com/jeonsworld/ViT-pytorch ,.

T. Kailath. The divergence and Bhattacharyya distance measures in signal selection. IEEE Transactions on Communication Technology , 15(1):52-60, 1967. doi: 10.1109/TCOM.1967.1089532.

S. M. Kay. Fundamentals of Statistical Signal Processing, vol. 2: Detection Theory . Prentice Hall PTR, 1998.

D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4110-4124, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.324. URL https://aclanthology.org/2021.naacl-main.324 .

D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 , 2013.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

P. Kumar. Moments inequalities of a random variable defined over a finite interval. J. Inequal. Pure Appl. Math , 3(3):1-24, 2002.

J. Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory , 37(1):145-151, 1991. doi: 10.1109/18.61115.

T. Lissack and K.-S. Fu. Error estimation in pattern recognition via L α -distance between posterior density functions. IEEE Transactions on Information Theory , 22(1):34-45, 1976. doi: 10.1109/TIT.1976. 1055512.

S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems , 33:20331-20342, 2020.

G. Loizou and S. J. Maybank. The nearest neighbor and the Bayes error rates. IEEE Transactions on Pattern Analysis and Machine Intelligence , PAMI-9(2):254-262, 1987. doi: 10.1109/TPAMI.1987. 4767899.

H. Mania, J. Miller, L. Schmidt, M. Hardt, and B. Recht. Model similarity mitigates test set overuse. Advances in Neural Information Processing Systems , 32, 2019.

R. A. Maronna, R. D. Martin, V. J. Yohai, and M. Salibián-Barrera. Robust Statistics: Theory and Methods (with R) . John Wiley & Sons, 2019.

A. K. Menon, A. S. Rawat, S. Reddi, S. Kim, and S. Kumar. A statistical perspective on distillation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7632-7642. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/menon21a.html .

S. Minsker. Distributed statistical estimation and rates of convergence in normal approximation. Electronic Journal of Statistics , 13(2):5213 - 5252, 2019. doi: 10.1214/19-EJS1647. URL https://doi.org/10. 1214/19-EJS1647 .

E. A. Nadaraya. On estimating regression. Theory of Probability & Its Applications , 9(1):141-142, 1964.

N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari. Learning with noisy labels. Advances in Neural Information Processing Systems , 26, 2013.

A. S. Nemirovskij and D. B. Yudin. Problem Complexity and Method Efficiency in Optimization . WileyInterscience, 1983.

J. Neyman and E. S. Pearson. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 231(694-706):289-337, 1933.

M. Noshad, L. Xu, and A. Hero. Learning to benchmark: Determining best achievable misclassification error from training data. arXiv preprint arXiv:1909.07192 , 2019.

G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1944-1952, 2017.

J. C. Peterson, R. M. Battleday, T. L. Griffiths, and O. Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9617-9626, 2019.

H. Phan. PyTorch_CIFAR10, 2021. https://github.com/huyvnphan/PyTorch_CIFAR10 ,.

Z. Ren, R. Yeh, and A. Schwing. Not all unlabeled data are equal: Learning to weight data in semisupervised learning. Advances in Neural Information Processing Systems , 33:21786-21797, 2020.

C. Renggli, L. Rimanic, N. Hollenstein, and C. Zhang. Evaluating Bayes error estimators on real-world datasets with FeeBee. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021. URL https://openreview.net/forum?id=uKv5inrWeld .

T. Sakai, M. C. Plessis, G. Niu, and M. Sugiyama. Semi-supervised classification based on classification from positive and unlabeled data. In International Conference on Mmachine Learning , pages 2998-3006. PMLR, 2017.

B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conference on World Wide Web , WWW'01, page 285-295, New York, NY, USA, 2001. Association for Computing Machinery. ISBN 1581133480. doi: 10.1145/371920.372071. URL https://doi.org/10.1145/371920.372071 .

S. Y. Sekeh, B. Oselio, and A. O. Hero. Learning to bound the multi-class Bayes error. IEEE Transactions on Signal Processing , 68:3793-3807, 2020. doi: 10.1109/TSP.2020.2994807.

S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014.

J. Shlens. A tutorial on principal component analysis. arXiv preprint arXiv:1404.1100 , 2014.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. pages 1-14. Computational and Biological Learning Society, 2015.

A. Singh, R. Nowak, and J. Zhu. Unlabeled data: Now it helps, now it doesn't. Advances in Neural Information Processing Systems , 21, 2008.

H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , pages 1-19, 2022. doi: 10.1109/TNNLS.2022.3152527.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2818-2826, 2016.

M. S. Tanveer, M. U. Karim Khan, and C.-M. Kyung. Fine-tuning darts for image classification. In 2020 25th International Conference on Pattern Recognition , pages 4789-4796, 2021. doi: 10.1109/ICPR48806. 2021.9412221.

R. Theisen, H. Wang, L. R. Varshney, C. Xiong, and R. Socher. Evaluating state-of-the-art classification models against Bayes optimality. Advances in Neural Information Processing Systems , 34:9367-9377, 2021.

R. L. Thorndike. Who belongs in the family? Psychometrika , 18(4):267-276, 1953. doi: 10.1007/ BF02289263. URL https://doi.org/10.1007/BF02289263 .

G. T. P. Toussaint. Feature Evaluation Criteria and Contextual Decoding Algorithms in Statistical Pattern Recognition . PhD thesis, University of British Columbia, 1972.

K. Tumer and J. Ghosh. Bayes error rate estimation using classifier ensembles. International Journal of Smart Engineering System Design , 5(2):95-109, 2003. doi: 10.1080/10255810305042. URL https: //doi.org/10.1080/10255810305042 .

J. Unnikrishnan, S. Haghighatshoar, and M. Vetterli. Unlabeled sensing with random linear measurements. IEEE Transactions on Information Theory , 64(5):3237-3253, 2018.

X. Wang, Y. Hua, E. Kodirov, D. A. Clifton, and N. M. Robertson. ProSelfLC: Progressive self label correction for training robust deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 752-761, 2021.

G. S. Watson. Smooth regression analysis. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 359-372, 1964.

Q. Wei, L. Feng, H. Sun, R. Wang, C. Guo, and Y. Yin. Fine-grained classification with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1165111660, June 2023.

R. Werpachowski, A. György, and C. Szepesvári. Detecting overfitting via adversarial examples. Advances in Neural Information Processing Systems , 32, 2019.

X. Xia, T. Liu, B. Han, C. Gong, N. Wang, Z. Ge, and Y. Chang. Robust early-learning: Hindering the memorization of noisy labels. In International Conference on Learning Representations , 2021.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Y. Yao, T. Liu, M. Gong, B. Han, G. Niu, and K. Zhang. Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems , 34:4409-4420, 2021.

Y. Yao, L. Peng, and M. Tsakiris. Unlabeled principal component analysis. Advances in Neural Information Processing Systems , 34:30452-30464, 2021.

K. Yi and J. Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7017-7025, 2019.

H. Yuan, N. Xu, Y. Shi, X. Geng, and Y. Rui. Learning from biased soft labels. arXiv preprint arXiv:2302.08155 , 2023.

C.-B. Zhang, P.-T. Jiang, Q. Hou, Y. Wei, Q. Han, Z. Li, and M.-M. Cheng. Delving deep into label smoothing. IEEE Transactions on Image Processing , 30:5984-5996, 2021.

H. Zhang, P. Koniusz, S. Jian, H. Li, and P. H. Torr. Rethinking class relations: Absolute-relative supervised and unsupervised few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9432-9441, 2021.

L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3713-3722, 2019.

H. Zhou, L. Song, J. Chen, Y. Zhou, G. Wang, J. Yuan, and Q. Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=gIHd-5X324 .

Appendices

A Related work

It is well known that the BER of a binary classification problem can be achieved by a likelihood ratio test [66] and the BER of a multiclass classification problem can be attained by a Maximum a Posteriori (MAP) classifier [50]. The problem of estimating the BER using a dataset has been extensively investigated from the middle of the 20 th century [16] to the present [13], leveraging different approaches, such as the k -nearest neighbors (NN) method, probabilistic divergence, and bounds on the BER. In the early stage, the main approach was to derive upper and lower bounds on the BER, while recent work mainly focuses on directly estimating the BER rather than deriving bounds on it.

BER bounds. Several bounds on the BER in binary classification tasks have been proposed using probability divergence and distance metrics. Notable approaches use the Mahalanobis distance [20, 86], the Bhattacharyya distance between two class conditional distributions [20, 49, 86], the Chernoff coefficient (a generalization of the Bhattacharyya distance) [14, 25, 29], the Henze-Penrose divergence [6], the Jensen-Shannon divergence [55], bounds for the min { a, b } function [2, 39] (a generalization of the equivocation bound [41] and of the Bayesian bound [12]), the M 0 -distance [85], the Fano's inequality [15], and the L 2 norm of posterior probabilities [19]. Some of these bounds are tight, but they still require estimating the data distribution (e.g., class conditional distributions or posterior probabilities), which may not be practical, particularly for modern data (e.g., images and videos). 10 Bounds on the BER were also derived using non-parametric approaches [16, 21, 30, 33], such as the k -NN classifier. A remarkable advantage of a k -NN classifier for this problem is its distributionfree property to bound the BER (i.e., it works under no assumption on the data distribution). A convergence rate analysis for universal BER estimators was developed in [1, 22].

Direct estimation of the BER. As an alternative approach to using bounds on the BER, some recent research directly estimates the BER [13, 46, 67, 83]. In particular, the authors in [67] expressed the BER as a function of the f -divergence and proposed a density ratio estimator to evaluate such an f -divergence that yields the BER. Under various label assumptions, a direct BER estimator was proposed in [46]; this estimator is unbiased and consistent for soft labels, and it has desirable properties. Another direct estimator for the BER was proposed in [13]. In particular, this estimator uses the relationship between the BER and the miss-classified samples.

Extension to multi-class classification. The extension of the BER in binary classification to multi-class classification has been extensively investigated both in terms of generalizing existing bounds and deriving new approaches. For example, the Chernoff bound [34], the Jensen-Shannon divergence [55], the Fano's inequality [15], the general mean distance [8], the L 2 norm of posterior probabilities [19], and the generalized Henze-Penrose divergence [75] can be used to derive bounds on the BER in multi-class classification problems. Non-parametric methods [31, 58], such as the k -NN classifier, can also bound the BER in multi-class classification settings. Bounds based on the error probability of a k -NN classifier in the asymptotic regime were studied in [16], and some bounds in the non-asymptotic regime can be found in [10, 32].

B Proofs and auxiliary results

B.1 Proof of Theorem 1

Proof. By taking the expectation of ψ soft in (4) with respect to P X , Y , we obtain

where the second equality follows from (3), and the last equality is due to (2). This shows that ψ soft in (4) is an unbiased estimator of P e .

Next, let U k i.i.d. ∼ Y M : M for all k ∈ [ n ] , and note that U k ∈ [ 1 M , 1 ] := U . Define a function f : U n → [ 0 , 1 -1 M ] as f ( U 1 , . . . , U n ) := 1 n ∑ n k =1 (1 -U k ) , which is equivalent to ψ soft in (4). Then, for any u k ∈ U and for all k ∈ [ n ] , we have that

By using McDiarmid's inequality [23], together with (16) and (17), we arrive at

By taking n →∞ , we obtain

which shows that ψ soft in (4) is a consistent estimator of P e .

By setting ε = √ ( 1 -1 M ) 2 2 n ln 2 δ in (18) where δ ∈ (0 , 1) , we obtain

This shows the convergence rate.

The estimator ψ soft in (4) can be viewed as the sample mean of 1 -max j ∈ [ M ] y j in D . Since max j ∈ [ M ] y j ∼ Y M : M , from the central limit theorem, as n →∞ , we obtain that

This concludes the proof of Theorem 1.

B.2 Proof of Proposition 1

Proof. We have that Y M : M ∼ max j ∈ [ M ] Y j . Then, by letting U k i.i.d. ∼ Y M : M for all k ∈ [ n ] , we obtain

Moreover, by using the Bhatia-Davis inequality [7], we obtain Var ( Y M : M ) = Var (1 -Y M : M ) ≤ ( 1 -1 M -P e ) P e since E [ ψ soft ] = P e , which gives

Finally, we can maximize the term ( 1 -1 M -P e ) P e over 0 ≤ P e ≤ 1 -1 M . The maximum is indeed ( 1 -1 M ) 2 4 attained by P e = 1 2 ( 1 -1 M ) . Hence,

which concludes the proof of Proposition 1.

B.3 Proof of Theorem 2

Proof. We start by noting that, by viewing ψ soft in (4) as a sample mean of (1 -Y M : M ) , the estimator MoB K ( ψ soft , D ) is a median-of-means estimator. It has been shown in [62] that, provided that E [ | 1 -Y M : M -P e | 3 ] < ∞ , a median-of-means estimator satisfies

with probability at least 1 -4 e -2 s , for all s ≲ K . From [54, eq. (3.8)], since (1 -Y M : M ) ∈ [ 0 , 1 -1 M ] we have that

where the last inequality follows by maximizing P e ( 1 -1 M -P e ) ( 1 -1 M -2 P e ) with respect to 0 ≤ P e ≤ 1 -1 M . Substituting (26) into (25) leads to

Under an appropriate choice of n and K (e.g., K = o ( n ) ), the upper bound in (27) converges to 0 , which proves the consistency of MoB K ( ψ soft , D ) .

For the breakdown point of MoB K ( ψ soft , D ) , we start by observing that, if we have ⌊ K +1 2 ⌋ outliers, then at most ⌊ K +1 2 ⌋ base estimators ψ soft ( D k ) 's in (6) can be affected by these outliers, i.e., they can be bad. In this case, MoB K ( ψ soft , D ) gives a bad estimate since more than half of the base estimators ψ soft ( D k ) 's are bad. However, if the number of outliers is smaller than ⌊ K +1 2 ⌋ , e.g., ⌊ K +1 2 ⌋ -1 , then the median of ψ soft ( D k ) , k ∈ [ K ] is not a bad estimate since the number of outliers is smaller than half of the base estimators. Thus, from Definition 3 with τ = ⌊ K +1 2 ⌋ and κ + τ = n , we obtain B ( MoB K ( ψ soft , D )) = ⌊ K +1 2 ⌋ 1 n .

The proof of the asymptotic normality of MoB K ( ψ soft , D ) follows directly from [62, Theorem 4]. In particular, if K →∞ and K = o ( √ n ) as n →∞ , it holds that

This concludes the proof of Theorem 2.

B.4 Random permutation noise

One of the most common type of label noises is a noise that randomly permutes/shuffles the data labels [27], i.e., we have ( x , ˜ y ) ∈ ˜ D P where ˜ D P denotes the dataset with randomly permuted labels. This noise model can be categorized into two types [27], namely instance-independent and instancedependent . The former perturbs the label y with a random permutation according to some probability distribution (the extreme noise case, namely unlabeled data or data without correspondence, where the label is permuted with probability one using a random permutation matrix, has also been studied extensively [36, 71, 73, 79, 87, 95]), which is independent of the input features. For example, a corruption by an instance-independent noise to y = [0 . 1 , 0 . 7 , 0 . 2] ⊤ may result in ˜ y = [0 . 1 , 0 . 2 , 0 . 7] ⊤ . The other type of noise, the instance-dependent noise, flips/permutes the labels depending on the instance. For example, the label 'car' is more likely to be flipped to 'truck' rather than 'cat'.

We note that ψ soft in (4) only depends on the maximum value of y , which does not change after that a random permutation is applied on y . This property ensures that ψ soft is still an effective estimator of P e even when the labels are randomly permuted, as stated in the next theorem.

Theorem 4. Let ˜ D P = { ( x i , ˜ y i ) } n i =1 be a dataset that consists of noisy soft labels ˜ y i = P i y i where P i , i ∈ [ M ] is any random permutation matrix of size M × M . Then, ψ soft ( ˜ D P ) satisfies all of the properties in Theorem 1 and Proposition 1.

Proof. Since max j ∈ [ M ] y j is permutation-invariant with respect to y , it follows that

where the first equality follows since ˜ y i = P i y i , and the second equality is due to the permutationinvariance property of the max function. The results in Theorem 1 and Proposition 1 then prove Theorem 4.

B.5 Proof of Theorem 3

Proof. Without loss of generality, we let X = { u 1 , u 2 , . . . , u K } , where K = |X| is the cardinality of X , and u k is the unique feature vector (or tensor). We denote by v k ∈ Y the soft label corresponding to u k , i.e., v k = [Pr( C = 1 | X = u k ) , . . . , Pr( C = M | X = u k )] ⊤ . With these definitions, we observe that due to the law of large numbers, we have

where (a) follows from the law of total expectation and (b) is due to the fact that E [ ˜ Y | Y ] = Y since the noise is zero mean. Now, since for any i ∈ [ n ] there exists a k ∈ [ K ] such that x i = u k , we can write that

where the equality in (a) follows from (30). Then, we can write ψ DN ( ˜ D A ) in (12) as

When n →∞ , it follows that

where the labeled equalities follow from: (a) the law of large numbers, the assumption that ( x i , ˜ y i ) ∼ P X , ˜ Y , and using (31); (b) using the law of total expectation; and (c) the fact that E [ ˜ Y | Y ] = Y since the noise is zero mean. This proves the consistency of ψ DN .

The asymptotic unbiasedness directly follows from the consistency together with the assumption of bounded support for the noise. Specifically, since Z j ∈ [ a, b ] for all j ∈ [ M ] , we have that

where the first equality follows by the dominated convergence theorem that can be applied since | ψ DN ( ˜ D A ) | ≤ max {| a | , | 1 -1 M + b |} , where a and b are the minimum and the maximum value in the support of the noise, respectively, and they are both finite. This shows the asymptotic unbiasedness of ψ DN ( ˜ D A ) .

We are left to prove the denoising consistency. The denoised label s ( x i ) in (11) is an unbiased estimator of y i since

where the labeled equalities follow from: (a) letting X j , Y j , and ˜ Y j for all j ∈ [ n ] be independent copies of X , Y , and ˜ Y , respectively; (b) introducing a random variable N ∈ [ n ] that counts the number of X j such that X j = X i and using the law of total expectation; (c) re-indexing ˜ Y k , k ∈ [ N ] for the N pairs ( X j , ˜ Y k ) such that X i = X j ; and (d) the fact that the noise is zero mean.

Note that s ( x i ) is the average of n x i noisy labels ˜ y 's corresponding to x i . By Hoeffding's inequality, we then obtain

Setting ε = √ ( 1 -1 M -a + b ) 2 2 n x i ln 2 δ leads to

This concludes the proof of Theorem 3.

B.6 Properties of ψ genie ( ˜ D A )

The following proposition defines ψ genie and provides some properties of it.

Proposition 2. Let ˜ D A = { ( x i , ˜ y i ) } n i =1 be a dataset that consists of noisy soft labels ˜ y i = y i + z i with z i i.i.d. ∼ P Z such that E [ Z ] = 0 . Let c i = arg max j ∈ [ M ] ( y i ) j . Then, the following estimator

satisfies the following properties:

(Unbiasedness): It is an unbiased estimator of the BER;

(Consistency): It is a consistent estimator of the BER;

(Variance) : Var ( ψ genie ( ˜ D A ) ) = Var ( Y M : M )+ Var ( Z ) n ;

(Asymptotic Normality) : √ n ( ψ genie ( ˜ D A ) -P e ) d →N (0 , Var ( Y M : M )+ Var ( Z )) as n →∞ .

Proof. Define the random variable C = arg max j ∈ [ M ] Y j that indicates the index of Y having the largest value, i.e., Y C = max j ∈ [ M ] Y j . We have that

where the labeled equalities follow from: (a) the law of total expectation; and (b) the fact that E [ ˜ Y | Y ] = Y since E [ Z ] = 0 . This shows the unbiasedness of ψ genie ( ˜ D A ) .

Now, we show the consistency of ψ genie ( ˜ D A ) . Due to the law of large numbers, as n → ∞ , we obtain that

which demonstrates the consistency of ψ genie ( ˜ D A ) .

We now compute the variance of ψ genie ( ˜ D A ) . We have that

where (a) follows from the independence between Y and Z , and the fact that Z j 's (with j ∈ [ M ] ) are i.i.d.; and (b) follows by Proposition 1.

This yields

Finally, to show the asymptotic normality, we observe that from (41), we have that

where the last equality follows since ( z i ) k d = ( z i ) ℓ for all ( k, ℓ ) ∈ [ M ] 2 . Since (43) is the sample mean of n realizations from 1 -Y M : M -Z , the central limit theorem leads to

where the last equality is due to the independence between Y M : M and Z . This concludes the proof of Proposition 2.

C Details on existing BER estimators / Additional experiments

C.1 Details on existing BER estimators

C.1.1 k-NN BER bounds [16]

Let P NN e be the error rate of the 1-nearest neighbor (NN) classifier with n samples. Then, for binary classification, it follows that for n →∞ [16],

The above bounds were proved by showing the convergence of the conditional posterior probability of the nearest neighbor to the true conditional posterior probability. For details, we refer an interested reader to [16].

To generalize (45) to M -classification problems, it was shown that for n →∞ [16],

In our experiments, we chose k = 1 as this value provides the tightest bounds among larger values of k [16, 72], and we plotted the upper and lower bounds defined in (46). In particular, in order to evaluate P NN e on given data samples, we generated true one-hot labels for all data samples from their soft labels by taking the index having the maximum value in the soft labels.

C.1.2 Generalized Henze-Penrose (GHP) divergence BER bounds [75]

Consider a parameter p ∈ (0 , 1) and two distributions f 0 and f 1 . Then, the Henze-Penrose (HP) divergence between f 0 and f 1 is defined as,

An appealing feature of D p is that it can be estimated directly from the data using the generalized Friedman-Rafsky (FR) statistic [28, 75], which uses the Euclidean minimal spanning tree.

Then, for f 0 and f 1 with prior probabilities p and 1 -p , the BER is bounded as [6],

where

and D p ( f 0 , f 1 ) can be estimated based on the minimal spanning tree [6].

To generalize the HP bounds in (48), consider an M -class classification problem with p 1 , . . . , p M as class prior probabilities and class conditional probability densities given by f k ( x ) := f ( x | c = k ) , k = 1 , . . . , M . Let f ( M ) ( x ) := ∑ M k =1 p k f k ( x ) . Then, the generalized Henze-Penrose (GHP)integral is defined as,

where S is the support of f ( M ) ( x ) . Let δ ( M ) i,j := ∫ p i p j f i ( x ) f j ( x ) f ( M ) ( x ) d x . Then, it was shown in [75] that the BER upper bound is

and the BER lower bound is

In our experiments, we plotted the GHP BER upper and lower bounds defined in (51) and (52). Similar to the k -NN BER bounds, we generated true one-hot labels for all data samples before evaluating the GHP divergence D p . Computing the above bounds requires to evaluate δ ( M ) i,j and this can be done by using the generalized FR statistics (for more details, see [75, Section IV.]).

C.2 Additional experiments

10

CIFAR-10

CIFAR-10H

Misclassification rate (%)

8

6

4

2

0

VGG19

ViT-16B

ResNet19

ResNet50

DenseNet169

Estimated BER

Models

In Figure 5, we compare the estimated BER with the misclassification error rate of a few popular neural network models on the CIFAR10 and CIFAR10 Hdatasets. We chose ResNet [40], VGG [78],

BER

k-NN upper bound

★

GHP upper bound

MoBK(c)

GHP lowerbound

→

k-NN lower bound

0.4

0.4

p°

p°

0.2

0.2

0.0

0.0

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.10

0.10

2

1

一

BER

0.05

|BER

0.05

+

0.00

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

(a) Soft labels.

(b) Noisy soft labels.

DenseNet [43], and Vision Transformer (ViT) [24] as classifiers to compare with and we trained these models on the CIFAR10 dataset. 11 For the error rate evaluated on the CIFAR-10H dataset, we followed the same experiment setup as in [46], under which we generated the ground-truth labels for testset images from the soft labels (i.e., the probabilities for each class being associated with the input image) and evaluated the testset error rate. We iterated this procedure 20 times and plotted the average of the error rates. Figure 5 shows that the error rates on CIFAR10 from the models are worse than the estimated BER, with the exception of ViT. As noted in [46], the reasons for having a lower error rate for ViT than the estimated BER are mainly due to 1) testset over-fitting and 2) various labeling difficulties on the dataset CIFAR10 H that is used for estimating the BER. After compensating for such difficulties on CIFAR10 H, all models' error rates (evaluated on CIFAR10 H) are worse than the estimated BER; this suggests that the estimated BER may be a good estimation since the error rate of any classifier is larger than the BER.

Figure 6 shows the comparison of MoB K ( ψ C ) with the GHP and k -NN bounds on the BER. For each class c ∈ { (1 , 1) , ( -1 , 1) , ( -1 , -1) , (1 , -1) } , we generated 500 Gaussian samples N ( c , σ 2 I 2 ) to have a total of 2 , 000 samples. The label noise was generated according to Z ∼ N ( 0 , 1 5 I 4 ) . The case of no outliers is considered in this experiment. We observe that our BER estimator outperforms the other estimators over σ 2 ∈ [0 . 1 , 2] in both cases of noiseless soft labels and noisy soft labels.

We also made performance comparisons varying different parameters (i.e., µ, σ 2 , r ) in Figures 7, 8, 9, and 10. Similar to the experiment setting for Figure 6, we generated a total of 2 , 000 Gaussian samples with different parameters that are specified in the caption of each figure. The effect of µ and σ 2 on the BER estimation is shown in Figures 7 and 8 under different choices of the radius r . We also provide performance comparisons of our estimator with the other estimators for different values of the noise power σ 2 Z in Figures 9 and 10. Choosing an appropriate value for r is critical in the estimation of the BER as shown in Figure 10. An appropriate value for r can be empirically determined by using the elbow method [84], which is commonly used for choosing the number of clusters in k -means clustering algorithms. Using Figure 10, we selected r = 0 . 2 (elbow point) for the main experiment in Figure 2. Overall, from these figures, we observe that our estimator outperforms the GHP and the 1 -NN classifier BER bounds in various settings.

(a)

r

= 0

.

3

.

(b)

r

= 0

.

4

.

BER

K-NN upper bound

GHP upper bound

MoBK(c)

GHP lowerbound

k-NN lower bound

0.6

0.6

Q 0.4

0.2

0.2

*++

0.0

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.10

0.10

|BER -

一

|BER

0.05

0.05

000

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

μ

μ

(a)

r

= 0

.

3

.

(b)

r

= 0

.

4

.

BER

k-NN upper bound

GHP upper bound

MoBk(c)

GHPlowerbound

←k-NNlowerbound

0.6

0.6

0.4

0.4

p°

p°

0.2

0.2

0.0

0.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.10

0.10

|-

|BER

BER

0.05

0.05

0.00

0.00

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

g²

02

(a)

σ

2

Z

= 0

.

3

.

(b)

σ

2

Z

= 0

.

4

.

BER

k-NN upper bound

MoBK(c)

GHP lowerbound

k-NNlowerbound

0.6

0.6

Q 0.4

0.2

0.2

0.0

0.0

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.10

0.10

一

|BER

|BER

0.05

0.05

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

μ

μ

(a)

σ

2

Z

= 0

.

3

.

(b)

σ

2

Z

= 0

.

4

.

BER

k-NN upper bound

GHP upper bound

MoBK(c)

GHPlowerbound

1

k-NN lower bound

0.04

0.1

0.02

p°

p°

0.00

0.0

0.02

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.20

0.04

0.15

|BER

0.10

0.02

0.05

0.00

000

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the important problem of performance estimation in a multi-class setting. The contributions come in the form of several Bayesian Error Rate estimators for different multi-class variants. Several theoretical guarantees are provided throughout the paper, as well as empirical results on synthetic and real dataset.\n\nSTRENGTHS:\n# Motivation\nThe problem of estimating and validating the performance of a classifier in a multi-class setting has been a hot topic in Machine Learning. As such, the results provided in this paper are of prime interest for the ML community. Estimating the BER can lead to a better understanding of the multi-class problem and to novel methods for such setting.\n\n# Completness\nThe main strength of this paper resides in providing theoretical results for various settings. The first result is provided for the vanilla case of multi-class learning, where the data is drawn i.i.d. from a hidden distribution. The authors then provide a robust version of the previous result. Finally the paper provides several adaptation of the previous results to the noisy setting, i.e. when noise is added to the labels.\n\n# Experimental results\nThe experimental section provides extensive empirical validation of the theoretical results. Several synthetic and real-world datasets are analyzed and discussed in Section 5 thus shedding light on the benefits of the proposed estimates.\n\nWEAKNESSES:\n# Clarity\nWhile the main motivations are easy to follow, and the paper is overall very-well organized, in several occasions definitions and notations are poorly introduced. In lines 76-78, the definition of Y is misleading since it uses the same notation as in multi-class multi-label learning, particularly since no assumption is made on c and how it is retrieved from y. In line 79, it's not clear what $e$ in $P_e$ stands for and how it is related to the definition that follows.\n\nIn line 86, \"an optimal classifier.\" implies that several classifiers can achieve BER. In other words, looking at the problem from an optimization problem point of view, it means that there are several global minima exist. I'm not quite sure I fully grasp the interest of such an assumption.\n\nIn line 111 the notation $Y_{M:M}$ is used without being properly introduced before or in the remainder of the paper.\n\nIn Definition 4, it's unclear wether an assumption on class distribution exists when defining the K subsets. The absence of such an assumption might explain the trade-off discussed in line 163, in which case it should be clearly noted in the paper.\n\nI'm not quite sure I understand the role of paragraph Notation (lines 65 to 71), when most of the notations are loosely introduced in the remainder of the paper.\n\nEquation (11) is quite puzzling. If I'm reading it right, the denoise estimator is based on samples that have the same exact $\\bf x$, i.e. it implies that there are duplicated data in the learning dataset, which seems odd compared to the i.i.d. assumption of line 76. The denoise estimator of Equation (14) seems like a better and more realistic choice.\n\n# Conclusion\nWhile the experimental section provides several discussions on the proposed estimators, I strongly think that a Conclusion section is needed.\n\nQUESTIONS:\n* Can the framework be extended to multi-label classification?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the estimation of the Bayes error rate (BER) in the multiclass classification problem. BER is the best classification error (in terms of expectation) that can be achieved by the Bayes optimal classifier. First, this paper studies the soft labels case, where the direct extension of the estimator proposed by Ishida et al. [ICLR2023] can be shown to be valid in the multiclass case. Next, this paper studies the case where the soft labels are noisy and proposed the median-of-means estimator to improve the robustness. Furthermore, the denoise estimator cluster-based BER estimators are also proposed. Experiments are also provided to justify the usefulness of the proposed methods.\n\nSTRENGTHS:\n1. Writing is easy-to-follow: paper motivation, objectives, and solutions are clear.\n2. Proposed methods are intuitive, easy to implement, and theoretically guaranteed. Many estimators are provided for different use cases.\n3. I found the discussion about the robustness of the estimator $\\psi_{\\mathrm{soft}}$ interesting and the proposed median-of-BERs estimator to mitigate this problem is theoretically guaranteed and practically effective in experiments. In theory, I found the discussion in Lines 158-166 provides interesting comparisons between $\\psi_{\\mathrm{soft}}$ and its MoB estimator.\n4. Experiments are convincing to show that the extension beyond straightforward estimator $\\psi_\\mathrm{soft}$ (MoB, cluster, denoising)can significantly outperform $\\psi_\\mathrm{soft}$.\n\nWEAKNESSES:\n1. Experiments to compare other methods may not be sufficient. In my understanding, only experiments with synthetic datasets with limited hyperparameter choice are conducted to justify the superior of the proposed methods (result in Figure 2). Additional experiments in Appendix C are also only synthetic datasets with the same hyperparameter choice as Figure 2.\n2. Evaluation of the non-synthetic dataset is difficult to know if the Bayes error rate is well estimated, as also suggested in the paper that the labels themselves can be unreliable because of human errors (as discussed in Lines 312-314). But this issue is very problematic by itself and I have no idea how to better deal with this. \n3. Soft labels can be restrictive in many scenarios which may make the proposed method prohibitive. Nevertheless the cluster estimator sounds in (15) reasonable as a good heuristic for BER estimation.\n\nQUESTIONS:\n1. Regarding Section 3.1, it is still not clear to me why having a higher value of the breakdown point guarantees more robustness of the estimator (Lines 139-141). What is the meaning of robust here? Does it mean a more accurate estimator under high noise? \n2. I feel Definition 3 is quite difficult to interpret and more discussion could be useful for readers to grasp the importance and how to use Definition 3. I failed to fully appreciate it and would like to ask some questions. \n- 2.1 What is definition of $\\mathcal{D}^{(\\kappa + \\tau)}$?     \n- 2.2 How to interpret the meaning of Eq. (5)? Is there an intuitive explanation of the value of the breakdown points?\n- 2.3 Why $B(\\psi_\\mathrm{soft}) = \\frac{1}{n}$ is not robust? Why we can conclude that this makes the estimator not robust (Line 143)? What number of $B(\\psi)$ is robust?\n3. Questions about Figure 2 experiments:\n- 3.1 I don't see $\\psi_\\mathrm{soft}$ but rather $MoB_K(\\psi_c)$, is this a typo? If so, which methods are compared? I think it is also useful to see the performance of other proposed estimators as well. \n- 3.2  Is it reasonable to use $k=1$ for $k-NN$ bound? I am not familiar with this method but I feel $1$-nearest neighbor may not be robust enough.\n- 3.3 How hyperparameter selection is carried out? I feel the choice of $K=44$ for the proposed estimator is quite unintuitive and was wondering about the sensitivity of this hyperparameter.\n4. Is there a theory to justify the statistical properties of the cluster estimator (15)? If not, is there a way to theoretically validate this estimator?\n5. It is difficult to know the accuracy of BER estimation from Table 1. Is it possible to also put a reference of a model trained for these classification tasks similar to Figure 4 in Ishida et al? \n6. It is discussed that MoB estimator is more robust and therefore gives lower value in Table 1. I was wondering if it is possible to have an outlier that has a low value also? \n\nTypo:\n1.Line 229: nosy -> noisy?\n2. In some places $MoB$ is used instead of $MoB_K$, e.g., legend in Figure 2 and some other places. Maybe we should unify it if this is not intentional.\n\nLIMITATIONS:\nI have no further things to add here. Please see the weaknesses and questions section for some discussions on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a few methods for estimating Bayes error rates of multi-class classification in different scenarios. The proposed method generalize the previous soft-label approach for binary classification [Ishida et al., 2023] and also extends it for robustness to label noise and outliers. The first extension uses the median-of-means in place of the ordinary average for better robustness, and the second one uses the feature-wise average assuming the features are discrete. The third one uses a Nadaraya-Watson estimator with the Parzen kernel for incorporating data points near each point. The paper provides theoretical analyses for the estimation errors, convergence rates, and the robustness guarantees. Finally, the paper presents experiments using synthetic data and three benchmark datasets demonstrating the superiority of the proposed methods.\n\nSTRENGTHS:\n- The paper is well-written and pleasant to read.\n- The extensions to the noisy cases including the one with hard (one-hot) labels may be practically useful because many datasets do not have soft labels.\n- The use of the median of means to this problem seems interesting. The theoretical results about this part are also insightful.\n- The experiments show the clear superiority of the proposed method over the baseline methods.\n\nWEAKNESSES:\n- The multi-class extension when soft labels are provided seems straightforward.\n- The denoising method in Eq. (11) is a bit disappointing because it heavily relies on the assumption that the features are discrete.\n- The denoising method in Definition 6 could be applied to any $\\mathcal{X}$ with a distance metric, but this essentially estimates the conditional probability $P_{Y|X}$, which contradict the following claim: \"This is an appealing feature of our work, different from taking a plug-in approach that first estimates the istribution from which the data is drawn, and then evaluates the BER. Indeed, our BER estimators, which are proved to be unbiased, consistent and robust to label noise and outliers, do not require the estimation of the data probability density to perform an effective BER estimation.\"\n- Regarding the previous point, if the method in Definition 6 is no better than the SOTA method in the conditional probability estimation, the estimate will be an upper bound of the SOTA score (and the Bayes error). However, it is not convincing to say the proposed method, which looks like a classical method, is better than the SOTA method.\n\nQUESTIONS:\n### Major comments\n- How does one choose $r$? More importantly, how did the authors choose it in the experiments?\n- The proposed denoising method essentially estimates the class posterior probability using the kernel method. What is the motivation for using this method when there are many other classification methods performing well?\n- The footnote on page 2 says \"We assume that $\\mathcal{X}$ is a finite set, but several of our results easily extend to the case when $\\mathcal{X}$ is an infinite set.\" Having a finite $\\mathcal{X}$ is very restrictive. Why not work on the infinite case in the first place if the extension is easy? What about the continuous case?\n\n### Minor comments\n- Line 137, \"$\\operatorname{rad}(\\theta)$ is the vector consisting of $L$ radiuses along the main axes of the largest $L$-dimensional ellipsoid in replaced by outliers.\": I could not understand this. What are the $L$ radiuses and the vector consisting of them, more precisely?\n- Lines 164-165, \"For example, setting $K = \\ln n$ leads to a higher breakdown point (and hence, is more robust) than setting $K = \\sqrt{n}$.\" Doesn't a larger $K$ have better robustness?\n- Line 178, \"We assume that z has i.i.d. components each with zero mean (without loss of generality).\" I think this does lose the generality.\n- In footnote 3 on page 5, is the definition of sub-Gaussianity correct?\n- I am not very comfortable to call the method in Definition 6 a \"nearest-neighbor\" method because it uses a fixed kernel (with a fixed $r$). I would call it a Nadaraya-Watson estimator with the Parzen window kernel.\n- I suggest using different letters for $P_e$ and $P_e(\\cdot)$.\n\nLIMITATIONS:\nA limitation is that the paper assumes that the set of features is finite.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to design a tighter estimate for the Multi-class classification error. The paper analyzes several theoretical aspects of the suggested Bias Error rate  estimator, including its consistency, unbiasedness, convergence rate, variance, and robustness. Moreover the authors utilize a denoising method to reduce label noise if such is present and improve robustness to the outliers by incorporating the median-of-means estimator. They validate the effectiveness of our theoretical results via experiments both on synthetic data under various noise settings and on real data (CIFAR-10H, Fashion-MNIST-H and MovieLens -- movie recommendation dataset)\n\nSTRENGTHS:\nThe paper raises an important issue of estimating performance limit of the multi-class classifier.  It provides thorough theoretical analysis, backed up by some experimentation.\n\nWEAKNESSES:\nI'm struggling to see an impact of this paper. It aims to demistify optimal performance of multi-class classification, but I did not see any illustration of this in the paper. I would expect to see performance of SOTA method on a complex task (e.g., ImageNet) vs. the provided estimator, but the paper uses relatively easy classification tasks for illustration of the bounds. So not clear what is the applicability of this paper for more challenging tasks of Computer Vision and NLP. Also I would expect some theoretical derivation on what is a good feature space for which the suggested denoising holds. Note that the denoising itself is not novel, and seems a bit ad-hoc solution to improve the accuracy of the estimator.\n\nQUESTIONS:\nI would love the authors to explain what is the impact of their work and experiment (maybe in Discussion/ Conclusion section that is currently missing).\n\nLIMITATIONS:\nMy main concern is that the paper does not have a real impact on more complex Deep Learning tasks of NLP and Computer Vision.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}