Optimal Binary Classification Beyond Accuracy

Abstract: The vast majority of statistical theory on binary classification characterizes performance in terms of accuracy. However, accuracy is known in many cases to poorly reflect the practical consequences of classification error, most famously in imbalanced binary classification, where data are dominated by samples from one of two classes. The first part of this paper derives a novel generalization of the Bayes-optimal classifier from accuracy to any performance metric computed from the confusion matrix. Specifically, this result (a) demonstrates that stochastic classifiers sometimes outperform the best possible deterministic classifier and (b) removes an empirically unverifiable absolute continuity assumption that is poorly understood but pervades existing results. We then demonstrate how to use this generalized Bayes classifier to obtain regret bounds in terms of the error of estimating regression functions under uniform loss. Finally, we use these results to develop some of the first finite-sample statistical guarantees specific to imbalanced binary classification. Specifically, we demonstrate that optimal classification performance depends on properties of class imbalance, such as a novel notion called Uniform Class Imbalance, that have not previously been formalized. We further illustrate these contributions numerically in the case of $k$-nearest neighbor classification.

Reviews and author responses:

 Dear Reviewer rWrH,

We wanted to express our sincere thanks for your careful review of the paper, especially the proof of Theorem 3. It is rare to have a reviewer check the mathematical details so carefully, and the points you raised definitely clarified the writing.

Since you carefully checked the mathematical details, we also invite you to raise your confidence score (currently 1), if you feel more confident about the correctness of the paper, and perhaps your "Presentation" score (currently 1) if you feel the revisions have significantly clarified the paper.

Best regards,

The Authors

 Dear Reviewer rWrH,

Thank you for the question. We just wanted to clarify that we are not simply jumping from
$$\operatorname{esssup} f([0, z]) > \operatorname{essinf} f((z, 1]).$$
to
$$\inf_{z \in A} f(z) > \sup_{z \in B} f(z)$$
(which we agree, is not obvious). Rather, we give the following argument:

Suppose
$$\operatorname{esssup} f([0, z]) > \operatorname{essinf} f((z, 1]).$$
Then, there exist $a > b$ such that
$$\operatorname{esssup} f([0, z]) > a > b > \operatorname{essinf} f((z, 1]).$$
Since $\operatorname{esssup} f([0, z]) > a$,
it follows that
$$P_Z([0, z] \cap \\{z : f(z) \geq a\\}) > 0.$$
Similarly, since $\operatorname{essinf} f((z, 1]) < b$, it follows that
$$P_Z((z, 1] \cap \\{z : f(z) \leq b\\}) > 0.$$
Hence, letting $A = [0, z] \cap \\{z : f(z) \geq a\\}$ and $B = (z, 1] \cap \\{z : f(z) \leq b\\}$, we have
$$\inf_{z \in A} f(z) \geq a > b \geq \sup_{z \in B} f(z).$$
As the reviewer notes, we had provided this argument as footnote 2 (to avoid interrupting the overall proof), but we will make this more explicit by separating it into a small Lemma.

 Thank you. The reviewer changed his score accordingly.

 The authors want to prove
$$ \mathrm{esssup} f([ 0,z] ) \le \mathrm{essinf} f( (z,1 ] )$$

they do it by a contradiction: Lets assume 
$$  \mathrm{esssup} f([ 0,z] ) > \mathrm{essinf} f( (z,1 ] )$$

that means, that there exists an $A \subseteq  [0,z]$  with $P_Z(A) >0$, a $B \subseteq (z,1]$ with $P_Z(B) >0$ such that
$$\sup_{z \in A} f(z)  > \inf_{z \in B} f(z)$$

line 696 states the something different, namely
$$ \inf_{z \in A} f(z)  > \sup_{z \in B} f(z)$$

To make it more clear:
What we get out from the assumption above, namely,

$$\sup_{z \in A} f(z)  > \inf_{z \in B} f(z)$$  
does **NOT** imply your statement $\inf_{z \in A} f(z)  > \sup_{z \in B} f(z)$

but your statement  $\inf_{z \in A} f(z)  > \sup_{z \in B} f(z)$ implies $\sup_{z \in A} f(z)  > \inf_{z \in B} f(z)$ (whch is what one gets from the  assumption between esssup and essinf)

Therefore, as it is stated now,  you are using a stronger statement 
( namely 
 $\inf_{z \in A} f(z)  > \sup_{z \in B} f(z)$ ])
than what you obtain from the assumption between esssup and essinf, in order to arrive at a contradiction.

What you need to do, is to prove that there exists two sets used to construct the esssup, where additionally your intended stronger statement holds ( $\inf_{z \in A} f(z)  > \sup_{z \in B} f(z)$).

Lets take footnote 2 as a proof.

Since the supplement is unlimited, making this tightening step clear (from sup > inf to inf > sup)would be nice.


 Related Work: Of course, work on statistical guarantees is more relevant than the works I suggested. However, they do have a sound theoretical underpinning, whereas quite a few of the works that you cite (e.g., SMOTE), seem to be more practical. So I do think there should have been a place for them.

I found the discussion under point 3. enlightening. This is essentially what I had missed in the paper.

 Dear Reviewer p1uP,

Since the discussion period is ending soon, we wanted to check whether we addressed your concerns, or whether you had any further questions.

As you suggested in Question 2, we have re-written the abstract to clarify that our contributions go beyond imbalanced classification. We also hope our responses below have answered your Questions 3 and 4.

Regarding your Question 1, we agree that experiments with real data can strengthen the paper's usefulness for practitioners and are currently working on an experiment using a credit card fraud dataset to demonstrate (a) performance benefits of stochastic classifiers (as in the current Experiment 1) and (b) the prediction from our theory that optimal classification under Uniform Class Imbalance requires $k$NN methods to use larger values of $k$. We will include these results in the final version of the paper.

Nevertheless, we feel that the paper's contributions address significant gaps in the theoretical literature that are important for theoreticians to fill even if the relevance to practicioners is not immediately apparent. For this reason, we hope that the reviewer will consider raising their score based on our current revisions and responses.

Best regards,

The Authors

 Dear Reviewer c1ro,

Since the discussion period is ending soon, we wanted to check whether we were able to address your concerns regarding the paper's contributions.

Specifically, we hope our response below expressed that the paper's contributions are more specific than simply showing that every confusion matrix measure can be optimized by a regression-thresholding classifier, and that the literature discussed in the paper captures the most directly relevant prior work.

We also hope you find that our revised abstract and additions to the Introduction and Related Work sections (highlighted in blue) make these specific contributions clearer to the reader.

Please do not hesitate to make any further suggestions or ask us any questions before the discussion period ends.

Best regards,

The Authors

 Dear Reviewer rWrH,

Since the discussion period is ending soon, we wanted to check whether you had any further questions regarding the paper. If you have a chance, please take a look at the revision we uploaded (with changes highlighted in blue) and let us know if we can further clarify anything.

We really appreciate the time you took to verify the details of the proofs. We believe we have addressed all of the points you raised and want to make sure you find the revised paper clear and convincing! If so, we hope the reviewer will consider raising their score.

Best regards,

The Authors

 Thank you for your questions, helpful comments about the motivation of the paper, and appreciation of the paper’s strengths. We respond to your specific comments below.

> The numerical experiments provide certain confidence on the correctness of the theoretical results presented in the paper. However, it will be much more useful to see experiments in actual imbalanced datasets....

We agree that experiments on real-world data could help to demonstrate the practical significance of the paper’s theoretical contributions, and we are keeping this in mind as we pursue further work in this area. However, the main goals of this paper were to fill specific gaps in existing theory, and we felt that numerical simulations, in which the reader could fully understand the behavior of the data, were the best way to illustrate these gaps.

> Currently, the abstract focuses primarily on imbalanced classification even though the generalization of the Bayes optimal classifier seems like a more general result. Furthermore, it implies that classifier behavior is studied in the context of different types of class imbalance, which is not true. I strongly suggest rewriting the abstract in much cleaner way to clarify the contributions of this line of work and place it in the right context.

Our original motivation for pursuing this work was to address a specific gap in existing literature, namely the lack of statistical theory for imbalanced classification. This is why, as you described, the abstract focuses on imbalanced classification, while Theorem 3 could be of more general interest. However, we agree that this framing fails to highlight the more general significance of some of our results. We have thus rewritten the entire abstract to better explain the paper’s contributions.

> The definition of uniform class imbalance is unclear in the sense that it seems to be defined in terms of the behavior of a regression function. I may have misunderstood the focus of the definition, but shouldn't class imbalance types be defined in a more generic way with respect to the structure of the data itself? Also, why not in terms of a general classification function?

As is typical for statistical assumptions, uniform class imbalance is an assumption about the *data-generating process*. The regression function models the true conditional relationship between X and Y. This is always defined (in terms of the joint distribution of X and Y) and is independent of any particular classifier or dataset. In this sense, uniform class imbalance is indeed an assumption about the structure of the data itself. On a real dataset, the prescribed imbalance generally does not manifest exactly (due to sampling variability), and statistical methodology takes this into account.

> Finally, it is unclear what is the motivation behind the particular type of class imbalance and how often this is encountered in practice.

As we discuss on lines 264-271, uniform class imbalance is likely the main type of class imbalance in problems such as rare disease detection or fraud detection, where the covariate X provides only partial information about the class Y (or, equivalently, there is a large amount of irreducible noise in the labels).

> The motivation behind the study of the k-nearest neighbor classifier is unclear. Why this classifier? Is it commonly used in real-world imbalanced classification problems?

The kNN classifiers and its variants are among the most common approaches for classifying imbalanced data (see, e.g., the textbooks by He and Ma [2013] and Fernández et al. [2018] on imbalanced classification). It is also perhaps the most extensively studied classifier in the theoretical literature, because of its long history, intuitive nature, and mild assumptions. Practically, kNN is widely used in industry because it is easy to implement and understand without specific domain knowledge (in contrast, e.g., to deep neural networks).

> How difficult it is to analyze the behavior of other types of classifiers, which are commonly used in practice for these types of problems?

Our results in Section 4 apply to all regression-thresholding classifiers, which also include, e.g., logistic regression and random forests, and we believe that our analysis of uniform class imbalance could also be adapted to these classifiers. For example, the key novel steps of the analysis based on the UCI assumption, which include (1) using Bernstein’s inequality rather than Chernoff bounds and (2) bounding the Lipschitz constant of the true regression function, can be incorporated into the analysis of Gao & Zhou (2020) for random forests.

References:
Gao, W., & Zhou, Z. H. (2020). Towards convergence rate analysis of random forests for classification. Advances in neural information processing systems, 33, 9300-9311.

 We thank you for your detailed review and helpful suggestions for improving the paper’s clarity. We agree with and have implemented most of your notational suggestions. In other cases, your comments alerted us to potential points of confusion in the paper, and we have revised the paper to clarify these. Please let us know if we can further clarify any proofs or notation, to make sure there are no doubts about the correctness of the paper. 

Itemized responses to your comments are below:

> eq(1) is superfluous: 1{\eta^*(x)>0.5} is either 0 or 1. Thus putting a Bernoulli in front of it makes no sense.

Although the notation is (intentionally) redundant, we disagree that it “makes no sense”. Bernoulli(1{\eta^*(x)>0.5}) is simply a constant (deterministic) random variable that takes either the value 0 or the value 1 (depending on $\eta^*(x)$) with probability 1. Our reason for writing Eq. (1) in this way is to make it clear that the classical Bayes classifier is a Regression-Thresholding Classifier (Definition 2). In the revision, we have added sentences immediately after Eq. (1) explaining this notation.

> The paper makes a mistake in equation (3)... The true positive rate, however, is a conditional probability, conditioned on ground truth being positive: P(\hat{Y}=1| Y=1) = P( \hat{Y}=1, Y=1 ) / P(Y=1) = \frac{ E_X[ P(Y=1|X=x) \hat{Y}(X=x) ] }{ E_X[ P(Y=1|X=x)] }

We agree that we mean to refer to the joint probability, while “true positive rate” usually refers to the conditional probability. To fix this, we have changed “rate” to “probability” (e.g., “true positive probability”) throughout the paper.

> Theorem 3 is wrong, but there might be a typo and they might mean stochastic classifiers.

We agree that the wording “any classifier” in Theorem 3 should be “any stochastic classifier”. We have fixed this in the revision.

> The statement of lemma 18 is trivially true if we assume that f and \eta can be anything.… Lemma 18 would make sense if one considers \eta(x) to be P(Y=1|X=x) and thus \eta^*(x). Then the proof makes more sense.

Indeed, throughout Lemma 18 and its proof, $\eta$ should be replaced with $\eta^*$. Thank you for catching that!

> The proof of Lemma 18 contains a problem: E_{Z \sim P_X} [ \hat{Y}(Z) | \eta(Z)=\eta(x) ] is a conditioned expectation, conditioned on the set P( { z: \eta(z) = \eta(x)}) - this probability can be 0, then this conditional expectation can become undefined.

To clarify this, we have expanded the proof of Lemma 18 in terms of the $\sigma$-field generated by $\eta^*(Z)$. The conditional expectations are then defined as in Chapter 5 of Durrett (2010). We have also expanded on the justifications for each step of the proof. Since the formal $\sigma$-field justification requires lengthy setup and a few conditional expectation equalities, we have reorganized the presentation to isolate this formality into a proof of the new equation (7). 

References:
Durrett, R. (2010). Probability Theory and Examples. Fourth Edition. Cambridge University Press.

> Lemma 19: Moreover, since the actual proportions of positives and negatives are fixed - why does that hold?

The probability of a positive or negative label depends only on the true regression function $\eta^*$ and the distribution of $X$, not on the classifier. To clarify this, we have reworded this to “since the proportions of positive and negative true labels are independent of the chosen classifier”.

> line 639 of the supplement: could it be sup_{z \in A} f(z) > inf_{z \in B} f(z) instead ? As far as the reviewer remembers, the essential supremum is up to removal of a set of measure 0.

We believe this is correct in the paper. To make this clearer, we have added the following explicit formula for the sets $A$ and $B$:
$$A := \left\\{ z' \in [0, z] : f(z') > \frac{3g(z) + h(z)}{4} \right\\} \quad \text{ and } \quad B := \left\\{ z' :  f(z') < \frac{g(z) + 3h(z)}{4} \right\\}.$$
By the assumption $g(z) > h(z)$, $\operatorname{esssup} f([0, z]) = g(z) > \frac{3g(z) + h(z)}{4}$, and so, by definition of $\operatorname{esssup}$, $A$ has non-zero probability (i.e., $P_Z(A) > 0$). Similarly, since $\frac{g(z) + 3h(z)}{4} > h(z)$, $B$ has non-zero probability (i.e., $P_Z(B) > 0$).
By construction of $A$ and $B$ and the assumption $g(z) > h(z)$,
$$\inf_{z \in A} f(z) \geq \frac{3g(z) + h(z)}{4} > \frac{g(z) + 3h(z)}{4} \geq \sup_{z \in B} f(z).$$

We hope these responses address your concerns. Please let us know if we can further clarify any proofs or notation or fix any typos to make sure there are no doubts about the correctness of the paper. 

 Thank you for your positive feedback on the submission!

Regarding your question:
> For the special case of the classical Bayes classifier, is this a good strategy? Is this SOTA?

As noted on lines 173-174 of the paper, the classical Bayes classifier always sets $(p, t) = (0, 0.5)$. So, in this special case, tuning $(p,t)$ isn’t necessary. We apologize if we are misunderstanding the question.

Perhaps the reviewer is suggesting that one could train an RTC to maximize accuracy (i.e., to estimate the classical Bayes classifier), and then re-tune the classifier threshold to optimize another performance metric? If so, then yes, our results suggest that, statistically speaking, this should work just as well as training an RTC to optimize the second metric directly.

 Thank you for your comments and suggested additional references. Broadly speaking, we agree that there has been a great deal of related work on selecting classification thresholds, but the main contribution of this paper was intended to be much more specific than that summarized by the reviewer, motivating our choice of literature to discuss. We have tried to clarify this in the responses below.

>The paper discusses related work (> 2 pages of references), but does not discuss concepts that are obviously related, such as calibrating classifiers, isotonic regression, Brier scores, etc. 

Binary classification is probably the most thoroughly studied problem in machine learning. As such, the closely related literature is huge, and we had to be selective about which literature to fit in a conference paper. The goal of this paper is to advance *statistical guarantees* for binary classification beyond accuracy, and so we focused on discussing the most relevant literature towards this goal. In particular, the papers of Narasimhan et al. [2014, 2015], Koyejo et al. [2014], Yan et al. [2018], and Wang et al. [2019a], discussed on Lines 179-185, all use a similar idea of finding a threshold that optimizes a general class of performance metrics, and our contributions relative to these papers relative to these papers are more specific, as discussed below. While the concepts of calibrating classifiers, Brier scores, etc. are certainly relevant in practice, the literature on these topics is methodological, rather than theoretical, in nature.

> Note that the class distribution (skewness) can also be viewed as a cost ratio.

Indeed, we made note of a similar equivalence on lines 69-27. We have edited this to refer to both costs *and cost ratios*, and to reference Flach (2003).

> The paper investigates the connection between 2x2 confusion matrices and shows that any measure that is defined on those, can be optimized with a threshold optimization method. The strength of this paper is its formal rigor…. Yet, I did not get the impression that I have learned something new here. The fact that any 2x2 matrix can be optimized with a threshold is… already known from ROC analysis… Similarly, ROC analysis is briefly mentioned in section 4.1., but restricted to the area under curve, which is quickly done away with the argument that it is not a confusion matrix measure (which is correct). However, ROC analysis does exactly what the authors set out to do, namely to find optimal regression thresholding classifiers (in the authors' terminology).... How does your work relate to works on finding optimal performance under different skew/cost distributions using ROC analysis and related methods?

As the reviewer notes, the idea that thresholding classifiers can optimize confusion matrices is well known. Prior work already discussed in the paper, e.g., by Narasimhan et al. [2014, 2015], Koyejo et al. [2014], Yan et al. [2018], and Wang et al. [2019a], also utilizes a similar idea. Theorem 3’s novel contributions are to refine this in the following ways:
1. We show that stochastic thresholds sometimes offer better performance than the best deterministic threshold. The closest existing result (Corollary 2 of Wang et al. [2019a]) shows only that stochastic combinations of deterministic classifiers are optimal; our result is strictly stronger and is useful in practice since it only requires learning a threshold, rather than training multiple classifiers.
2. Together with Example 4, Theorem 3 clarifies the role of the absolute continuity assumption that is ubiquitous but poorly understood in existing generalizations of the Bayes optimal classifier. Since this assumption is unverifiable in practice, this is important for understanding whether existing results are relevant to real data. We also note that removing this assumption requires a completely novel proof approach, based on variational methods, from prior work.
3. The ROC analysis of Hernandez-Orallo et al. (2013) and Flach (2003) is complementary to the results of the present paper. They study how general performance metrics change in terms of parameters such as the label distribution and the regression threshold. Meanwhile, the present paper aims to understand statistical properties: specifically, how well a regression-thresholding classifier trained on finite training data will perform on test data. ROC analysis informs the selection of a regression threshold, but it does not provide any guarantee of performance on test data. In contrast, our results illustrate how test performance under an arbitrary confusion matrix measure depends on various parameters (sample size, Lipschitz constants, choice of $k$ in knn, degree of uniform class imbalance, etc.).

 The paper investigates the connection between 2x2 confusion matrices and show that any measure that is defined on those, can be optimized with a threshold optimization method.  The strength of this paper is its formal rigor. Most of the points are also very well explained, the non-trivial math is reserved for the appendices.

Yet, I did not get the impression that I have learned something new here. The fact that any 2x2 matrix can be optimized with a threshold is, in my opinion, already known from ROC analysis. You can analyze arbitrary measures with their isometric structure, and use a cost-threshold for finding the optimal point. Note that the class distribution (skewness) can also be viewed as a cost ratio. This has been investigated by Peter Flach in the early 2000s in a series of papers. Maybe the following are good starting points:

* Peter A. Flach:
Classifier Calibration. Encyclopedia of Machine Learning and Data Mining 2017: 210-217
* José Hernández-Orallo, Peter A. Flach, César Ferri:
ROC curves in cost space. Mach. Learn. 93(1): 71-91 (2013)
* Peter A. Flach:
The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics. ICML 2003: 194-201

In the reviewer-discussion of the paper, the following has also popped up:
Optimal Thresholding of Classifiers to Maximize F1 Measure
https://link.springer.com/content/pdf/10.1007/978-3-662-44851-9_15.pdf

(BTW: In case you are wondering: none of the above is a self citation... :-))

The paper discusses related work (> 2 pages of references), but does not discuss concepts that are obviously related, such as calibrating classifiers, isotonic regression, Brier scores, etc. Similarly, ROC analysis is briefly mentioned in section 4.1., but restricted to the area under curve, which is quickly done away with the argument that it is not a confusion matrix measure (which is correct). However, ROC analysis does exactly what the authors set out to do, namely to find optimal regression thresholding classifiers (in the authors' terminology). 

Now, I am not 100% sure whether the main result of the paper, namely that every confusing matrix measure can be optimized with a regression-thresholding classifier, has formally been proven before (probably not), and this may very well be a contribution that is worth-while to be reported, but that this is possible seemed to be clear to me from ROC analysis.

In any case, I think a discussion of the relation of this work to the above work on finding optimal performance under different skew/cost distributions would be essential for this paper. 

The authors have responded to that point, and shed some light on the relation of the above-mentioned works to this paper. I do think that this is much more relevant to this work than some of the works that are currently discussed in the related works section (such as SMOTE et al.). They have not been very clear about whether they intend to actually incorporate this, but I give them the benefit of the doubt and trust that they will do so.

A minor point is that the practical relevance of the experimental results is not entirely clear. I do think that they are interesting, and also well selected, and I am also fine without a real-world case study, but I do think it would be important to discuss whether, e.g., Experiment 1 is an interesting artifact or something that we can also expect to see in practice. * How does your work relate to works  on finding optimal performance under different skew/cost distributions using ROC analysis and related methods? The authors briefly state some obvious limitations at the end. But I also would not know what else they should do, so fine with me.

 Motivated by imbalanced classification problems, the paper generalizes the analysis of Bayes-optimal classifier to any performance metric supported in the confusion matrix. 
In a second part, the paper analysis the specific case of the k-NN classifier under a specific setting (uniform loss) of the previous generalization.
 Strengths
+good theoretical analysis
+good theoretical insights
+well-written paper and technically sound

Weaknesses
-not explored enough how the theoretical knowledge can impact the design of models in practice.




 "Such an approach has many practical advantages. For example, as we show in Appendix D, a simple algorithm can exactly optimize the threshold (p, t) over large datasets in O(n log n) time. Additionally, one can address covariate shift or retune a classifier trained under one
CMM to perform well under another CMM, simply by re-optimizing (p, t), which is statistically and computationally much easier than retraining a classifier from scratch."

For the special case of  the classical Bayes classifier, is this a good strategy? Is this SOTA? yes

 The authors consider general measures over confusion matrices and propose a generalized Bayes classifier. This is claimed to be an upper bound for any stochastic binary classifier of type P(Y=1|X=x) for the given measure over confusion matrices.
They prove bounds on differences of confusion matrices for differently parameterized generalized Bayes classifiers. strength: if they results hold, it would be of  theoretical interest.
weakness: there are lots of small notational issues in the paper.

The paper can be correct or wrong. it is hard to judge due to the brevity of the proofs.
The reviewer believes that many proofs needs to be extended to be readable.

eq(1) is superfluous: 1{\eta^*(x)>0.5} is either 0 or 1. Thus putting a Bernoulli in front of it makes no sense.

The paper makes a mistake in equation (3):
Confusion matrices contain joint probabilities
P( Y, \hat{Y} ), as also stated in line 141:
C_{1,1} + C_{1,2} + C_{2,1} + C_{2,2} =1
The true positive **rate**, however, is a conditional probability, conditioned on groundtruth being positive:
P(\hat{Y}=1| Y=1)  = P( \hat{Y}=1, Y=1 ) / P(Y=1) = \frac{ E_X[ P(Y=1|X=x) \hat{Y}(X=x)   ] }{  E_X[ P(Y=1|X=x)] }

the empirical true positive rate is given as

\frac{\sum_i Y_i \hat{Y}(X_i) }  {\sum_i Y_i}

not as (wrongly) stated in eq(3).

Their TN, TP terms inside the C matrices are joint probabilities:

TN + FP = P(Y=0)
FN + TP = P(Y=1)


TN + TP = P(Y=1,\hat{Y}=1) + P(Y=0,\hat{Y}=0)

ACC = P( Y= \hat{Y}  )

Theorem 3 is  wrong, but there might be a typo and they might mean stochastic classifiers.
 
take \hat{Y}(x) as the perfect oracle which can look at the ground truth label y_i in a pair (x_i,y_i) and use this to predict. This has accuracy 1 independent of P(Y=1|X=x). Therefore, theorem 3 does not hold for *any* classifier $\hat{Y}$.

Theorem 3 in  the appendix states it for a stochastic classifier, not for any classifier.

Lemma 18 in the supplement / line 579:

The statement of lemma 18 is trivially true if we assume that f and \eta can be anything: set f(\eta(x)) = P(\hat{Y}=1|X=x). Done.

The proof of Lemma 18 contains a problem:
E_{Z \sim P_X} [ \hat{Y}(Z) | \eta(Z)=\eta(x) ] is a conditioned expectation, conditioned on the set
P( \{ z: \eta(z) = \eta(x)\}) - this probability can be 0, then this conditional expectation can become undefined.

Lemma 18 would make sense if one considers \eta(x) to be P(Y=1|X=x) and thus \eta^*(x). Then the proof makes more sense.

line 589 in the supplement is wrong: it does not hold for any classifier, but for a stochastic classifier.



Lemma 20 and 21 seem to be correct - except for likely mixing up sup_{z \in A} f(z) > inf_{z \in B} f(z) in line 639 of the supplement Lemma 19: Moreover, since the actual proportions of positives and negatives are fixed - why does that hold?
 line 639 of the supplement: could it be sup_{z \in A} f(z) > inf_{z \in B} f(z) instead ? As far as the reviewer remembers, the essential supremum is up to removal of a set of measure 0. N/A

 This paper studies the problem of imbalanced binary classification. First, the Bayes-optimal classifier is generalized to any performance metric determined from the confusion matrix. This result is then employed to obtain regret bounds in terms of the error of estimating the class probability function under uniform loss. Second, a specific type of class imbalance (Uniform Class Imbalance) is proposed and analysis of how this affects the optimal classifier performance is carried out. Two numerical examples are provided to illustrate the theoretical results in the case of k-nearest neighbor classification.

The contributions of the paper are as follows:
1. A novel characterization of classifiers optimizing general performance metrics that are functions of a classifier's confusion matrix is derived.
2. The authors show that in general, a Bayes classifier always exists if one considers stochastic classifiers, but not if one considers only deterministic classifiers.
3. Performance guarantees are derived under more general performance measures in terms of the error of estimating the class probability function under uniform loss.
4. Analysis of k-nearest neighbor classification under uniform loss is provided. Strengths:
+ Generalization of the Bayes optimal classifier to any performance metric computed from the confusion matrix.
+ Regret bounds in terms of the error of estimating the class probability function under uniform loss.
+ Definition and analysis of specific type of class imbalance.
+ Analysis and insights on the performance of k-nearest neighbor classification under uniform loss.

Weaknesses:
- Numerical experiments are provided only for synthetically generated datasets and not real-world imbalanced datasets.
- The motivation behind the proposed uniform class imbalance and the study of the k-nearest neighbor classifier is unclear.
- The abstract implies that classifier behavior is studied in the context of different types of class imbalance, which is not true.
- The definition of uniform class imbalance is unclear. 1. The numerical experiments provide certain confidence on the correctness of the theoretical results presented in the paper. However, it will be much more useful to see experiments in actual imbalanced datasets. Given the prevalence of imbalanced classification problems in practice and the plethora of algorithmic solutions, there is a large number of datasets available in the public domain. I understand that the numerical results included in the paper serve the role of verifying the theoretical contributions and sharing important insights in a more palatable way. However, providing actual experiments on real-world imbalanced datasets can provide unique insights and help practitioners understand on how to use the findings of this paper when developing their algorithmic solutions.

2. Currently, the abstract focuses primarily on imbalanced classification even though the generalization of the Bayes optimal classifier seems like a more general result. Furthermore, it implies that classifier behavior is studied in the context of different types of class imbalance, which is not true. I strongly suggest rewriting the abstract in much cleaner way to clarify the contributions of this line of work and place it in the right context.

3. The definition of uniform class imbalance is unclear in the sense that it seems to be defined in terms of the behavior of a regression function. I may have misunderstood the focus of the definition, but shouldn't class imbalance types be defined in a more generic way with respect to the structure of the data itself? Also, why not in terms of a general classification function? Finally, it is unclear what is the motivation behind the particular type of class imbalance and how often this is encountered in practice.

4. The motivation behind the study of the k-nearest neighbor classifier is unclear. Why this classifier? Is it commonly used in real-world imbalanced classification problems? How difficult it is to analyze the behavior of other types of classifiers, which are commonly used in practice for these types of problems? The authors have adequately addressed the limitations of their work. However, they haven't discussed any potential negative societal impact of their work, and thus, they are encouraged to do so.

Meta-review: After discussion with the authors, reviewers are positive about this paper. Some loose ends remain, but it is impressive that the authors have discovered something new about binary classification, the single most intensively studied topic in all of ML.

Decision: Accept