A Closer Look at AUROC and AUPRC under Class Imbalance

Authors: matthew b mcdermott
haoran  zhang
lasse hyldig hansen
giovanni  angelotti
jack  gallifant
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 21:36:17

Abstract: In machine learning (ML), a widespread claim is that the area under the precision-recall curve (AUPRC) is a superior metric for model comparison to the area under the receiver operating characteristic (AUROC) for tasks with class imbalance. This paper refutes this notion on two fronts. First, we theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, establishing clearly that AUPRC is not generally superior in cases of class imbalance. We further show that AUPRC can be a harmful metric as it can unduly favor model improvements in subpopulations with more frequent positive labels, heightening algorithmic disparities. Next, we empirically support our theory using experiments on both semi-synthetic and real-world fairness datasets. Prompted by these insights, we conduct a review of over 1.5 million scientific papers to understand the origin of this invalid claim, finding that it is often made without citation, misattributed to papers that do not argue this point, and aggressively over-generalized from source arguments. Our findings represent a dual contribution: a significant technical advancement in understanding the relationship between AUROC and AUPRC and a stark warning about unchecked assumptions in the ML community.

Main paper: 
 introduction: 
 Machine learning (ML), especially in critical domains like healthcare, necessitates careful selection and application of evaluation metrics to guide appropriate model choices and understand performance nuances [150] . Model evaluation can happen in one of two settings: (1) a methodological/model comparison setting, which occurs outside of a specific deployment setting and target model usage workflows, optimal decision thresholds, or specific false-positive (FP) and false-negative (FN) costs are typically not known, or (2) an application/deployment setting, where reasonably specific estimates of model usage workflows and FP/FN costs can be made. In both settings, appropriate metric choice is critical-and inappropriate selection can hinder innovation when used for model comparison and lead to significant real-world costs (e.g., misdiagnosis in a medical setting) in deployment settings.Claim 1. Let f be a model which outputs continuous probabilistic predictions trained to solve a binary classification task for which the prevalence of negative labels is significantly higher than the prevalence of positive labels. For this problem, the AUPRC will yield a "better" or "more accurate" or "fairer" evaluation of f than the AUROC. Claim 1 is made widely in both the scientific literature [399, 71, 159, 124] , in ML educational content like notable textbooks [119, 141] , and in popular press sources [80, 254] . It is so widespread that even basic search results for queries relating to AUROC and AUPRC 1 and large language model assistants like ChatGPT or Github Co-pilot will profess its veracity. 2 Throughout these sources, it has been justified on numerous, often imprecise grounds (see Section 5) , but despite this extensive attention, we show in this work that this claim is, in fact, wrong, and may be dangerous from a model fairness perspective; further, many of its justifications are invalid or misapplied in common ML settings. More specifically, we show the following:1) AUROC and AUPRC only differ with respect to model-dependent parameters in that AUROC weighs all false positives equally, whereas AUPRC weighs false positives at a threshold τ with the inverse of the model's likelihood of outputting any scores greater than τ (Theorem 1). This result shows that we can reason about the suitability of optimizing or selecting AUROC vs. AUPRC based on whether we care more about reducing false positives above low thresholds or high thresholds. In particular,2) AUROC favors model improvements uniformly over all positive samples, whereas AUPRC favors improvements for samples assigned higher scores over those assigned lower scores (Theorem 2). This indicates that the key factor differentiating the utility of AUROC or AUPRC as an evaluation metric is not class imbalance at all, but it is rather based on the target use case of the model in question. See Figure 1 for a visual explanation. It also reveals that AUPRC can amplify algorithmic biases. In particular, 3) AUPRC can unduly prioritize improvements to higher-prevalence subpopulations at the expense of lower-prevalence subpopulations, raising serious fairness concerns in any multi-population use cases (Theorem 3).In this work, we establish these three claims theoretically via synthetic experiments and real-world validation on popular public fairness datasets. In addition, we demonstrate through an extensive, large-language model aided, literature review of over 1.5 million scientific papers that Claim 1 has been used to motivate numerous improper uses of AUPRC relative to AUROC across high-stakes domains like healthcare and in several established venues, including AAAI, NeurIPS, ICML, ICLR, Cancer Cell, Nature Journals, PNAS, and more. Through this paper, we hope to shed light on the nuances of appropriate evaluation and provide key guidance to limit future misuse of evaluation metrics in the scientific and machine learning communities. 
theoretical analyses: 
 Please note that all notation used is defined in Appendix Section C.In this section, we introduce Theorem 1, which is as follows: Theorem 1. Let X , Y = 0, 1 represent a paired feature and binary classification label space from which i.i.d. samples (x, y) ∈ X × Y are drawn via the joint distribution over the random variables x, y. Let f : X → (0, 1) be a binary classification model outputting continuous probability scores over this space. Then,AUROC(f ) = 1 -E t∼f (x)|y=1 [FPR(f, t)] AUPRC(f ) = 1 -P y (y = 0)E t∼f (x)|y=1FPR(f, t) P x (f (x) > t) dataset consisting of two distinct subpopulations, A ∈ {0, 1}. If we order samples in ascending order of output score, each misordered pair of samples (e.g., mistake 1-4) represents an opportunity for model improvement.Theorem 2 shows that a model's AUROC will improve by the same amount no matter which mistake you fix, while the model's AUPRC will improve by an amount correlated with the score of the sample. b) When comparing models absent a specific deployment scenario, we have no reason to value improving one mistake over another, and model evaluation metrics should therefore improve equally regardless of which mistake is corrected. c) When false negatives have a high cost relative to false positives, evaluation metrics should favor mistakes that have lower scores, regardless of any class imbalance. d) When limited resources will be distributed among a population according to model score, in a manner that requires certain subpopulations to all be offered commensurate possible benefit from the intervention for ethical reasons, evaluation metrics should prioritize the importance of within-group, high-score mistakes such that the highest risk members of all subgroups receive interventions. e) When false positives are expensive relative to false negatives and there are no fairness concerns, evaluation metrics should favor model improvements in decreasing order with score.We provide the proof in Appendix Section D. The two key intuitions are that integrating over the TPR is equivalent to taking the expectation over the induced distribution of positive sample scores, and that via Bayes rule, Prec(f, τ ) = 1 -P y (y = 0) FPR(f,τ ) Px(f (x)>τ ) . Despite its simplicity, Theorem 1 has far-reaching implications. Namely, it reveals that the only difference between AUROC and AUPRC with respect to model dependent parameters (i.e., omitting the dependence of AUPRC on the fixed prevalence of the dataset, which is not model varying) is that optimizing AUROC equates to minimizing the expected false positive rate over all positive samples in an unweighted manner (equivalently, in expectation over the distribution of positive sample scores) whereas optimizing AUPRC equates to minimizing the expected false positive rate over all positive samples weighted by the inverse of the model's "firing rate" (P x (f (x) > τ )) at the given positive sample score. This preference can be crystallized when we examine how AUROC vs. AUPRC would prioritize correcting indivisible units of model improvements, termed "mistakes" which we will discuss next.Understanding how a given evaluation metric prioritizes correcting various kinds of model mistakes or errors offers significant insight into when that metric should be used for optimization or model selection. To examine this topic for AUROC and AUPRC, consider the following definition of an "incorrectly ranked adjacent pair", which we will colloquially refer to as a "model mistake": Definition 2.1. Let f, X , Y, x, y be defined as in Theorem 1. Further, let us suppose we have sampled a static dataset from x, y for evaluation which will be denoted X, y = {(x 1 , y 1 ), . . . , (x N , y N )}, for x i ∈ X , y i ∈ {0, 1}, and N ∈ N. We assume for convenience that f is an injective map and all x i are distinct (i.e., ∀(i, j) :x i ̸ = x j which, by injectivity of f , implies that f (x i ) ̸ = f (x j )).We say that (x i , x j ) are an incorrectly ranked adjacent pair and thus that the model makes a "mistake" at samples (x i , x j ) if:1. y i = 1 and y j = 0 2. f (x i ) < f (x j ) 3. ∄x k such that f (x i ) < f (x k ) < f (x j ).Essentially, Definition 2.1 states that a mistake occurs when a model assigns adjacent probability scores to a pair of samples with discordant labels, as shown in Figure 1 . With this in mind, we can then introduce Theorem 2 which states that AUROC improves by a constant amount regardless of which mistake is corrected for a given model and dataset whereas AUPRC improves more when the mistake corrected occurs at a higher score than when it occurs at a lower score: Theorem 2. Define f, X , X, y and N as in Definition 2.1. Further, suppose without loss of generality that the dataset X is ordered such that f (x i ) < f (x i+1 ) for all i. Then, let us define M = {i|(x i , x i+1 ) is an incorrectly ranked adjacent pair for model f }. Define f ′ i to be a model that is identical to f except that the probabilities assigned to x i and x i+1 are swapped:f ′ i :    f (x) if x / ∈ {x i , x i+1 } f (x i+1 ) if x = x i f (x i ) if x = x i+1 . Then, AUROC(f ′ i ) = AUROC(f ′ j ) for all i, j ∈ M , and AUPRC(f ′ i ) < AUPRC(f ′ j ) for all i, j ∈ M such that i < j.The proof for Theorem 2 can be found in Appendix E. This proof simply stems from the fact that correcting a single mistake (x i , x j ) (as defined in Definition 2.1) always changes the false positive rate by the same amount, and only changes it at the threshold f (x i ). This, combined with the formalization of AUROC and AUPRC in Theorem 1, establishes the proof. Note that this Theorem can be trivially extended to include a case where ties are possible simply by noting that "swapping" two samples x i and x j in the manner of the theorem results in no change to either AUROC or AUPRC, and similarly by the same reasoning separating any tie in the appropriate direction will improve AUROC uniformly over samples and will improve AUPRC in a manner monotonic with model score.The reliance on a model's firing rate revealed in Theorem 1 and the optimization behavior in Theorem 2 reveals significant issues with the fairness of AUPRC. In particular, in this section we introduce Theorem 3: Theorem 3. Let f, X , X, y, N, M, and f ′ j all be defined as in Theorem 2. Further, suppose that in this setting the domain X now contains an attribute defining two subgroups, A = {0, 1}, such that for any sample (x i , y i ), a i denotes the subgroup to which that sample belongs. Let f be perfectly calibrated for samples in subgroup a = 0, such that P y|a,x (y = 1|a = 0, f (x) = t) = t. Then,lim P y|a (y=1|a=0)→0 P a i = a i+1 = 1 i = arg max j∈M AUPRC(f ′ j ) = 1.Essentially, Theorem 3 (proof provided in Appendix F) shows the following. Suppose we are training a model f over a dataset with two subpopulations: Population a = 0 and a = 1. If the model f is calibrated and the rate at which y = 1 for population a = 0 is sufficiently low relative to the rate at which y = 1 for population a = 1, then the mistake that, were it fixed, would maximally improve the AUPRC of f will be a mistake purely in population a = 1. This demonstrates that AUPRC provably favors higher prevalence subpopulations (those with a higher base rate at which y = 1) under sufficiently severe prevalence imbalance between subpopulations.Note that this property is, generally speaking, not desirable. In particular, this property establishes that in settings where model fairness among a set of subpopulations in the data is important, AUPRC should not be used as an evaluation metric due to the risk that it will introduce biases in favor of the highest prevalence subpopulations. We validate this result empirically over both synthetic and real-world data in Section 3, demonstrating that the import of Theorem 3 is not merely limited to an analytical curiosity but can have real-world impact on algorithmic disparities in practice. Furthermore, note that this theorem does not indicate that AUPRC will be superior to AUROC for differentiating a low prevalence (or low risk) subpopulation relative to a high-risk subpopulation, a property that is sometimes attributed to AUPRC in the literature. Rather, Theorem 3 shows that maximizing AUPRC will be more likely to optimize solely within the high-risk subgroup, rather than optimizing to differentiate across subgroups, as low-risk subgroup samples will only occur in lower-score regions under severe class imbalance. 
experimental validation: 
 In this section, we establish via synthetic and real-world experiments that Theorem 3 is not merely an analytical effect but has real world consequences on the implications of optimizing or performing model selection via AUPRC.In this section, we use a carefully constructed synthetic optimization procedure to demonstrate that, when all other factors are equal, optimizing by or performing model selection on the basis of AUPRC vs. AUROC risks excacerbating algorithmic disparities in the manner predicted by Theorem 3. For analyses under more realistic conditions with more standard models, see our real-world experiments in Section 3.2.Experimental Setup. Let y ∈ {0, 1} be the binary label, s ∈ [0, 1] be the predicted score, and a ∈ {1, 2} be the subpopulation. We fix P y|a (y = 1|a = 1) = 0.05 and P y|a (y = 1|a = 2) = 0.01. We sample a dataset for each group D a = {(s 1 , y 1 ), ..., (s na , y na )}, such that AUROC(D 1 ) ≈ AUROC(D 2 ) ≈ AUROC(D 1 ∪ D 2 ) = 0.85 (See Appendix G.1; A target AUROC of 0.65 was also profiled in Appendix Figure 5 ).Our main experimental challenge is to determine how to simulate "optimizing" or "selecting" a model by AUROC or AUPRC. Simulating optimizing by these metrics allows us to explicitly assess how the use of either AUPRC or AUROC as an evaluation metric in model selection processes such as hyperparameter tuning or architecture search, can translate into model-induced inequities in dangerous ways. We explore two approaches here. First, we can simply correct the atomic mistake that maximally improves AUROC or AUPRC in each optimization iteration. In our experiments, we use n 1 = n 2 = 200 and optimize for 50 steps for this experiment. This is the most straightforward optimization procedure to analyze, but it is unrealistic. In real optimization scenarios, larger model changes will be made at once, and a model will have an opportunity to degrade performance in some regions in order to improve it in others.Next, we profile an optimization procedure that randomly permutes all the (sorted) model scores up to 3 positions (See Appendix G.3 for details). This has the effect of randomly adjusting all model scores, and can worsen model performance under some random permutations, but offers precisely the same "optimization capacity" to the low and high prevalence subgroups. To ensure the model is under some optimization constraint (and therefore does not always find the "perfect" permutation to maximize both metrics identically), we allow the model to sample only 15 possible permutations before choosing the best option. This means the system will be forced to navigate optimization trade-offs between which permutations improve the right regions of the score most effectively among its limited set. We use n 1 = n 2 = 100 for these experiments and optimize for 25 total steps. Across both settings, we run these experiments across 20 randomly sampled datasets and show the mean and an empirical 90% confidence interval around the mean in Figure 2 . We present a formal mathematical formulation of these perturbations, as well as profile a third random perturbation method, in Appendix G.3.Our results demonstrate the impact of the optimization metric on subpopulation disparity. In particular, in Figure 2 , we observe a notable disparity introduced when optimizing under the AUPRC metric regardless of the optimization procedure. This is evident in the performance metrics across the high and low prevalence subpopulations, which exhibit significant divergence as the optimization process favors the group with higher prevalence. In the more realistic, randompermutation optimization procedure (Figure 2d ), this even results in a decrease in the AUROC for the low prevalence subgroup. In comparison, when optimizing for overall AUROC, the AUROC of both groups increase together. Note that we show the effect of this optimization on the AUPRC metric, which shows very similar trends, in Appendix Figure 4 . These results demonstrate explicitly that not only does optimizing for AUPRC differ greatly than for AUROC, as has been noted historically by researchers developing explicit AUPRC optimization schemes [409] , but it in fact does in an explicitly discriminatory way in very realistic scenarios.To demonstrate the generalizability of our finding to the real world, we evaluate fairness gaps induced by AUROC and AUPRC selection on four common datasets in the fairness literature [441, 99, 205] .Datasets. We use the following four tabular binary classification datasets: adult [17] , compas [14] , lsac [413] , and mimic [178] . In each dataset, we consider both sex and race as sensitive attributes. To mimic the setting of our theorems, we balance each dataset by the sensitive attribute during both training and test, by randomly subsampling the majority group. Further details about each dataset, as well as preprocessing steps, can be found in Appendix H.Experimental setup. We train XGBoost models [65] on each dataset. For each task, we iterate over a grid of per-group weights in order to create a diverse set of models that favor different groups.For each setting of task and per-group weight, we conduct a random hyperparameter search [37] with 50 runs. Though more complex hyperparameter search methods such as BOHB [100] or TPE [36] might lead to better performance, random searches are far more popular and practical, and have been used in popular benchmarking libraries [131, 337] .We evaluate the validation set overall AUROC and AUPRC. We also evaluate the test set AUROC gap and AUPRC gap between groups, where gaps are defined as the value of the metric for the higher prevalence group minus the value for the lower prevalence group. Based on our theorems, our hypothesis is that overall AUPRC should be more positively correlated with the signed AUROC gap than overall AUROC, indicating that it better favors the higher prevalence group, especially when the prevalence ratio between groups is high. To test this hypothesis, we evaluate the Spearman correlation coefficient between these quantities. We repeat this experiment 20 times, with different random data splits, to obtain a 95% confidence interval.Results. In Figure 3 , we plot the difference in the Spearman correlation coefficient of the AUROC gap versus the overall AUPRC, and AUROC gap versus overall AUROC. We observe mixed results in datasets with low prevalence ratio. In dataset with higher prevalence ratio, we find that overall AUPRC is more positively correlated with the AUROC gap than overall AUROC, indicating that AUPRC more aggressively favors the higher prevalence group. We emphasize that the prevalence ratios observed in these real-world datasets is much lower than the ratio of 5 used in our synthetic experiments, which may account for the mild effect observed. To see raw results from these experiments, see Appendix Figure 7 . Similar results for neural network classifiers can be found in Appendix Section H.3.Next, in Appendix Figure 8 , we plot the difference in the Spearman's ρ from Figure 3 , versus the prevalence gap. We find that there is a statistically significant correlation between the two (Spearman's ρ = 0.905, p = 0.002). Thus, while our power to detect a prevalence mediated AUPRC bias amplification effect is limited due to the limited prevalence disparities in these datasets, we nonetheless observe a strong positive correlation between the extent of the prevalence mismatch between the low and high prevalence group and the amount that AUPRC favors the high prevalence group over AUROC. In other words, our results show that across these fairness datasets and attributes, as the prevalence disparity grows more extreme, we observe a statistically significant corresponding increase in the extent to which AUPRC introduces algorithmic bias, exactly in accordance with what Theorem 3 suggests.4 When Should One Use AUPRC vs. AUROC?In Sections 2 and 3, we have shown that AUPRC is not universally superior in cases of class imbalance (and that instead, it merely preferentially optimizes high-score regions over low-score regions) and that it also poses serious risks to the model fairness in settings where subgroup prevalences differ. In light of this, how should we revise Claim 1 to reflect when we actually should use AUPRC instead of AUROC or vice versa? Below, we explore this question and provide practical guidance on metric selection for binary classification, building on our theoretical results and highlight specific contexts in which one may be favorable (Figure 1 ). Note that while we provide guidance below on situations in which AUROC vs. AUPRC is more or less favorable, this is not to suggest that authors should not report both metrics, or even larger sets of metrics or more nuanced analyses such as ROC or PR curves; rather this section is intended to offer guidance on what metrics should be seen as more or less appropriate for use in things like model selection, hyperparameter tuning, or being highlighted as the 'critical' metric in a given problem scenario.For context-independent model evaluation, use AUROC: For model evaluations conducted outside of specific deployment contexts, where the differential costs of errors are undefined, the necessity for a metric that impartially values improvements across the entire model output space becomes paramount. As shown in Figure 1a . As it is not known in advance where samples of interest will live in the output space, nor are particular cost ratios known, there should be no preference for mistake correction. Therefore, in this setting AUROC is favorable as it uniformly accounts for every correction, offering a fair assessment irrespective of decision thresholds.For deployment scenarios with elevated false negative costs, use AUROC: In applications where the consequences of false negatives are especially high, such as in the early screening for critical illnesses like cancer (Figure 1c ), a primary goal of the model will be to achieve the fewest missed cancer cases; equating to prioritizing model recall. In such a scenario, the most important mistakes to correct occur at lower score thresholds, as high-score mistakes will not change which positive samples are missed in deployment settings (as chosen thresholds are likely to be low). This behavior is the inverse of what AUPRC prioritizes, demonstrating that in such situations, AUROC should be preferred over AUPRC.For ethical resource distribution among diverse populations, use AUROC: When faced with the challenge of ethically allocating scarce resources across a broad population, necessitating equitable benefit distribution among subgroups (Figure 1d ), one must avoid prioritizing model improvements that selectively favor one subpopulation. As AUPRC will target high-score regions selectively, it risks unduely favoring high-prevalence subpopulations, as shown in Theorem 3 and Figures 2 and 3 . Even though in this resource distribution problem, high-score regions are selectively important compared to low-score regions, the fact that in this problem, we must prioritize across all subpopulations equally means that AUPRC's global preference is untenable as it could induce bias.For reducing false positives in high-cost, single-group intervention prioritization or information retrieval settings, use AUPRC: In scenarios where the cost associated with false positives significantly outweighs that of false negatives, absent of equity concerns-such as in selecting candidate molecules from a fragment library for drug development trials, where only the most promising molecules will proceed to costly experimental validation (Figure 1e )-the metric of choice should facilitate a reduction in high-score false positives. This necessitates a focus on correcting high-score errors, for which AUROC might not be ideal due to its uniform treatment of errors across the score spectrum, potentially obscuring improvements in critical high-stake decisions.5 Literature Review: Examining how Claim 1 Became so WidespreadClaim 1 states that "AUPRC is better than AUROC in cases of class imbalance" and is widespread in the literature. Via both a manual literature search and an automated search of over 1.5M arXiv papers (see Appendix I for methodology), we observed 424 publications making this claim. 3 This widespread disemmination of Claim 1 has a significant impact on the validity of scientific discourse. We observed examples of high-profile papers operating in medically critical settings where high-recall is a key priority evaluating ML systems via AUPRC due to their task's underlying class imbalance [399] ; papers focusing on fairness critical applications relying on AUPRC due to this claim, even while our results demonstrate AUPRC has major problems in the fairness regime [366, 306] , and numerous other papers perpetuating this source of scientific misinformation.Among the 424 papers we discovered referencing this claim, 167 did so with no associated citation. These papers were published in a wide range of venues, including high profile venues such as NeurIPS, ICML, and ICLR. This reflects not only the widespread belief in this claim, but also that we may be too comfortable making seemingly "correct" assertions without appropriate attribution in ML today. Further, Among the 257 that reference this claim and cite a source for this assertion, 135 do not cite any papers that actually make this claim in the first place. Most often, papers erroneously attribute this claim to [83] , which was cited as a source for this claim 144 times. While [83] makes many interesting, meaningful claims about the ROC and PR curves, and does argue that the precision-recall curve may be more informative than the ROC in cases of class imbalance it never asserts that the area under the PR curve should be preferred over the area under the ROC in cases of class imbalance. This distinction is important, because while curves can be used to simultaneously communicate many different performance statistics to their viewers across different FPR/TPR or Precision/Recall criteria, and therefore should be assessed primarily as communication tools to deployment experts, areas under these curves are single-point summarizations which are primarily used for deployment-agnostic model comparison, which is a very different use case.Even when appropriate papers are cited, the valid settings in which AUPRC should be preferred (see Section 4 for examples) are often over-shadowed by significant over-generalizations to preferring AUPRC in all settings featuring class imbalance. For example, claims such as that "precision-recall curves are more informative of deployment metrics" are often used to justify why AUPRC should be used in all cases of class imbalance, rather than just in cases where the relevant deployment metrics are most directly associated with the PR curve. Another class of arguments made in favor of Claim 1 is rooted in claims that AUROC is poor in cases of class imbalance because its scores are misleadingly high. While this argument can reflect a meaningful limitation of the communication value of the ROC or the AUROC, comments about singleton metric results (rather than model comparison through metric values) are inherently orthogonal to the goal of model evaluation. In other words, what matters for model evaluation is not how high a given metric is, but rather the extent to which the metric meaningfully captures the right improvements in the model in the right ways. The widespread nature of Claim 1 has also led researchers astray when exploring new optimization procedures for AUPRC, by advocating for the importance of AUPRC when processing skewed data, even in domains such as medical diagnoses that often have high false negative costs relative to false positive costs [409] . For a more extensive breakdown of the arguments we observed in the literature and the sources making them, see Appendix Tables 2 and 3 . 
limitations and future works: 
 There are still a number of areas for further improvement and future work. Firstly, our theoretical findings can be refined and generalized to, e.g., take into account the difficulty of the target task (which may differ between subgroups), not require models to be calibrated (in the case of Theorem 3), or specifically take into account more than 2 subpopulations for more nuanced comparisons beyond what can be inferred through pairwise comparisons between subpopulations, where our results would naturally apply. Further, extending our real-world experiments to more fairness datasets and identifying more nuanced ways to probe the impact of metric choice on disparity measures would significantly strengthen this work. These analyses can also be extended to consider other metrics, such as the area under the precision-recall-gain curve [104] , the area under the net benefit curve [384, 307] , and single-threshold, deployment centric metrics as well. In addition, one of the largest limitations of Theorem 3 is its restrictive assumptions, in particular the requirement of perfect calibration. A ripe area of future work is thus to investigate how we can soften our analyses for models with imperfect calibration or to determine whether or not our results imply anything about the viability or safety of post-hoc calibration of models optimized either through AUPRC or AUROC. 
conclusion: 
 This study interrogates the pervasive assumption within the ML community that AUPRC is a better evaluation metric than AUROC in class-imbalanced settings. Our empirical analyses and literature review reveal several concrete findings that challenge this notion. In particular, we show that while optimizing for AUROC equates to minimizing the model's FPR in an unbiased manner over positive sample scores, optimizing for AUPRC equates to minimizing the FPR specifically for regions where the model outputs higher scores relative to lower scores. Further, we show both theoretically and empirically over synthetic and real-world fairness datasets that AUPRC can be an explicitly discriminatory metric through favoring higher-prevalence subgroups.In summary, our research advocates for a more thoughtful and context-aware approach to selecting evaluation metrics in machine learning. This paradigm shift, favoring a balanced and conscientious approach to metric selection, is essential in advancing the field towards developing not only technically sound, but also equitable and just models. 
 
 Appendix: This research paper challenges the conventional wisdom regarding the superiority of the AUPRC over AUROC in binary classification tasks with class imbalance and has several ethical implications and impacts.Our analysis reveals that the preference for AUPRC in certain ML applications may not be empirically justified and could inadvertently amplify algorithmic biases. This calls for a re-examination of prevalent metrics within ML, especially in high-stakes domains like healthcare, finance, and criminal justice where biased models can have profound societal repercussions. The tendency of AUPRC to disproportionately favor models with higher prevalence of positive labels could exacerbate existing disparities, underscoring the ethical need for rigorous validation and scrutiny of evaluation metrics.Additionally, our use of large language models for literature analysis demonstrates a novel approach in scrutinizing and re-evaluating long-standing assumptions in ML. This method could set a precedent for more comprehensive and robust scientific investigations in the field, fostering a culture of empirical rigor and ethical awareness.The ethical dimension of our work lies in the spotlight it casts on metric selection in ML model evaluation. The potential of metrics like AUPRC to skew model performance favoring certain groups raises pressing concerns about fairness in algorithmic decision-making. This is particularly critical when algorithms influence key decisions affecting individuals and communities.While we use the COMPAS dataset for recividism prediction in this work, we recognize the many societal issues with automated predictions of recidivism [93] . We utilize this dataset as it is a commonly used dataset in the fairness literature, but do not advocate for deployment of these models in any way.Our study contributes to the technical discourse on metric behaviors in ML and serves as a cautionary tale against uncritically embracing established norms. It underscores the imperative for careful metric selection aligned with ethical principles and fairness objectives in ML, highlighting the far-reaching consequences of these choices in shaping societal outcomes and advancing the field of ML.All code is available at https://github.com/hzhang0/auc_bias and https://github.com/ Lassehhansen/ArxivMLClaimSearch.Let X , Y = 0, 1 represent a paired feature and binary classification label space from which i.i.d. samples (x, y) ∈ X × Y are drawn via the joint distribution over the random variables x, y. Let f θ : X → (0, 1) be a binary classification model parametrized by θ ∈ R d for some d ∈ N outputting continuous probability scores over this space.We define random variable s = f θ (x) to be the distribution of scores output by the model over input samples. Throughout the paper, θ may be omitted if it is clear from context. We will occasionally also use the notation s + and s -to reflect the conditional distributions of model outputs conditioned on the label being 1 or 0, respectively:s + = f (x)|y = 1 s -= f (x)|y = 0.Let N P be the number of data points with a positive label and N N the number with a negative label. Further, given a threshold τ , defineTP θ (τ ) = {x i ∈ X|p (θ) i ≥ τ, y i = 1} FN θ (τ ) = {x i ∈ X|p (θ) i < τ, y i = 1} TN θ (τ ) = {x i ∈ X|p (θ) i < τ, y i = 0} FP θ (τ ) = {x i ∈ X|p (θ) i ≥ τ, y i = 0} FR(f, τ ) = P p (p > τ ) TPR θ (τ ) = TP θ (τ ) TP θ (τ ) + FN θ (τ ) = P x|y=1 (f (x) > τ ) = P s|y=1 (s > τ ) = P (s + > τ ) FPR θ (τ ) = FP θ (τ ) FP θ (τ ) + TN θ (τ ) = P x|y=0 (f (x) > τ ) = P s|y=0 (s > τ ) = P (s -> τ ) Prec θ (τ ) = TP θ (τ ) TP θ (τ ) + FP θ (τ ) = P y|f (x)>τ (y = 1) = P y|s>τ (y = 1)Lastly, recallAUROC θ = 1 0 TPR θ dFPR θ dτ dτ = 1 0 TPR θ dFPR θ = 1 - 1 0 FPR θ dTPR θ AUPRC θ = 1 0 Prec θ dTPR θ dτ dτ = 1 0 Prec θ dTPR θ D Proof of Theorem 1Recall that all notation is defined formally in Appendix C.Here, we prove Theorem 1, which states Theorem 1. Let X , Y = 0, 1 represent a paired feature and binary classification label space from which i.i.d. samples (x, y) ∈ X × Y are drawn via the joint distribution over the random variables x, y. Let f : X → (0, 1) be a binary classification model outputting continuous probability scores over this space. Then,AUROC(f ) = 1 -E t∼f (x)|y=1 [FPR(f, t)] AUPRC(f ) = 1 -P y (y = 0)E t∼f (x)|y=1 FPR(f, t) P x (f (x) > t)Proof. Recall that AUROC and AUPRC are as follows:AUROC = 1 0 TPR dFPR = 1 - 1 0 FPR dTPR AUPRC = 1 0Prec dTPR However, we can further clarify these by leveraging the fact that TPR(τ ) = P s+ (s + > τ ) = 1 τ s + (t)dt, as below:1 0 g(τ )d(TPR(τ )) = 0 1 g(τ ) dTPR(τ ) dτ dτ = 0 1 g(τ ) d dτ (P s+ (s + > τ ))dτ = 0 1 g(τ ) d dτ 1 τ s + (t)dt dτ = 0 1 g(τ )(-s + (τ ))dτ = E s+ [g] So, AUROC = 1 -E s+ [FPR] & AUPRC = E s+ [Prec].To further simplify, we expand Prec via Bayes rule:Prec = 1 -P y|s>τ (y = 0) = 1 -P s|y=0 (s > τ ) FPR(τ ) P y (y = 0) P s (s > τ )Thus,AUROC(f ) = 1 -E t∼s+ [FPR(f, t)] = 1 -E t∼f (x)|y=1 [FPR(f, t)] AUPRC(f ) = E t∼s+ [Prec(f, t)] = 1 -P y (y = 0)E t∼s+ FPR(f, t) P s∼s (s > t) = 1 -P y (y = 0)E t∼f (x)|y=1 FPR(f, t) P x (f (x) > t)as desired.Synthetic validation of Theorem 1 can also be found in our public code. Note that this formulation of AUPRC reflects earlier, different formulations of AUPRC, such as those found in the AUPRC optimization literature [409] .Proof. As this model is calibrated perfectly, we have that p + (t) = P s|y=1 (s = t) = P y|s=t (y = 1)p s (t) P y (y = 1) = t P y (y = 1)p + (t) + P y (y = 0)p -(t) P y (y = 1)= tp + (t) + t P y (y = 0) P y (y = 1) p -(t).Thus, p + (t) = t 1-t Py(y=0)Py(y=1) p -(t) as desired. Lemma 2. Let a model f be perfectly calibrated and yield score distributions for positive and negative samples from probability density functions p + and p -, with overall distribution given by p(t) = P y (y = 1)p + (t) + P y (y = 0)p -(t). Then for all τ ∈ (0, 1), FR(f, τ ) ≤ Py(y=1) τ.Proof. By definition, we haveFR(f, τ ) = 1 τ P y (y = 1)p + (t) + P y (y = 0)p -(t)dt = 1 τ P y (y = 1)p + (t) + P y (y = 1) 1 -t t p + (t)dt = P y (y = 1) 1 τ 1 t p + (t)dt,where step two leverages the fact that f is perfectly calibrated and the result in Lemma 1.As t ≥ τ , 1 t ≤ 1 τ . Then, as p + (t) ≥ 0,1 τ 1 t p + (t)dt ≤ 1 τ 1 τ p + (t)dt. Finally, as 1 0 p + (t)dt = 1, we see that 1 τ p + (t)dt ≤ 1. Therefore, FR(f, τ ) = P y (y = 1) 1 τ 1 t p + (t)dt ≤ P y (y = 1) • 1 τ • 1 = P y (y = 1) τ .Theorem 3. Let f, X , X, y, N, M, and f ′ j all be defined as in Theorem 2. Further, suppose that in this setting the domain X now contains an attribute defining two subgroups, A = {0, 1}, such that for any sample (x i , y i ), a i denotes the subgroup to which that sample belongs. Let f be perfectly calibrated for samples in subgroup a = 0, such that P y|a,x (y = 1|a = 0, f (x) = t) = t. Then,lim P y|a (y=1|a=0)→0 P a i = a i+1 = 1 i = arg max j∈M AUPRC(f ′ j ) = 1.Proof. Given Theorem 2, the atomic mistake that would, upon correction, result in the largest improvement to AUPRC is the mistake which occurs at maximal score (as this minimizes the firing rate, which is the denominator in the weighting term for AUPRC). Suppose that at threshold τ , the probability that a mistake will occur above score τ in subgroup 1 with N samples drawn is at least δ ∈ (0, 1]. As the parameters for subgroup 1 are fixed as we vary the prevalence for subgroup 2, τ can be seen as a constant with respect to the limit we are taking.But, by Lemma 2 and by the fact that f is perfectly calibrated for subgroup 2, we know that the probability that f will output a score for sample 2 regardless of its label that exceeds τ is upper bounded by p (2) y τ . In the limit as py tends to zero, the probability that any probabilities will be observed at our greater than τ from subgroup 2 likewise tends to zero. This means that while the probability that we observe a mistake from subgroup 1 stays fixed at at least δ > 0, the probability that we could observe any mistake that involves any sample from subgroup 2 (either a cross-group mistake or a purely subgroup 2 mistake) tends to zero, establishing the claim.A key component of our synthetic experiments is the ability to sample a set of model scores and labels randomly that will have a target AUROC. To do this, we use the following procedure (which may or may not be previously known; we derived it from scratch for this work, but make no claim about its novelty). Let N be the number of points we are sampling overall, and N + be the number of positive points being sampled (which is dictated by the user given prevalence). , we can count the number of positive samples that have scores less than any value in this window (i) and the number that have scores greater than any value in this window (which will be N + -i).3. As the target AUROC is the probability that a randomly sampled negative will be ranked more highly than a randomly sampled positive, we can leverage the number of less-than positive scores i and greater than positive scores N + -i to compute the probability that a randomly sampled negative score will live in the window (pEQUATION) via the binomial distribution. 4. Now, to sample a random negative, we simply first sample a random window (pEQUATION) with the probabilities assigned above, then uniformly sample a value p -within that window. We can repeat this process to the target number of negative samples N -N + to form our final set of scores. 5. If desired, the output scores can further be scaled to have expectation given by the dataset's prevalence or can be adjusted via a calibration method to be calibrated given the assigned labels. Both procedures can be done without affecting the AUROC. Note that as any calibrated model will have expected probability given by the label's prevalence (See Appendix G.2), the former condition is strictly weaker than the latter.The procedure outlined above guarantees that, in expectation, the AUROC of the generated set of scores and labels will be precisely the target AUROC. However, if you apply this procedure indpendently across different sample subpopulations, this guarantee can only be applied on each subpopulation individually, and not necessarily on the overall population due to the unspecified xAUC term. However, in practice, for the experiments we ran here, that impact neither meaningfully impacts our experiments nor were the joint AUROCs sufficiently different from the target AUROC to warrant a more complex methodology.Let p be a random variable describing the probabilities (not scores) output by the model over the input distribution defined by the data generative function. If a model is calibrated, this means that P y|p (y = 1|p = q) = q -that the probability that the label for a given point is 1 is given precisely by the models output probability for that sample. With that in mind, we have:E p [q] = E p P y|p (y = 1|p = q) = 1 0 P y|p (y = 1|p = q)p p (q)dq = 1 0 P y,p (y = 1, p = q)dq = P y (y = 1)G.3 Details on optimization procedures M1. Adding Random Noise. We sample a vector ϵ ∈ R n , where each element is uniformly drawn from [-δ, δ]. We compute the selection metric for S ′ = S + ϵ. We repeat this procedure 100 times, and return the S ′ that achieves the maximum value for the selection metric. We vary the maximum magnitude of the perturbation δ ∈ [0, 0.1] in a grid. Results for this setting are shown in Figure 6 . We note that this approach is subtly biased in favor of the lower-prevalence group. In particular, because scores for the low-prevalence group tend to be "squished" into a smaller region of the probability space, a random perturbation of fixed magnitude will proportionally induce more score permutations in the low-prevalence group than the high-prevalence group, which affords the system greater capacity to improve the model for the low-prevalence group independent of the choice of AUROC or AUPRC. M2. Sequentially Fixing Atomic Mistakes. We sequentially correct atomic mistakes, as defined in Figure 1 . At each step, we first discover the set of all atomic mistakes M . To maximize AUROC, we randomly select a pair (S i , S j ) ∈ M , and swap their scores in S, i.e. S ′ i = S j , S ′ j = S i . To maximize AUPRC, we swap the scores for the pair (S i , S j ) = arg max (si,sj )∈M s j . We repeat this process for 50 steps, with each one sequentially fixing another atomic mistake in S. Results for this setting are shown in Figures 4b and 4a . M3. Sequentially Permuting Nearby Scores. We first sort S and Y such that S is in ascending order. We apply a random permutation to S by re-indexing it using a random ordering, but such that scores are not shuffled too far from their original index. Let σ be the ordered sequence (1, 2, ..., n). Define Ω to be the set of all permutations of σ, such that for all ω ∈ Ω, |ω i -σ i | ≤ γ for i ∈ {1, ..., n}. At each step, we sample ω ∈ Ω with γ = 3 twenty times, where each ω corresponds to a new candidate ordering of S. We compute the selection metric for each of the twenty orderings, and return S ′ to be the score permutation that achieves the maximum value for the selection metric. We repeat this procedure for 25 steps, setting S at each step to be the S ′ output from the previous step. Results for this setting are shown in Figures 4d and 4c . Low-prevalence Group High-prevalence GroupStepStepStepStep AUROC (5%-95%) AUROC (5%-95%) Low-prevalence Group High-prevalence GroupStepStepStep The initial phase of our comprehensive literature search involved the acquisition of datasets from both the ArXiv preprint server (through the RedPajama dataset on Hugging Face), as well as from a subset of years of NeurIPS, ICML, ICLR, ACL, and CVPR conference proceedings (all scraped manually). 1. Preliminary Analysis with GPT-3.5: We utilized OpenAI's GPT-3.5 model for an initial round of AI-assisted analysis for the arXiv dataset. This model identified and extracted papers making explicit claims regarding the comparative effectiveness of AUPRC over AUROC in scenarios of class imbalance, reducing our dataset from Arxiv to 2,728 papers.2. Further Refinement Using GPT-4.0 Turbo: To refine our dataset further, we employed the GPT-4.0 Turbo model. Out of the 2,728 papers scrutinized from Arxiv using this model, 201 were found to be relevant. For NeurIPS, our focused search with GPT-4 resulted in identifying 2 papers of particular relevance to our thesis from the initial set that contained keywords related to both AUPRC and AUROC.• Shared Document for Collaborative Analysis: We compiled all pertinent papers, along with their respective Arxiv IDs and the claims identified by GPT-4.0 Turbo, into a shared Google document for team review. Claims made in papers were found manually, and the specific quote of the claim they made was highlighted along with whether or not they had a citation for this claim.Precision-recall curves or other associated metrics may more appropriately reflect deployment objectives than the receiver operating characteristic.[ 78, 212, 428, 279, 85, 137, 6] While this claim is true, the informativeness of the PR curve for target deployment metrics is not sufficient to conclude that the AUPRC is superior to the AUROC in all cases of class imbalance. Despite this, it is often taken to assert this more general claim without caveat. AUPRC does not depend on the number of true-negatives, so will be less optimistic than the AUROC [212, 204, 2, 258, 275, 356, 79, 90, 325, 159, 180, 301, 329, 395] As shown in Theorem 1, AUROC and AUPRC can both be naturally expressed as a function of the expectation of the model's false positive rate. More generally, lack of dependence on one quadrant among the mutually dependent four quadrants of a confusion matrix is not an informative property for the AUROC and AUPRC metrics. AUPRC will often be significantly lower, farther from optimality, and/or will grow more non-linearly as model performance improves than AUROC for low-prevalence tasks [212, 428, 257, 79] Metric utility for model comparison depends on how appropriately it prioritizes model improvements, and is therefore less about the raw magnitude of the metric and more about the situations in which the order of a set of models will differ under one metric vs. another. One could easily make AUROC yield smaller values or grow more quickly near optimality by simply exponentiating it, but this would not yield a better metric. AUPRC depends on prevalence, which is a desirable property [282] This statement is too vague to be formally evaluated; whether or not this dependence on prevalence is desirable depends on the context. For model comparison in general, we argue it is not desirable in this form as it induces the biases inhere in AUPRC previously discussed. AUPRC better captures differentiating a positive sample with high score from a "hard" negative sample ("hard" meaning one also with high score) [193] While this claim is true by Theorem 2, it is not clear why this would be desired in general; this implicitly favors comparing "hard" negatives against "easy" positives as opposed to "easy" negatives against "hard" positives. AUROC is otherwise "optimistic" in lowprevalence settings [78, 4, 417, 261, 70, 134, 425, 251, 363, 238, 6, 448, 236, 335, 58, 317, 12, 202, 183, 149] This claim is underspecified, and un-true. AUROC always means the same thing, probabilistically, and that meaning independent from class imbalance. AUPRC focues more on the positive (minority) class [393, 417, 361, 281, 22, 115, 85, 10, 275, 356, 296, 293, 82, 193, 239, 319, 288] This is unfounded; both AUROC and AUPRC are weighted expectations over the model's false positive rate-AUPRC cares more about samples in regions of low firing rate, not explicitly about positive or minority samples. AUROC can not appropriately detect models with poor recall [282] This claim is unfounded; the AUROC clearly depends on the model's recall. Besides, if recall is the measure of interest, then that should be measured explicitly.Table 2 : Various arguments and our responses to them present on a subset of papers for this claim in the literature.All code pertaining to the literature review search can be found in the following GitHub repository: https://github.com/Lassehhansen/ArxivMLClaimSearch• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Our work is about correcting a major misunderstanding in the ML community regarding a widely used evaluation metric, and the potential fairness implications of this misunderstanding. In that sense, our entire work is clearly about the potential societal impacts of this misunderstanding, and how it should be corrected. We also clearly discuss the limitations of our work in Section 6.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.See https://archive.is/qXPKu, which shows results dominated by those making Claim 1See https://chat.openai.com/share/f8f7fddb-1553-41a5-976d-789e2f3a90d6Note that throughout this section, full citations for all of the larger lists of papers we reference will be relegated to Appendix Section I for concision. 


Decision: This paper provides a comprehensive theoretical and empirical rebuttal of the folklore assertion that area under the precision-recall curve (AUPRC) is a superior metric for model comparison as compared to the more standard area under the receiving operator characteristic (AUROC) in situations where there is class imbalance. Ultimately, all reviewers were in support of acceptance of this paper and found the contributions to be innovative and valuable to the ML community. I concur with this recommendation.

Please address all suggestions made by the reviewers to improve readability and figure formatting in the camera-ready version.

Reviews: review: paper_summary: paper_summary: The paper considers a common in literature claim that AUPRC is “better” to use than AUROC for class imbalance datasets and attempts to prove it wrong based on theoretical results, empirical observations, and real-world experiments. The major focus of the paper is on the fairness gap that AUPRC exhibits. The authors provide guidance on when each of the metrics should be used and advocate for careful metric selection.
strength_weakness: strength_weakness: 1. It is unusual to see in the synthetic example in Section 3 that AUPRC is being optimized for the classification setting (as it is more common to use AUPRC as an evaluation metric). Could you please elaborate on when it can potentially be useful? 
2. For the synthetic dataset example, since everything is controllable, including the number of mistakes, does it make sense to learn boosted trees or any other model of the authors' choice for different levels of mistakes and report AUROC/AUPRC instead of the proposed optimized procedure?
3. Since AUPRC focuses on the positive class and does not use true negatives, it is "expected" by definition of this metric to increase existing fairness gaps in the positive (minority) class. Could you please elaborate on what Theorem 3 adds beyond the definition?
4. The incorrectly ranked adjacent pair mistake model seems to favor AUROC over AUPRC. By definition, AUROC measures the ability of the model to distinguish between positives and negatives across all possible thresholds, so fixing this mistake improves AUROC uniformly. By definition of AUPRC, correcting such a mistake can have a significant or minor impact depending on the threshold.
5. While p-values and checking for for statistical significance is important, in addition for the experimental results on real-world datasets, could you please provide actual values of AUROC and AUPRC (mean and variance), for example in a bar plot, similar to what is typically reported in fairness literature?
6. The broader idea of the paper—to advocate for the careful selection of metrics—is highly welcomed. AUROC and AUPRC are different metrics with different goals, just as ROC and PR curves are. They can provide different insights when evaluating data with class imbalance. Wouldn’t it make sense to encourage readers to explore both AUROC and AUPRC, as well as ROC and PR curves, under class imbalance, given the authors' findings and reference [83] instead of proposing to use one metric as in Section 4?

Minor:
Some figure legends seem to be cut off in the appendix figures.
questions: questions: 1. What causes the variance to be so high in Figure 2?
2. Could you please elaborate more on what 'Prevalence (Higher)' and 'Prevalence (Lower)' measure in Table 1? If class imbalance, shouldn’t they sum up to 1?
limitations: limitations: The authors discuss the limitations of the paper.
paper_summary: paper_summary: The paper proves that AUPRC weights mistakes in higher score ranges higher, while AUROC weights all mistakes uniformly. This property of AUPRC can be underable in many real-life settings. It goes against the widespread belief that AUPRC is somehow “better” than AUROC in low-prevalence domains, which is a common belief both in academia and industry. 

The paper also proves a theorem that states that AUPRC is be discriminatory against low-scoring subpopulations, and it backs up this theoretical finding by a series of experiments that confirm that these discriminatory effects occur in practice when selecting models based on AUPRC
strength_weakness: strength_weakness: A minor critique goes to the experimental section. In both the synthetic and the real-world evaluation the confidence intervals appear rather wide. While the Figures 2 and 3 do appear to support the message that the authors claim that it does, it could be more convincing. For example,
Is it really true that the AUROC for the low-prevalent sub-group decreases in Figure 2d, or is this noise?
The AUROC of the high-prevalence group in Figure 2b appears to trend up while increasing steps while the one of the low-prevalence group appears to stay flat, as the theory would suggest. However, the confidence intervals do still overlap everywhere.
Most of the CIs for the correlations in Figure 3 include 0.0.

This appears to simply be a matter of sample size: 20 randomly sampled datasets in the synthetic evaluation and relatively small tabular datasets for the real-life data evaluations.
It appears that Figure 2 may become more convincing simply by simulating more randomly sampled datasets (reducing the standard error of the mean), while for Figure 3 we may obtain a more convincing plot by simply including a larger real-life dataset.

That said, I consider this to be a minor point. The main contributions of the paper are theoretical, and despite my critique, the current figures do support the theory. Just not as convincingly as could have been, and it appears easily solvable.
questions: questions: - Line 199/200: “We also evaluate the test set AUROC gap and AUPRC gap between groups”. I believe this may be a mistake, as Figures 3, 7, and 8 only appear to report AUROC gaps and not AUPRC gaps. I believe it is the right choice to refrain from reporting AUPRC gaps: the fact that AUPRC is known to be prevalence-dependent would imply that prevalence-gaps by themselves may already explain AUPRC-gaps, even without existence of underlying fairness issues. The experimental setup based on AUROC gaps appears correct.
- Line 154: "such that $AUROC(D1) \approx AUROC(D2) \approx AUROC(D1\cup D2) = 0.85$. What is the algorithmic procedure that is used to obtain scores at the target AUROC, and how precisely does this achieve the target AUROC?
- Line 163: "Next, we profile an optimization procedure that randomly permutes all the (sorted) model scores up to 3 positions." This description seems imprecise, what is the precise reordering procedure that is applied?
limitations: limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: The paper challenges the claim that area under the precision-recall curve (AUPRC) is a better metric for model comparison to the area under the receiver operating characteristic (AUROC) when it comes to tasks with class imbalance. The paper offers three formal results, proving i) a characterization of the two metrics in terms of how they relate to FPR, ii) a description of how the two metrics optimize for the correction of mistakes and iii) in fairness-sensitive settings, AUPRC introduces biases in favor of the highest prevalence subpopulations wrt AUROC. These findings are supported by synthetic and real-world experiments corroborating the findings.
They conclude that the aforementioned claim is unfounded and AUROC is preferable in several settings. Moreover AUPRC is potentially harmful in fairness-sensitive scenarios due to its bias towards the correction of mistakes in the highest prevalence subpopulations. Finally, the paper contains a review on the literature generating the claim and some guidance on when using which of the two metrics.
strength_weakness: strength_weakness: 1.     Section 4 has a fair amount of repetition wrt Figure 1. I suggest compressing one or the other.
2.     Figure 2 is corrupted and the results cannot be assessed. This needs to be fixed.
3. The explanation is line 129-133 is not clear, I suggest re-writing it. 
4.     A minor point, but I would suggest the authors to not use both boldface and italic at the same time (e.g. in the introduction). It feels unnecessarily intense.
5. Another minor point, but I would discourage the authors from using self-praising expressions such as “our analyses are thorough and compelling”. Let the reader be the judge of that.
6. In the caption of Figure 1, repeating the definition of mistake is not really necessary I believe.
7. The theoretical results have some strong assumptions, e.g. perfect calibration. 
 Typos:
line 290: significnat -> significant
line 310: These -> these
questions: questions: 1.     For the relaxation of perfect calibration in Theorem 3, have the authors tried to consider different models with ascending levels of calibration, to see to what extent the property expressed by Theorem 3 holds?
limitations: limitations: The limitations seem to be properly discussed.
paper_summary: paper_summary: A widespread claim in machine learning that AUPRC is a superior metric than AUCROC for tasks with class imbalance is not strictly true. Based on this statement, this paper challenges this claim from two perspectives. On the one hand, the authors theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, showing that optimizing for AUROC equates to minimizing the model’s FPR in an unweighted manner, whereas optimizing for AUPRC equates to minimizing the FPR in a weighted manner. On the other hand, experiments on both semi-synthetic and real-world fairness datasets support their theory.
strength_weakness: strength_weakness: - The presentation needs to improve. For example, the term “high prevalence subgroup” is introduced without explanation, hindering my understanding of the theorem. A more detailed explanation of the high and low-prevalence subgroups should be provided. Besides, In theorem 3, the authors say that there exists a prevalence disparity sufficiently severe, but I can't see it directly from this theorem and suggest that the authors provide a clearer explanation.
- I find this paper (Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability, NeurIPS 2022) also has a similar form (equation 2) and conclusion about AUCPRC. Can you explain the difference between your method and this one? The corresponding citation is also necessary.
- The caption of Figure 1 is too long. It is recommended to make it more concise.
questions: questions: - Theorem 3 is based on two subgroups but real-world datasets generally have more than subgroups. Is recommended that the authors provide analysis based on multiple subgroups.
limitations: limitations: Yes

Comments: title: Request for any additional feedback or concerns given our rebuttal
Link to image file sent to AC
Image corruption
Image Corruption Reproduced; we will address
Response to rebuttal
Additional comments (in addition to the official rebuttal) for reviewer XQD9's questions
comment: Thank you for your response, for your excellent review, and for going through our clarifications!
I thank the authors for the clarifications. My score, which was already very positive, remains unchanged.
Thank you for going through our rebuttal so clearly and for raising your score! Per your suggestion, we will definitely add a summary of the raw results as well to our work, and will add clarifying text to the manuscript to help readers better understand the mistake model and why it shows such different performance under AUROC vs. AUPRC. Thank you for both of these excellent suggestions!
Thank you to the authors for the rebuttal. After carefully reading the reply, the paper’s contributions are clearer to me, so I will increase my score and recommend acceptance.

I encourage the authors to review the paper carefully to improve its clarity.

> Accordingly, the number of "actual AUROC and AUPRC values" we have is quite large. 

For Figure 3, you can report the mean and variance over different text/train splits for cross-validated parameters. I believe this should strengthen the paper's results.

> We agree that correcting adjacent pair mistakes can have varying impacts on AUPRC depending on the threshold. 

Adding this clarrification to the paper will help readers better understand the mistake model and its role.
Thank you again for your time and valuable feedback! As the rebuttal period comes to a close, we were wondering if our response has adequately addressed your concerns. If there are any remaining questions or comments, we would be happy to discuss!
Absolutely; we're happy to send the figure in a different format. Per the [NeurIPS 2024 FAQ for authors](https://neurips.cc/Conferences/2024/PaperInformation/NeurIPS-FAQ#:~:text=the%20author%20rebuttal%3F-,No.,all%20linked%20files%20are%20anonymized), we're told not to send links in any part of the response, and unfortunately we do not have the ability to edit the official rebuttal anymore, only post comments, so we can't upload a PDF with the image in a different format now. But, we've sent a message to the AC with an anonymized link to the image in a png format directly that we can confirm we see in Safari in full; I believe the AC should be able to share that link with you (this is how we are instructed to share links in the instructions). Thank you again for helping us debug this and for your understanding!
Thanks for looking into this. I would appreciate if you could upload the image to your general rebuttal in a format that I can visualize in Safari. This way I am sure I have seen the paper in its entirety.
Thank you for your impressively quick response, and for providing the additional details on the image corruption in Figure 2. We have successfully replicated this issue by viewing it in Safari, and can confirm that this is very different and clearly corrupted in comparison to how we see the image in, for example, Chrome. Now that we can reproduce this issue, we will debug and correct it promptly. Thank you again for providing these additional details!
I thank the authors for the rebuttal, which addresses my points. Concerning Figure 2: this is visualized as corrupted when the file is opened on Safari; I see this was not a problem for the other reviewers but I am nonetheless surprised since this never happened with other papers on OpenReview. I recommend the authors double-check the formatting of the image in their iteration on the paper.
Thank you for your many insightful and helpful comments to improve our work! Note that, for space reasons, while we have responded to all your identified weaknesses individually in our official "rebuttal" (which may or may not be released to reviewers at the time this comment is visible, depending on openreview settings), in this "comment" we respond to your questions instead. Please see the rebuttal as well for our full response to your excellent review.

### Q1: What causes the high variance in Figure 2?
Two factors contribute:
1. We report confidence intervals spanning the 5th to 95th percentiles, not standard errors of the mean; these confidence intervals will therefore remain wide no matter how many samples we run, and not shrink towards the mean as other measures of variance do.
2. Our procedure samples from an extremely wide space of possible "models" resulting in high true variance by design. This is important so we can make sufficiently general conclusions about the validity of our theory.

We've added to Figure 2's caption: "_Synthetic experiment per-group AUROC, showing a confidence interval spanning the 5th to 95th percentile of results observed across all seeds_" to help clarify this.

### Q2: What do 'Prevalence (Higher)' and 'Prevalence (Lower)' mean?
These refer to the fraction of samples with label 1 in the subpopulations with the highest and lowest prevalence, respectively. They don't sum to 1 as they're for different subpopulations. E.g., if we are predicting the likelihood a patient has had a heart attack based on their symptoms for a population containing both men and women, both subgroups will have different rates of heart attacks, and those rates will not sum to one.

To clarify, we've added to Table 1's caption:
"_Here, ``Prevalence (Higher)'' refers to the rate at which the prediction label $y=1$ for the subpopulation with a higher such rate, and ``Prevalence (Lower)'' refers to the same rate but over the subpopulation of the dataset with a lower rate of $y=1$._"