A Closer Look at AUROC and AUPRC under Class Imbalance
Authors: Matthew B.A. McDermott
Haoran Zhang
Lasse Hyldig Hansen
Giovanni Angelotti
Jack Gallifant
Venue: NeurIPS 2024 poster (2024)
Keywords: AUROC
AUPRC
Area under the receiver operating characteristic
Area under the precision recall curve
evaluation
fairness
disparities
bias
Primary area: fairness
TL;DR: Contrary to popular belief, we show that AUPRC should *not* be generally preferred over AUROC in cases of class imbalance, and may even be a harmful metric as it can risk exacerbating algorithmic disparities.

Abstract: In machine learning (ML), a widespread claim is that the area under the precision-recall curve (AUPRC) is a superior metric for model comparison to the area under the receiver operating characteristic (AUROC) for tasks with class imbalance. This paper refutes this notion on two fronts. First, we theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, establishing clearly that AUPRC is not generally superior in cases of class imbalance. We further show that AUPRC can be a harmful metric as it can unduly favor model improvements in subpopulations with more frequent positive labels, heightening algorithmic disparities. Next, we empirically support our theory using experiments on both semi-synthetic and real-world fairness datasets. Prompted by these insights, we conduct a review of over 1.5 million scientific papers to understand the origin of this invalid claim, finding that it is often made without citation, misattributed to papers that do not argue this point, and aggressively over-generalized from source arguments. Our findings represent a dual contribution: a significant technical advancement in understanding the relationship between AUROC and AUPRC and a stark warning about unchecked assumptions in the ML community.

A Closer Look at AUROC and AUPRC under Class Imbalance
Matthew B. McDermott
Haoran Zhang
Harvard Medical School
Massachusetts Institute of Technology haoranz@mit edu
matthew_mcdermot Chms hwrvard. edu
Lasse Hyldig Hansen Aarhus University
Giovanni Angelotti
IRCCS Humanitas Research Hospital
20 201903623@post.atu au.dk
giovanni. angelotti @humani tas. it
Jack Gallifant
Massachusetts Institute of Technology
jgally@mit.ede
Abstract
In machine learning (ML), widespread claim is that the area under the precision- recall curve (AUPRC) is superior metric for model comparison to the area under the receiver operating characteristic (AUROC) for tasks with class imbalance. This paper refutes this notion on two fronts. First, we theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, establishing clearly that AUPRC is not generally superior in cases class imbalance. We further show that AUPRC can be harmful metric as it can unduly favor model improvements subpopulations with more frequent positive labels, heightening algorithmic disparities. Next, empirically support our theory using experiments on both semi-synthetic and real-world fairness datasets. Prompted by these insights, reviewed over 1.5 million scientific papers to understand the origin of this invalid claim-finding it is often made without citation, misattributed to papers that do not argue this point, and aggressively overgeneralized from source arguments. Our findings represent a dual contribution: significant technical advancement understanding the relationship between AUROC and AUPRC and stark warning about unchecked assumptions the MI. community.
Introduction
Machine learning (ML), especially in critical domains like healthcare, necessitates careful selection and application of evaluation metrics to guide appropriate model choices and understand performance nuances [150]. Model evaluation happen in one of two settings: (1) methodologicallloode comparison setting, which occurs outside a specific deployment setting and in which target model usage workflows, optimal decision thresholds, specific false-positic (FP) and false-negative (FN) costs are typically not known, or (2) an application/deployneen estimates of model usage workflows and FP/EN costs be made. In both settings, appropriate metric choice is critical, as inappropriate selection can hinder innovation when used for model comparison and lead to significant real-world costs (e.g., misdiagnosis in medical setting) in deployment settings.
setting, where reasonably specific
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

This study focuses on two widely used metrics for binary classification tasks across both evaluation contexts: Area Under the Precision-Rese Recall Curve (AUPRC) and Area Under the Receiver Operating Characteristic (AUROC). Central to this paper is the following key claim:
Claim 1. Let f be a model which outputs continuous probabilistic predictions trained to solve a binary classification task for which the prevalence of negative labels is significantly higher than the prevalence of positive labels. For this problem, the AUPRC will yield a "better" or "more accurate" or "fairer" evaluation of f than the AUROC
Claim is made widely in both the scientific literature [399 71 159 124], in ML educational content 119 [41], and in popular press sources 80, 224 It is so widespread that even basic search results for queries relating to AUROC and AUPRO and large language model assistants like Throughout these sources, it has been justified on numerous, often imprecise grounds (see Section SD. but despite this extensive attention, we show in this work that this claim is, in fact, wrong, and may be dangerous from a model fairness perspective; further, many of its justifications are invalid or misapplied in common ML settings. More specifically, we show the following:
ChatGPT or Github Co-pilot will profess its veracity P
1) AUROC and AUPRC only differ with respect to model-dependent parameters in that AUROC weighs all false positives equally, whereas AUPRC weighs false positives at a threshold 7 with the inverse of the model's likelihood of outputting any scores greater than 7 (Theorem I This result shows that we can reason about the suitability of AUROC vs. AUPRC based on whether we care more about reducing false positives above low thresholds high thresholds. In particular,
2) AUROC favors model improvements uniformly over all positive samples, whereas AUPRC favors improvements for samples assigned higher scores over those assigned lower scores (Theorem2) 2 This indicates that the key factor differentiating the utility of AUROC or AUPRC as an evaluation metric is not class imbalance at all, but it is rather based on the target use case of the model in question. See Figure [ for a visual explanation. It also reveals that AUPRC can amplify algorithmic biases. In particular,
3) AUPRC can unduly prioritize improvements to higher-prevalence subpopulations at the ex- pense of lower-preval prevalecee subpopulations, raising serious fairness concerns in any multi-populan napti-popplltion use cases (Theorem
In this work, we establish these three claims both theoretically and empirically via synthetic exper- iments and real-world validation on popular public fairness datasets. In addition, we demonstrate through an extensive, large-language model aided literature review of over 1.5 million scientific papers that Claim I haa been used to motivate numerous improper uses of AUPRC relative to AUROC across high-stakes domains like healthcare and in several established venues, including AAAI, NeurIPS, ICML, ICLR, Cancer Cell, Nature Journals, PNAS, and more. Through this paper, we hope shed light on the nuances of appropriate evaluation and provide key guidance to limit future misuse of evaluation metries in the scientific and machine learning communities.
2 Theoretical Analyses
Please note that all notation used is defined in Appendix Section C
2.1 Relationship between AUROC and AUPRC
In this section, we introduce Theorem I which is as follows:
Theorem 1. Let X, y = {0, 1} represent paired feature and binary classification label space from which i.i.d. samples (z. y) E X ) are drawn via the joint distribution over the random variables
y. Let f X - (0, 1) be binary classification model outputting continuous probability scores over this space. Then,
AUROC(f)
E,- f(0 t))
FPR(S,) Px(f(x) > f)
AUPRC()) =1 0)
'See htaps://aarchiv.is/gp/PPa which shows results dominated by those making Claim
"Se https::/chat.opena.... www.ap.enn.......oo... wtw.ppaww....oooaaaae/////ttaaaa 111a5-9766-78922f3335

---

a)
Data
Model
Model output (score)
=
a=1
Mistake #:
2
3
contribution to AUROC/AUPRC
b) Methodology Comparison
Favor mistakes equally
Random 0 ff
c) Cancer Screening
Favor low-score region to catch all disease
Further tests
To No follow up
d) Public Health Intervention a==1 top-k,
Favor per-group regions differently
a= 0: top-ka
e) Small Molecule Screening
Favor high score region to optimize top- k
of+ Screen top-k k hits
Figure I: a) Consider a model f yiclding continuous output scores for a binary classification task applied to dataset consisting of two distinct subpopulations, A E {0, 1}. we order samples in ascending order of output score, each misordered pair of samples (e.g., mistake 1-4) represents opportunity for model improvement. Theorem [ shows that model's AUROC will improve by same amount no matter which mistake you fix, while the model's AUPRC will improve by an amount correlated with the score of the sample. b) When comparing models absent specific deployment scenario, we have no reason value improving one mistake over another, and model evaluation metrics should therefore improve equally regardless of which mistake corrected. e) When mistakes that have lower scores, regardless of any class imbalance. d) When limited resources will be distributed among population according to model score, in manner that requires certain subpopulations to all be offered commensurate possible benefit from the intervention for ethical reasons, evaluation metrics should prioritize the importance of within-group, high-score mistakes such that the highest risk members of all subgroups receive interventions. e) When false positives are expensive relative false negatives there are no fairness concerns, evaluation metrics should favor model improvements in decreasing order with score.
negatives have high cost relative to false positives, evaluation metrics should favor
3

---

We provide the proof in Appendix Sc The two key intuitions are that integrating over the TPR equivalent to taking the expectation over the induced distribution of positive sample scores, and that via Bayes rule, Prec(f.T) = 1y - =0) P(((()>>
FPR().T)
Despite its simplicity, Theorem [ has far-reaching implications. Namely, it reveals that the only difference between AUROC and AUPRC with respect to model dependent parameters (i.e., omitting the dependence of AUPRC on the fixed prevalence of the dataset, which is not model varying) is that optimizing AUROC equates to minimizing the expected false positive rate over all positive samples in an unweighted manner (equivalently, in expectation over the distribution of positive sample scores) whereas optimizing AUPRC equates to minimizing the expected false positive rate over all positive samples weighted by the inverse of the model's "firing rate" (P2((()) > 7)) at the given positive sample score. This preference be crystallized when we examine how AUROC vs. AUPRC would prioritize correcting indivisible units model improvements, termed "mistakes" which we will discuss next.
2.2 AUPRC prioritizes high-score mistakes, AUROC treats all mistakes equally
Understanding how a given evaluation metric prioritizes correcting various kinds of model mistakes or errors offers significant insight into when that metric should be used for optimization or model selection. To examine this topic for AUROC and AUPRC, consider the following definition of an "incorrectly ranked adjacent pair", which we will colloquially refer to as "model mistake":
Definition 2.1. Let f. X,V, x, y be defined as in Theorem I Further, let us suppose we have sampled a static dataset from x, y for evaluation which will be denoted X, y = {211 U1)..... (x yN )}. for I; E X' y/ E {0. and N E N. We assume for convenience that f is an injective map and all m; are distinct (i.e., (il #j:z 2j which, by injectivity of f. implies that f(x,) + f(z)))
We say that (I:; my) are an incorrectly ranked adjacent pair and thus that the model makes a "mistake" samples (Ii, Wrj if:
1. 8/4 = and yj =
2. f fxx) f(z))
3. Azg such that f(x.) < f(xx) < f(x)).
Essentially, Definition 2.1 states that mistake occurs when model assigns adjacent probability scores to pair of samples with discordant labels, as shown in Figure [ With this in mind, we can then introduce Theorem which states that AUROC improves by constant amount regardless of which mistake is corrected for a given model and dataset whereas AUPRC improves more when the mistake corrected occurs
higher score than when it occurs a lower score:
Theorem 2. Define f.X, X, X, y and N as Definition 2.1 Further, suppose without loss of generality that the dataset X is ordered such that f(x;) < f(1+11) for all i. Then, let us define M = (i), 2141) is an incorrectly ranked adjacent pair for model f). Define f to be a model that is identical to except that the probabilities assigned to I; and x++1 are swapped:
((x)
if x & (11, if n;
f:
f(1+1)) f(xz)
if ri+1.
Then, AUROC(f)) = AUROC(F)) for all i.j e M. and AUPRC(F)) AUPRC(F)) for all i.j E M such that < j.
The proof for Theorem can be found in Appendix E This proof simply stems from the fact that correcting single mistake (z,, (as defined in Dehnition 2.1 always changes the false positive rate by the same amount, and only changes it at the threshold f(:.). This, combined with the formalization of AUROC and AUPRC in Theorem I. establishes the proof. Note that this Theorem can be trivially extended to include case where ties are possible simply by noting that "swapping" two samples v; and xj where f(x;) = f(x;) ) in the manner of the theorem results no change to either AUROC or AUPRC, and similarly by the same reasoning separating any tie in the appropriate direction will improve AUROC uniformly over samples and will improve AUPRC a manner monotonic with model score.

---

2.3 AUPRC is explicitly discriminatory in favor of high-scoring subpopulations
The reliance on model's firing rate revealed in Theorem I and the optimization behavior in Theorem reveals significant issues with the faimess of AUPRC. In particular, in this section we introduce Theorem
Theorem 3. Let X, X, y, N, M, and f all be defined as in Theorem 2 Further, suppose that in this setting the domain X' now contains an attribute defining two subgroups, A = {0. 1), such that for any sample (zi, y;), denotes the subgroup to which that sample belongs. Let f be perfectly calibrated for samples in subgroup a = O, such that (yxx(( 1/a = 0, f(x) = () Then,
lim Popa 1a 0) -0
(1: = di+1 i arg max (AUPRO(S)) 1. JEM
Essentially, Theorem 3 (proof provided in Appendix F) shows the following. Suppose we are training model f over dataset with two subpopulations: Population O is sufficiently low relative to the rate at 1, then the mistake that, were it fixed, would maximally improve the AUPRC of f will be a mistake purely in population a = 1. This demonstrates that AUPRC provably favors higher prevalence subpopulations (those with higher base rate at which y 1) under sufficiently severe prevalence imbalance between subpopulations.
and = 1. If the model is
calibrated and the rate at which y = for population which y 1 for population
Note that this property is, generally speaking, not desirable. In particular, this property establishes that in settings where model fairness among a set of subpopulations in the data is important, AUPRC evaluation metric due to the risk that it will introduce biases in favor of
should not be used the highest prevalence subpopulations. We validate this result empirically over both synthetic and real-world data in Section 3 demonstrating that the import of Theorem is not merely limited an analytical curiosity but can have real-world impact on algorithmic disparities in practice. Furthermore, note that this theorem does not indicate that AUPRC will be superior to AUROC for differentiating low prevalence (or low risk) subpopulation relative to a high-risk subpopulation, a property that is sometimes attributed to AUPRC in the literature. Rather, Theorem B shows that maximizing AUPRC will be more likely to optimize solely within the high-risk subgroup, rather than optimizing to differentiate across subgroups, as low-risk subgroup samples will predominantly occur in lower-score regions under severe class imbalance.
3 Experimental Validation
In this section, we establish via synthetic and real-world experiments that Theorem is not merely an analytical effect but has real world consequences on the implications of optimizing or performing model selection via AUPRC.
3.1 Synthetic optimization experiments demonstrate AUPRC-induced disparities
In this section, we use a carefully constructed synthetic optimization procedure to demonstrate that, when all other factors equal, optimizing by or performing model selection the basis of AUPRC vs. AUROC risks excacerbating algorithmic disparities in the manner predicted by Theorem 3 For analyses under more realistic conditions with more standard models, see our real-world experiments in Section 3.2
Experimental Setup. Let y E {0, }} be the binary label, E [0. 1] be the predicted score, and a E (1 be the subpopulation. We fix Pyl = 1/a 1) = 0.05 and (y = la = 2) = 0.01. We sample a dataset for each group D. = {31. 91).... (Sp.. yr.)), such that AUROC(D)) P
AUROC(D2) ss AUROC(D, L L22 = 0.85 (See Appendix G.I A target AUROC of 0.65 was also profiled Appendix 5).
Our main experimental challenge is to determine how to simulate "optimizing" or "selecting" a model by AUROC or AUPRC. Simulating optimizing by these metries allows us to explicitly assess how the use of either AUPRC or AUROC as evaluation metric in model selection processes such as hyperparameter tuning or architecture search, can translate into model-induced inequities in dangerous ways. We explore two approaches here. First, can simply correct the atomic mistake
5

---

that maximally improves AUROC or AUPRC in each optimization iteration. In our experiments, we use n2 n2 200 and optimize for 50 steps for this experiment. This the most straightforward optimization procedure to analyze, but it is unrealistic. In real optimization scenarios, larger model changes will be made at once, and model will have an opportunity to degrade performance some regions in order to improve it in others.
Next, we profile optimization procedure that randomly permutes all the (sorted) model scores up to 3 positions (See Appendix G.3 for details). This has the effect of randomly adjusting all model scores, and can worsen model performance under some random permutations, but offers precisely the same "optimization capacity" to the low and high prevalence subgroups. To ensure the model is under some optimization constraint (and therefore does not always find the "perfect" permutation to maximize both metrics identically). we allow the model to sample only 15 possible permutations before choosing the best option. This means the system will be forced navigate optimization trade-offs between which permutations improve the right regions of the score most effectively among its limited set. We use n1 n2 100 for these experiments and optimize for 25 total steps.
a) Mistakes Fixed by AUROC
b) Mistakes Fixed by AUPRC
0.9
0.88
AUROC (5%-959)
AUROC (5%-959%)
0.88
0.86
0.84
0.82
0.82
0
10
40
30
40
Permutation optimization by AUROC
d) Permutation optimizatiin by AUPRO
AUROC (5%-955%
AUROC (5%-995%)
0.95
0.85
0.8
0.75
Step
Step
Low-prevalence Group
High-prevalence Group
Figure 2: Synthetic experiment per-group AUROC, showing confidence interval spanning the 5th to 95th percentile of results observed across all seeds, after successively either fixing individual mistakes, defined in Definition 2.I (a) and b)) successively choosing the optimal score permutation (c) and d)) in order to optimize either AUROC (a) and c)) or AUPRC (b) and d)). is clear across both forms of optimization that AUPRC definitively favors the higher prevalence subpopulation, whereas AUROC treats subgroups approximately equally. Similar patterns were observed when comparing per-group AUPRCs over the same experimental procedures, shown in Appendix Figure 4
Across both settings, the mean and an empirical 90% confidence interval around the mean in Figure [ We present a formal mathematical formulation of these perturbations, as well profile third random perturbation method, in Appendix G.3
these experiments across 20 randomly sampled datasets and show
Results. Our results demonstrate the impact of the optimization metric on suhpopulation dispar- ity, In particular, in Figure 2 we observe notable disparity introduced when optimizing under

---

0.100
Difference Spearman P TAUROC AUPRC
AUROC Gep vs. AUROCI
-0.075
Isac
mimic sex
compas
Isac
campas
mimic
adul:
adult
sex
sex
race
race
race
race
sex
(1.0)
(1.0)
(1.3)
(1.3)
(1.4)
(1.4)
(2.2)
(2.7)
Figure 3: Difference in the Spearman's between the test-set signed AUROC gap versus the validation overall AUPRC, and the AUROC gap versus the overall AUROC. Numbers in parentheses are the prevalence ratios between the two groups for the particular attribute, and datasets are sorted by this quantity. Error bars are 95% confidence intervals from 20 different random data splits.
the AUPRC metric regardless of the optimization procedure. This evident the performance metrics across the high and low prevalence subpopulations, which exhibit significant divergence as the optimization process favors the group with higher prevalence. In the more realistic, random- permutation optimization procedure (Figure 21), this even results in decrease in the AUROC for the low prevalence subgroup. In comparison, when optimizing for overall AUROC, the AUROCs of both groups increase together. Note that we show the effect of this optimization on the AUPRC metric, which shows very similar trends, in Appendix Figure 4 These results demonstrate explicitly that not only does optimizing for AUPRC differ greatly than for AUROC, as has been noted historically by researchers developing explicit AUPRC optimization schemes [409], but in fact does so in an explicitly discriminatory way in realistic scenarios.
3.2 Real-world experimental validation
To demonstrate the generalizability our findings to the real world, we evaluate fairness gaps induced by AUROC and AUPRC selection four common datasets in the fairness literature [41 99 205
Datasets. We use the following four tabular binary classification datasets: adult 17 compas [4. lsac 4 and mimic [178. In each dataset, we consider both sex and race sensitive attributes. To mimic the setting of our theorems, we balance each dataset by the sensitive attribute during both training and test, by randomly subsampling the majority group. Further details about each dataset, as well as preprocessing steps, can be found in Appendix H
Experimental setup. We train XGBoost models [65] on each dataset. For each task, we iterate grid of per-group weights in order create diverse set of models that favor different groups
For each setting of task and per-group weight, we conduct random hyperparameter search (37] with 50 runs. Though might lead to better performance, random searches are far more popular and practical, and have been used in popular benchmarking libraries [131 (37).
complex hyperparameter search methods such as BOHB (100) TPE (36]
We evaluate the validation set overall AUROC and AUPRC. We also evaluate the test set AUROC gap and AUPRC gap between groups, where gaps defined the value of the metric for the higher prevalence group minus the value for the lower prevalence group. Based on our theorems, our hypothesis is that overall AUPRC should be more positively correlated with the signed AUROC gap than overall AUROC, indicating that better favors the higher prevalence group, especially when the prevalence ratio between groups is high. To test this hypothesis, evaluate the Spearman correlation coefficient between these quantities. We repeat this experiment 20 times, with different random data splits, to obtain 95% confidence interval.
we plot the difference in the Spearman correlation coefficient of the AUROC gap versus the overall AUPRC, and AUROC gap versus overall AUROC. We observe mixed results in datasets with low prevalence ratio. In dataset with higher prevalence ratio, we find that overall AUPRC
Results. In Figure 3
7

---

is more positively correlated with the AUROC gap than overall AUROC, indicating that AUPRC more aggressively favors the higher prevalence group. We emphasize that the prevalence ratios observed in these real-world datasets is much lower than the ratio of 5 used in our synthetic experiments, which may account for the mild effect observed. To see raw results from these experiments, see Appendix Figure Similar results for neural network classifiers can be found in Appendix Section H.3
Next, in Appendix Figure 8 we plot the difference in the Spearman's p from Figure the prevalence gap. We find that there is a statistically significant correlation between the two (Spearman's p= 0.905, p = 0.002). Thus, while our power to detect a prevalence mediated AUPRC bias amplification effect is limited due to the limited prevalence disparities in these datasets, we nonetheless observe a strong positive correlation between the extent of the prevalence mismatch between the low and high prevalence group and the amount that AUPRC favors the high prevalence group over AUROC. In other words, our results show that across these fairness datasets and attributes, as the prevalence disparity grows more extreme, we observe statistically significant corresponding increase in the extent which AUPRC introduces algorithmic bias, exactly in accordance with what Theorem 3 suggests.
3 versus
When Should One Use AUPRC vs. AUROC?
In Sections 2 andd] we have shown that AUPRC is not universally superior in cases of class imbalance (and that instead, it merely preferentially optimizes high-score regions over low-score regions) and that it also poses serious risks to the model fairness in settings where subgroup prevalences differ. In light of this, how should we revise Claim to reflect when we actually should use AUPRC instead of AUROC or vice versa? Below, we explore this question and provide practical guidance metric selection for binary classification, building on our theoretical results and highlight specific contexts in which one may be favorable (Figure D). Note that while we provide guidance below situations in which AUROC vs. AUPRC is more or less favorable, this is not to suggest that authors should not report both metrics, or even larger sets of metrics more nuanced analyses such ROC PR curves; rather this section is intended to offer guidance on what metrics should be seen as more or less appropriate for use in things like model selection, hyperparameter tuning, or being highlighted as the 'critical' metric in a given problem scenario.
For context-independent model evaluation, use AUROC: For model evaluations conducted outside of specific deployment contexts, where the differential costs of errors undefined, the necessity for a metric that impartially values improvements across the entire model output space becomes paramount (Figure [p). As it is not known in advance where samples of interest will live in the output space, nor are particular cost ratios known, there should be no preference for mistake correction. Therefore, in this setting, AUROC is favorable as it uniformly accounts for every correction, offering a fair assessment irrespective of decision thresholds.
For deployment scenarios with elevated false negative costs, use AUROC: In applications where the consequences of false negatives are especially high, such as in the early screening for critical illnesses like cancer (Figure Ib), primary goal of the model will be to achieve the fewest missed cancer cases; equating to prioritizing model recall. In such a scenario, the most important mistakes to correct occur at lower score thresholds, as high-score mistakes will not change which positive samples are missed in deployment settings (as chosen thresholds are likely to be low). This behavior is the inverse of what AUPRC prioritizes, demonstrating that in such situations, AUROC should be preferred over AUPRC.
For ethical resource distribution among diverse populations, use AUROC: When faced with the challenge of ethically allocating scarce resources across a broad population, necessitating equitable benefit distribution among subgroups (Figure Tt), one must avoid prioritizing model improvements that selectively favor one subpopulation. As AUPRC will target high-score regions selectively, it risks unduely favoring high-prevalence subpopulations, as shown in Theorem 3 and Figures Even though in this resource distribution problem, high-score regions are selectively important compared to low-score regions, the fact that in this problem, we must prioritize across all subpopulations equally means that AUPRC's global preference is untenable it could induce bias.
8

---

For reducing false positives in high-cost, single-group intervention prioritization or information retrieval settings, use AUPRC: In scenarios where the cost associated with false positives signifi- cantly outweighs that of false negatives, absent of equity concerns- molecules from a fragment library for drug development trials, where only the most promising molecules will proceed to costly experimental validation (Figure ( - the metric of choice should facilitate a reduction in high-score false positives. This necessitates a focus on correcting high-score errors, for which AUROC might not be ideal due to its uniform treatment of errors across the score spectrum, potentially obscuring improvements in critical high-stake decisions.
as in selecting candidate
5 Literature Review: Examining how Claim BBeame so Widespread
Claim Jttates that "AUPRC is better than AUROC in cases of class imbalance" and is widespread in the literature. Via both manual literature search and an automated search of over 1.5M arXiv papers (see Appendix methodology), we observed 424 publications making this claim This widespread disemmination of T a significant impact on the validity of scientific discourse. We observed examples of high-profile papers operating medically critical settings where high-recall a key priority evaluating MI systems via AUPRC due to their task's underlying class imbalance [399]; papers focusing on fairness critical applications relying on AUPRC due to this claim, even while our results demonstrate AUPRC has major problems in the fairness regime [366 (06), and numerous other papers perpetuating this source of scientific misinformation.
Among the 424 papers we discovered referencing this claim, 167 did so with no associated citation. These papers were published in a wide range of venues, including high profile venues such as NeurIPS, ICML, and ICLR. This reflects not only the widespread belief in this claim, but also that we may be too comfortable making seemingly "correct" assertions without appropriate attribution MI. today. Further, Among the 257 that reference this claim and cite source for this assertion, 135 do not cite any papers that actually make this claim in the first place. Most often, papers erroneously attribute this claim to (83), which was cited as source for this claim 144 times. While [83] makes many interesting, meaningful claims about the ROC and PR curves, and does argue that the precision- curve may be more informative than the ROC in cases of class imbalance it never asserts that the area under the PR curve should be preferred over the area under the ROC in cases of class imbalance. This distinction is important, because while curves can be used to simultaneously communicate many different performance statistics to their viewers across different FPR/TPR or Precision/Recall criteria, and therefore should be assessed primarily as communication tools deployment experts, areas under these curves are single-point summarizations which are primarily used for deployment-agnostic model comparison, which is very different use case.
Even when appropriate papers are cited, the valid settings in which AUPRC should be preferred (see Section 4 for examples) are often over-shadowed by significant over-generalizations to preferring AUPRC in all settings featuring class imbalance. For example, claims such as that "precision- curves more informative of deployment metries" are often used to justify why AUPRC should be used in all cases of class imbalance, rather than just in cases where the relevant deployment metrics most directly associated with the PR curve. Another class of arguments made in favor of Claim I is rooted in claims that AUROC is poor in cases of class imbalance because its scores are misleadingly high. While this argument can reflect meaningful limitation of the communication value of the ROC or the AUROC, comments about singleton metric results (rather than model comparison through metric values) are inherently orthogonal to the goal of model evaluation. In other words, what matters for model evaluation is not how high given metric is, but rather the extent to which the metric meaningfully captures the right improvements in the model in the right ways. The widespread nature of Claim I has also led researchers astray when exploring new optimization procedures for AUPRC, by advocating for the importance of AUPRC when processing skewed data, even in domains such as medical diagnoses that often have high false negative costs relative to false positive costs [409]. For a more extensive breakdown of the arguments we observed in the literature and the sources making them, see Appendix Tables 2 and
'Note that throughout this section, full citations for all of the larger lists of papers we reference will be relegated to Appendix Section concision.

---

Fully 6 Limitations and Future Works
There are still a number of areas for further improvement and future work. Firstly, our theoretical findings can be refined and generalized to, e.g., take into account the difficulty of the target task (which may differ between subgroups), not require models to calibrated (in the case of Theorem B or specifically take into account more than 2 subpopulations for more nuanced comparisons beyond what can be inferred through pairwise comparisons between subpopulations. where our results would naturally apply. Further, extending our real-world experiments to more fairness datasets and identifying more nuanced ways to probe the impact of metric choice on disparity measures would significantly strengthen this work. These analyses can also be extended to consider other metrics, such as the area under the precision-rees recall- curve [04], the area under the net benefit curve [34 and tingle-thresood, deployment centric metrics well. In addition, one of the largest limitations of Theorem 3 is its restrictive assumptions, in particular the requirement of perfect calibration. A ripe area of future work is thus investigate how calibration or to determine whether or not our results imply anything about the viability or safety of post-hoc calibration of models optimized either through AUPRC or AUROC.
soften our analyses for models with imperfect
7 Conclusion
This study interrogates the pervasive assumption within the ML community that AUPRC is better evaluation metric than AUROC in class-imbalanced settings. Our empirical analyses and literature review reveal several concrete findings that challenge this notion. In particular, we show that while optimizing for AUROC equates to minimizing the model's FPR in an unbiased manner over positive sample scores, optimizing for AUPRC equates to minimizing the FPR specifically for regions where the model outputs higher scores relative to lower scores. Further, we show both theoretically and empirically over synthetic and real-world fairness datasets that AUPRC can be explicitly subgroups.
discriminatory metric through favoring
In summary, our research advocates for a more thoughtful and context-aware approach to selecting evaluation metrics in machine learning. This paradigm shift, favoring balanced and conscientious approach to metric selection, is essential in advancing the field towards developing not only technically sound, but also equitable and just models.
Acknowledgements
MBAM gratefully acknowledges support from a Berkowitz Postdoctoral Fellowship. JG is funded by the National Institute of Health through DS-I Africa US4 TW012043-01 and Bridge2AI OT2OD032701.
References
[1] Nasim Abdollahi, Seyed Ali Madani Tonekaboni, Jay Huang, Bo Wang, and Stephen MacKin- non. Nodecoder: graph-based machine learning platform to predict active sites of modeled protein structures, 2023.
[2] Avraham Adler. Using machine learning techniques to identify key risk factors for diabetes and undiagnosed diabetes, 2021.
[3] B Thomas Adler, Luca De Alfaro, Santiago M Mola- Velasco, Paolo Rosso, and Andrew G West. Wikipedia vandalism detection: Combining natural language, metadata, and reputation features. In Computational Linguistics and Intelligent Text Processing: 12th International Conference, CICLing 2011, Tokyo, Japan, February 20-26, 201 Proceedings, Part II 12, pages 277-288. Springer, 2011.
[4] Sergey Afanasiev, Anastasiya Smirnova, and Diana Kotereva. Itsy bitsy spidernet: connected residual network for fraud detection, 2021.
[5] Karan Aggarwal, Onur Atan, Ahmed K Farahat, Chi Zhang, Kosta Ristovski, and Chetan Gupta. Two birds with one network: Unifying failure event prediction and time-to-failure
10

---

modeling. In 2018 IEEE international conference on big data (Big Data), pages IEEE, 2018.
[6] Faruk Ahmed and Aaron Courville. Detecting semantic anomalies. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04)3314-3112, Apr. 2020.
[7] Sajid Ahmed, Farshid Rayhan, Asif Mahbub, Md Rafsan Jani, Swakkhar Shatabda, and De- wan Md Farid. Liuboost: locality informed under-boosting for imbalanced data classification. Emerging Technologies in Data Mining and Information Security: Proceedings of IEMIS 2018, Volume pages 133-144. Springer. 2019.
[8] Giambattista Albora and Andrea Zaccaria. Machine learning to assess relatedness: the advantage of using firm-level data. Complexity, 2022, 2022.
[9] Jr. Allam, Tarek and Jason D McEwen. Paying attention to astronomical transients: introducing the time-series transformer for photometric classification. RAS Techniques and Instruments, 3(1):209-223) 223, 10 2023.
[10] Maxime Alvarez, Jean- CaanCCharees Verdier, D'Jeff K. Nkashama, Marc Frappier, Pierre-Marthn Tardif, and Froduald Kabanza. A revealing large-scale evaluation of unsupervised anomaly detection algorithms, 2022.
[11] Ilya Amburg, Jon Kleinberg, and Austin R Benson. Planted hitting set recovery in hypergraphs. Journal of Physics: Complexity, 2(3):035004, 2021.
[12] Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbalanced data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part 10, pages 770-785. Springer, 2017.
[13] Renan Andrades and Mariana Recamonde-Mendone Machine learning methods for prediction of cancer driver genes: a survey paper. Briefings in Bioinformatics, 23(3):bbac062, 2022.
[14] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of data and analytics, pages 254-264. Auerbach Publications, 2022.
[15] Nino Antulov Fantuuln,, Dijana Tolic, Matija Piskorec, Zhang Cc, and Irena Vodenska. Inferring short-term volatility indicators from the bitcoin blockchain. In Luca Maria Aiello, Chantal Cherifi, Hocine Cherifi, Renaud Lambiotte, Pictro Lió, and Luis M. Rocha, editors, Complex Networks and Their Applications VII, pages 508-520, Cham, 2019. Springer International Publishing.
[16] Bruno Arcanjo, Bruno Ferrarini, Michael Milford, Klaus McDonald-Maier, and Shoaib Ehsan. An efficient and scalable collection of fly-inspired voting units for visual place recognition in changing environments. IEEE Roboties and Automation Letters, 7(2):2527- 2534, 2022.
[17] Arthur Asuncion and David Newman, Uci machine learning repository, 2007.
[18] Sezin Kircali Ata, Yuan Fang, Min Wu, Jiaqi Shi, Chee Keong Kwoh, and Xiaoli Li. Multi- view collaborative network embedding. ACM Transactions Knowledge Discovery from Data, 15(3):1-18, April 2021.
[19] Anand Avati, Kenneth Jung, Stephanic Harman, Lance Downing, Andrew Ng, and Nigam H Shah. Improving palliative care with deep learning. BMC medical informatics and decision making, 18(4):55-66) 64, 2018.
[20] Kleanthis Avramidis, Shanti Stewart, and Shrikanth Narayanan. On the role of visual context in enriching music representations. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[21] Simon Axelrod and Rafael ensembles. Machine Learning: Science and Technology, 4(3):035025, 2023.
Molecular machine learning with conformer

---

[22] Kasra Babaei, Zhi Yuan Chen, and Tomas Maul. Aegr: simple approach to gradient reversal in autoencoders for network anomaly detection. Soft Computing. (5(24):5529--15288 2021.
[23] Kasra Babaei, Zhi Yuan Chen, and Tomas Maul. Data augmentation by autoencoders for unsupervised anomaly detection, 2019.
[24] Van Bach Nguyen, Kanishka Ghosh Dastidar, Michael Granitzer, and Wissam Siblini. The importance of future information in credit card fraud detection. In Gustau Camps- Valls, Fran- cisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference Artificial Intelligence and Statisties, volume 151 of Proceedings of Machine Learning Research, pages 10067-10077. PMLR, 28-30 Mar 2022.
[25] Bart Baesens, Sebastiaan H" H oppner, Irene Ortner, and Tim Verdonek, robrose: robust ap- proach for dealing with imbalanced data in fraud detection. Statistical Methods & Applicatiins, 30(3):841-861, 2021.
[26] Leen De Baets, Joeri Ruyssinek, Thomas Peiffer, Johan Decruyenaere, Filip De Turck, Femke Ongenae, and Tom Dhaene. Positive blood culture detection in time series data using a bilstm network, 2016.
[27] Vassileios Balntas, Karel Lene, Andrea Vedaldi, and Krystian Mikolajezyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5173-5182, 2017.
[28] Amitava Banerjee, Joseph D Hart, Rajarshi Roy, and Edward Otr. Machine learning link inference of noisy delay-coupled networks with optoelectronic experimental tests. Physical Review X, 11(3):031014, 2021.
[29] Batuhan Bardak and Mehmet Tan. Using clinical drug representations for improving mortality and length of stay predictions. In 2021 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), pages 1-8. IEEE, 2021.
[30] Ioannis Bargiotas, Argyris Kalogeratos, Myrto Limnios, Pierre-Paul Vidal, Damien Ricard, and Nicolas Vayatis. Revealing posturographic profile of patients with parkinsonian syndromes through novel hypothesis testing framework based on machine learning. PLOS ONE, 16(2):1- 22, 02 2021.
[31] Jens Bayer, David Münch, and Michael Arens. Image-based principles on graph-based input data in human action recognition. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings. Part I, page 26-40, Berlin, Heidelberg, 2021. Berinner-Verrage
[32] Neslihan Bayramoglu, Miika Nieminen, and Simo Saarakkala. Automated detection of patellofemoral osteoarthritis from knee lateral view radiographs using deep learning: data from the multicenter osteoarthritis study (most). Osteoarthritis and Cartilage, 29(10):1432-1447,
2021.
[33] Andreas Beger. Precision-recall curves, April 2016.
[34] Hafida Benhidour, Lama Almeshkhas, and Said Kerrache. An approach for link prediction in directed complex networks based on asymmetric similarity-populariyy 2022.
[35] Austin R Benson and Jon Kleinberg. Found graph data and planted vertex covers. Advances in Neural Information Processing Systems, 31, 2018.
[36] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances in neural information processing systems, 24, 2011.
[37] James Bergstra and Yoshua Bengio. Random search for pyper-parameter optimizztionn Journal of machine learning research, 13(2). 2012.
[38] Victor Besnier, Andrei Bursue, David Picard, and Alexandre Briot. Triggering failures: Out- of-distribution detection by learning from local adversarial attacks in semantic segmentation. In Proceedings of the IEFF/CVF international conference on computer vision, pages 15701-
107 2021.
12

---

[39] Debayan Bhattacharya, Benjamin Tobias Becker, Finn Behrendt, Marcel Bengs. Dirk Beyers- dorff, Dennis Eggert, Elina Petersen, Florian Jansen. Marvin Petersen, Bastian Cheng, et al. Supervised contrastive learning to classify paranasal anomalies in the maxillary sinus. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pages 429-438. Springer, 2022.
[40] Kevin Bleakley, Gérard Biau, and Jean-Philippe Vert. Supervised reconstruction of biological networks with local models. Bioinformatics, 23(13):157-165, 2007.
[41] D Blevins, Moriano, R Bridges, M Verma, M lannacone, and S Hollifield. Time-based can intrusion detection benchmark. In Workshop on Automotive and Autonomous Vehicle Security (AutoSec), 2021.
[42] Nathaniel J Bloomfield, Susan Wei, Bartholomew A. Woodham, Peter Wilkinson, and An- drew P Robinson. Automating the assessment of biofouling in images using expert agreement gold standard. Scientific reports, 11(1):2739, 2021.
[43] Hermann Blum, Paul- Bhul-Edouard Sarlin, Juan Nieto, Roland Siegwart, and Cesar Cadena. The fishyscapes benchmark: Measuring blind spots in semantic segmentation. International Journal of Computer Vision,
2021.
[44] Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A benchmark. IEEE transactions on image processing, 2A(12):5706-5722, 2015.
[45] Kendrick Boyd, Vitor Santos Costa, Jesse Davis, and C David Page. Unachievable region in precision-recat reccal space and its effect on empirical evaluation. In Proceedings of the... Inter- national Conference on Machine Learning. International Conference Machine Learning, volume 2012, page 349. NIH Public Access, 2012.
[46] Kendrick Boyd, Kevin Eng. and C David Page. Area under the precision- -ecall curve: point estimates and confidence intervals. In Machine Leurning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pages 451 551-466. Springer, 2013.
[47] Jakob Božič, Domen Tabernik, and Danijel Skočaj. End-to-end training of a two-stage neural network for defect detection. In 2020 25th International conference on pattern recognition (ICPR), pages 5619-50000 5626. IEEE, 2021.
[48] Jan Brabec, Tomáš Komárek, Vojtëch Franc, and Lukáš Machlica. On model evaluation under non-constant class imbalance. In Computational Science-ICCS 2020: 20th International Conference, Amsterdam, The Netherlands, June 3-5, 2020, Proceedings, Part IV 20, pages 74-87. Springer, 2020.
[49] Jonathan Brophy and Daniel Lowd. Eggs: A flexible approach to relational modeling of social network spam, 2020.
[50] Nicolas Brosse, Carlos Riquelme, Alice Martin, Sylvain Gelly, and éric Moulines. On last- layer algorithms for classification: Decoupling representation from uncertainty estimation, 2020.
[51] Jason Brownlee. Tour of evaluation metrics for imbalanced classification, January 2020.
[52] Jonathan Bryan and Pablo Moriano. Graph- based machine learning improves just-in-time defect prediction. Plos one, (8(4))0028407,,222
[53] Céline Budding, Fabian Eitel, Kerstin Ritter, and Stefan Haufe. Evaluating saliency methods artificial data with different background types, 2021.
[54] Marcin Budka, Akanda Wahid UI Ashraf, Matthew Bennett, Scott Neville, and Alun Mackrill. Deep multilabel cnn for forensic footwear impression descriptor identification. Applied Soft Computing, 109:107496. 2021.
[55] Tian Cai, Li Xie, Muge Chen, Yang Liu, Di He, Shuo Zhang, Cameron Mura, Philip E. Bourne, and Lei Xie. Exploration of dark chemical genomies space via portal learning: Applied to targeting the undruggable genome and covid-19 anti-infective polypharmacology. 2021.
13

---

[56] Chen Chai, Juanwu Lu, Xuan Jiang, Xiupeng Shi, and Zeng Zeng, An automated machine learning (automl) method for driving distraction detection based on lane-keeping performance. arXiv preprint arXiv:2103.08.0 2021.
[57] Satrajit Chakrabarty, Pamela LaMontagne, Joshua Shimony, Daniel Marcus, and Aristeidis Sotiras. Mri-based classification of idh mutation and 1p/19q codelction status of gliomas using a 2.5 hybrid multi-task convolutional neural network. Neuro-Oncology Advances, 5(1):vdad023, 2023.
[58] Neeloy Chakraborty, Aamir Hasan, Shuijing Liu, Tianchen Ji, Weihang Liang, D. Livingston McPherson, and Katherine autoencoder for highway vehicle anomaly detection. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23, page 1125 1134, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems.
Structural attention- recurrent variational
[59] Saptarshi Chakraborty, Colin B Begg. and Ronglai Shen. Using the "hidden" genome to improve classification of cancer types. Biometrics, 77(4):1445-144 1455, 2021.
[60] Saptarshi Chakraborty, Zoe Guan, Colin B Begg, and Ronglai Shen. Topical hidden genome: Discovering latent cancer mutational topics using a bayesian multilevel context-learning approach. arXiv preprint arXiv:2212. 2022.
[61] Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Robust, deep and inductive anomaly detection. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part 10, pages 36-51. Springer, 2017.
[62] Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta classification for out-of-distributoon detection in semantic segmentation. In Proceedings of the ieee/cvf international conference on computer vision, pages 5128-5137, 2021.
[63] Hugh Chen, Scott Lundberg, and Su-In Lee. Checkpoint ensembles: Ensemble methods from single training process, 2017.
[64] Hugh Chen, Scott Lundberg, and Su-In Lee. Hybrid gradient boosting trees and neural networks for forecasting operating room data, 2018.
[65] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining. pages 785-794, 2016.
[66] Yu Chen, Jiaqi Jin, Hui Zhao, Pengjie Wang, Guojun Liu, Jian Xu, and Bo Zheng. Asymptoti- cally unbiased estimation for delayed feedback modeling via label correction. In Proceedings of the ACM Web Conference 2022, pages 369-379, 2022.
[67] Zhuohao Chen, James Gibson, Ming-Chang Chiu, Qiaohong Hu, Tara K Knight, Daniella Meeker, James A Tulsky, Kathryn I Pollak, and Shrikanth Narayanan. Automated empathy detection for oncology encounters. In 2020 IEEE International Conference on Healthcare Informatics (ICHI), pages 1-8. IEEE, 2020.
[68] Davide Chicco. Ten quick tips for machine learning in computational biology. BioData mining, 10(1):35, 2017.
[69] Julien Chiquet, Stephane Robin, and Mahendra Mariadassou. Variational inference for sparse network reconstruction from count data. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1162-1171. PMLR, 09-15 Jun 2019.
[70] Brian Y Cho, Tucker Hermans, and Alan Kuntz. Planning sensing sequences for subsurface 3d tumor mapping. 202/ International Symposium on Medical Robotics (ISMR), pages 1-7, IEEE, 1.21.
14

---

[71] Edward Choi, Cao Xiao, Walter F. Stewart, and Jimeng Sun. Mime: Multilevel medical embedding of electronic health records for predictive healthcare. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 1552-4562,, Red Hook, NY, USA, 2018. Curran Associates Inc.
[72] Eunji Chong, Katha Chanda, Zhefan Ye, Audrey Southerland, Nataniel Ruiz, Rebecca M. Jones, Agata Rozga, and James M. Rehg. Detecting gaze towards eyes in natural social interactions and its use in child assessment. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 1(3), sep 2017.
[73] Peter Christen. Application of advanced record linkage techniques for complex population reconstruction. arXiv preprint arXiv:1612.0-288 2016.
[74] Xu Chu, Yang Lin, Jingyue Gao, Jiangtao Wang, Yasha Wang, and Leye Wang, Multi-label robust factorization autoencoder and its application in predicting drug-drug interactions, 2018.
[75] Marc Claesen, Frank De Smet. Johan A.K. Suykens, and Bart De Moor. A robust ensemble approach to learn from positive and unlabeled data using svm base models. Neurocomputing. 160:73 84, July 2015.
[76] Johanna Clauser, Judith Maas, Jutta Arens, Thomas Schmitz-Rode, Ulrich Steinseifer, and Benjamin Berkels. Automation of hemocompatibility analysis using image segmentation and a random forest. arXiv preprint arXiv:2010.066255, 2020.
[77] Felipe Colombelli, Thayne Woycinck Kowalski, and Mariana Grramonde-Mendore A hybrid ensemble feature selection design for candidate biomarkers discovery from transcriptome profiles. Showledge-Basee Based Systems, 254: 254:109655, 2022.
[78] Jonathan Cook and Vikram Ramadas. When to consult precision-recal curves. The Stata Journal, 20(1):131-148, 2020.
[79] Skyler J. Cranmer and Bruce A. Desmarais, What can we learn from predictive modeling?, 2016.
[80] Jakub Czakon. Fl score vs roc auc vs accuracy vs pr auc: Which evaluation metric should you choose?, July 2022.
[81] Anusha Damodaran, Fabio Di Troia, Corrado Aaron Visaggio. Thomas H Austin, and Mark Stamp. A comparison of static, dynamic, and hybrid analysis for malware detection. Journal of Computer Virology and Hacking Techniques, 13:1-12, 2017.
[82] Armin Danesh Pazho, Ghazal Alinezhad Noghre, Babak Rahimi Ardabili, Christopher Neff, and Hamed Tabkhi. CHAD: Charlotte Anomaly Dataset, page 50-66. Springer Nature Switzerland, 2023.
[83] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 233 240-240, New York, NY, USA, 2006. Association for Computing Machinery.
[84] Alice Del Vecchio, Andreea Deac, Pietro Liò, and Petar Veličković. Neural message passing for joint paratope-epitope prediction. arXiv preprint arXiv:2106.0077 2021.
[85] Jianyuan Deng, Zhibo Yang, Hehe Wang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. Unraveling key elements underlying molecular property prediction: A systematic study, 2023.
[86] Amit G. Deshwar, Shankar Vembu, Christina K. Yung. Gun Ho Jang, Lincoln Stein, and Quaid Morris. Reconstructing subclonal composition and evolution from whole genome sequencing of tumors, 2015.
[87] Giancarlo Di Biase, Hermann Blum, Roland Siegwart, and Cesar Cadena. Pixel-wise anomaly detection complex driving scenes. Proceedings of the IEEF/CV conference on computer 2021.
vision and pattern recognition, pages
15

---

[88] Daisy Yi Ding, Chloé Simpson, Stephen Pfohl, Dave C Kale, Kenneth Jung, and Nigam H Shah. The effectiveness of multitask learning for phenotyping with electronic health records data. In BIOCOMPUTING 2019: Proceedings of the Pacific Symposium, pages 18-29. World Scientific, 2018.
[89] Jonas C. Ditz, Bernhard Reuter, and Nico Pfeifer. Convolutional motif kernel networks, 2023.
[90] Rémi Domingues, Pietro Michiardi, Jérémie Barlet, and Maurizio Filippone. comparative evaluation of novelty detection algorithms for discrete sequences. Artificial Intelligence Review, 53:3787-3812, 2020.
[91] Shangjia Dong, Tianbo Yu, Hamed Farahmand, and Ali Mostafavi. A hybrid deep learning model for predictive flood warning and situation awareness using channel network sensors data. Computer-Aided Civil and Infrastructure Engineering, 36(4):402-420, 2021.
[92] Wen Dong, Tong Guan, Bruno Lepri, and Chunming Qiao. Pocketcare: Tracking the flu with mobile phones using partial observations of proximity and symptoms. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(2):1-23, 2019.
[93] Julia Dressel and Hany Farid. The accuracy, faimess, and limits of predicting recidivism. Science advances, 4(1 :e)aso5555 2018.
[94] Guodong Du, Liang Yuan, Kong Joo Shin, and Shunsuke Managi. Enhancement of land-use change modeling using convolutional neural networks and convolutional denoising autoen- coders, 2018.
[95] Marco Dueñas, Víctor Ortiz, Massimo Riccaboni, and Francesco Serti. Assessing the Impact of COVID-19 Trade: Machine Learning Counterfactual Analysis. Working papers 79, Red Investigadores de Economía, April 2021.
[96] Witold Dyrka, Mateusz Pyzik, François Coste, and Hugo Talibart. Estimating probabilistic context-free grammars for proteins using contact map constraints. PeerJ, 7:e6559, March 2019.
[97] Gabriel Erion, Hugh Chen, Scott M. Lundberg, and Su-In Lee. Anesthesiologisttevell forecast- ing of hypoxemia with only spo2 data using deep learning, 2017.
[98] Sirag Erkol. Satyaki Sikdar, Filippo Radicchi, and Santo Fortunato. Consistency pays off in science. Quantitative Science Studies, 4(2):491-500, 2023.
[99] Alessandro Fabris, Stefano Messina, Gianmaria Silvello, and Gian Antonio Susto. Algorithmic fairness datasets: the story so far. Data Mining and Knowledge Discovery, 2676)2277-- 2022.
[100] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In International conference on machine learning, pages 1437-1446. PMLR, 2018.
[101] Yang Feng, Min Zhou, and Xin Tong. Imbalanced classification: A paradigm-based review. Statistical Analysis and Data Mining: The ASA Data Science Journal, 14(5):383-406 2021.
[102] Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, Bartosz Krawczyk, and Francisco Herrera. Learning from imbalanced data sets, volume 10. Springer, 2018.
[103] Andres Ferraro, Dmitry Bogdanov, Xavier Serra Jay, Ho Jeon, and Jason Yoon. How low can you go? reducing frequency and time resolution in current enn architectures for music auto- tagging. In 2020 28th European signal processing conference (EUSIPCO), pages 131-135. IEEE, 2021.
[104] Peter Flach and Meelis Kull. Deccilio-rrecallgee gain curves: Pr analysis done right. Advances in neural information processing systems, 28, 2015.
[105] Carlos Floyd, Herbert Levine, Christopher Jarzynski, and Garegin A Papoian. Understanding cytoskeletal avalanches using mechanical stability analysis. Proceedings of the National Academy of Sciences, 118(4))e22112231111 2021.
16

---

[106] Marina Fomicheva, Piyawat Lertvitayakkujjon, Wei Zhao, Steffen Eger, and Yang Gao. The eval4nlp shared task on explainable quality estimation: Overview and results. In Yang Gao, Steffen Eger, Wei Zhao, Piyawat Lertvitayakumjorn, and Marina Fomicheva, editors, Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 165-178, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[107] Eduardo Fonseca, Xavier Favory. Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: An open dataset of namann-labeled sound events. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 30:829-08880 852, dec 2021.
[108] Fabrizio Frasca, Diego Galeano, Guadalupe Gonzalez, Ivan Laponogov, Kirill Veselkov, Al- berto Paccanaro, and Michael M. Bronstein. Learning interpretable disease self-representations for drug repositioning, 2019.
[109] Marco Frasca and Nicolo Cesa Bianchi. Multitask protein function prediction through task dis- similarity. IEEE/ACM transactions on computational biology and bioinformatics, 16(5):1550- 1560, 2017.
[110] Marco Frasca and Nicolò Cesa-Bianchi Positive and unlabeled learning through negative selection and imbalance-awe aware classification, 2019.
[111] Yuming Fu, Xue-Bing Wu, Qian Yang, Anthony GA Brown, Xiaotong Feng, Qinchun Ma, and Shuyan Li. Finding quasars behind the galactic plane. i. candidate selections with transfer learning. The Astrophysical Journal Supplement Series, 254(1):6, 2021.
[112] Silvio Galesso, Maria Alejandra Bravo, Mchdi Naouar, and Thomas Brox. Probing contextual diversity for dense out-of-distributiin detection, 2022.
[113] Varun Gangal, Abhinav Arora, Arash Einolghozati, and Sonal Gupta. Likelihood ratios and generative classifiers for unsupervised oft-of-domain detection in task oriented dialog. Proceedings of the AAAI Conference Artificial Intelligence, 34(05))7764-7711 Apr. 2020.
[114] Dario Careia-Gasulla, Eduard Ayguadé, Jesús Labarta, Ulises Cortés, and Toyotaro Suzumura. Hierarchical hyperlink prediction for the www, 2016.
[115] Matthicu Garcin and Samuel Stéphan. Credit scoring using neural networks and sure posterior probability calibration, 2021.
[116] Jean-Gabriel Gaudreault, Paula Branco, and João Gama. An analysis of performance metrics for imbalanced classification. In International Conference on Discovery Science, pages 67-77. Springer, 2021.
[117] Sachin Gavali, Chuming Chen, Julie Cowart, Xi Peng. Shanshan Ding, Cathy Wu, and Tammy Anderson. Understanding the factors related to the opioid epidemic using machine learning. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1309-1314, 14. IEEE, 2021.
[118] Isaac D Gerg and Vishal Monga. Structural prior driven regularized deep learning for sonar image classification. IEEE Transactions on Geoscience and Remote Sensing, 60:1-16, 2021.
[119] Aurélien Géron. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow. "
O' O'Reilly Media, Inc.", 2022.
[120] Djordje Gligorijevic, Jelena Gligorijevic, and Aaron Flores. Prospective modeling of users for online display advertising via deep time-aware model. In Proceedings of the 29th ACM Inter- national Conference on Information & Knowledge Management, CIKM '20, page 2461-2468, New York, NY, USA, 2020. Association for Computing Machinery.
[121] Mark Goadrich, Louis Oliphant, and Jude Shavlik. Gleaner: Creating ensembles of first-order clauses improve recall-precision curves. Machine Learning, 64:231-261, 2006.
[122] Izhak Golan and Ran El- Deep anomaly detection using geometric transformations. Advances in neural information processing systems, 31, 2018.
17

---

[123] Samuel Goldman, Ria Das, Kevin K. Yang, and Connor W. Coley, Machine learning mod- eling of family wide enzyme-subsiee substrate specificity screens. PLOS Computational Biology. February 2022.
[124] Hongyu Gong, Alberto Valido, Katherine M Ingram, Giulia Fanti. Suma Bhat, and Dorothy Espelage. Abusive language detection in heterogeneous contexts: Dataset collection and the role of supervised attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14804 14812, 2021.
[125] Yuan Gong, Yu-An Chung, and James Glass, Psla: Improving audio tagging with pretraining sampling, labeling, and aggregation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3292-323 -3062, 2021.
[126] Eric Goodman, Chase Zimmerman, and Corey Hudson. Packet2vec: Utilizing word2vec for feature extraction in packet data. Technical report, Sandia National Lab((SNL,NM), Albuquerque, NM (United States); Sandia 2019.
[127] Julia Sophia Gottfriedsen, Max Berrendorf, Pierre Gentine, Birgit Hassler, Markus Reichstein, Katja Weigel, and Veronika Eyring. On the generalization of agricultural drought classification from climate data. thirty-fifth conference on neural information processing systems (neurips) workshop 2021 tackling climate change with machine learning"". 2021.
[128] Anil Goyal and Jihed Khiari. Diversity-aware weighted majority vote classifier for imbalanced data. In 2020 International Joint Conference on Neural Networks (IJCNN, pages 1-8, 2020.
[129] Eric Granger, Madhu Kiran, Louis-Antoine Blais-Morin, et al. A comparison of cnn-based face and head detectors for real-time video surveillance applications. In 2017 seventh international conference on image processing theory, tools and applications (IPTA), pages 1-7. IEEE, 2017.
[130] Zoe Guan, Giovanni Parmigiani, Danielle Braun, and Lorenzo Trippa. Prediction of hereditary cancers using neural networks. The annals of applied statistics, 16(1):495, 2022.
[131] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint Sttv:2007.0143:1 2020.
[132] Ekaterina Gurina, Nikita Klyuchnikov, Alexey Zaytsev, Evgenya Romanenkova, Ksenia Antipova, Igor Simon, Victor Makarov, and Dmitry Koroteev. Application of machine learning to accidents detection at directional drilling. Journal of Petroleum Science and Engineering, 184:106519, 2020.
[133] Guillaume Haben, Sarra Habchi, Mike Papadakis, Maxime Cordy, and Yves Le Traon. Dis- cerning legitimate failures from false alerts: A study of chromium's 's continuous integration, 2021.
[134] Tom Rolandus Hagedoorn and Gerasimos Spanakis. Massive open online courses temporal profiling for dropout prediction. In 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI), pages 231-236 IEEE, 2017.
[135] Alexander Hgagel Jonas Rothfuss, Lars Lorch, Vignesh Ram Somnath, Bernhard Sch" olkopf, and Andreas Krause. Bacadi: Bayesian causal discovery with unknown interventions. In International Conference on Artificial Intelligence and Statistics, pages 1411-1436. PMLR, 2023.
[136] Melissa Hall, Bobbie Chern, Laura Gustafson, Denisse Ventura, Harshad Kulkarni, Candace Ross, and Nicolas Usunier. Towards reliable assessments of demographic disparities in multi-label image classifiers, 2023.
[137] Jacob A. Harer, Louis Y. Kim, Rebecca Russell, Onur Ozdemir, Leonard R. Kosta, Akshay Rangamani, Lei H. Hamilton, Gabriel I. Centeno, Jonathan R. Key, Paul M. Ellingwood, Erik Antelman, Alan Mackay, Mare W. McConley, Jeffrey M. Opper, Peter Chin, and Tomo Lazovich. Automated software vulnerability detection with machine learning, 2018.
18

---

[138] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data, Scientific data, 6(1):96, 2019.
[139] Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1,, June 2019.
[140] Seyed Raein Hashemi, Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, Sanjay P Prabhu, Simon K Warfield, and Ali Gholipour. Asymmetric loss functions and deep densely-connecter networks for highly- gigla--ibbllenced medical image segmentation: Application multiple sclerosis lesion detection. IEEE Access, 7:1721-1733 2018.
[141] Haibo He and Yunqian Ma. Imbalanced learning: foundations, algorithms, and applications. 2013.
[142] Jieyue He, Xinxing Yang. Zhuo Gong, and Ibrahim Zamit. Hybrid attentional memory network for computational drug repositioning, BMC bioinformatics, 21:1-17, 2020.
[143] Jörn Hees, Dayananda Herurkar, and Mario Meier. Recol: Reconstruction enror columns for outlier detection, 2021.
[144] Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real- world settings. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 8759-8773. PMLR, 17-23 Jul 2022.
[145] Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. In International Conference on Learning Rep- rexentations, 2017.
[146] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations. 2019.
[147] Jens Henriksson, Christian Berger, Markus Borg, Lars Tornberg, Cristofer Englund, Sankar Ra- man Sathyamoorthy, and Stig Ursing. Towards structured evaluation of deep neural network supervisors. In 2019 IEEE International Conference On Artificial Intelligence Testing (AlTest), pages 34.34. IEEE, 2019.
[148] Ulysse Herbach. Gene regulatory network inference from single-cell data using a self- consistent proteomic field, 2021.
[149] Justus I. Hibshman and Tim Weninger. Inherent limits on topology-based link prediction, 2023.
[150] Steven A Hicks, Inga Strümke, Vajira Thambawita, Malek Hammou, Michael A Riegler, Pål Halvorsen, and Sravanthi Parasa. On evaluation metries for medical applications of artificial intelligence. Scientific reports, 12(1):5979, 2022.
[151] Anna Himmelhuber, Mitchell Joblin, Martin Ringsquandl, and Thomas Runkler. Demystifying graph neural network explanations. In Join! European Conference on Machine Learning and Knowledge Discovery in Databases, pages 67-75. Springer, 2021.
[152] David Hin, Andrey Kan, Huaming Chen, and M Ali Babar. Linevd: Statement-level vulnerabil- ity detection using graph neural networks. In Proceedings of the 19th international conference on mining software repositories, pages 596-607 2022.
[153] Kaoutar Daoud Hiri, Matjaž Hren, and Tomaž Curk. NIp-based classification of software tools for metagenomics sequencing data analysis into edam semantic annotation, 2022.
[154] Ryohei Hisano, Didier Sornette, and Takayuki Mizuno. Prediction of esg compliance using a heterogencous information network. Journal of Big Data, 7(1), March 2020.
19

---

[155] Charmgil Hong and Milos Hauskrecht. Mcode: Multivariate conditional outlier detection. arXiv preprint arXiv:1505.0409: 2015.
[156] Shenda Hong, Cao Xiao, Trong Nghia Hoang, Tengfei Ma, Hongyan Li, and Jimeng Sun. Rdpd: Rich data helps poor data via imitation. In 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, pages 5895-5901. International Joint Conferences on Artificial Intelligence, 2019.
[157] Shenda Hong, Cao Xiao, Tengfei Ma. Hongyan Li, and Jimeng Sun. Mina: Multilevel knowledge-guided attention for modeling electrocardiograpyy signals. In Proceedings of the Eventy-Eights International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5888- 5894. International Joint Conferences on Artificial Intelligence Organization, 7 2019.
[158] Julia Hornauer and Vasileios Belagiannis. Heatmap-based dittt--distribttion detection. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, January 2023.
[159] Chao-Chun Hsu, Shantanu Karnwal, Sendhil Mullainathan, Ziad Obermeyer, and Chenhao Tan. Characterizing the value of information in medical notes. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2062-2072, Online, November 2020. Association for Computational Linguistics.
[160] Stanley Bryan Z. Hua, Mandy Rickard, John Weaver, Alice Xiang, Daniel Alvarez, Kyla N. Velear, Kunj Sheth, Gregory E. Tasian, Armando Lorenzo, Anna Goldenberg, and Lauren Erdman. From single-visit to multi-visit image-based models: single-visit models are enough predict obstructive hydronephrosis. Marius G. Linguraru, Letícia Rittner, and Eduardo Romero Castro M.D., editors, 18th Inter- national Symposium on Medical Information Processing and Analysis, volume 12567, page 1256710 10. International Society for Optics and Photonics, SPIE, 2023.
Jorge Brieva, Pamela Guevara, Natasha Lepore.
[161] Yiqing Hua, Armin Namavari, Kaishuo Cheng, Mor Naaman, and Thomas Ristenpart. In- creasing adversarial uncertainty scale private similarity testing. In 31st USENIX Security Symposium (USENIX Security 22), pages 1777-1794, 2022.
[162] Benjamin Hughes and Tilo Burghardt. Automated visual fin identification of individual great white sharks. International Journal of Computer Vision, 122:542-557, 2017.
[163] Zepeng Huo, Xiaoning Qian, Shuai Huang, Zhangyang Wang, and Bobak J Mortazavi. Density- aware personalized training for risk prediction in imbalanced medical data. In Machine Learning for Healthcare Conference, pages 101-122. PMLR, 2022.
[164] Fantine Huot, R. Lily Hu, Nita Goyal, Tharun Sankar, Matthias Ihme, and Yi-Fan Chen. Next day wildfire spread: A machine learning dataset to predict wildfire spreading from remote-sensing data. IEEE Transactions Geoscience and Remote Sensing, 60:1-13, 2022.
[165] Vân Anh Huynh- Thu and Guido Sanguinetti. Gene regulatory network inference: an introduc- tory survey. Gene regulatory networks: Methods and protocols, pages 1-23, 2019.
[166] Zina M Ibrahim, Daniel Bean, Thomas Searle, Linglong Qian, Honghan Wu, Anthony Shek, Zeljko Kraljevic, James Galloway, Sam Norton, James TH Teo, et al. A knowledge distillation ensemble framework for predicting short-and long term hospitalization outcomes from elec- tronic health records data. IEEE Journal of Biomedical and Health Informatics, 26(1) 26(1):423-44 435, 2021.
[167] Yotam Intrator, Gilad Katz, and Asaf Shabtai. Mdgan: Boosting anomaly detection using multi-discriminn generative adversarial networks, 2018
[168] Mary Isangediok and Kelum Gajamannage. Fraud detection using optimized machine learning tools under imbalance classes. In 2022 IEEE International Conference on Big Data (Big Data), pages 4275-4284. IEBE, 2022.
[169] Olga Isupova, Danil Kuzin, and I Lyddmia Mihaylova. Learning methods for dynamic topic modeling in automated behavior analysis. IEEE transactions on neural networks and learning systems, 2989))3880-399 2017.
20

---

[170] Jay Jacobs, Sasha Romanosky, Octavian Suciu, Benjamin Edwards, and Armin Sarabi. En- hancing vulnerability prioritization: Data-driven exploit predictions with community-driver insights, 2023.
[171] Lucas Jaffe, Michael Zelinski, and Wesam Sakla. Remote sensor design for visual recognition with convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing, 57(1 1):9090-9108, November 2019.
[172] Shantanu Jain, Martha White, and Predrag Radivojac. Recovering true classifier performance in positive-unlabeled learning. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017.
[173] Dong-ju Jeong, Insung Hwang, and Nam Ik Cho. Co-salient object detection based on deep saliency networks and seed propagation over an integrated graph. IEEE Transactions on Image Processing, 27(12):886-5887 2018.
[174] Biaobin Jiang. Kyle Kloster, David F Gleich, and Michael Gribskov. Aptrank: an adaptive pagerank model for protein function prediction on bi-relational graphs. Bioinformatics. 13(12)1882-1883, 2017.
[175] Wei Jiang, Jing-Hap Xue, and Weichuan Yu. Estimating reproducibility in genome-wide association studies, 2015.
[176] Yunjiang Jiang, Yue Shang, Rui Li, Wen-Yun Yang. Guoyu Tang, Chaoyi Ma, Yun Xiao, and Eric Zhao. A unified neural network approach to e-commerce relevance learning. In Pro- ceedings of the 1st International Workshop on Deep Learning Practice for High- high-DDimessioal Sparse Data, pages 1-7, 2019.
[177] Di Jin, Shuyang Gao, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tür. Towards textual out-of-domain detection without in-domain labels. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:1386-1399 2022.
[178) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[179] Eric Jonas, Monica Bobra, Vaishaal Shankar, J Todd Hoeksema, and Benjamin Recht. Flare prediction using photospheric and coronal image data. Solar Physics, 293(3):48, 2018.
[180] Cheng Ju, James Li, Bram Wasti, and Shengbo Guo. Semisupervised learning on heterogeneous graphs and its applications facebook news feed, 2018
[181] Sanghun Jung, Jungsoo Lee, Dachoon Gwak, Sungha Choi, and Jaegul Choo. Standardized max logits: A simple yet effective approach for identifying unexpected road obstacles in urban-scene segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 55425-15434 2021.
[182] Ruthwik R Junuthula, Kevin S Xu, and Vijay K Devabhaktuni. Evaluating link prediction accuracy in dynamic networks with added and removed edges. In 2016 IEEE international conferences on big data and cloud computing (BDCloud), social computing and networking (SocialCom), sustainable computing and communications CSust -omial SustainCom), pages 377-384. IEEE, 2016.
[183] Georgi Karadzhov, Tom Stafford, and Andreas Vlachos. What makes you change your mind? conversations, 2022.
an empirical investigation in online group
[184] Md Rezaul Karim, Michael Cochez, Joao Bosco Jares, Mamtaz Uddin, Oya Beyan, and Stefan Decker. Drug-drug interaction prediction based on knowledge graph embeddings and convolutional-Istm network. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pages 113- 2019.
[185] Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Explainable deep relational networks for predicting compound-prot protein affinities and contacts. Journal of chemical infor- mation and modeling, 61(1):46-66, 2020.
21

---

[186] Kleomenis Katevas, Katrin H" ansel, Richard Clegg, Ilias Leontiadis, Hamed Haddadi, and Laurissa Tokarchuk. Finding dory in the crowd: Detecting social interactions using multi- modal mobile sensing. In Proceedings of the 1st Workshop on Machine Learning on Edge in Sensor Systems, pages 37-42, 2019.
[187] Michal Kazmierski, Mattea Welch, Sejin Kim. Chris Mcintosh, Princess Margaret Head, Neck Cancer Group. Katrina Rey-Melntyre. Shao Hui Huang, Tirth Patel. Tony Tadic, Michael Milosevic, Fei-Fei Liu, Andrew Hope, Scott Bratman, and Benjamin Haibe- A machine learning challenge for prognostic modelling head and neck cancer using multi-modal data, 2021.
[188] Said Kerrache and Hafida Benhidour. A complex network based graph embedding method for link prediction, 2022.
[189] Kathleen R Kerwin and Nathaniel D Bastian. Stacked generalizations in imbalanced fraud data sets using resampling methods. The Journal of Defense Modeling and Simulation, 18(3):175-192, 2021.
2021.
[190] Refilwe Kgoadi, Chris Engelbrecht, Ian Whittingham, and Andrew Tkachenko. General classification of light curves using extreme boosting, 2019.
[191] Jang-Hyun Kim, Sangdoo Yun, and Hyun Oh Song. Neural relation graph: unified framework for identifying label noise and outlier data. Advances in Neural Information Processing Systems, 36, 2024.
[192] Minkyung Kim, Junsik Kim, Jongmin Yu, and Jun Kyun Choi. v. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pages 39 39-46, 2022.
[193] B. Ravi Kiran, Dilip Mathew Thomas, and Ranjith Parakkal. An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. Journal of Imaging, 4(2), 2018.
[194] Ivan Kiskin, Marianne Sinka, Adam D. Cobb, Waqas Rafique, Lawrence Wang, Davide Zilli, Benjamin Gutteridge, Rinita Dam, Theodoros Marinos, Yunpeng Li, Dickson Msaky, Emmanuel Kaindoa, Gerard Killeen, Eva Herreros-Moya, Kathy Willis, and Stephen J. Roberts. Humbugdb: A large-scale acoustic mosquito dataset, 2021.
[195] Ross S. Kleiman, Paul S. Bennett, Peggy I.. Peissig, Richard L. Berg, Zhaobin Kuang, Scott J. Hebbring, Michael D. Caldwell, and David Page. High-throughput machine learning from electronic health records, 2019.
[196] Nikita Klyuchnikov, Alexey Zaytsev, Arseniy Gruzdev, Georgiy Ovehinnikov, Ksenia An- tipova, Leyla Ismailova, Ekaterina Muravleva, Evgeny Burnaev, Artyom Semenikhin, Alexey Cherepanov, et al. Data-driven model for the identification of the rock type at a drilling bit. Journal of Petroleum science and Engineering, 178:506-55 516, 2019.
[197] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xic, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5637-5664. PMLR, 18-24 Jul 2021.
[198] Yunchuan Kong and Tianwei Yu. forgenet: a graph deep neural network model using tree- based ensemble classifiers for feature graph construction. Bioinformatics, 36(11):3507-311 03 2020.
[199] Anna-Kate Kata-Kathrinn Kopetzki, Bertrand Charpentier, Daniel Z"ugner, Sandhya Giri, and Stephan G G"unnemann. Evaluating robustness of predictive uncertainty estimation: Are dirichlet-based models reliable? In International Conference on Machine Learning, pages 5707-5718. PMIR, 2021.
22

---

[200] Denis KrompaB, Stephan Baier, and Volker Tresp. Type-constrained representation learning in knowledge graphs. In The Semantic Web-ISWC 2015: 14th International Semantic Web Conference, Bethlehem, PA, USA, October I 11-5, 2015, Proceedings, Part 14, pages 640-655. Springer, 2015.
[201] Sofia Ira Ktena, Alykhan Tejani, Lucas Theis, Pranay Kumar Myana, Deepak Dilipkumar, Ferenc Huszár, Steven Yoo, and Wenzhe Shi. Addressing delayed feedback for continuous training with neural networks in ctr prediction. In Proceedings of the 13th ACM Conference Recommender Systems, RecSys '1,, page 187-195, New York, NY, USA, 2019. Association for Computing Machinery.
[202] Viraj Kulkarni, Manish Gawali, and Amit Kharat. Key technology considerations in developing and deploying machine learning models in clinical radiology practice. JMIR Med Inform. 9()):22877, Sep 2021.
[203] Aditya Kunar. Effective and privacy preserving tabular data synthesizing, 2021.
[204] Trent Kyono, Fiona Gilbert, and Mihacla van der Schaar. Mammo: A deep learning solution for facilitating radiologist- aadillggis-mmachne
machine collaboration in breast cancer diagnosis, 2018.
[205] Precthi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Faimess without demographics through adversarially reweighted learning. Advances in neural information processing systems, 33:728 18002 2020.
[206] Conrad Lee, Bobo Nick, Ulrik Brandes, and Pádraig Cunningham. Link prediction with social vector clocks. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages
2013.
[207] Hao-Chih Lee, Matteo Danieletto, Riccardo Miotto, Sarah Cherng, and Joel Dudley. Scaling structural learning with no-bears infer causal transcriptome networks. In Pacific Symposium on Biocomputing 2020, pages 391-402, World Scientific, 2019.
[208] I-Ta Lee, Manish Marwah, and Martin Arlitt. Attention-based self-supervised feature learning for security data, 2020.
[209] Joongoo Lee and Min-Su Shin. Estimation of photometric redshifts. ii identification of out-of-distribution data with neural networks. The Astronomical Journal, 163(2):98, 2022.
[210] Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, al. Predicting deep zero-shot convolutional neural networks using textual descriptions. In Proceedings of the IEEE international conference on computer vision, pages 4247-4255. 2015.
[211] Jussi Leinonen, Ulrich Hamann, and Urs Germann. Seamless lightning nowcasting with recurrent-convolle conoouttoollnlll deep learning. Artificial Intelligence for the Earth Systems, 1(4):e22004,, 2022.
[212] Daniel E Leisman. Rare events in the icu: an emerging challenge in classification and prediction. Critical care medicine, 46(3):418-424, 2018.
[213] Jake Lever. Classification evaluation: It is important understand both what a classification metric expresses and what it hides. Nature methods, 13(8):603-66) 605, 2016.
[214] Jingyi Jessica Li and Xin Tong. Statistical hypothesis testing versus machine learning binary classification: Distinctions and guidelines. Patterns, 1(7): October 2020.
[215] Longyuan Li, Junchi Yan, Haiyang Wang, and Yaohui Jin. Anomaly detection of time series with smoothness-inducee sequential variational auto-encoder. IEEE transactions neural networks and learning systems, 32(3): 177-11911 2020.
[216] Qiujia Li, PM Ness, Anton Ragni, and Mark JF Gales. Bi-directional lattice recurrent neural networks for confidence estimation. In ICASSP 2019-2019 JEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6755-6759. IEEE, 2019.
23

---

[217] Qiujia Li, Yu Zhang, David Qiu, Yanzhang He, Liangliang Cao, and Philip C Woodland. Improving confidence estimation on out-of-domain data for end-to-end speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 6537-6541, IEEE, 2022.
[218] Wenyuan Li, Yunlong Wang, Yong Cai, Corey Arnold, Emily Zhao, and Yilian Yuan. Semi- supervised rare disease detection using generative adversarial network, 2018.
[219] Xiaoxiao Li, Rabah Al-Zaidy, Amy Zhang. Stefan Baral, Le Bao, and C. Lee Giles. Automating document classification with distant supervision to increase the efficiency systematic reviews, 2020.
[220] Xiaoya Li, Jiwei Li, Xiaofei Sun, Chun Fan, Tianwei Zhang, Fei Wu, Yuxian Meng. and Jun Zhang. kfolden: k-fold ensemble for out-of-distribution detection. In Maric-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih, editors. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3102-3115 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[221] Yufei Li, Simin Chen, and Wei Yang. Estimating predictive uncertainty under program data distribution shift. arXiv preprint arXiv:2107. 10989, 2021.
[222] Ryan Lichtnwalter and Nitesh V Chawla. Link prediction: fair and effective evaluation. In 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pages 376-383. IEEE, 2012.
[223] Bryan Lim and Mihaela van der Schaar. Disease-atlas: Navigating disease trajectories using deep learning. In Machine Learning for Healthcare Conference, pages 137-160. PMLR, 2018.
[224] Krzysztof Lis, Sina Honari, Pascal Fua, and Mathieu Salzmann, Detecting road obstacles by erasing them. IEEE transactions pattern analysis and machine intelligence, 2023.
[225] Bin Liu, Konstantinos Blekas, and Grigorios Tsoumakas. Multi-label sampling based on local label imbalance. Puttern Recognition, 122: 122: 108294, 2022.
[226] Bin Liu, Dimitrios Papadopoulos, Fragkiskos D Malliaros, Grigorios Tsoumakas, and Aposto- los N Papadopoulos. Multiple similarity drug-targed interaction prediction with random walks and matrix factorization. Briefings in Bioinformatics, 23(5):bbac333, 2022.
[227] Bin Liu, Jin Wang, Kaiwei Sun, and Grigorios Tsoumakas. Fine-grained selective similarity integration for drug-target interaction prediction. Briefings in Bioinformatics, 24(2):bbad085, 2023.
[228] Hanyang Liu, Michael Montana, Dingwen Li, Chase Renfroe, Thomas Kannampallil, and Chenyang Lu. Predicting intraoperative hypoxemia with hybrid inference sequence autoen- coder networks. In Proceedings of the 31 ACM International Conference on Information & Knowledge Management, pages 1269 1278, 2022.
[229] Yang Liu, Dingkang Yang, Yan Wang, Jing Liu, Jun Liu, Azzedine Boukerche, Peng Sun, and Liang Song. Generalized video anomaly event detection: Systematic taxonomy and comparison of deep models, 2023.
[230] Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. Mesa: boost ensemble imbalanced learning with meta-sampler. Advances in neural information processing systems, 33:14463-11444 14474, 2020.
[231] Artyom Lobanov, Egor Bogomolov, Yaroslav Golubev, Mikhail Mirzayanov, and Timofey Bryksin. Predicting tags for programming tasks by combining textual and source code data, 2023.
[232] Daniel Larez-Martinez, Christina Chen, and Ming-Jun Chen. Machine learning for dynami- cally predicting the onset of renal replacement therapy in chronic kidney disease patients using claims data. International Workshop on Applications of Medical AI, pages 18-28. Springer, 2022.
24

---

[233]
Daniel Lopez-Martinees Alex Yakubovich, Martin Seneviratne, Adam D. Lelkes, Akshit Tyagi. Jonas Kemp. Ethan Steinberg. N. Lance Downing, Ron C. Li, Keith E. Morse, Nigam H. Shah, and Ming-Jun Chen. Instability in clinical risk stratification models using deep learning. In Antonio Parziale, Monica Agrawal, Shalmali Joshi, Irene Y. Chen, Shengpu Tang, Luis Oala, and Adarsh Subbaswamy, editors, Proceedings of the 2nd Machine Learning for Health symposium, volume 193 of Proceedings of Machine Learning Research, pages 552-565. PMLR, 28 Nov 2022.
[234] Guan-Rong Lu, Yueh-Cheng Liu, Tung-I Chen, Hung-Ting Su, Tsung-Han Wu, and Winston H Hsu. Anomaly-aware semantic segmentation by leveraging synthetic-unknen data. arXiv 2021.
preprint
[235] Yvan Lucas,
Portier, Léa Laporte, Sylvie Calabretto, Olivier Caelen, Liyun He- Ele-Guelton, and Michael Granitzer. Multiple perspectives hmm-based feature engineering for credit card fraud detection. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC 11, page 1359-1361, New York, NY, USA, 2019. Association for Computing Machinery.
[236] Jeffiey Lund, Piper Armstrong, Wilson Fearn, Stephen Cowley, Emily Hales, and Kevin Seppi. Cross-referencing using fine-grained topic modeling. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3978-3987, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[237] Julia Lust and Alexandru Paul Condurache. A survey on assessing the generalization envelope of deep neural networks: Predictive uncertainty, dittiiitibtttion and adversarial samples, 2021.
[238] Yingzhe Lyu, Gopi Krishnan Rajbahadur, Dayi Lin, Boyuan Chen, and Zhen Ming (Jack) Jiang. Towards consistent interpretation of aiops models. ACM Trans. Softw. Eng. Methodol., (1(]), nov 2021.
[239] Victoria Lopez, Alberto Fernández, Salvador Gareía, Vasile Palade, and Francisco Herrera. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics. Information Sciences, 250: 13-141, 2013.
[240] Liantao Ma, Junyi Gao, Yasha Wang. Chaohe Zhang. Jiangtao Wang, Wenjie Ruan, Wen Tang, Xin Gao, and Xinyu Ma. Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 825- 832, 2020.
[241] Liantao Ma, Chaohe Zhang, Junyi Gao, Xianfeng Jiao. Zhihao Yu. Yinghao Zhu, Tianlong Wang, Xinyu Ma. Yasha Wang, Wen Tang, et al. Mortality prediction with adaptive feature importance recalibration for peritoneal dialysis patients. Patterns, 4(12), 2023.
[242] Liantao Ma, Chaohe Zhang, Yasha Wang. Wenjie Ruan, Jiangtao Wang, Wen Tang, Xinyu Ma, Xin Gao, and Junyi Gao. Concare: Personalized clinical feature embedding via capturing the healthcare context. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 340-840,22 2020.
[243] Xinyu Ma, Xu Chu, Yasha Wang, Hailong Yu, Liantao Ma, Wen Tang, and Junfeng Zhao. Medfact: Modeling medical feature correlations in patient health representation learning via feature clustering. 2022.
[244] Yiming Ma. Victor Sanchez, Soodeh Nikan, Devesh Upadhyay, Bhushan Atote, and Tanaya Guha. Real-time driver monitoring systems through modality and view analysis. arXiv preprint arXiv:2210.09441, 2022.
[245] Brielen Madureira and David Schlangen. Instruction clarification requests in multimodal collaborative dialogue games: Tasks, and an analysis of the CoDraw dataset. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2303-2319, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics.
25

---

[246] Lena Maier- Hein, Annika Reinke, Patrick Godau, Minu D Tizabi, Florian Buettner, Evangelia Christodoulou, Ben Glocker, Fabian Isensee, Jens Kleesiek, Michal Kozubek, et al. Metrics reloaded:
recommendations for image analysis validation. Nature methods, pages 1-18, 2024.
[247] Andrey Malinin and Mark Gales, Predictive uncertainty estimation via prior networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[248] Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. In International Conference on Learning Representations, 2020.
[249] Sunil Mallya, Marc Overhage, Sravan Bodapati, Navneet Srivastava, and Sahika Genc. Savehr: self attention vector representations for ehr based personalized chronic disease onset prediction and interpretability. arXiv preprint arXiv: 1eti::1911.053311 1.05370, 2019.
[250] Saurav Manchanda, Pranjul Yadav, Khoa Doan, and S Sathiya Keerthi. Targeted display advertising: the case of preferential attachment. In 2019 IEEE International Conference on Big Duta (Big Datu), pages 1868-1877. IEEE, 2019.
[251] Rafael B Mangolin, Rodolfo M Pereira, Alceu Britto Jr, Carlos N Silla Valéria D Feltrim, Diego Bertolini, and Yandre MG Costa. A multimodal approach for multi-label movie genre classification. Multimedia Tools and Applications, (111))19071 2022.
[252] Johan Markdahl, Nicolo Colombo, Johan Thunherg. and Jorge Gonçalves. Experimental design trade-offs for gene regulatory network inference: An in silico study of the yeast saccharomyces cerevisiae cell cycle. In 20/7 IEEE 56th Annual Conference Decision and Control (CDC), pages 423-428. IEEE, 2017.
[253] Mansour Zoubeirou A Mayaki and Michel Riveill. Multiple inputs neural networks for fraud detection. In 2022 International Conference Machine Learning, Control, and Robotics (MLCR), pages 8-13, 2022.
[254] Samuele Mazzanti. Why you should stop using the roc curve, September 2023.
[255] John McKay, Isaac Gerg, and Vishal Monga. Bridging the gap: Simultaneous fine tuning for data re-balancing. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pages 7062-7065. IEEE, 2018
[256] Matúš Medo, Daniel M Aebersold, and Michaela Medová. Protrank: bypassing the imputation of missing values in differential expression analysis of proteomic data. BMC bioinformatics, 20:1-12 2019.
[257] Aryan Mehboudi, Shrawan Singhal, and S. V. Sreenivasan. Squeeze flow of micro-droplets: convolutional neural network with trainable and tunable refinement, 2022.
[258] Julia A Meister, Khuong An Nguyen, and Zhiyuan Luo. Audio feature ranking for sound-hased covid-19 patient detection. In EPIA Conference on Artificial Intelligence, pages 146-158. Springer, 2022.
[259] David Merrell and Anthony Gitter. Inferring signaling pathways with probabilistic program- ming. Bioinformatics, 36(Supplement 1:1822--830. December 2020.
[260] Mika A Merrill and Tim Althoff. Self-supervised pretraining and transfer learning enable flu and covid-19 19 predictions in small mobile sensing datasets. In Bobak J. Mortazavi, Tasmie Sarker, Andrew Beam, and Joyce C. Ho, editors, Proceedings of the Conference on Health, Inference, and Learning, volume 209 of Proceedings of Machine Learning Research, pages 191-206. PMLR, 22 Jun-24 Jun 2023.
[261] Jiaju Miao and Wei Zhu. reciision-reaall curve (pre) classification trees. Evolutionary intelligence, 15(3):1545-156) 2022.
[262] Kai Middlebrook, Shyam Sudhakaran, and David Guy Brizan. Muslcat: Multi-scale multi-level convolutional attention transformer for discriminative music modeling on raw waveforms, 1021.
26

---

[263] Dimity Miller, Feras Dayoub, Michael Milford, and Niko S" underhauf.. Evaluating merging strategies for sampling- based uncertainty techniques in object detection. In 2019 international conference robotics and automation (icra), pages 2348-222 2354.. IEFE, 2019.
[264] Shahryar Minhas, Peter D Hoff, and Michael D Ward. Inferential approaches for network analysis: Amen for latent factor models. Political Analysis, 27(2):208 222, 2019.
[265] Amanda J Minnich, Kevin McLoughlin, Margaret Tse, Jason Deng, Andrew Weber, Neha Murad, Benjamin Madej. Bharath Ramsundar, Tom Rush, Stacie Calad-Thomson, al. Ampl: a data-driven modeling pipeline for drug discovery. Journal of chemical information and modeling, 60(4):1955-1965 2020.
[266] Yisroel Mirsky, Tomer Golomb, and Yuval Elovici. Lightweight collaborative anomaly detection for the iot using blockchain. Journal of Parallel and Distributed Computing, 145:75- 97, 2020.
[267] Pratik K Mishra, Alex Mihailidis, and Shehroz S Khan. Skeletal video anomaly detection using deep learning: Survey, challenges, and future directions. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024.
[268] Ali Mollahosseini, Behzad Hasani, and Mohammad H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18-33) 31, January 2019.
[269] Lena Mondrejevski, Ioanna Miliou, Annaclaudia Montanino, David Pitts, Jaakko Hollmén, and Panagiotis Papapetrou. Flicu: A federated learning workflow for intensive care unit mortality prediction. In 2022 IEEE 35th International Symposium Computer-Based Medical Systems (CBMS), pages 32-37. IEEE, 2022.
[270] Aanchal Mongia, Sanjay Kr Saha, Emilie Chouzenoux, and Angshul Majumdar. A compu- tational approach to aid clinicians in selecting anti-viral drugs for covid-19 trials. Scientific reports, (11):9907, 2021.
[271] Michael Moor, Max Horn, Bastian Rieck, Damian Roqueiro, and Karsten Borgwardt. Early recognition of sepsis with gaussian process temporal convolutional networks and dynamic time warping. In Finale Doshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens, editors, Proceedings of the 4th Machine Learning for Healthcare Conference, volume 106 of Proceedings of Machine Learning Research, pages 2-26. PMLR. 09-10 Aug 2019.
[272] José Morano, Álvaro S Hervella, N Barreira, Jorge Novo, and José Rouco. Multimodal transfer learning-based approaches for retinal vascular segmentation. In ECA/ 2020, pages 1866-1873. IOS Press, 2020.
[273] José Morano, Álvaro S. Hervella, Jorge Novo, and José Rouco. Simultaneous segmentation and classification of the retinal arteries and veins from color fundus images. Artificial Intelligence in Medicine, 118:102116 August 2021.
[274] Candelaria Mosquera, Luciana Ferrer, Diego Milone, Daniel Luna, and Enzo Ferrante. Im- pact of class imbalance on chest x-ray classifiers: towards better evaluation practices for discrimination and calibration performance, 2022.
[275] Pablo Mosteiro, Emil Rijcken, Kalliopi Zervanou, Uzay Kaymak, Floortje Scheepers, and Marco Spruit. Machine learning for violence risk assessment using dutch clinical notes. Journal of Artificial Intelligence for Medical Sciences, 2(1-2):44-55 54, 2021.
[276] Zaynab Mousavian, Sahand Khakabimamaghani, Kaveh Kavousi, and Ali Masoudi-Nejad. Drug-target interaction prediction from pssm based evolutionary information. Journal of pharmacological and toxicological methods, 78:42-51, 2016.
[277] Faezeh Movahedi, Rema Padman, and James F Antaki. Limitations of receiver operating characteristic curve on imbalanced data: assist device mortality risk scores. The Journal of Thoracic and Cardiovascular Surgery, 165(4): 2023.
27

---

[278] Christopher W Murphy. Class imbalance techniques for high energy physics. SciPost Physics, 7(6):076, 2019.
[279] Daniel Muthukrishna, Gautham Narayan, Kaisey S Mandel, Rahul Biswas, and Renée Hložek. Rapid: early classification of explosive transients using deep learning. Publications of the Astronomical Society of the Pacific, 131(1005): 18002, 2019.
[280] Anubhav Reddy Nallabasannagari, Madhu Reddiboina, Ryan Seltzer, Trevor Zeffiro, Ajay Sharma, and Mahendra Bhandari. All data inclusive, deep learning models to predict critical events in the medical information mart for intensive care iii database (mimic iii), 2020.
[281] Santhosh Narayanan, Carsten Maple, and Mark Hooper. A point process model for rare event detection, 2022.
[282] Jose M Navarro, Alexis Huet, and Dario Rossi. Human readable network troubleshooting based on anomaly detection and feature scoring. Computer Networks, 219:109447, 2022.
[283] Bijay Neupane, Torben Bach Pedersen, and Bo Thiesson. Utilizing device-level demand forecasting for flexibility markets- version. arXiv preprint arXiv:1805.00m 2018.
[284] Eli Newby, Jorge Gómez Tejeda Zañudo, and Réka Albert. Structure-based approach to identifying small sets of driver nodes biological networks. Chaos: An Interdisciplinary Journal of Nonlinear Science, 32(6):063102 06 2022.
[285] Minh-Nghia Nguyen and Ngo Anh Vien. Scalable and interpretable one-class svms with deep learning and random fourier features. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part 18, pages 157-172. Springer, 2019.
[286) Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11-33, January 2016.
[287] Zhale Nowroozilarki, Arash Pakbin, James Royalty, Donald KK Lee, and Bobak J Mortazavi. Real-time mortality prediction using mimic-iv icu data via boosted nonparametric hazards. In 2021 IEEE EMBS international conference on biomedical and health informatics (BHI), pages 1-4. IEBE, 2021.
[288]
Antonios Ntroumpogiannis, Michail Giannoulis, Nikolaos Myrtakis, Vassilis Christophides. Eric Simon, and loannis Tsamardinos. A meta-level analysis of online anomaly detectors. The VLDB Journal, pages 1-42, 2023.
[289] Chris Oates, Richard Amos, and Simon EF Spencer. Quantifying the multi-scale performance of network inference algorithms. Statistical Applications in Genetics and Molecular Biology, 13(5):611-631, 2014.
[290] Philipp Oberdiek, Matthias Rottmann, and Hanno Gottschalk. Classification uncertainty of deep neural networks based on gradient information. In Artificial Neural Networks in Pattern Recognition: 8th IAPR TC3 Workshop, ANNPR 2018, Siena, Italy, September 19-21, 2018, Proceedings 8, pages 1 113-125. Springer, 2018.
[291] María Öskarsdóttir; Waqas Ahmed, Katrien Antonio, Bart Baesens, Rémi Dendievel, Tom Donas, and Tom Reynkens. Social network analytics for supervised fraud detection in insurance. 2022.
Risk Analysis,
[292] Brice Ozenne, Fabien Subtil, and Delphine Maucort-Boulch. The precision-recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. Journal of clinical epidemiology, 68(8):855-88 859, 2015.
[293] Ozan Ozyegen, Devika Kabe, and Mucahit Cevik. Word-level text highlighting of medical texts for telehealth services. Artificial Intelligence in Medicine, 127:102284, 2022.
[294] José A. Padrón-Hidalgo. Valero Laparra, and Gustau Camps-Valls. Unsupervised anomaly and change detection with multivariate gaussianization. IEEE Transactions on Geoscience and Remote Sensing, 60: 60:1-10, 2022.
28

---

[295] Aniello Panariello, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Consistency- based self-supervised learning for temporal anomaly localization. In European Conference on Computer Vision, pages 338-349, Springer, 2022.
[296] Guansong Pang, Chunhua Shen, Huidong Jin, and Anton van den Hengel. Deep weakly- supervised anomaly detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 1795-1807, New York, NY, USA, 2023, Association for Computing Machinery.
[297] Guansong Pang, Chunhua Shen, and Anton Van Den Hengel. Deep anomaly detection with deviation networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 353 353-362,22 2019.
[298] Guansong Pang, Anton van den Hengel, Chunhua Shen, and Longbing Cao. Toward deep supervised anomaly detection: Reinforcement learning from partially labeled anomaly data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, KDD '21. ACM, August 2021.
[299] Fragkiskos Papadopoulos and Kojja Kleineberg, Link persistence and conditional distances multiplex networks. Physical Review E, (9(1):01232,, 2019.
[300] Min Sue Park, Hyeontae Jo, Haeun Lee, Se Young Jung, and Hyung Ju Hwang. Machine learning-based covid-19 patients triage algorithm using patient-generated health data from nationwide multicenter database. Infectious Diseases and Therapy, 11(2):787-805, February 2022.
[301] Ilya N Pashchenko, Kirill V Sokolovsky, and Panagiotis Gavras. Machine learning search for variable stars. Monthly Notices of the Royal Astronomical Society, 475(2):2326-2343, 2018.
[302] Ali Payani and Faramarz Fekri. Inductive logie programming via differentiable deep neural logic networks. arXiv preprint arXiv: 2019.
[303] Xueping Peng. Guodong Long, Tao Shen, Sen Wang, and Jing Jiang. Self-attention enhanced patient journey understanding in healthcare system. In Frank Hutter, Kristian Kersting, Jefrey Lijftijt, and Isabel Valera, editors, Machine Learning and Knowledge Discovery in Databases, pages 719-735, Cham, 2021. Springer International Publishing.
[304] Xueping Peng, Guodong Long, Tao Shen, Sen Wang, Jing Jiang, and Michael Blumenstein. Temporal self elffaateention network for medical concept embedding. In 2019 IEEE international conference data mining (ICDM), pages 498 507. IEEE, 2019.
[305] Xueping Peng, Guodong Long, Tao Shen, Sen Wang, Jing Jiang, and Chengqi Zhang. Bitenet: bidirectional temporal encoder network to predict medical outcomes. In 2020 IEEE Interna- tional Conference Data Mining (ICDM), pages 412 12-422. IEEE, 2020.
[306] Eike Petersen, Melanie Ganz, Sune Holm, and Aasa Feragen. On (assessing) the faimess of risk score models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT '23, page 817-829, New York, NY, USA, 2023. Association for Computing Machinery.
[307] Stephen Pfohl, Yizhe Xu, Agata Foryciarz, Nikolaos Ignatiadis, Julian Genkins, and Nigam Shah. Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1039-1052, 2022.
[308] Dario Piermarini, Antonio M. Sudoso, and Veronica Piccialli. Predicting municipalities in financial distress: a machine learning approach enhanced by domain expertise, 2023.
[309] Walter Hugo Lopez Pinaya,
Tudosiu, Robert Gray, Geraint Rees, Parashkev Nachev, Sébastien Ourselin, and M. Jorge Cardoso. Unsupervised brain anomaly detection and segmentation with transformers. Mattias Heinrich, Qi Dou, Marleen de Bruijne, Jan Lellmann, Alexander Schläfer, and Floris Ernst, editors, Proceedings of the Fourth Conference on Medical Imaging with Deep Learning, volume 143 of Proceedings of Machine Learning Research, pages 596-617. PMLR, 07-09 Jul 2021.
29

---

[310] Adrian Alan Pol, Thea Aarrestad, Katya Govorkova, Roi Halily, Tal Kopetz, Anat Klempner, Vladimir Loncar, Jennifer Ngadiuba, Maurizio Pierini, Olya Sirkin, et al. Jet single shot detection. In EPJ Web of Conferences, volume 251, page 04027. EDP Sciences, 2021.
[311] Jordi Pons, Oriol Nieto, Matthew Prockup, Erik Schmidt, Andreas Ehmann, and Xavier Serra. End-to-end learning for music audio tagging at scale. In Proceedings of the 19th International Society for Music Information Retrieval Conference, 2018.
[312] Ioannis Prapas, Akanksha Ahuja, Spyros Kondylatos, Ilektra Karasante, Eleanna Panagiotou, Lazaro Alonso, Charalampos Davalas, Dimitrios Michail, Nuno Carvalhais, and Ioannis Papoutsis. Deep learning for global wildfire forecasting, 2023.
[313] Sam Preston, Mu Wei, Rajesh Rao, Robert Tinn, Naoto Usuyama, Michael Lucas, Roshanthi Weerasinghe, Soohee Lee, Brian Piening, Paul Tittel, Naveen Valluri, Tristan Naumann, Carlo Bifulco, and Hoifung Poon. Towards structuring real-world data at scale: Deep learning for extracting key oncology information from clinical text with patient-level supervision, 2022.
[314] Qi Qi, Youzhi Luo, Zhao Xu, Shuiwang Ji, and Tianbao Yang. Stochastic optimization of areas under rrcall curves with provable convergence. Advances in neural information processing systems, 34:1752-1765, 2021.
[315] Gwenolé Quellec, Hassan AI Hajj. Mathieu Lamard, Pierre-Henri Conze, Pascale Massin, and Béatrice Cochener. Explain: Explanatory artificial intelligence for diabetic retinopathy diagnosis. Medical Image Analysis, 72: August 2021.
[316] Mahmudur Rahman, Tanay Kumar Saha, Mohammad AI Hasan, Kevin S. Xu, and Chandan K. Reddy. Dylink2vec: Effective feature representation for link prediction in dynamic networks, 2018.
[317] Farzaneh Rajabi and Jack Siyuan He. Click-through rate prediction using graph neural networks and online learning. 2021.
[318] Korbinian Randl, Núria Lladós Armengol, Lena Mondrejevski, and Ioanna Miliou. Early prediction of the risk of ieu mortality with deep federated learning. In 2023 IEEE 36th International Symposium on Computer-Baun Based Medical Systems (CBMS), pages 706-711. IEEE, 2023.
[319] Susie Xi Rao, Clémence Lanfranchi, Shuai Zhang. Zhichao Han, Zitao Zhang, Wei Min, Mo Cheng. Yinan Shan, Yang Zhao, and Ce Zhang. Modelling graph dynamics in fraud detection with Attention, 2022.
[320] Gopikrishna Rathinavel, Nikhil Muralidhar, Timothy O' O'Shea, and Naren Ramakrishnan. Detecting irregular network activity with adversarial learning and expert feedback. In 2022 IEEE International Conference on Data Mining (ICDM), pages 1161-1166. IEEE, 2022.
[321] Farshid Rayhan, Sajid Ahmed, Zaynab Mousavian, Dewan Md Farid, and Swakkhar Shatabda. Frnet-dti: Deep convolutional neural network for drug-target interaction prediction. Heliyon, 6(3), 2020.
[322] Farshid Rayhan, Sajid Ahmed, Swakkhar Shatabda, Dewan Md Farid, Zaynab Mousavian, Abdollah Dehzangi, and Sohel Rahman. idti-esboost: identification of drug target interaction using evolutionary and structural features with boosting. Scientific reports, 7(1):1773,, 2017.
[323] Haroldas Razvadauskas, Evaldas Vaiciukynas, Kazimieras Buskus, Lukas Drukteinis, Lukas Arlauskas, Saulius Sadauskas, and Albinas Naudziunas. Exploring traditional machine learning for identification of pathological auscultations, 2022.
[324] Ismat Ara Reshma, Sylvain Cussat-Blane, Radu Tudor Ionescu, Hervé Luga, and Josiane Mothe. Natural vs balanced distribution in deep learning whole slide images for cancer detection. In Proceedings of the 36th Annual ACM Symposium on Applied Computing, pages 18-25, 2021.
30

---

[325] Roonak Rezvani, Samanch Kouchaki, Ramin Nilforooshan, David J. Sharp, and Payam Barnaghi. Semi- Sume-supee learning for identifying the likelihood of agitation in people with dementia, 2021.
[326] Georgios Rizos, Jenna Lawson, Simon Mitchell, Pranay Shah, Xin Wen, Cristina Banks- Leite, Robert Ewers, and Björn W. Schuller. Propagating variational model uncertainty for bioacoustic call label smoothing. Patterns, 5(3):100332, 2024.
[327] Narjes Rohani and Changiz Eslahchi. Drug-drug interaction predicting by neural network using integrated similarity. Scientific reports, 9()):3665, 2019.
[328] Michal Romaszewski, Przemystaw Glomb, Arkadiusz Sochan, and Michal Cholewa. dataset for evaluating blood detection in hyperspectral images. Forensic science international, 10701, 2021.
[329] Miguel Romero, Oscar Ramírez, Jorge Finke, and Camilo Rocha. Feature extraction with spectral clustering for gene function prediction using hierarchical multi-label classification. Applied Network Science, 7(1):28, 2022.
[330] Caitlin Rose, Jeyhan Kartaltepe, Gregory F Snyder, Vicente Rodriguez-Gomez, LY Aaron Yung, Pablo Arrabal Haro, Micaela B Bagley, Antonello Calabró, Nikko J Cleri, MC Cooper, et al. Identifying galaxy mergers in simulated ceers nircam images using random forests. The Astrophysical Journal, 942(1):54, 2023.
[331] Daniel Rosenberg. Imbalanced data? stop using roc-auc and use aupre instead, June 2022.
[332] Sankardas Roy, Jordan DeLoach, Yuping Li, Nic Herndon, Doina Caragea, Xinming Ou, Venkatesh Prasad Ranganath, Hongmin Li, and Nicolais Guevara. Experimental study with real-world data for android app security analysis using machine learning. In Proceedings of the 31st Annual Computer Security Applications Conference, pages 81-90, 2015.
[333] Timothy N Rubin, America Chambers, Padhraic Smyth, and Mark Steyvers. Statistical topic models for multi-label document classification. Machine learning, 18:157-208, 2012.
[334] Clemente Rubio-Manzano, Tomás Lermanda, Claudia Martinez-Arameda, Christian Vidal, and Alejandra Segura. Teach me to play, gamer! imitative learning in computer games via linguistic description of complex phenomena and decision trees. Soft Computing, 776))3023-3033 2023.
[335] Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 15555775567755 2021.
[336] Victor Saase, Holger Wenz, Thomas Ganslandt, Christoph Groden, and Máté E. Maros. Simple statistical methods for unsupervised brain anomaly detection on mri are competitive deep learning methods, 2020.
[337] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et Extending the wilds benchmark for unsupervised adaptation. arXiv preprint arXiv:2 /2.iv:2//20050900 2021.
[338] Berkman Sahiner, Weijie Chen, Aria Pezeshk, and Nicholas Petrick. Comparison of two classifiers when the data sets are imbalanced: the power of the area under the precision-recall curve as the figure of merit versus the area under the ROC curve. In Matthew A. Kupinski and Robert M. Nishikawa, editors, Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, volume 10136, page 101360G, International Society for Optics and Photonics, SPIE, 2017.
[339] Takaya Saito and Mare Rehmsmeier. The precision-recall plot more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3)e011880 18432, 2015.
[340] Takaya Saito and Mare Rehmsmeier. Precrec: fast and accurate precision-recall and ROC curve calculations in R. Bioinformatics, 33(1):145-147, 09 2016.
31

---

[341] Amir Sarabadani, Aaron Halfaker, and Dario Taraborelli. Building automated vandalism detection tools for wikidata. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 1647-1654, 2017.
[342] Mehrzad Saremi and Maryam Amirmazlaghani. Reconstruction of gene regulatory networks using multiple datasets. IEEF/ACM transactions computational biology and bioinformatics, 19(3):1827-1839 2021.
[343] Hamed Sarvari, Carlotta Domeniconi, Bardh Prenkaj, and Giovanni Stilo. Unsupervised boosting- beastng--assed autoencoder ensembles for outlier detection. Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 91-103. Springer, 2021. 2021.
[344] Hamed Sarvari, Carlotta Domeniconi, and Giovanni Stilo. Graph-based selective outlier ensembles. In Proceedings of the 34th ACM/SIGAPP Symposium Applied Computing, pages 318-525, 2019.
[345] Thomas Schlegl, Hrvoje Bogunovic, Sophie Klimscha, Philipp Seeböck, Amir Sadeghipour, Bianca Gerendas, Sebastian M. Waldstein, Georg Langs, and Ursula Schmidt-Erfurte Fully automated segmentation of hyperreflective foci in optical coherence tomography images, 2018.
[346] Thomas Schlegl, Heiko Stino, Michael Niederleithner, Andreas Pollreisz, Ursula Schmidt- Erfurth, Wolfgang Drexler, Rainer A. Leitgeb, and Tilman Schmoll. Data-centric ai approach to improve optic nerve head segmentation and localization in oct en face images, 2022.
[347] Oliver Schulte, Zhensong Qian, Arthur E Kirkpatrick, Xiaoqian Yin, and Yan Sun. Fast learning of relational dependency networks. Machine Learning. 103:377 4067 2016.
[348] Kyriakos Schwarz, Ahmed Allam, Nicolas Andres Perez Gonzalez, and Michael Krautham- mer. Attentionddi: Siamese attention based deep learning method for drug drug interaction predictions. BMC bioinformatics, 22(1):1-19, 2021.
[349] Kyriakos Schwarz, Alicia Pliego-Mendieta, Amina Mollaysa, Lara Planas-Paz, Chantal Pauli, Ahmed Allam, and Michael Krauthammer. Ddos: A graph neural network based drug synergy prediction algorithm, 2024.
[350] Eugene Seo, Rebecca A Hutchinson, Xiao Fu, Chelsea Li, Tyler A Hallman, John Kilbride, and W Douglas Robinson. Stateconet: Statistical ecology neural networks for species distribution modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 513-521, 2021.
[351] Hrishikesh Sharma, Prakhar Pradhan, and Balamuralidhar P. Scnet: a generalized attention- based model for crack fault segmentation. In Proceedings of the twelfth indian conference on computer vision, graphics and image processing, pages 1-9, 2021.
[352] Mradul Sharma, Jitadeepa Nayak, Maharaj Krishna Koul, Smarajit Bose, and Abhas Mitra. Gamma/hadron segregation for ground based imaging atmospheric cherenkov telescope using machine learning methods: Random forest leads. Research Astronomy and Astrophysics, 14(11):1491, 2014.
[353] Hongda Shen and Eren Kursun. Label augmentation via time-based knowledge distillation for financial anomaly detection, 2021.
[354] Xiupeng Shi, Yiik Diew Wong, Michael Zhi-Feng Li, Chandrasekar Palanisamy, and Chen Chai. A feature learning approach based on xgboost for driving assessment and risk prediction. Accident Analysis & Prevention, 129:170-179, 2019.
[355] Yishai Shimoni, Ehud Karavani, Sivan Ravid, Peter Bak, Tan Hung Ng, Sharon Hensley Alford, Denise Meade, and Yaara Goldschmidr. An evaluation toolkit to guide model selection and cohort definition in causal inference, 2019.
[356] Samuel Showalter and Zhixin Wu. Minimizing the societal cost of credit card fraud with limited and imbalanced data, 2019.
32

---

[357] Hai Shu and Hongtu Zhu. Sensitivity analysis of deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01)::4433 4950, July 2019.
[358] Satya Narayan Shukla and Benjamin Marlin. Interpolation-prrdicttinn networks for irregularly sampled time series. In International Conference on Learning Representations, 2019.
[359] Satya Narayan Shukla and Benjamin M. Marlin. Modeling irregularly sampled clinical time series, 2018.
[360] Yaniv Shulman. Unsupervised contextual anomaly detection using joint deep variational generative models, 2019.
[361] Yuqi Si and Kirk Roberts. Three-level hierarchical transformer networks for long-sequence and multiple clinical documents classification, 2021.
[362] Amit Kumar Sikder, Hidayet Aksu, and A Selcuk Uluagac. A context-aware framework for detecting sensor-base based threats on smart devices. IEEE Transactions on Mobile Computing, 19(2):245-26) 261, 2019.
[363] Marília Costa Rosendo Silva, Felipe Alves Siqueira, João Pedro Mantovani Tarrega, João Vi- tor Pataca Beinotti, Augusto Sousa Nunes, Miguel de Mattos Gardini, Vinícius Adolfo Pereira da Silva, Nádia Félix Felipe da Silva, and André Carlos Ponce de Leon Ferreira de Car- valho. No pattern, no recognition: survey about reproducibility and distortion issues of text clustering and topic modeling, 2022.
[364] Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, lasonas Kokkinos, and Francese Moreno- Noguer. Fracking deep convolutional image descriptors, 2015.
[365] Aditya Kumar Singh and B. Uma Shankar. Multi-label classification on remote-sensing images, 2022.
[366] Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. Fairness violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 3-13, 2021.
[367] Joakim Skarding, Bogdan Gabrys, and Katarzyna Musial. Foundations and modeling of dynamic networks using dynamic graph neural networks: A survey. IEEE Access, 9:79143- 79168, 2021.
[368] Johannes De Smedt and Jochen De Weerdt. Predictive process model monitoring using recurrent neural networks, 2023.
[369] Anna I. Smith, Tian Zheng, and Andrew Gelman. Prediction scoring of data-driven discoveries for reproducible research. Statistics and Computing, 33(1):11, 2023.
[370] Artur Sokolovsky, Luca Arnaboldi, Jaume Bacardit, and Thomas Gross. Volume-centred range bars: Novel interpretable representation of financial markets designed for machine learning applications, 2022.
[371] Roghayeh Soleymani. Eric Granger, and Giorgio Fumera. Progressive boosting for class imbalance and its application to face re-identification. Expert Systems with Applications, 101:271-291, 2018.
[372] Janne Spijkervet and John Ashley Burgoyne. Contrastive learning of musical representations. In Proceedings of the 22nd International Society for Music Information Retrieval Conference. pages 673-681. ISMIR, October 2021.
[373] Saurabh Srivastava, Vinay P. Namboodiri, and T. V. Prabhakar. Putworkbench: Analysing privacy in ai-intensive systems, 2019.
[374] Benjamin Stadnick, Jan Witowski, Vishwaesh Rajiv, Jakub Chiędowski, Farah E. Shamout, Kyunghyun Cho, and Krzysztof Geras. Meta-repository screening mammography classi- fiers, 2022.
33

---

[375] Christoph Stanik, Marlo Haering, and Walid Maalej. Classifying multilingual user feedback using traditional machine learning and deep learning. In 2019 IEEE 27th international requirements engineering conference workshops (REW), pages 220-226. IEFF, 2019.
[376] Georg Steinbuss and Klemens Böhm. Benchmarking unsupervised outlier detection with realistic synthetic data. ACM Trans. Knowl. Discov. Data, 15(4), apr 2021.
[377] Oliver Stephenson. Tobias Köhne. Eric Zhan, Brent E. Cahill, Sang-Ho Ho Yun, Zachary Ross, and Mark Simons. Deep learning- based damage mapping with insar coherence time series. IEFE Transactions on Geoscience and Remote Sensing, 60:1-17, 2022.
[378] Callum L. Stewart, Amos Folarin, and Richard Dobson. Personalized acute stress classification from physiological signals with neural processes, 2020.
[379] Andrew Stolman, Caleb Levy, C Seshadhri, and Aneesh Sharma. Classic graph structural features outperform factorization-base graph embedding methods on community labeling. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), pages 388-396. SIAM, 2022.
[380] Nils Strodthoff, Baris Goktepe, Thomas Schierl, Cornelius Hellge, and Wojciech Samek. Enhanced machine learning techniques for early harq feedback prediction in 5g. IEEE Journal on Selected Areas in Communications,
((11))2553-2587 November 2019.
[381] Olly Styles, Tanaya Guha, and Victor Sanchez. Multi-camera trajectory forecasting with trajec- tory tensors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 1):4422- 8491, 2021.
[382] Sujanya Suresh, Savitha Ramasamy, P. N. Suganthan, and Cheryl Sze Yin Wong. Incremental knowledge tracing from multiple schools, 2022.
[383] Lionel Tabourier, Daniel F Bernardes, Anne-Sophie Libert, and Renaud Lambiotte. Rankmerg- ing: a supervised learning-to- rank framework to predict links in large social networks. Machine Learning. 10881729-1777 2019.
[384] Rajesh Talluri and Sanjay Shete. Using the weighted area under the net benefit curve for decision curve analysis. BMC medical informatics and decision making, 16:1-9, 2016.
[385] Fei Tan, Zhi Wei, Jun He, Xiang Wu, Bo Peng, Haoran Liu, and Zhenyu Yan. blended deep learning approach for predicting user intended actions. In 2018 IEEE international conference on data mining (ICDM), pages 487-496. IEEE, 2018.
[386] Federico Tavella, Alberto Giaretta, Mauro Conti, and Sasitharan Balasubramaniam. A ma- chine learning- based approach to detect threats in bio-cyber dna storage systems. Computer Communications, 187:59-700 -0, 2022.
[387] Vajira Thambawita, Debesh Jha, Hugo Lewi Hammer, Håvard D Johansen, Dag Johansen, Päl Halvorsen, and Michael A Riegler. An extensive study on cross cross-dataset bias and evalua- tion metries interpretation for machine learning applied to gastrointestinal tract abnormality classification. ACM Trunsactions Computing for Healthcare, 1(3):1-29, 2020.
[388] Santosh Thoduka, Juergen Gall, and Paul G. Ploger. Using visual anomaly detection for task execution monitoring. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS). IEEE, September 2021.
[389] Maud Thomas and Holger Rootzén. Real-time prediction of severe influenza epidemics using extreme value statistics. Journal of the Royal Statistical Society Series C: Applied Statistics, 71 (2)3377633 2022.
[390] Aleksei Tiulpin, Stefan Klein, Sita MA Bierma-Zeinstra, Jérôme Thevenot, Esa Rahtu, Joyce van Meurs, Edwin HG Oei, and Simo Saarakkala. Multimodal machine learning- based knee osteoarthritis progression prediction from plain radiographs and clinical data. Scientific reports, 91):20003, 2019.
34

---

[391] Amirsina Torñ, Edward A Fox, and Chandan K Reddy. Differentially private synthetic medical data generation using convolutional gans. Information Sciences, 586:485-500, 2022.
[392] Meredith V. Trotter, Cuong Q. Nguyen, Stephen Young, Rob T. Woodruff, and Kim M. Branson. Epigenomic language models powered by cerebras, 2021.
[393] Hasan Md Tusfiqur, Duy M. H. Nguyen, Mai T. N. Truong, Triet A. Nguyen, Binh T. Nguyen, Michael Barz, Hans-Juergen Profitlich, Ngoc T. T. Than, Ngan Le, Pengtao Xie, and Daniel Sonntag. Drg-net: Interactive joint learning of multi-lesion segmentation and classification for diabetic retinopathy grading, 2022.
[394] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
[395] Celine Vens, Jan Struyf, Leander Schietgat, Saão Džeroski, and Hendrik Blockeel. Decision trees for hierarchical multi-label classification. Machine learning, 73:185-214, 2008.
[396] Pim Verschuuren, Serena Palazzo, Tom Powell, Steve Sutton, Alfred Pilgrim, and Michele Faucci Giannelli. Supervised machine learning techniques for data matching based similarity metries. arXiv preprint 2XXv:2007.04001,, 2020.
[397] V Vijayan, D Critchlow, and T Milenković, Alignment of dynamic networks. Bioinformatics, 33(14): 33(14)i180-i189) 07 2017.
[398] Gaurav Vishwakarma, Aditya Sonpal, and Johannes Hachmann. Metrics for benchmarking and uncertainty quantification: Quality, applicability, and best practices for machine learning in chemistry. Trends in Chemistry, 3(2): 2021.
[399] Sophia J. Wagner, Daniel Reisenbüühher,, Nicholas West, Jan Moritz Niehues, Jiefu Zhu, Sebastian Foersch, Gregory Patrick Veldhuizen, Philip Quirke, Heike I. Grabsch, Pict A. van den Brandt, Gordon G. A. Hutchins, Susan D. Richman, Tanwei Yuan, Rupert Langer, Josien C. A. Jenniskens, Kelly Offermans, Wolfram Mueller, Richard Gray, Stephen B. Gruber, Joel K. Greenson, Gad Rennert, Joseph D. Bonner, Daniel Schmolze, Jitendra Jonnagaddala, Nicholas J. Hawkins, Robyn L. Ward, Dion Morton, Matthew Seymour, Laura Magill, Marta Nowak, Jennifer Hay. Viktor H. Koelzer, David N. Church, David Church, Enric Domingo, Joanne Edwards, Bengt Glimelius, Ismail Gogenur, Andrea Harkin, Jen Hay, Timothy Iveson, Emma Jaeger, Caroline Kelly, Rachel Kerr, Noori Maka, Hannah Morgan, Karin Oien, Clare Orange, Claire Palles, Campbell Roxburgh, Owen Sansom, Mark Saunders, Ian Tomlinson, Christian Matek, Carol Geppert, Chaolong Peng, Cheng Zhi, Xiaoming Ouyang, Jacqueline A. James, Maurice B. Loughrey, Manuel Salto-Tellez. Hermann Brenner, Michael Hoffmeister, Daniel Truhn, Julia A. Schnabel, Melanie Boxberg, Tingying Peng, and Jakob Nikolas Kather. Transformer- based biomarker prediction from colorectal cancer histology: A large-scale multicentric study, September 2023. Publisher: Elsevier.
[400] Guanghui Wang, Ming Yang. Lijun Zhang, and Tianbao Yang. Momentum accelerates the convergence of stochastic aupre maximization. In International Conference on Artificial Intelligence and Statistics, pages 3753-3771. PMLR, 2022.
[401] Ruoyu Wang, Syed Ali Khurram, Amina Asif, Lawrence Young, and Nasir Rajpoot. Rank the triplets: A ranking- based multiple instance learning framework for detecting hpv infection in head and neck cancers using routine h& images, 2022.
[402] Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. Mimic-extract: data extraction, preprocessing, and represen- tation pipeline for mimic-iii. In Proceedings of the ACM conference on health, inference, and learning, pages 222-235 2020.
[403] Siyue Wang, Giles R. S. Atkinson, and Wayne B. Hayes. Sana: cross-species prediction of gene ontology go annotations via topological network alignment. upj Systems Biology and Applications, 8(1), July 2022.
35

---

[404] Wei-Chen Wang and Jonas Mueller. Detecting label errors in token classification data. arXiv preprint arXiv:2210.03920, 2022.
[405] Xuhong Wang. Ying Du, Shijie Lin, Ping Cui, Yuntian Shen, and Yupu Yang. advae: A self-adversarial variational autoencoder with gaussian anomaly prior knowledge for anomaly detection. Knowledge Based Systems, 190: 187, 2020.
[406] Yaqing Wang, Yifan Ethan Xu, Xian Li, Xin Luna Dong, and Jing Gao. Automatic validation of textual attribute values in e-commerce catalog by learning with limited labeled data. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2533 2541, 2020.
[407] Michael Weiss and Paolo Tonella. Fail-safe execution of deep learning based systems through uncertainty monitoring. 2021 14th IEEE conference on software testing, verification and validation (ICST), pages 24-35. IEEE, 2021.
[408] Michael Weiss and Paolo Tonella. Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines. Software Testing, Verification and Reliability, 33 1840, 2023.
[409] Peisong Wen, Qianqian Xu, Zhiyong Yang, Yuan He, and Qingming Huang. Exploring the algorithn-dependent generalization of aupre optimization with list stability. In S. Koyejo, S. Mohamed, Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 28335-283493 Curran Associates, Inc., 2022.
[410] Fiorella Wever, T Anderson Keller, Laura Symul, and Victor Garcia. As easy apc: coming missing data and class imbalance in time series with self-supervised learning. 2nd Workshop on Self-Supervised Learning: Theory and Practice of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Sydney, Australia. 2021.
[411] Julian Wiederer, Julian Schmidt, Ulrich Kressel, Klaus Dietmayer, and Vasileios Belagiannis. A benchmark for unsupervised anomaly detection in multi-agent trajectories. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pages 130-137. IEEE, 2022.
[412] Sarah Wiegreffe, Edward Choi, Sherry Yan, Jimeng Sun, and Jacob Eisenstein. Clinical concept extraction for document-level coding. In Dina Femner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii, editors, Proceedings of the 18th BioNLP Workshop and Shared Task, pages 261-272, Florence, Italy, August 2019. Association for Computational Linguistics.
[413] Linda Wightman. Lsac national longitudinal bar passage study. Isac research report series. 1998.
[414] Minz Won, Keunwoo Choi, and Xavier Serra. Semi-supervised music tagging transformer, 2021.
[415] Minz Won, Sanghyuk Chun, and Xavier Serra. Toward interpretable music tagging with self-attention, 2019.
[416] Minz Won, Andres Ferraro, Dmitry Bogdanov, and Xavier Serra. Evaluation of cnn-based automatic music tagging models. arXiv preprint arXiv:2006.00751, 2020.
[417] Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In Proceedings of the European Conference on Computer Vision, pages 322-339. Springer, 2020.
[418] Yifan Wu, Min Zeng, Ying Yu, Yaohang Li, and Min Li. A pseudo label-wise attention network for automatic ied coding. IEFE Journal of Biomedical and Health Informatics, 26(10):500155212, 2022.
36

---

[419] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9()):113-330 2018.
[420] Chengyuan Xu, Curtis McCully, Boning Dong, D Andrew Howell, and Pradeep Sen. Cosmic- conn: A cosmic-ray detection deep-learning framework, data set, and toolkit. The Astrophysical Journal, 942(2):73, 2023.
[421] Nuo Xu, Pinghui Wang, Long Chen, Jing Tao, and Junzhou Zhao. Mr-gon: multi-resolution and dual graph neural network for predicting structured entity interactions. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI' 19, page 3968-3974. AAAI Press, 2019.
[422] Weihuang Xu, Guohao Yu, Alina Zare, Brendan Zurweller, Diane L. Rowland. Joel Reyes- Cabrera, Felix B. Fritschi, Roser Matamala, and Thomas E. Juenger. Overcoming small minirhizotron datasets using transfer learning. Computers and Electronics Agriculure, 175:105466, August 2020.
[423] Yanhua Xu and Dominik Wojtezak. Multi-channel neural networks for predicting influenza virus hosts and antigenic types. arXiv preprint arXiv:2206.03823 2022.
[424] Tianbao Yang. Deep auc maximization for medical image classification: Challenges and opportunities, 2021.
[425] Xinxing Yang, Genke Yang, and Jian Chu. The computational drug repositioning without negative sampling. IEEE/ACM Transactions on Computational Biology and Bioinformatics, ((12):1506-117, 2022.
[426] Xinxing Yang, Genke Yang, and Jian Chu. The neural metric factorization for computational drug repositioning. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 20(1):731-741) 2022.
[427] Xinxing Yang, Genke Yang, and Jian Chu, Self-supervised learning for label sparsity in computational drug repositioning. /EEE/ACM Transactions Computational Biology and Bioinformatics, 2023.
[428] Yang Yang, Ryan N Lichtenwalter, and Nitesh V Chawla. Evaluating link prediction methods. Knowledge and Information Systems, 45:751-782, 2015.
[429] Zi-Yi Yang, Zhao-Feng Ye, Yi-Jia Xiao, Chang-Yo Yu Hsich, and Sheng- Yu Zhang. Spidextratrees: robust machine learning approach for predicting kinase inhibitor resistance. Briefings in Bioinformatics, 23(3):bbac050, 2022.
[430] Shota Yasui, Gota Morishita, Fujita Komei, and Masashi Shibata. A feedback shift correction in predicting conversion rates under delayed feedback. Proceedings of The Web Conference 2020, pages 2740-2746, 2020.
[431] Chun-Kit Yeung and Dit-Yan Yeung. Incorporating features learned by an enhanced deep knowledge tracing model for stem/non-stem job prediction. International Journal of Artificial Intelligence in Education, 29(3):317-341 2019.
[432] Xin Yi, Scott Adams, Robert DE Henderson, and Paul Babyn. Computcr-aided assessment of catheters and tubes radiographs: How good is artificial intelligence for assessment? Radiology: Artificial Intelligence, (1))=1990822, 2020.
[433] Yilin Yin and Chun-An Chou, Early icu mortality prediction and survival analysis for respira- tory failure, 2021.
[434] Shuhan Yuan and Xintao Wu. Deep learning for insider threat detection: Review, challenges and opportunities. Computers & Security, 104:102221, 2021.
[435] Yan Yuan, Wanhua Su, and Mu Zhu. Threshold-free free measures for assessing the performance of medical screening tests. Frontiers in public health, 3:57, 2015.
37

---

[436] Mubariz Zaffar. Sourav Garg, Michael Milford, Julian Kooij, David Flynn, Klaus McDonald- Maier, and Shoaib Ehsan. Vpr-bench: An open-source source visual place recognition evaluation framework with quantifiable viewpoint and appearance change. International Journal of May 2021.
Computer Vision, 129(7):216--1174
[437] Roee Zamir, Shai Bagon, David Samocha, Yael Yagil, Ronen Basri, Miri Sklair-Levy, and Meirav Galun. Segmenting microcalcifications mammograms and its applications. In Medical Imaging 2021: Image Processing, volume 1 1596, pages 788-795. SPIE, 2021.
[438] Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Draem-a discriminatively trained re- construction embedding for surface anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8330-8339, 2021.
[439] Vitjan Zavrtanik. Matej Kristan, and Danijel Skočaj. Dsr-a dual subspace re- re-projection network for surface anomaly detection. In European conference on computer vision, pages 539-554. Springer, 2022.
[440] Ido Zehori, Nevo Itzhak, Yuval Shahar, and Mia Dor Schiller. Towards user privacy-aware mobile gaming app installation prediction model, 2023.
[441] Brian Hu Zhang. Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335-340, 2018
[442] Chaohe Zhang, Xu Chu, Liantao Ma, Yinghao Zhu, Yasha Wang, Jiangtao Wang, and Junfeng Zhao. M3care: Learning with missing modalities in multimodal healthcare data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22.
ACM, August 2022.
[443] Chaohe Zhang, Xin Gao, Liantao Ma, Yasha Wang. Jiangtao Wang, and Wen Tang. Grasp: Generic framework for health status representation learning based incorporating knowledge from similar patients. Proceedings of the AAAI Conference on Artificial Intelligence, 35(1):715-
723, May 2021.
[444] Dingwen Zhang, Huazhu Fu, Junwei Han, Ali Borji, and Xuelong Li. A review of co-saliency detection technique: Fundamentals, applications, and challenges, 2017.
[445] Weijia Zhang. Non-i.i.. multi-instance learning for predicting instance and bag labels using variational auto-encoder, 2021.
[446] Weijia Zhang, Xuanhui Zhang, hanwen deng, and Min-Ling Zhang. Multi-instance causal representation learning for instance label prediction and oft-s--distiibttioo generalization. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 34940-3493 34953. Curran Associates, Inc., 2022.
[447] Wencan Zhang, Mariella Dimiccoli, and Brian Y Lim. Debiased-cam to mitigate image perturbations with faithful visual explanations of machine learning. In Proceedings of the 2022 CHI Conference Human Factors in Computing Systems, pages 1 1-32, 2022.
[448] Wenning Zhang, Ryohei Hisano, Takaaki Ohnishi, and Takayuki Mizuno. Nondiagonal mixture of dirichlet network distributions for analyzing stock ownership network. In Complex Networks & Their Applications IX: Volume Proceedings of the Ninth International Conference Complex Networks and Their Applications COMPLEX NETWORKS 2020, pages 75-86. Springer, 2021.
[449] Xuan Zhang, Shiyu Li, Xi Li, Ping Huang, Jiulong Shan, and Ting Chen. Destseg: Segmenta- tion guided denoising student-teacher for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3914-3923, 2023.
[450] Yan Zhang and Howard D. Bondell, Variable Selection via Penalized Credible Regions with Dirichlet-Laplace Global-Local Shrinkage Priors. Bayesian Analysis, 13(3):823 844, 2018.
38

---

[451] Qian M. Zhou, Zhe Lu, Russell J. Brooke, Melissa M Hudson, and Yan Yuan. Is the new model better? one metric says yes, but the other says no. which metric do use?, 2020.
[452] Zhou, Boyang Wang, Lei Huang, Shanshan Cui, and Ling Shao. A benchmark for studying diabetic retinopathy: Segmentation, grading, and transferability. IEEE Transactions on Medical Imaging, 40(3):818 818-888, March 2021.
[453] Qianqian Zou, Claus Brenner, and Monika Sester. Gaussian process mapping of uncertain building models with gmm prior. IEEE Robotics and Automation Letters, 8(10):557--658 2023.
[454] Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the- difference self-supervised pre-training for anomaly detection and segmentation. In European Conference on Computer Vision, pages 392 Springer, 2022.
[455] Ali Burak Ünal, Nico Pfeifer, and Mete Akgün. ppaurora: Privacy preserving area under receiver operating characteristic and precision- curves, 2023.
39

---

A Broader Impact and Ethical Considerations
This research paper challenges the conventional wisdom regarding the superiority of the AUPRC over AUROC in binary classification tasks with class imbalance and has several ethical implications and impacts.
Our analysis reveals that the preference for AUPRC in certain ML applications may not be empirically justified and could inadvertently amplify algorithmic biases. This calls for re-examination of prevalent metrics within ML, especially in high-stakes domains like healthcare, finance, and criminal justice where biased models can have profound societal repercussions. The tendency of AUPRC to disproportionately favor models with higher prevalence of positive labels could exacerbate existing disparities, underscoring the ethical need for rigorous validation and scrutiny of evaluation metrics.
Additionally, our use large language models for literature analysis demonstrates novel approach in scrutinizing and re-evaluating long- -tanddin assumptions in ML. This method could set precedent for more comprehensive and robust scientific investigations in the field, fostering a culture of empirical rigor and ethical awareness.
The ethical dimension of our work lies in the spotlight it casts on metric selection in ML model evaluation. The potential of metrics like AUPRC to skew model performance favoring certain groups raises pressing concerns about fairness in algorithmic recision-making This is particularly critical when algorithms influence key decisions affecting individuals and communities.
While we use the COMPAS dataset for recividism prediction in this work, recognize the many societal issues with automated predictions of recidivism (93). We utilize this dataset as is commonly used dataset in the fairness literature, but do advocate for deployment of these models in any way.
Our study contributes to the technical discourse metric behaviors in ML and serves cautionary tale against uncritically embracing established norms. It underscores the imperative for careful metric selection aligned with ethical principles and fairness objectives in ML, highlighting the far-reaching consequences of these choices shaping societal outcomes and advancing the field of ML.
B
Code Availability
All code available at http://gi..uu..mm//hhghg///aaa. bias and https: https://github.cco com/ Lasechaausen//xwvvillCCaaaaaaa
C
Notation
Let X, 0. represent paired feature and binary classification label space from which i.i.d. samples (I. v) E X x y are drawn via the joint distribution over the random variables x, y. Let fe : X + (0, be a binary classification model parametrized by 0 E Rd for some E outputting continuous probability scores over this space.
We define random variable = fo (x) be the distribution of scores output by the model over input samples. Throughout the paper, 0 may be omitted if is clear from context. We will occasionally also use the notation s, and s. reflect the conditional distributions of model outputs conditioned on the label being 1 or 0, respectively:
S s(x)ly =
s_ f(x)ly = O.
40

---

Let Np be the number of data points with positive label and Ny the number with negative label. Further, given a threshold 7, define
(P)(T)
fog E X X|p: (0)
2 T, = 1}
(To =
{24 E X Xpp
< T,3; = 1}
(No(r)
{2t E X
(0)
T, 0)
(Pe(T) (2t E X Pi
(0)
2 T, 0}
FR(f.T) = Pp(p > T)
TPp(T)
(PRo(T) = (Po(T) + (N)
= PPy==1ff(2) > T)
-Pal==18>>> > T)
P(s+ TT
FPo(T)
FPR FPa(T) + TTN(T)
(f(z=(((z) > T)
= Paly=o(s > T)
= -P(s_ > T)
(P
(T) = TPo(T) + FPP(T)
Pyls(i)ST( (y = 1)
Pylssr(1 1)
Lastly, recall
TPRo dFPRo dr dr
AUROCo
TPRadFPRe
FPReCTPRe
dTPRo dr
AUPRCo
Prece dr
PrecedTPRo
D Proof of Theorem 1
Recall that all notation is defined formally in Appendix C
Here, we prove Theorem [ which states
Theorem 1. Let X', {0, }} represent paired feature and binary classification label space from which i.i.d. samples (x, y) E X' y are drawn via the joint distribution over the random variables
y. Let f X + (0, 1) be binary classification model outputting continuous probability scores
41

---

over this space. Then,
AUROC(S) 1 /w)yy=-== [PR((,E)
FPR(F.))
AUPRC(f) Py(y f(0
((ff())t)
Proof. Recall that AUROC and AUPRC are follows:
AUROC = L. TPR dFPR =1
FPR dTPR
AUPRC- [
Prec dTPR
However, we can further clarify these by leveraging the fact that TPR(T) = Ps. (s+ > T) J 8. (t)dt, below:
Lo ((F)d((PR(+))- 1 g(T)
(TPR(()) dr
dr
g(T) d
(Pss (8+ > T))dTT
dr
d
(T: ([(de) dr
9(T)) dr
(-+)(-a(()))d
Es, [g]
So. AUROC=1 E,, [FPR] & AUPRC = Es, [Prec]. To further simplify, expand Prec via
Bayes rule:
Prec = 1 - Py/ssr (y =0)
= - Psly=o(8 > 7) (y = 0)
P.(s 7)
FPR(F)
Thus,
AUROC(f)
EENs, [FR(S,()
= -1 - (P [FP((F.))
AUPRO(f) = Er-sy pree((f (f,
FPR(F.1)
1 - Py 0)E
Pens(s t)
FPR(F.))
1 - Py(y 0 1)EEFFO0))
Px(f(x) > t)
desired.
Synthetic validation of Theorem can also be found in our public code. Note that this formulation of AUPRC reflects earlier, different formulations AUPRC, such those found the AUPRC optimization literature [409].
42

---

E Proof of Theorem 2
Here, we prove Theorem 2 which states
Theorem 2. Define f, X, and N as in Definition 2.1 Further, suppose without loss of generality that the dataset X is ordered such that f(z:) < f(1++1) for all Then, let us define M (1x, 1i++1) 1) is an incorrectly ranked adjacent pair for model f}. Define I to be model that is identical f except that the probabilities assigned to n, and 1441 are swapped:
[(x)
if $ (11 if = 32
M: +
[(1+1) [(zi)
if T:41.
Then, AUROC(!)) = AUROC(F)) for all i, E M, and AUPRO(F) < AUPRO(F)) for all i M such that j.
Proof. Suppose/ ʃ has given, non-empty set M of atomic mistakes, such that, without loss of generality, (Iy, Fi+1) E M. Suppose we construct new model I" with empirical distributions y4- and p'_ by replicating the scores assigned by the model f with x; and 2++1 swapped (i.e., we correct the mistake (I), 21+1), so x I++1 and 2'41 24).
For which thresholds drawn from the original distribution p+ will the number of false positives of f' differ from the number of false positives of f at that same threshold? For any threshold 7 < Zy. fixing the mistake (2,, F++1) will not change the number of false positives with threshold т, because both 2; and 1;+1 are above т. For any threshold > Z++1, the number will likewise not change as both 2; and x++1 are below т. The only that will have an impact empirical distribution p+ which contains T; and by the definition of atomic mistakes, there are no samples in f with scores between x; and 11+1). ). J, the fact that 1;+1 > I3 yet has negative label means that there will be one false positive corresponding to sample i + greater than in addition to all those that exist with scores greater than 24+1- For f', however, the samples have swapped, so o 4 and thus there is no false positive corresponding to sample i + 1 at the positive score threshold corresponding to x². Therefore, the number of false positives will only change to decrease by one for the threshold x; when the mistake (I,, 2++1) is corrected.
I; (recall that this is for an
As AUROC weights the false positive rate at all positive samples equally and the false positive rate is proportional to the number of false positives, this shows that AUROC will improve by constant amount no matter which atomic mistake is fixed. In contrast, as AUPRC weights false positives inversely by the model's firing rate, it will improve by an amount that is directly linearly correlated with the inverse of the model's s firing rate, implying that it favors mistakes with higher scores and disfavors mistakes with lower scores.
Note that as we use strict inequalities in our definition of the decision rule underlying the FPR here, pair of scores that tied but have different labels will not induce false positive at the corresponding positively labeled sample's 's threshold, so separating such ties will have no impact on AUROC whatsoever. It would similarly not impact AUPRC neither the FPR nor the model firing rate will decrease when the negative sample within the tie is perturbed to be strictly below the positive sample.
Synthetic empirical validation of Theorem can also be found in our public code.
F Proof of Theorem 3
In this section, we formally prove Theorem 2 We begin by establishing Lemma I annd 2
Lemma Let a model f be perfectly calibrated and yield score distributions for positive and
negative samples from probability density functions p+ and p.. Then (t) = I- P)
-Ply=0)
43

---

Proof. As this model is calibrated perfectly, we have that
pt(t) P/yy=1 (s py|==1 (11 = 11)
P2( = 1)
Py(v (t)+ Py (tp-(t)
Py(y =
1p+ + Py(y = 0)
Py = 1) p- (B).
1. Py(y=0) Thus, (1) I-t P- as desired.
Lemma Let a model f be perfectly calibrated and yield score distributions for positive and negative samples from probability density functions p+ and p-, with overall distribution given by p(t) (y (t)+ + Py (t)- Then all (0,1), FR(f,r) B(v=1)
Proof. By definition, we have
J. Py(y = (t)) = D)p-(ttdtt
FR(f.T) =
[ Py(U 11)p+(t) Py( t (t)dt
1) f. (t)(tt
Pyly = Pf(y=1)
where step leverages the fact that perfectly calibrated and the result in Lemmal I As 2 T. 1 S J. Then, as (t) 2 f (t) S 7 S p+ (t) Finally, as So p+ (t) see that S p+ < 1. Therefore,
FR(f, : = 1) [ p+)(t
$ Py(v = 1) 1
1)
Theorem 3. Let f, X, y, N. M, and I all be defined as in Theorem) 2 Further, suppose that in this setting the domain X' now contains attribute defining two subgroups, A = {0, 1), such that for any sample (z,, y:), a denotes the subgroup to which that sample belongs. Let f be perfectly calibrated for samples in subgroup 0, such that Pyla,(y 1|a 0. f(x) t) Then,
lim P=)(v=1la=0)000
P ai+1 1 i = arg max (AUPRO())) =
JEM
Proof. Given Theorem the atomic mistake that would, upon correction, result in the largest improvement to AUPRC is the mistake which occurs at maximal score (as this minimizes the firing rate, which is the denominator in the weighting term for AUPRC). Suppose that at threshold 7, the probability that a mistake will occur above score in subgroup with N samples drawn least 8 E (0, 1]. As the parameters for subgroup are fixed we vary the prevalence for subgroup 2, can be seen as a constant with respect to the limit we are taking.
But, by Lemma 2 and by the fact that is perfectly calibrated for subgroup 2, we know that the probability that f will output a score for sample regardless of its label that exceeds 7 is upper bounded by 2/2) - In the limit as (2) tends to zero, the probability that any probabilities will be observed at our greater than 5 from subgroup 2 likewise tends to zero.
44

---

This means that while the probability that we observe mistake from subgroup 1 stays fixed at least 8 the probability that we could observe any mistake that involves any sample from subgroup 2 (either cross-group mistake or purely subgroup 2 mistake) tends to zero, establishing the claim.
45

---

G Details for Synthetic Experiments
G.1 Sampling a random model with given AUROC
key component of our synthetic experiments is the ability to sample a set of model scores and labels randomly that will have target AUROC. To do this, may or may not be previously known; we derived it from scratch for this work, but make no claim about its novelty). Let N be the number of points we are sampling overall, and N+ be the number of positive points being sampled (which is dictated by the user given prevalence).
the following procedure (which
1. Uniformly sample a random collection of positive-label sample scores between zero and one.
2. Between each (ascending) model positive score indexed from 1 p and p+ count the number of positive samples that have scores less than any value in this window
((+1) we can
(i) and the number that have scores greater than any value in this window (which will be
N+ i).
3. As the target AUROC is the probability that a randomly sampled negative will be ranked more highly than a randomly sampled positive, we can leverage the number tean positíve scores i and greater than positive scores N+- i to compute the probability that a randomly sampled negative score will live the window (p P4 ((+1)) via the binomial distribution.
4. Now, to sample a random negative, we simply first sample a random window (p_), P. with the probabilities assigned above, then uniformly sample value p- within that window. We can repeat this process to the target number of negative samples N N to form our final set of scores.
((+t))
5. If desired, the output scores can further be scaled to have expectation given by the dataset's s prevalence or can be adjusted via calibration method be calibrated given the assigned la- bels. Both procedures can be done without affecting the AUROC. Note that as any calibrated model will have expected probability given by the label's prevalence (See Appendix G.2). the former condition is strictly weaker than the latter.
The procedure outlined above guarantees that, in expectation, the AUROC of the generated set of scores and labels will be preciscly the target AUROC. However, if you apply this procedure indpendently across different sample subpopulations, this guarantee can only be applied on each subpopulation individually, and not necessarily on the overall population due the unspecified xAUC term. However, in practice, for the experiments we ran here, that impact neither meaningfully impacts our experiments nor were the joint AUROCS sufficiently different from the target AUROC warrant a more complex methodology.
G.2 Calibration includes prevalence matching
Let be a random variable describing the probabilities (not scores) output by the model over the input distribution defined by the data generative function. If model is calibrated, this means that (y = 1/p q) 9 that the probability that the label for given point is I is given precisely by the models output probability for that sample. With that in mind, we have:
Ep [a]
Ep [Pyp(y 11p-= 4)]
[/ Pylply 11p (g)d
[ Pyp( = 1,p = b)dg
Py(y =
G.3 Details optimization procedures
MI. Adding Random Noise. We sample vector 6 E TR.,, where each element uniformly drawn from [-8, 6]. We compute the selection metric for S' S e. We repeat this
46

---

procedure 100 times, and return the S' that achieves the maximum value for the selection metric. We vary the maximum magnitude of the perturbation 8 [0, 0.1] grid. Results for this setting are shown in Figure
We note that this approach is subtly biased in favor of the lower-prevalence group. In particular, because scores for the low-prevalence group tend to be "squished" into smaller region of the probability space, random perturbation of fixed magnitude will proportionally induce more score permulations in the low-prevalence group than the high-prevalence group, which affords the system greater capacity to improve the model for the low-prevalence group independent of the choice of AUROC or AUPRC.
M2. Sequentially Fixing Atomic Mistakes. We sequentially correct atomic mistakes, as de- fined in Figure [ At each step, first discover the set of all atomic mistakes M. To maximize AUROC, we randomly select a pair (S,, Sy) E M, and swap their scores in S. i.c. S = Sj S. To maximize AUPRC,
swap the scores for the pair (S,, S2) = arg (s.,is. $j. We repeat this process for 50 steps, with each one sequen- tially fixing another atomic mistake S. Results for this setting are shown in Figures 4b and
M3. Sequentially Permuting Nearby Scores. We first sort S and Y such that S is in ascending order. We apply a random permutation to S by re-indexing using a random ordering, but such that scores are not shuffled too far from their original index. Let be the ordered sequence (1,2, .... n). Define 12 to be the set all permutations of a, such that for all w P, w4 0 S y for i twenty times, where each w corresponds to a new candidate ordering of S. We compute the selection metric for each of the twenty orderings, and return S' to be the score permutation that achieves the maximum value for the selection metric. We repeat this procedure for 25 steps, setting S at each step to be the S' output from the previous step. Results for this setting are shown in Figures 4d and 4c
nT). At cach step, sample E Q with ? 3
47

---

AUROC Oseei
AUROI
AUROC (RV-294)
0.08
GroUp
600
Shop
AUPRC: Overal
AUPRC: Per GrOup
Saep
Step
AUPRC: Overall
AUPRC: Per Group
AUTRIC (99-220)
Uivinll
AUPRIC 29% 0996
(a) Fixing individual mistakes to optimize overall AUROC
(b) Fixing individual mistakes optimize overall AUPRC
Ovaral
Parr Group
PURIOC 2999 9899
AUPOC
0.0
St2p
AUPRC: Overal
AUPRC: GHOUP
ALPRC: Overal
AUPRC: Per Group
Ovell
Owival
AUPRIC FLPFCC09-228 20
0.5
2
(c) Randomly permuting scores to optimize overall AUROC
(d) Randomly permuting scores optimize overall AUPRC
Figure 4: Comparison of the impact of optimizing for overall AUROC and overall AUPRC on the per-group AUROC and AUPRCs of two groups in a synthetic setting, using both the sequentially fixing individual mistakes optimization procedure (M2; top) and the sequentially permuting nearby scores optimization procedure (M3: bottom) described in Section 3.1 Note that the prevalence of Y the high prevalenee group and the low-prevalence group are 0.05 and 0.01 respectively.
48

---

AUROC CR 201-00004
AUROC: Overall
AUROC: Per Group
AUROC 02--994
0.00
64
Step
Sanp
AUPRC: Overall
AUPRC: Per GrOup
AUPRC: Overall
AUPRC: Per
AUPRIC 206-009
AUPRIN 1MP 999-899
8.4
22
22
20
(a) Fixing individual mistakes to optimize overall
AUROC
(b) Fixing individual mistakes optimize overall AUPRC
AURDC: Overall
AUROC: Group
AURICC Ovaral
AURROC Per Group
AUROC/22-2500
AUROC 190-8890000 (99)
Overall
Overal
AUPRC:
AURIC (NN-0N
AURIC 2890
Step
(c) Randomly permuting scores to optimize overall AUROC
(d) Randomly permuting scores to optimize overall AUPRC
Figure 5: Comparison the impact of optimizing for overall AUROC and overall AUPRC on the per-group AUROC and AUPRCs of two groups in synthetic setting where the initial AUROC was set to 0.65 rather than 0.85, using both the sequentially fixing individual mistakes optimization procedure (M2; top) and the sequentially permuting nearby scores optimization procedure (M3; bottom) described in Section 3.1 Note that the prevalence in the high- prevalene group and the low-prevalence group are 0.05 and 0.01 respectively.
49

---

AUROC: Overall
NUROC:
AUROC 29V-29)
AURICC (5A (PP)
3.02
1.54
0.05
003
Saggiture
Max Magniaalc
Max Mug Studo
AUPRC: Overall
AUPRC: Per Gmup
Hight
ALPRCC Overal
AUPRC: Per Group
AUTRI (06-001
2.80
2.28
1106
0128
1108
Magnitad
Max Magnllate
Max Mageiaade
Max Magritudo
(a) Optimizing for Overall AUROC
(b) Optimizing for Overall AUPRC
Figure 6: Comparison of the impact of optimizing for overall AUROC and overall AUPRC on the per-group AUROC and AUPRCs of two groups synthetic setting, using the adding random noise optimization procedure (MI) described in Section 3.1 Note that the prevalence of Y in G₁ and G2 0.05 and 0.01 respectively.
50

---

H Additional Details on Real World Experiments
H.1 Dataset Details
We use the following four datasets, In all datasets, we use sex and race protected attributes.
adult [7 The UCI Adult dataset, where the goal is predict whether an individual's income is $50k.
compas 14 The task to predict two-year recidivism. We only select samples helonging to Affican-AAmercaan and "Caucasian", leading binary variable.
1sac 413 The task is to predict whether law school applicant will pass the bar. We only select samples belonging to White and Black applicants.
mimic [78]:
We use the in-hospital mortality task proposed by 138 where the goal is to predict whether patient will die in the ICU given labs and vitals from the first 48 hours of their hospital stay. We only select samples belonging to White and Black patients.
In each dataset, balance the groups by subsampling the majority group. We then split each dataset into 50% training, 25% validation, 25% test sets, stratified by the group. Dataset statistics can be found in [able
Table 1: Dataset statistics for the four binary classification datasets used in this study. Note that n
refers to the number of samples after balancing by the corresponding attribute. Here, "Prevalence (Higher)" refers to the rate at which the prediction label = for the subpopulation with higher such rate, and "Prevalence (Lower)" refers to the same rate but over the subpopulation of the dataset with lower rate of y =
Dataset
Attribute
n
Features
Prevalence (Higher)
Prevalence (Lower)
adult
Sex
20,394
30.1%
10.7%
adult
Race
6,248
12
24.6%
12.2%
compas
Sex
2,438
6
52.0%
39.4%
compas
Race
4,908
6
55,4%
42.3%
lsac
Sex
15,906
8
96.0%
95.0%
lsac
Race
2,396
8
96.5%
77.2%
mimic
Sex
15,632
49
12.5%
11.9%
mimic
Race
4,030
49
13.9%
9.3%
H.2 Hyperparameter Grid
We use the following hyperparameter grid for our experiments:
max depth: [1. 2......))
learning rate: [0,01, 0.3]
number of estimators: [50, 1000]
min child weight: (1, 2, .... 9}
use protected attribute input feature: (yes, no]
group weight of higher prevalence group: {1, 2, 3, 4, 5, 10, 20, 25,
H.3 Additional Results
All raw AUC results can be found in the code repository for these experiments, at wothtts::/// //github
com/ homm/hzangg// Baapnng////uubbaa aaaauuu_baa bias Additionally, summarized results different views can be found in
Figures and
51

---

0.6
Overall AUROC
Overall AUFPC
0.4
Sparman's (AUROC
0.2
0.0
-0.2
compas: (1.30
Compas: (1.4)
mimic:
Figure 7: Spearman's between the test-set signed AUROC gap versus the validation set overall AUPRC, and the AUROC gap versus the overall AUROC. Numbers in parentheses are the prevalence ratios between the two groups for the particular attribute, and datasets are sorted by this quantity. Error bars are 95% confidence intervals from 20 different random data splits.
AUROC Gap vs, AUROC)
Difference in Spearman
(AUROC Gap Vs. AUPRC
0.10
0.05
0.00
-0.05
-0.10
1.0
1.5
2.0
2.5
Prevalence Ratio
Figure 8: Correlation between the prevalence ratio, and the difference between the Spearman's p of the AUROC gap versus AUROC and the AUROC gap versus AUPRC. Each point represents dataset and attribute combination. This correlation üself has a Spearman's 's of 0.9.00 (p = 0.002).
I Literature Review Methodology
L.1 Paper Acquisition
The initial phase of our comprehensive literature search involved the acquisition of datasets from both the ArXiv preprint server (through the RedPajama dataset Hugging Face), well from a subset of years of NeurIPS, ICML, ICLR, ACL, and CVPR conference proceedings (all scraped manually). The ArXiv dataset, approximately 93.8 GB in size, encompassed over 1.5 million texts JSONL format. For NeurIPS, we developed script to scrape conference papers from 1987 to 2019 (9680 texts), aiming to enrich our search. Other venues contributed fewer papers our assessment process.
1.2 Keyword-Driven Filtering Process
Keyword List Development: We developed two distinct keyword lists to systematically identify papers relevant to our research on AUROC (Area Under the Receiver Operating Characteristic) and AUPRC (Area Under the Precision-Recal Curve) in our initial screening phase. The keyword lists can be accessed here for AUPRC and here for AUROC
2. Automated Script-Based Search: Python scripts were employed to traverse the Arxiv and NeurIPS datasets. These scripts detected occurrences of our predefined keywords, allowing efficient parsing of a vast number of texts from both sources.
3. Dual Mention Selection Criterion: We focused on papers discussing both AUROC and AUPRC. This criterion ensured the relevance of the papers to research question. Through
52

---

this process, we narrowed the pool from 16,022 texts (containing either set of keywords) to 8,244 texts mentioning both in the Arxiv dataset. In the NeurIPS dataset, out of 9,680 texts reviewed, 78 were found to contain keywords from AUPRC and AUROC.
I.3 Al-Assisted Screening and Refinement
1. Preliminary Analysis with GPT-3.5: We utilized OpenAl's GPT-3.5 model for an initial round of Al-assisted analysis for the arXiv dataset. This model identified and extracted papers making explicit claims regarding the comparative effectiveness of AUPRC over AUROC in scenarios of class imbalance, reducing our dataset from Arxiv 2,728 papers.
2. Further Refinement Using GPT-4.0 Turbo: To refine our dataset further, we employed the GPT-4.0 Turbo model. Out of the 2,728 papers scrutinized from Arxiv using this model, 201 were found to be relevant. For NeurlPS, our focused search with GPT-4 resulted in identifying 2 papers of particular relevance to our thesis from the initial set that contained keywords related to both AUPRC and AUROC.
1.4 Manual Review
Shared Document for Collaborative Analysis: We compiled all pertinent papers, along with their respective Arxiv IDs and the claims identified by GPT-4.0 Turbo, into shared Google document for team review. Claims made in papers were found manually, and the specific quote of the claim they made was highlighted along with whether or not they had citation for this claim.
1.4.1 Final Papers
After manual review, we identified 424 papers that make or reference some version of the claim that "AUPRC is better than AUROC in cases of class imbalance." "
iB
B23
LTd
22
FON
550
DO FOU
AST AT
(35)
[3]
HD
K0S
550
59
GEB
50
5351
BNT
358
159
369
7] [80
101 K
190
533
505
337
319
149
23
40%
46
B0
411
233
101
24
M
23 BX
198
[0
410
151
124
Abd
120
204
19]
54
[19
562 316
220
17:
[2]
39
38
2011
419
309
224
100
197
51
24
[85
[5]
[E
ESB
22
805
3M
100
266
Mik
603
Ebl
[33
Lkl
k4
ki
kill
bill
1&51
Mi
211
16d
100
100
kM
200
b03
Ki
18t1
midd
19
30
100
Hill
Min
bidd
Mis
Dxll
100
133
51)
555
bux
bad
KSS
kid Muy
150
15
[05
1B
R33
[2
12
583
210
99
F63
FW
[ot]
150
6000
833
331
[50
9:61
500
259
315
X
153
30m
To0
30
2200
AD
035
146
[1B
100
31
50
[E3)
14
1EH
[03
456
F951
226
195
13
850)
KTM
15
108
3334
83
63
Those papers that reference this claim without citation include
10)
843
[68]
210
B0d
304
E50
[50 154
58
B13
858 En9 (80 BSS
Edy
T0
459
BST
ESC
52
P58
[1]
203
Sc
159
[6]
100
124
440
208
313
2r
Exc
[33
BX
NK
140
288 4003 1hdd 183 AM IMM Mil
Hil HMI
bi 200
[Dk
M01
233 MI
kiil
16
00
191
150
Ubd
too
03
WX
kid
16st
165
2000
5d
k
200
201
(53) 201 ir Lk
209
K0
[M
8811 233
(00 Ubii
bai
8331
50
55g
3
37 [33 5000 202
Ang 200 1200
23
13
[os
235
(22)
130
[53
[85] 250
AID
GV
[bd
259
136
5000
15y
335
269
PAU)
239
[i DE 51
330
830
105
250
[95]
800
5531
ES
[00
13N
Those that do so while citing only other papers that themselves never reference argue Claim
include 5
k0d
E50
150
[E4
A54 ES]
B50 M0
558 41 [59 300 P4
538 0 124 B50 B8
200
[e
530
373
20
20 1I
P0N BS 5
LoB
4X 200 [9]
250 310
100
13
2X6
150
96
572
50
23
500 231 285
[8]
[R
59
B57
534 208 y 08 FR
I8
LB
122 [st) E33 200
216
289
43
[5]
5P3
01
10
MIl
All papers identified, manual screening results, and extracted quotes will be made available upon publication.
53

---

Claim
Refereniess
Commentary
Precision recall curres other asociated metrics more appropriately reflect de- ployuent objectives than the recetver op- crating charactristee.
ZM
35
137
While this claim is true, the infomativeness of the PK curve for lageet deployment merrics is not sufficsent conclude that the AUROC in this more general claim
AUPRC
imbollnee..
Desgie this, without cavest.
AU does not depend on the number of negalives. will be less optimistic AUROC
RIN
2
55A
10
XX
As shown Theorent [ AUROC and AUPRC can both be naturally as function the
the model's false guadrant dependence
generally. assng the ausuaally dependent four quidrants of confusion monix not an infammstive propeity for the AUROC and AUPRC metrics. comparison depends therefore approprintll
positive
AUPPC will sigaificantly optimality, will grow more non-lincarly as model performance improves than AUROC for lou-prevalence tasks
Menrie geioritizes model insenvements, raw magnitude of the metric and more about the vitutions in which
the order of set of models will differ under one metri another. make AURUC smaller
quickly near optimulity by simply exponeniuting not yield better metric.
AUPRC depends prevalence, which desirable property
782
This striement too this dependence prevalencc desinable dapends model comparison in general. argue form induces the bisses inhere in AUPRC previonsly discuvsed. While this claim tnee Theorem it is not clear why this would desired general, implicily
be fommally evalinted: whother
ATPPCC better captures differentiating posiiiv sample with high from "hard" negative sample ("hand" meaning
LDX
comparing "hard"
negatives apainst "easy" positive opposed "easy" nngaaiives againet "hard" positives.
one also with high soore)
AUROC otherwise "optimistie" low-
7W
417
20
[54
Eda
This claim is underspcified, and un-roe. AUROC always means probubiliticaall. and meaning independent imbalance.
presalence settings
AUPRC focues more on the posiive (mi- nority) class
103
3SSS
10 Pa5
This is unfouned:; both AUROC and AUPRC ire weighted expec- tatins over the model's false posilive mtc-AtIPRC cares mone samples regions low firing pusilive or minority samples.
201
103
28U
MBV
explicirly
AUROC can not appropiinely detect mod- cls with recall
This claim is unfounded: the AUROC clearly depends on the model's recall. Resides, if rocall is the measure of intereat, then that should
explicitly.
Table 2: Various arguments and our responses to them present subset of papers for this claim in the literature.
1,5 Code Availability
All code pertaining to the literature review search can be found in the following GitHub repository: https: wtt.be/www..sseeelaeeeeeeerrttta
54

---

Claim
References
Valid?
Commentary
Precision-recall
77
212
339
435
40
292
While this claim is the informativeness the PR for target deployment metrics is insufficient conclude superior to the AUROC in all cases of class imbalance. Despite this, it is often taken to this more general claim without As shown Theorem D AUROC and AUPRC can both be naturally expressed as a function of the expectation of the model's false positive rate. More generally, the lack of dependence on quadrant among the mutually dependent four quadrants of con- fusion matrix is not an informative property for the AUROC and AUPRC metrics.
other associated metries
331
451
222
428
may more appropriately re- flect deployment objectives than the receiver operating characteristic.
the AUPRC
AUPRC does not depend on the number of true neg- atives, so will be less opti- mistic than the AUROC
212
AUPRC will often be signif- icantly lower, farther from optimality, and/or will grow more non-lincarly as model performance improves than AUROC for low-prevalence tasks
212
435
12
254
331
Metric utility for model comparison depends on how appropriately it prioritizes model im- provements. Therefore, is less about the raw magnitude of the metric and more about the situations in which the order of set of models will differ under
451
222
428
metric an- other. One could easily make AUROC yield smaller values or grow more quickly near op- timality by simply exponentiating it, but this would not yield better metric.
AUPRC depends on preva- lence, which a desirable property
339
121
435
This statement is too vague be formally evaluated; whether or not this dependence on prevalence is desirable depends on the context. For model comparison in general, we argue it desirable in this form it induces biases AUPRC previously discussed.
AUPRC better captures dif- ferentiating positive sam- plc with high score from a "hard" negative sample ("hard"' meaning one also with high score)
331
While this claim is true by Theorem 2 it is not clear why this would be desired in gen- cral; this implicitly favors comparing "hard" negatives against "easy" positives as opposed to "easy" negatives against "hard" positives.
Table Various arguments and our responses to them present on a subset of papers for this claim in the literature.
55

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in the abstract reflect this paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations and future work in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
56

---

Justification:
The requisite assumptions and associated proofs for all theorems are provided in full technical detail in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the experiments completely and fully release our code. All datasets used are either synthetic and reproducible in the code itself or publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
57

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As stated above, all data used is either synthetic and fully reproducible or publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
Sublic/guidess http://ooeouiiissoooooo.l.......
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
cttpssccccuublccc www.ll///guu.deedddddbbbbissses wtw.oodeeSubmmsssoooooo.ooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: While we do have some model training results to assess metrics (and for such results all training and test details are fully described in this paper and the full set of parameters and code publicly released), this study is not actually a modelling study, but rather study of machine learning metrics, main contribution is not modeling result that is directly dependent on released
details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We perform appropriate statistical significance tests and report them in this
work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
58

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Our synthetic experiments be replicated in colab notebook with the provided Jupyter notebook file, and the real data experiments are adequately described with released code.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the http://wwwrrprccccppullllllt/ata
NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work is about correcting major misunderstanding in the ML community regarding a widely used evaluation metric, and the potential fairness implications this misunderstanding. In that sense, our entire work is clearly about the potential societal impacts of this misunderstanding, and how should be corrected. We also clearly discuss the limitations of our work in Section
59

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release new data or models in this work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: Only public datasets, appropriately cited, are used in this work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
60

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets are released in this work.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing experiments were done in this work, nor was any research with human subjects done. The manual annotation of reviewed papers in this work was solely performed by authors this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects research was performed in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
61

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
62


Reviews:
[{"review_id": "bOhVXgzy4b", "reviewer": "Reviewer_XQD9", "summary": "The paper considers a common in literature claim that AUPRC is “better” to use than AUROC for class imbalance datasets and attempts to prove it wrong based on theoretical results, empirical observations, and real-world experiments. The major focus of the paper is on the fairness gap that AUPRC exhibits. The authors provide guidance on when each of the metrics should be used and advocate for careful metric selection.", "questions": "1. What causes the variance to be so high in Figure 2?\n2. Could you please elaborate more on what 'Prevalence (Higher)' and 'Prevalence (Lower)' measure in Table 1? If class imbalance, shouldn’t they sum up to 1?", "limitations": "The authors discuss the limitations of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a thorough review of the literature, aiming to get to the root cause of Claim 1, and advocates for more responsible practices in choosing evaluation metrics.\n2. The authors attempt to examine the problem from different perspectives, including theoretical analysis, synthetic, and empirical experiments.\n3. The paper includes extensive details on notation, definitions, experimental details, and figures.", "weaknesses": "1. It is unusual to see in the synthetic example in Section 3 that AUPRC is being optimized for the classification setting (as it is more common to use AUPRC as an evaluation metric). Could you please elaborate on when it can potentially be useful? \n2. For the synthetic dataset example, since everything is controllable, including the number of mistakes, does it make sense to learn boosted trees or any other model of the authors' choice for different levels of mistakes and report AUROC/AUPRC instead of the proposed optimized procedure?\n3. Since AUPRC focuses on the positive class and does not use true negatives, it is \"expected\" by definition of this metric to increase existing fairness gaps in the positive (minority) class. Could you please elaborate on what Theorem 3 adds beyond the definition?\n4. The incorrectly ranked adjacent pair mistake model seems to favor AUROC over AUPRC. By definition, AUROC measures the ability of the model to distinguish between positives and negatives across all possible thresholds, so fixing this mistake improves AUROC uniformly. By definition of AUPRC, correcting such a mistake can have a significant or minor impact depending on the threshold.\n5. While p-values and checking for for statistical significance is important, in addition for the experimental results on real-world datasets, could you please provide actual values of AUROC and AUPRC (mean and variance), for example in a bar plot, similar to what is typically reported in fairness literature?\n6. The broader idea of the paper—to advocate for the careful selection of metrics—is highly welcomed. AUROC and AUPRC are different metrics with different goals, just as ROC and PR curves are. They can provide different insights when evaluating data with class imbalance. Wouldn’t it make sense to encourage readers to explore both AUROC and AUPRC, as well as ROC and PR curves, under class imbalance, given the authors' findings and reference [83] instead of proposing to use one metric as in Section 4?\n\nMinor:\nSome figure legends seem to be cut off in the appendix figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and comprehensive review! Note that, for space reasons, we respond to your two questions in a \"comment\" rather than here in our official \"rebuttal.\"\n\n### W1: Why optimize for AUPRC?\nAUPRC is implicitly used as an optimization metric in cases where it is the target for hyperparameter tuning or model selection (as we explore in our real-world experiments). It also can be an explicit optimization target via specialized algorithms (e.g., [here](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b5dc49f44db2fadc5c4d717c57f4a424-Abstract-Conference.html), [here]( https://proceedings.neurips.cc/paper_files/paper/2021/file/0dd1bc593a91620daecf7723d2235624-Paper.pdf)).\n\nOur work shows that optimizing for AUPRC favors subpopulations with higher outcome base rates. We've added clarification to Section 3: \"_Simulating optimizing by these metrics allows us to explicitly assess how the use of either AUPRC or AUROC as an evaluation metric in model selection processes such as hyperparameter tuning can translate into model-induced inequities in dangerous ways._\"\n\n### W2: Why not use boosted trees for the synthetic example?\nWhile in our real-world experiments, we use boosted trees for exactly the reasons you highlight, in our synthetic experiments we need to use more controlled optimization procedures so that we can precisely attribute the results to the choice of optimizing via AUROC or AUPRC in isolation. In particular, using a classical model risks confounding the impact of AUROC vs. AUPRC with factors such as the ease or difficulty of optimizing towards the target for various subpopulations, optimization algorithms, or output score regions.\n\nTo make this clearer, we have added to Section 3.1: \"_In this section, we use a carefully constructed synthetic optimization procedure to demonstrate that, when all other factors are equal, optimizing by or performing model selection on the basis of AUPRC vs. AUROC risks exacerbating algorithmic disparities in the manner predicted by Theorem 3. For analyses under more realistic conditions with more standard models, see our real-world experiments in Section 3.2._\"\n\n### W3: What does Theorem 3 add beyond the definition of AUPRC?\nOur work's theoretical analyses provide several key insights:\n  1. Theorem 1 demonstrates that AUPRC & AUROC can be expressed as very similar linear functions of the expectation of the model's FPR over positive class scores. This reveals that the key difference between AUROC & AUPRC is not precisely that AUPRC focuses more on the positive class or doesn't care about true negatives, but rather that AUPRC weights model errors more heavily when they occur in high-score output regions vs. low-score regions, whereas AUROC weights all errors equally.\n  2. Theorem 3 formalizes the intuition that you rightly identify: that given AUPRC's focus on the high-score region, subpopulations with a higher prevalence of the label will be preferentially optimized by AUPRC. To the best of our knowledge, this has not been previously proven formally.\n  3. Critically, these theoretical formulations demonstrate both the widespread misconception in the ML community that AUPRC is generally superior for skewed data (Section 5) and formally prove why this misconception is dangerous to model fairness. In particular, despite the intuitive recognition that some top experts like you may have about AUPRC's fairness risks, many authors have used AUPRC as a selection and principal evaluation metric in fairness-centric settings, as noted in a variety of the papers in Section 5.\n\n### W4: Doesn't the mistake model favor AUROC over AUPRC?\nWe agree that correcting adjacent pair mistakes can have varying impacts on AUPRC depending on the threshold. This behavior--and critically, its consequences on model fairness and appropriate use cases--is precisely what we want to highlight to readers in our work. In particular, this model makes it clear that in high-score regions and retrieval contexts without fairness constraints, AUPRC is appropriate. For optimization problems that heavily depend on lower-score regions, AUPRC is less appropriate. This holds _regardless of class imbalance_, despite the widespread belief in the community that AUPRC should be generally preferred in cases of class imbalance.\n\n\n### W5: Can you provide actual AUROC and AUPRC values?\nYes, we will make these results available in our revision. However, we have two notes about these raw values:\n  1. Our experimental analyses are focused on the impact of performing model selection or other optimization by AUROC vs. AUPRC under varying levels of prevalence imbalance. This question is therefore inherently not about the precise AUROC and AUPRC disparities in a single model run, but rather about how optimizing by AUROC vs. by AUPRC impacts fairness gaps in aggregate across many hyperparameter settings and prevalence disparities.\n  2. Accordingly, the number of \"actual AUROC and AUPRC values\" we have is quite large. For our experiments, we train a very large number of models (500+) over different hyperparameters and across many seeds so that we can assess the impact of the evaluation choice with sufficient statistical power. Nevertheless, we will absolutely make all these results available in our revised manuscript.\n\n### W6: Shouldn't readers explore a variety of metrics?\nYou're right; to clarify this, we have added the following to the top of Section 4: \"_Note that while we provide guidance below on situations in which AUROC vs. AUPRC is more or less favorable, this is not to suggest that authors should not report both metrics, or even larger sets of metrics or more nuanced analyses such as ROC or PR curves; rather this section is intended to offer guidance on what metrics should be seen as more or less appropriate for use in things like model selection, hyperparameter tuning, or being highlighted as the 'critical' metric in a given problem scenario._\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqtrNdL8c5", "reviewer": "Reviewer_2iwp", "summary": "The paper proves that AUPRC weights mistakes in higher score ranges higher, while AUROC weights all mistakes uniformly. This property of AUPRC can be underable in many real-life settings. It goes against the widespread belief that AUPRC is somehow “better” than AUROC in low-prevalence domains, which is a common belief both in academia and industry. \n\nThe paper also proves a theorem that states that AUPRC is be discriminatory against low-scoring subpopulations, and it backs up this theoretical finding by a series of experiments that confirm that these discriminatory effects occur in practice when selecting models based on AUPRC", "questions": "- Line 199/200: “We also evaluate the test set AUROC gap and AUPRC gap between groups”. I believe this may be a mistake, as Figures 3, 7, and 8 only appear to report AUROC gaps and not AUPRC gaps. I believe it is the right choice to refrain from reporting AUPRC gaps: the fact that AUPRC is known to be prevalence-dependent would imply that prevalence-gaps by themselves may already explain AUPRC-gaps, even without existence of underlying fairness issues. The experimental setup based on AUROC gaps appears correct.\n- Line 154: \"such that $AUROC(D1) \\approx AUROC(D2) \\approx AUROC(D1\\cup D2) = 0.85$. What is the algorithmic procedure that is used to obtain scores at the target AUROC, and how precisely does this achieve the target AUROC?\n- Line 163: \"Next, we profile an optimization procedure that randomly permutes all the (sorted) model scores up to 3 positions.\" This description seems imprecise, what is the precise reordering procedure that is applied?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "It has long been well-established in the literature that the AUROC is prevalence-invariant (or as they put it in concept drift-focused ML subcommunities: it is invariant to prior probability shift), while the AUPRC is not [1,2,3,4]. In the light of this, the argument that AUPRC would somehow be “better” than AUROC in low-prevalence domains has always appeared paradoxical to me (how could a prevalence-invariant metric be unsuitable in low-prevalent domains?). \n\nThis paper does an excellent job in theoretical analysis of AUPRC and AUROC, an engaging read, and is clear on the practical implications. Figure 1 provides a compelling overview of the main results and manages to convey the implications clearly in one glance. I really enjoyed this read and consider this to be an important paper.", "weaknesses": "A minor critique goes to the experimental section. In both the synthetic and the real-world evaluation the confidence intervals appear rather wide. While the Figures 2 and 3 do appear to support the message that the authors claim that it does, it could be more convincing. For example,\nIs it really true that the AUROC for the low-prevalent sub-group decreases in Figure 2d, or is this noise?\nThe AUROC of the high-prevalence group in Figure 2b appears to trend up while increasing steps while the one of the low-prevalence group appears to stay flat, as the theory would suggest. However, the confidence intervals do still overlap everywhere.\nMost of the CIs for the correlations in Figure 3 include 0.0.\n\nThis appears to simply be a matter of sample size: 20 randomly sampled datasets in the synthetic evaluation and relatively small tabular datasets for the real-life data evaluations.\nIt appears that Figure 2 may become more convincing simply by simulating more randomly sampled datasets (reducing the standard error of the mean), while for Figure 3 we may obtain a more convincing plot by simply including a larger real-life dataset.\n\nThat said, I consider this to be a minor point. The main contributions of the paper are theoretical, and despite my critique, the current figures do support the theory. Just not as convincingly as could have been, and it appears easily solvable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, expertise, recognition of the impact of our work, and helpful suggestions. Below, we address each of your questions or concerns individually.\n\n### W1: Why is the variance so high in the experiments?\nTwo factors contribute:\n1. We report confidence intervals spanning the 5th to 95th percentiles, not standard errors of the mean; these confidence intervals will, therefore, remain wide no matter how many samples we run and not shrink towards the mean as some other measures of variance do.\n2. Our procedure samples from an extremely wide space of possible models, resulting in high true variance by design. This is important so we can make sufficiently general conclusions about the validity of our theory.\n\nWe've added the following to Figure 2's caption to help clarify: \"_Synthetic experiment per-group AUROC, showing a confidence interval spanning the 5th to 95th percentile of results observed across all seeds_\" to help clarify this.\n\nFor the real-world data, these experiments are actually somewhat expensive to run, as we sample a very large number of models, seeds, and hyperparameter options in order to sufficiently assess the correlations of the fairness disparities under AUROC and AUPRC, so using larger datasets does pose an additional challenge there as well. Relatedly, the datasets we have chosen are some of the more common fairness datasets, so without branching into much more intensive modeling domains, we have limited options for additional datasets that are widely used.\n\nThat said, your point is still well taken and we will explore other ways to present these results in our revision to better demonstrate the consistency and statistical reliability of aggregate results across many samples, while still communicating the raw variance.\n\n### Q1: There is a mistake on line 199/200.\nThank you for catching this typo! We have corrected this sentence to be _\"We also evaluate the test set AUROC gap between groups, where gaps are defined as the value of the metric for the higher prevalence group minus the value for the lower prevalence group.\"_\n\n### Q2: What Algorithmic Procedure Obtains Scores at the Target AUROC?\nWe outline the procedure in Appendix G.1. Briefly, we randomly sample scores for positively labeled samples, then draw scores for negatively labeled samples under a distribution that ensures, in expectation, the count of positive samples with scores above or below any given negative sample will be precisely the target AUROC. While exact for single-group AUROC, it is not necessarily exact for multi-group AUROC constraints, though it works well in practice.\n\nTo improve clarity, we have added the following text to our paper:\n  1. On line 154, the sentence now reads _\"... such that $\\text{AUROC}(\\mathcal D_1) \\approx \\text{AUROC}(\\mathcal D_2) \\approx \\text{AUROC}(\\mathcal D_1 \\cup \\mathcal D_2) = 0.85$ (See Appendix G.1 for technical details; ...).\"_\n  2. In Appendix G.1, we have added a new paragraph at the end which states _\"The procedure outlined above guarantees that, in expectation, the AUROC of the generated set of scores and labels will be precisely the target AUROC. However, if you apply this procedure independently across different sample subpopulations, this guarantee can only be applied on each subpopulation individually, and not necessarily on the overall population due to the unspecified xAUC term. However, in practice, for the experiments we ran here, that impact neither meaningfully impacts our experiments nor were the joint AUROCs sufficiently different from the target AUROC to warrant a more complex methodology.\"_\n\n### Q3: What is the precise reordering procedure applied?\nThis procedure is described in detail in Appendix G.3 of the submitted Manuscript, subsection \"M3. Sequentially Permuting Nearby Scores.\" In short, the reordering procedure selects a random permutation of scores, realized as a permutation matrix, such that that permutation matrix has no entries more than 3 positions off the diagonal (thus ensuring that samples cannot change position by more than 3 places). We have amended line 163 to reference this, stating _\"... (sorted) model scores up to 3 positions (See Appendix G.3 for details).\"_ The implementation of this procedure is released along with the rest of our experimental code at the link included in the manuscript. If requested, we are happy to add more details about the procedure we use to the text or the comments here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPCwzAlgCF", "reviewer": "Reviewer_PmDV", "summary": "The paper challenges the claim that area under the precision-recall curve (AUPRC) is a better metric for model comparison to the area under the receiver operating characteristic (AUROC) when it comes to tasks with class imbalance. The paper offers three formal results, proving i) a characterization of the two metrics in terms of how they relate to FPR, ii) a description of how the two metrics optimize for the correction of mistakes and iii) in fairness-sensitive settings, AUPRC introduces biases in favor of the highest prevalence subpopulations wrt AUROC. These findings are supported by synthetic and real-world experiments corroborating the findings.\nThey conclude that the aforementioned claim is unfounded and AUROC is preferable in several settings. Moreover AUPRC is potentially harmful in fairness-sensitive scenarios due to its bias towards the correction of mistakes in the highest prevalence subpopulations. Finally, the paper contains a review on the literature generating the claim and some guidance on when using which of the two metrics.", "questions": "1.     For the relaxation of perfect calibration in Theorem 3, have the authors tried to consider different models with ascending levels of calibration, to see to what extent the property expressed by Theorem 3 holds?", "limitations": "The limitations seem to be properly discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper combines theoretical results with empirical investigations on both synthetic and real-world data.\n2. The paper does important work in challenging a claim that is widespread in the prediction model literature.\n3. The practical implications of the theoretical findings, especially when it comes to fairness, are thoroughly investigated. The authors also research the origin of the claim and provide guidance on when to use which metric.\n4. The text overall reads well and makes clear points. There are however some things to fix urgently in the presentation (which is why the presentation score is low and the overall score cannot be higher).", "weaknesses": "1.     Section 4 has a fair amount of repetition wrt Figure 1. I suggest compressing one or the other.\n2.     Figure 2 is corrupted and the results cannot be assessed. This needs to be fixed.\n3. The explanation is line 129-133 is not clear, I suggest re-writing it. \n4.     A minor point, but I would suggest the authors to not use both boldface and italic at the same time (e.g. in the introduction). It feels unnecessarily intense.\n5. Another minor point, but I would discourage the authors from using self-praising expressions such as “our analyses are thorough and compelling”. Let the reader be the judge of that.\n6. In the caption of Figure 1, repeating the definition of mistake is not really necessary I believe.\n7. The theoretical results have some strong assumptions, e.g. perfect calibration. \n Typos:\nline 290: significnat -> significant\nline 310: These -> these", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and valuable comments!\n\n### W1: Can you reduce the repetition in Section 4 and Figure 1?\nThank you for this suggestion! We've condensed Figure 1's caption and Section 4. However, we maintain some overlap to ensure clear presentation of our key takeaway: understanding when to prefer AUROC or AUPRC as optimization metrics. To demonstrate these changes within the space limitations of the \"official rebuttal\", we provide our updated Figure 1 caption in response to your concern specifically about its redundancy later in this rebuttal.\n\n\n### W2: Why does Figure 2 appear corrupted?\nWe do not see any issues with Figure 2 on OpenReview, using Google Chrome's built-in PDF viewer. Could you please provide more details about the issue and your PDF viewer/OS in a comment? We will follow up promptly to ensure all figures are properly rendered.\n\n### W3: Can you clarify the explanation in lines 129-133?\nIn essence, we show that if one group of samples has a much higher rate of the outcome than another (e.g., men are more likely than women to receive a correct diagnosis for a heart attack), optimizing for AUPRC will provably preferentially optimize to fix mistakes that affect the high prevalence group over the low prevalence group (e.g., the model will preferentially learn to identify heart attack symptoms for men at the expense of learning those for women). In settings where such a model is used to determine who should receive some limited resource (e.g., to be evaluated by a cardiac specialist after an ED visit), this preferential optimization procedure will translate into a disparity in resource allocation between the two groups, which in many cases may be undesirable.\n\nIn the text we have reworded this key result:\n_\"Essentially, Theorem 3 (proof provided in Appendix F) shows the following.\nSuppose we are training a model $f$ over a dataset with two subpopulations: Population $a=0$ and $a=1$. If the model $f$ is calibrated and the rate at which $y=1$ for population $a=0$ is sufficiently low relative to the rate at which $y=1$ for population $a=1$, then the mistake that, were it fixed, would maximally improve the AUPRC of $f$ will be a mistake purely in population $a=1$. This demonstrates that AUPRC provably favors higher prevalence subpopulations (those with a higher base rate at which $y=1$) under sufficiently severe prevalence imbalance between subpopulations.\"_\n\n\n### W4: Can you avoid using boldface and italics simultaneously?\nThank you for this suggestion. We've removed all such instances.\n\n### W5: Can you remove self-praising expressions?\nThank you for this apt suggestion! We have removed all such instances.\n\n### W6: Can you reduce redundancy in Figure 1's caption?\nGreat suggestion. We have removed the repeated definition of the mistake from the figure caption. The whole caption of Figure 1 (which has been further condensed) is shown below:\n\n_\"a) Consider a model $f$ yielding continuous output scores for a binary classification task applied to a dataset consisting of two distinct subpopulations, $\\mathcal{A} \\in \\{0, 1\\}$. If we order samples in ascending order of output score, each misordered pair of samples (e.g., mistakes 1-4) represents an opportunity for model improvement. Theorem 3 shows that a model's AUROC will improve by the same amount no matter which mistake you fix, while the model's AUPRC will improve by an amount correlated with the score of the sample. b) When comparing models absent a specific deployment scenario, we have no reason to value improving one mistake over another, and model evaluation metrics should therefore improve equally regardless of which mistake is corrected. c) When false negatives have a high cost relative to false positives, evaluation metrics should favor mistakes that have *lower scores*, regardless of any class imbalance. d) When limited resources will be distributed among a population according to model score, *in a manner that requires certain subpopulations to all be offered commensurate possible benefit from the intervention for ethical reasons*, evaluation metrics should prioritize the importance of within-group, high-score mistakes such that the highest risk members of all subgroups receive interventions. e) When false positives are expensive relative to false negatives and there are no fairness concerns, evaluation metrics should favor model improvements in decreasing order with score.\"_\n\n### W7: How do you address the strong assumptions in theoretical results, e.g., perfect calibration?\nThis is true; We acknowledge this limitation and have extended our discussion in Section 6: _\"In addition, one of the largest limitations of Theorem 3 is its restrictive assumptions, in particular the requirement of perfect calibration. A ripe area of future work is thus to investigate how we can soften our analyses for models with imperfect calibration or to determine whether or not our results imply anything about the viability or safety of post-hoc calibration of models optimized either through AUPRC or AUROC\"_\n\n### W8: Can you fix the typos on lines 290 and 310?\nThank you for catching these! We have corrected both.\n\n### Q1: Have you considered models with ascending levels of calibration?\nThis is a great area for future work. We believe it's possible but requires careful specification of model miscalibration. For example, bounding a model's calibration error over any region of the output score space could help bound the extent to which prevalence gaps correspond to different prediction rates, translating into bounds on AUPRC's preference for fixing high-prevalence group mistakes. We've highlighted this in Section 6 with strengthened language.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GD9rsA2YMU", "reviewer": "Reviewer_pyKS", "summary": "A widespread claim in machine learning that AUPRC is a superior metric than AUCROC for tasks with class imbalance is not strictly true. Based on this statement, this paper challenges this claim from two perspectives. On the one hand, the authors theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, showing that optimizing for AUROC equates to minimizing the model’s FPR in an unweighted manner, whereas optimizing for AUPRC equates to minimizing the FPR in a weighted manner. On the other hand, experiments on both semi-synthetic and real-world fairness datasets support their theory.", "questions": "- Theorem 3 is based on two subgroups but real-world datasets generally have more than subgroups. Is recommended that the authors provide analysis based on multiple subgroups.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "-\tThe proposed ideas are novel and innovative. Theoretically, the authors explore the relationship between AUROC and AUPRC, revealing their key differences. Specifically, the authors show that while optimizing for AUROC equates to minimizing the model’s FPR in an unbiased manner over positive sample scores, optimizing for AUPRC equates to reducing the specifically for regions where the model outputs higher scores relative to lower scores. In addition, the authors propose that AUPRC is explicitly discriminatory in favor of high-scoring subpopulations. \n-\tThe authors validate the proposed theoretical findings with the help of numerical results. To rigorously confirm the findings of differences between AUROC and AUPRC, the authors conduct many synthetic experiments and real-world validation on popular public fairness datasets.\n-\tGuidance on how to choose AUROC and AUPRC is provided. In section 4, the authors offer detailed instructions for use in different scenarios. For context-independent model evaluation, deployment scenarios with elevated false negative cost, and ethical resource distribution among diverse populations, AUCROC is a more proper metric. However, for reducing false positives in high-cost, single-group intervention prioritization or information retrieval settings, AUPRC is a better choice.", "weaknesses": "- The presentation needs to improve. For example, the term “high prevalence subgroup” is introduced without explanation, hindering my understanding of the theorem. A more detailed explanation of the high and low-prevalence subgroups should be provided. Besides, In theorem 3, the authors say that there exists a prevalence disparity sufficiently severe, but I can't see it directly from this theorem and suggest that the authors provide a clearer explanation.\n- I find this paper (Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability, NeurIPS 2022) also has a similar form (equation 2) and conclusion about AUCPRC. Can you explain the difference between your method and this one? The corresponding citation is also necessary.\n- The caption of Figure 1 is too long. It is recommended to make it more concise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your apt and constructive review!\n\n### W1: Can the presentation be improved, particularly regarding the clarity of \"prevalence\"?\nIn essence, in Theorem 3, we show that if one group of samples has a much higher outcome rate (prevalence) than another (e.g., men are more likely than women to receive a correct diagnosis for a heart attack), optimizing for AUPRC will provably preferentially optimize to fix mistakes that affect the high prevalence group over the low prevalence group (e.g., the model will preferentially learn to identify heart attack symptoms for men at the expense of learning those for women). In settings where such a model is used to determine who should receive some limited resource (e.g., to be evaluated by a cardiac specialist after an ED visit), this translates into a disparity in resource allocation between the two groups, which in many cases may be undesirable.\n\nTo clarify this in our text, we have added: _\"Essentially, Theorem 3 (proof provided in Appendix F) shows the following. Suppose we are training a model $f$ over a dataset with two subpopulations: Population $a=0$ and $a=1$. If the model $f$ is calibrated and the rate at which $y=1$ for population $a=0$ is sufficiently low relative to the rate at which $y=1$ for population $a=1$, then the mistake that, were it fixed, would maximally improve the AUPRC of $f$ will be a mistake purely in population $a=1$. This demonstrates that AUPRC provably favors higher prevalence subpopulations (those with a higher base rate at which $y=1$) under sufficiently severe prevalence imbalance between subpopulations.\"_\n\nThis clarifies that prevalence relates to the probability of $y=1$ for a specific subgroup. The prevalence disparity is captured by the limit as $P(y=1|a=0)$ approaches zero while $P(y=1|a=1)$ remains fixed. If you have further feedback on how to further clarify this point, please let us know.\n\n\n### W2: How does your work relate to the NeurIPS 2022 paper on AUPRC optimization?\nThank you for pointing out this related work! We've included it as a reference and explained how our findings are synergistic with and extend upon this excellent prior work:\n  1. In our proof of Theorem 1: \"_Note that this formulation of AUPRC reflects earlier, different formulations of AUPRC, such as those found in the AUPRC optimization literature (Wen et al., 2022)_\"\n  2. In our Synthetic Results section, where we comment on the impact of optimizing for AUROC vs. AUPRC: \"_These results demonstrate explicitly that not only does optimizing for AUPRC differ greatly than for AUROC, as has been noted historically by researchers developing explicit AUPRC optimization schemes (Wen et al., 2022), but it in fact does in an explicitly discriminatory way in very realistic scenarios._\"\n  3. Finally, in Section 5 (our literature review), we note that \"_The widespread nature of Claim 1 has also led researchers astray when exploring new optimization procedures for AUPRC, by advocating for the importance of AUPRC when processing skewed data, even in domains such as medical diagnoses that often have high false negative costs relative to false positive costs (Wen et al., 2022)._\"\"\n\nTo further clarify the novelty of our work relative to that of Wen et al., note that while Wen et al. provide a related probabilistic expression for AUPRC designed to facilitate their optimization algorithm, our Theorem 1 (which presents a different formulation of AUPRC) is designed to clearly show the mathematical relationship between AUROC and AUPRC. Our impact comes from realizing the implications of this relationship on the strengths and weaknesses of these metrics, challenging the popular opinion that class imbalance is the defining factor in their distinct use cases.\n\n\n### W3: Can you shorten the caption of Figure 1?\nWe've shortened the caption considerably; it now reads: _\"a) Consider a model $f$ yielding continuous output scores for a binary classification task applied to a dataset consisting of two distinct subpopulations, $\\mathcal{A} \\in \\{0, 1\\}$. If we order samples in ascending order of output score, each misordered pair of samples (e.g., mistakes 1-4) represents an opportunity for model improvement. Theorem 3 shows that a model's AUROC will improve by the same amount no matter which mistake you fix, while the model's AUPRC will improve by an amount correlated with the score of the sample. b) When comparing models absent a specific deployment scenario, we have no reason to value improving one mistake over another, and model evaluation metrics should therefore improve equally regardless of which mistake is corrected. c) When false negatives have a high cost relative to false positives, evaluation metrics should favor mistakes that have *lower scores*, regardless of any class imbalance. d) When limited resources will be distributed among a population according to model score, *in a manner that requires certain subpopulations to all be offered commensurate possible benefit from the intervention for ethical reasons*, evaluation metrics should prioritize the importance of within-group, high-score mistakes such that the highest risk members of all subgroups receive interventions. e) When false positives are expensive relative to false negatives and there are no fairness concerns, evaluation metrics should favor model improvements in decreasing order with score.\"_\n\n### Q1: Can you provide analysis based on multiple subgroups?\nThis is a great question and a rich area for future work. We've added to our Future Work section:\n\"_Firstly, our theoretical findings can be refined and generalized to... specifically take into account more than 2 subpopulations for more nuanced comparisons beyond what can be inferred through pairwise comparisons between subpopulations, where our results would naturally apply_\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their insightful, comprehensive, and constructive feedback! We're particularly encouraged by the widespread recognition of our work's novelty, impact, and theoretical contributions, with reviewers noting things like 
  * "I really enjoyed this read and consider this to be an important paper." _(Reviewer 2iwp)_
  * "This paper does important work in challenging a claim that is widespread in the prediction model literature." _(Reviewer PmDV)_
  * "The proposed ideas are novel and innovative. Theoretically, the authors explore the relationship between AUROC and AUPRC, revealing their key differences" _(Reviewer pyKs)_

In addition, we found the various points of feedback extremely helpful and have, based on your comments, made the following key improvements:
  1. **Systemic Presentation Improvements**: While we were glad to see some aspects of our presentations highlighted positively, reviewers were generally aligned on a need for improvements to our presentation. Accordingly, we have
     - Extensively condensed the caption of Figure 1 and reduced redundancy between Figure 1 and Section 4, while maintaining their clarity and informative value _(as suggested by Reviewers PmDV and pyKS)_.
     - Clarified our usage of "prevalence" across multiple subpopulations in the context of Theorem 3, significantly enhancing the clarity of one of our key theoretical results _(as suggested by Reviewers 2iwp, PmDV, and pyKS)_.
     - Addressed any missing experimental details, typos, poor word/formatting choice, missing citations or references to appendix sections, and added raw experimental results _(as suggested by Reviewers XQD9, 2iwp, PmDV, and pyKS)_.

  2. **Clarification of the Novelty and Import of our Findings**: While reviewers 2iwp, PmDV, and pyKS all noted specifically the novelty and importance of our findings, we also made significant improvements to these areas as well in line with the many suggestions, including:
     - Expanding our commentary on how our findings extend upon the raw definition of AUPRC and on why it matters that AUROC and AUPRC behave differently under the "mistake correction" model of optimization _(as suggested by Reviewer XQD9)_.
     - Clarified the notion of "AUPRC optimization" in our work and our work's relationship to past works on AUPRC optimization _(As suggested by Reviewers XQD9 and pyKS)_.

We're confident these changes address the main concerns raised and strengthen the paper significantly. Each reviewer's concern is addressed in more detail below. Please do not hesitate to respond to our rebuttals or leave additional comments if you have more feedback or questions or want to see further revisions to address your concerns. Thank you to all reviewers again.

Meta-review: This paper provides a comprehensive theoretical and empirical rebuttal of the folklore assertion that area under the precision-recall curve (AUPRC) is a superior metric for model comparison as compared to the more standard area under the receiving operator characteristic (AUROC) in situations where there is class imbalance. Ultimately, all reviewers were in support of acceptance of this paper and found the contributions to be innovative and valuable to the ML community. I concur with this recommendation.

Please address all suggestions made by the reviewers to improve readability and figure formatting in the camera-ready version.
Decision: Accept (poster)