Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs
Authors: Gerardo Flores
Alyssa Hasegawa Smith
Julia Fukuyama
Ashia C. Wilson
Venue: NeurIPS 2025 poster ()
Keywords: binary classification
calibration
label shift
cost-sensitive evaluation
proper scoring rules
Primary area: evaluation
TL;DR: Clipped Log Scores measures miscalibration in units of outcomes, allowing for bounded but uncertain label shift as well as asymmetric costs.

Abstract: Machine learning-based decision support systems are increasingly deployed in clinical settings, where probabilistic scoring functions are used to inform and prioritize patient management decisions.
However, widely used scoring rules, such as accuracy and AUC-ROC, fail to adequately reflect key clinical priorities, including calibration, robustness to distributional shifts, and sensitivity to asymmetric error costs.
In this work, we propose a principled yet practical evaluation framework for selecting calibrated thresholded classifiers that explicitly accounts for uncertainty in class prevalences and domain-specific cost asymmetries.
Building on the theory of proper scoring rules, particularly the Schervish representation, we derive an adjusted variant of cross-entropy (log score)  that averages cost-weighted performance over clinically relevant ranges of class balance.
The resulting evaluation is simple to apply, sensitive to clinical deployment conditions, and designed to prioritize models that are both calibrated and robust to real-world variations.

Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs
Gerardo A. Flores* MIT
Alyssa H. Smith Northeastern University
Julia A. Fukuyama Indiana University
Ashia C. Wilson MIT
Abstract
Machine learning-base based decision support systems are increasingly deployed in clinical settings, where probabilistic scoring functions are used to inform and prior- itize patient management decisions. However, widely used scoring rules, such accuracy and AUC-ROC, fail to adequately reflect key clinical priorities, including calibration, robustness distributional shifts, and sensitivity asymmetric error costs. In this work, we propose a principled yet practical evaluation framework for selecting calibrated thresholded classifiers that explicitly accounts for uncertainty in class prevalences and domain-specific cost asymmetries. Building on the theory of proper scoring rules, particularly the Schervish representation, we derive an ad- justed variant of cross-entropy (log score) that averages cost- cost-weighted performance over clinically relevant ranges of class balance. The resulting evaluation is simple to apply, sensitive to clinical deployment conditions, and designed to prioritize models that are both calibrated and robust to real-workd variations.
Introduction
The field of medicine increasingly relies on machine learning tools for clinical decision support. For both diagnostic and prognostic applications, probabilistic scoring functions inform and prioritize decisions about patient management. As such, it is critical that these scoring functions reflect clinical priorities. We propose three principles that scoring functions used for clinical purposes should satisfy as closely as possible. First, scoring functions should be adapted to account for the known label shifts that commonly arise between development and deployment environments. Because medical datasets are often deliberately rebalanced during development to improve sensitivity to rare outcomes, standard metries computed under these distributions can overestimate performance deployment, where class prevalences are highly skewed. Second, the scores returned by scoring functions should be sensitive to the relative cost of errors that are clinically significant, such as the trade-off between the cost of misdiagnosis and the cost of failing to diagnose in any given setting. This supports patient-centered care by enabling the classifier's sensitivity to be calibrated to human feedback, rather than presuming fixed normative standard. Third, scores should be calibrated to ensure that their outputs align with a common interpretive frame, enabling practitioners to form consistent and reliable expectations about risk and outcomes.
This work focuses on evaluation; specifically, we examine how the field of medical machine learning assesses and compares scoring functions and the extent to which current evaluation practices reflect these aforementioned clinical priorities. We begin by showing that neither of the most commonly used metrics, accuracy and AUC-ROC, adequately captures all three priorities outlined above. Each abstracts away some considerations that are critical for clinical decision-making.
We structure the paper as follows. We first examine accuracy and its variants, as these remain the most widely used evaluation metries for classification tasks. Accuracy evaluates each decision
Corresponding author: mullset@mit.. edu. Code available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

independently and measures the overall proportion of correct predictions, abstracting away critical pepliiatioonspecifif considerations such as class imbalance and asymmetric evror costs. While this abstraction offers a form of neutrality, it obscures important aspects of clinical deployment, where decision thresholds must often be adapted to reflect evolving prevalence rates or varying tolerances for false positives and false negatives. As several works have noted [Bradley] [1997), Provost [ and Fawcett 1997]. Drummond and Holte (2006), accuracy fixes a single operating point and therefore fails to accommodate the flexibility required for robust clinical evaluation.
We then turn our attention to the Area Under the Receiver Operating Characteristic Curve (AUC- ROC), which is commonly viewed as solution to the rigidity of accuracy, as it evaluates classifier performance across all possible thresholds. However, AUC-ROC measures the expected performance of the ideally calibrated version of a scoring function, not the actual, potentially miscalibrated outputs of a model. Moreover, it ties evaluation to distribution over positive prediction rates that may not correspond to clinical contexts. These assumptions often lead AUC-ROC to overstate the real-world reliability of scoring functions, especially when calibration imperfect or deployment conditions differ from development data.
Recent work in the fairness literature has explored calibration more directly [Kleinberg et al., 2017 Corbett-Davies et al., 2017 HHbbrrtJJhhssn et al. 2018), but the literature lacks broad consensus on best practices for how calibration interacts with varying cost structures. particular, the definition of perfect calibration is widely agreed upon, but the correct way to measure degrees of miscalibration, taking into account label prevalence and asymmetric error costs, is not. As a consequence, the use of calibration based metrics has lagged behind that of accuracy and AUC-ROC, in clinical MI. settings. To address these concerns regarding current evaluation practices, we propose adapting a framework from the weather forecasting and belief elicitation literature known as the Schervish representation Schervish [199]]. This framework shows that any proper scoring rule (a measure of calibration that does not require binning) can be represented as an integral over discrete cost-weighted losses, directly linking calibration to decision-theored performance. We extend this framework to the setting of label shift and asymmetric costs, and we average cost-sensitive metrics over a bounded range of class balances.
In summary, this work makes two main contributions. First, we introduce a framing of scoring rule design that centers clinical priorities, namely calibration, robustness to distributional shift, and sensitivity to error costs. Second, we propose an adaptable scoring framework based on adjusted log scores that reflects clinical needs. It accommodates uncertainty in class balance, asymmetric cost structures, and the requirement for calibrated predictions. thereby offering more principled foundation for evaluating machine learning models in clinical decision support.
1.1 Problem Formulation
Given an input space X' and binary label space {0, 1}, the standard goal of binary classification is to learn decision rule that maps each input x X' to predicted label. To do so, scoring function s : X - IR is designed to assign a vellev score to each input, which is then converted into a binary prediction by thresholding. For threshold parameter E R, the predicted label is defined as
<(s()),7) 7) (c(x)27)
where (()) denotes the indicator function, equal to the argument is true and otherwise. We denote the dataset by D Droe consisting of input-label pairs (ær, y) drawn from an unknown distribution. We define the empirical class prevalence as To = P (y = 1), which represents the proportion of positive examples in the dataset and the possibly unknown targer or deployment class prevalence as π = PDD (y 1). To formalize evaluation objectives, we introduce three additional elements: (1) A value function where V(y, x(s(a)), r)) specifies the utility or loss associated with predicting k( k(s(x), 7) when the true label is and (2) parameter E (0. 1), which encodes the relative cost of false positives and false negatives and determines the threshold; and (3) a distribution H over possible data-generating distributions Dx., modeling uncertainty over the environment and potential
distribution shifts. We denote odds multiplication by @ ab+(1-0[1-B)
ab
We denote the
clipping operator by clip(z) max(o, min(b, z)). [a.0]

---

1.2 Related Work
We provide an extensive discussion of related literature which includes calibration techniques, robust- ness to distribution shift, cost-sensitive learning, visualization methods, and more in Appendix [A] Here, we focus on the most directly relevant conceptual threads that motivate our approach and situate our contributions. A growing body of work in Al emphasizes that evaluation metrics must reflect peployyeen-specifee decision contexts rather than relying on generic benchmarks that can obscure model behavior [Cruz Rivera et al., 2020 Liu al., 2020]. This view aligns with long-standing traditions in decision theory, including Decision Curve Analysis (DCA) [Vickers and Elkin 2006). which critiques metrics like AUC and the Brier score for lacking clinical relevance [Vickers and
Holland [
Assel et al. 2017].
The theory of proper scoring rules has similarly evolved from early formulations [Brier
1950 Good
1952 to rich characterizations linking scores to decision-theoretis utility [Schervish] 1989 Shen
2005]. These foundations have motivated context-sensitive adaptations, such as asymmetric scoring
rules [Hand and Anagnostopoulos 2014). although adoption remains uneven (see Appendix A for details on the difficulty of picking Beta parameters).
Recent efforts in calibration PPltt 1999 Guo et al. 2017 fairness-awarr evaluation [Hebert-Johnson al. 2018 and robustness under label shift MMoreno-Toree et al., 2012 Muandet et 2013] all aim to bridge the gap between formal guarantees and practical constraints. Meanwhile, AUC-based
metrics continue face scrutiny over interpretability and subgroup fairness (Kallus and Zhou 2019).
prompting exploration of cost-sensitive methods [Elkan 2001] and visualization- diagnostics
[Murphy] 1977
Ehm al. 2016].
Frequent suggestions are made to measure both AUC-ROC and Expected Calibration Error (ECE)
Pakdaman Naeini et al. [2015], , despite known critiques of ECE /Vaicenavicius al. 2019 Widmann
et al. 2019 Nixon et al. 2020 Futami and Fujisawa 2025 Roelofs et al. [022 Ferrer 2022 and
the lack of principled way combine the two scores.
Our work contributes to this evolving landscape by proposing an evaluation framework that preserves the rigor of proper scoring rule theory while introducing design flexibility to reflect bounded cost asymmetrics and decision-relevant dispersion. In doing so, we aim to retain the theoretical benefits of established metries while addressing their practical limitations in contexts where calibration, subgroup validity, or threshold sensitivity are paramount.
2 Accuracy: Calibration without Label Shift Uncertainty
The most popular metric for evaluating binary classifiers is the simplest: accuracy.
Definition 2.1 (Accuracy). Given a dataset D Day a score function 8, and a threshold t, the accuracy is defined as
Accuracy 8,
Σ 1 ((=x(((t))))))
Dool
(a.))EDT
Accuracy considers the binarized score, discarding the real-valued information necessary for assessing calibration uncertainty. It further assumes false positives and false negatives are equally costly. This is misaligned with most real-world decision problems where asymmetric stakes are the norm. Finally, the validity of the evaluation results presumes that the operational data setttaagneraing distribution matches the evaluation distribution, thereby ignoring the possibility of distribution shift. We describe existing extensions to address asymmetric costs, label shift, and calibration.
Asymmetric Costs In most practical decision problems, false positives and false negatives carry asymmetric consequences. Several extensions of accuracy have been proposed to account for this asymmetry. Two commonly used variants are net benefit and weighted accuracy. The notion of net benefit that we use originates from decision curve analysis (DCA) [Vickers and Elkin 2006 but is similar in structure to earlier formulations [Murphy] 1966). We use variation of net benefit parameterized by the benefit of true negatives rather than the costs false positives in order to be more directly comparable to accuracy.
3

---

Definition 2.2 (Net Benefit), Given a dataset Dmo a score function s, a threshold and cost parameter e E (0, 1), the net benefit is defined as
Net Benefit (D mo : 8, T, c)
1 D DTo
E (2.9)EDuo
1-9 I-c
1
((=x(((w)..))
However, the procedure by which the threshold should be set given what we now know about cost ratio deployment remains unclear: namely, will often not match the cost ratio used in training. If a score function s(x:) is well-calibrated, we can reliably threshold it to optimize binary decisions under any cost asymmetry. Specifically, the optimal threshold r satisfies P(Y = 1|s(z) = 7) = where V is the value function. See Appendix B. for details. Net benefit has the advantage that the interpretation of true positives remains consistent with standard accuracy. That is, true positives are rewarded uniformly regardless of the cost ratio, while false positives are penalized according to the cost ratio determined by c. Another popular metric is weighted accuracy, which corresponds to what Murphy [966) called relative utility, and is normalized so that a perfect classifier achieves
V(0.1)-V(0.0)
(0.1))-/(000)++(((1))))
of 1 regardless of class balance. We provide a definition in Appendix [ While net benefit and weighted accuracy are both widely used, both inherit critical limitations from the basic accuracy framework: they binarize the score, thereby discarding information about uncertainty and calibration, and they assume fixed data-generating distribution, thereby failing to account for distribution shift.
2.1 Label Shift
To model deployment scenarios, we adopt a causal perspective. Under the label shift assumption, represented by the causal structure Dr Y X, the conditional distribution P(X Y, >) P(X Y) remains invariant across domains. This assumption holds in many clinical contexts, where observed features (X) reflect underlying conditions (Y) whose prevalence varies across populations (D+). We focus on this structure because aligns with the intuition of identifying latent diagnostic classes and enables robust correction methods for distribution shift. In contrast, the alternative causal structure Dr + X + Y is often associated with timento-evet outcomes, requiring distinct modeling strategies such survival analysis.
The DT Y - X structure permits importance sampling to estimate deployment- expecta- tions. However, because prediction is performed via P(Y X) must also adjust the posterior using Bayes' rule to account for class prevalence changes [Meehl and Rosen 1955
Heckman 1979
Saerens al. 2002). This yields the adjusted posterior:
P(Y = [ss( xx)) = P(Y = 1|s(x), Dm,)) @ (1 %g n
We can then predict the net benefit attainable in the deployment environment if the score is adjusted using this formula. We define (I) A 1
2 s(x), and denote the adjusted binary classifier by K( & (1/2(3); 7). The full derivation is provided in Appendix B.3
Definition 2.3 (Prior-Adjuste Maximum Net Benefit). Given empirical dataset D mo. score function 8, a threshold т, the deployment distribution DT,, and the cost ratio parameter c E (0, 1), the prior-adjusted maximum nel benefit is defined as,
cl no ) 1 (1=N(T89))())))))
( C 1 - π
PAMNB m. 8, C,
Σ D
D
To
1
The above expression generalizes several familiar metrics. When c = 1/2 it reduces to the prior- adjusted maximum accuracy (PAMA) (see Definition C.3 recovering the standard decision-optimal accuracy under label shift. For arbitrary c, provides a cost-sensitive extension that supports domain adaptation under both class imbalance and asymmetric decision costs. While this approach can fully adjust for any fixed label shift, it still relies on binarized scores and thus cannot capture uncertainty over the true deployment prevalence.
2.2 Calibration
Beyond adapting to shifts in population prevalence, another crucial aspect of evaluating probabilistic model outputs, particularly in clinical decision-making, is calibration. Unlike threshold- based

---

decision-makee which focuses on classification accuracy, the goal of calibration is to ensure that predicted probabilittiss match observed frequencies: that is, P(Y 1[s(x)) s(x). This perspective, well-established in the weather forecasting literature, prioritizes reporting reliable probabilities over optimizing decisions directly. However, calibration alone does not guarantee utility.
Consider the ACC/AHA guidelines for primary prevention of cardiovascular disease. The guidelines recommend prescribing statins based 10-year cardiovascular disease risk, thresholded at 2.5%, 5%. 7.5%, and 20% [Amett et al., 2019). A calibration-only approach can be pathological: a model that predicts the population base risk (around 10%) for everyone would receive an excellent score from calibration-onle metrics like ECE (Ferrer and Ramos 2024), but it would be clinically unhelpful because recommends light dose of statins for every patient. It easiest to understand optimal thresholding if classifier has everywhere the property that P(Y = 1[s(z) = c) = c.
However, what is practically necessary for classifier support good thresholding at is that P(Y = 11s(x) > r) > and P(Y 0]s(z) < 7) < even if it is miscalibrated elsewhere. In the ACC/AHA example, the absence of deployment shift, we would need this property hold at 2.5%, 5%, 7.5%, and 20%; it is not necessary for it to hold elsewhere.
Uncertain label shift more complex and motivates need for broader sense of calibration. If we can bound the possible class balances, we will need the model to be calibrated within the whole corresponding range. A score function that well-calibrated only in this narrow region can, however, still support robust, cost-sensitive classification. This suggests more nuanced perspective: rather than enforcing global calibration, it may suffice to ensure calibration (by measuring miscalibration) within a threshold band. Part of the contribution of this paper is to formalize and operationalize this idea.
3 AUC-ROC: Label Shift Uncertainty without Calibration
The most common approach to integrate over range of operating conditions is to use AUC-ROC instead of accuracy. AUC-ROC an ordinal metric that still based on thresholding. but it discards information about the magnitudes of those thresholds and evaluates performance solely based on the relative ordering between positive and negative examples.
Definition 3.1 (AUC-ROC). Let 8 distribution. The AUC-ROC is given by
X ->
R be a scoring function on D the training data
AUC-ROC (DO no: 8)
y
Σ
V/ ([(otes20())) + (((0)))()))
D
TO
D Tu
(2.y)
(w',s')ED
At first glance, this formulation poses a challenge for decision-theoretic interpretation. Specifically, is not priori clear how to interpret AUC-ROC within framework where the metric corresponds to expected utility or decision quality under a specified loss function and distributional assumption. AUC-ROC resists this interpretation because it is invariant monotonic transformations of the score function and, therefore, is indifferent to the calibration or absolute values of the scores, which are central threshold-based decision-making, On the other hand, AUC-ROC does capture something that accuracy fails to: it aggregates performance across the full range of the score distribution, effectively summing population-level statistic over levels of the score.
There are numerous ways to interpret AUC-ROC, at least a dozen of which are enumerated in Appendix G We nevertheless offer a new formulation, whose proof is in Theorem H.55 that sheds particular light on its relationship to label shift, i.c. when the marginal distribution over labels differs between training and deployment.
Theorem 3.2 (AUC-ROC as Accuracy Averaged Across Label Shift). Let 8 be a scoring function that is calibrated on the evaluation distribution D Dono Then:
AUC-ROC(s)
8, 1/2, 1/2)
where Diy2 denotes a balanced reweighting of the dataset (i.e., class prior π = 1/2), and s s(Daya) denotes the distribution of model scores over this reweighted set.
This perspective reveals that AUC-ROC can be viewed as averaging thresholded accuracy across distribution of class prevalences, albeit one that is induced implicitly by the score distribution of
5

---

the model itself. This provides limited form of robustness to label shift in contrast to metrics like accuracy which are typically evaluated at a fixed class balance. Yet this same interpretation also exposes several critical limitations.
First, AUC-ROC entirely disregards calibration. By evaluating only the ordering of scores, it fails to assess whether predicted probabilities are well-aligned with empirical outcomes, which is a property essential in many domains where decisions depend on absolute risk levels. This issue is shared by other ranking metrics such as AUC-PR and net discounted cumulative gain, which similarly ignore score magnitudes. frequent suggestion made in response is report ranking metries together with binned calibration metrics, but there is in general no principled way to combine the two scores to make a decision. As shown in Table I real clinical data can produce cases where AUC-ROC and ECE together fail to indicate which subgroup performs better, underscoring the need for unified evaluation criteria.
Second, although AUC-ROC evaluates calibrated scores for their performance across varying class balances, the distribution over these prevalences is not user-specified or interpretable. It is instead byproduct of the model's score distribution on a hypothetical balanced dataset. Consequently, the underlying population over which AUC-ROC aggregates accuracy differs across models, making metric comparisons across models trained on the same data unreliable.
Finally, AUC-ROC does not allow the independent specification of label shift and asymmetric error costs. Although we can interpret varying prevalences as including varying cost ratios through the relationship xL with shifts in class balance.
π (Hernandez-Orallo al. 2011), doing so entangles cost asymmetry
In summary, AUC-ROC offers partial advantage over accuracy by aggregating across class balances, but its benefits are offset by its insensitivity to calibration, its implicit and model-dependent averaging distribution, and its inability to account for cost asymmetry. While it captures ranking performance, fails to reflect key aspects of real-world decision quality. The historical development of AUC- ROC provides important context. Ordinal metrics originated in fields like psychology, where class prevalences were fixed by design, and information retrieval, where results per page were fixed by the capacity constraint of the querying user regardless of quality. Their subsequent adoption in machine learning reflects a shift in evaluation priorities away from deployment evaluation and toward the abstract comparison of new architectures and optimization techniques. In such a setting, ordinal metrics offer a convenient, threshold-fis mode of comparison. However, such metrics are poorly aligned with the needs of real-world deployments, where thresholding, cost asymmetries, and calibration are often indispensable.
4
Log Score: Label Shift, Calibration and Cost Asymmetry
To evaluate the utility of a thresholded classifier under uncertain or varying class balance, it is critical to evaluate the calibration of its underlying score function across a range of label distributions. Calibration metrics are only meaningful insofar as they are expressed in units that reflect application- specific costs. In this context, cost asymmetry not a minor adjustment but a first-order concern that must be explicitly accounted for. Moreover, a persistent challenge in real-worke deployments is the difficulty of comparing the impact of miscalibration, measured in cost-aligned units, with the loss in performance attributable to poor sharpness uncertainty in ranking (Vickers and Holland 2021]. As we have seen, measuring both AUC-ROC and ECE separately does not solve this problem.
We build on the view that proper scoring rules can be interpreted as mixtures over a distribution H of data distributions Dr,, where each scoring rule evaluates cost-weighted errors V over the corresponding Dz Our approach does not have the ambiguity of combined cost / balance terms in Drummond and Holte [2006), nor does require the problematic double integration over both cost and prevalence as suggested in Hand and Anagnostopoulos [2023). Instead, we fix the cost ratio e and integrate over the variability of data distributions captured by H. yielding tools that are computationally simpler and semantically interpretable.
4.1 Background
Accuracy can be generalized to account for asymmetric costs and label shift, but it assumes exact knowledge of a single value of class balance in deployment. The log score (or cross entropy) can,

---

owing to the Schervish representation, be viewed as an average of accuracy over the full range of possible class balances, from 0 1. Unfortunately, as Assel et al. [2017] point out, averaging over so wide a range is not specific enough to be clinically useful. We need, therefore, to find a way to narrow that range. When there are only couple deployment settings, we can take discrete average over performance in each. But as uncertainty grows we need a simpler and more reproducible continuous approach to narrowing the distribution.
4.2 Clipped Cross Entropy
The core contribution of this paper is to propose: (1) a simple way to characterize uncertainty in label shift using lower and upper bounds on class balance, (2) a straightforward means to average accuracy over that range. and (3) natural extension to two standard approaches for handling asymmetric costs. As previously mentioned, there exists a duality between measures of calibration and mixtures of accuracy measures across different prevalences. This result is somewhat unintuitive; see Appendix for a derivation that sheds more light. We extend this result to average over only specific subinterval of prevalences. Our new contribution demonstrates how this can also be applied when costs are asymmetric. These formulas enable straightforward calculation of the average cost-sensitive performance of a classifier over specified range of prevalences. A major practical benefit of this approach is that it is based on a pointwise calculation of loss, so confidence intervals can be trivially bootstrapped by resampling calculated losses, and each new draw only requires weighted sum of these losses.
Bounding the Class Balance We begin by specifying a lower bound and an upper bound b on the class balance. These bounds can be obtained in direct consultation with domain experts, through surveys, utilizing previous estimates. Consider the range of syphilis prevalence in US states: between about 7 in 10,000 (South Dakota) and in 1,000,000 (Vermont) (U.S. Centers for Disease Control and Prevention 2024]. Though two orders of magnitude wide, this interval still substantially constrains the range of possible prevalences.
Intuitively, practitioners might expect about half of the interval should lie in the lower order of magnitude, and half in the higher. However, if prevalence is uniformly distributed between South Dakota and Vermont, the midpoint is as high as 3 in 10,000 (Mississippi). For midpoint around 7 in 100,000 (Connecticut), we instead want the log odds of the prevalence to be uniformly distributed. We represent this log odds transformation by saying that o (TT) Uniform(o
(a) a (b)).
Theorem 4.1 (Bounded DCA Log Score). Let s(x) E [o, 1] be a score function and e E (0, 1) be a cost parameter defining decision threshold and y & (2(1 - c) 1)=") (b) r (a)) be normalizing constant. Then, the expected net benefit over a logit-uniform prior on prevalence satisfies
E|PAMNB(DTr. T. r))) = <E[lg]]]
clip((1 c) @ sys(z)) - log/ - clip(1 v) [1-b,1-9)
[1-b119]
where the expectation on the left-hand side with respect to 0 (л) Uniform(o""(a) 0 (b)).
and on the right-hand side is with respect to (x, y) ~ D (1-o)-
Proof. See Theorem F.T
By clipping the score, this formula is able to produce closed form expression for the average net benefit over a range of class balances. We focus on net benefit, because it is scaled the same way at different class balances, so it makes sense to add and average this quantity. However, see Theorem E.2 for a related derivation that holds for weighted accuracy as well; this is effectively a rescaled version of net benefit for which the maximum possible score of a perfect classifier is always 1, regardless of the class balance.
4.3 Calibration, Label Shift and Asymmetric costs.
The interpretation of our newly introduced DCA log score as a mixture of cost- weightee accuracy is clear: the units are, all cases, units of true positives. The Schervish representation gives us simple way to describe what this is doing as a calibration metric as well: it is calibrating the model only over a particular bounded range of scores that is relevant to decisions and weighting scores in that range uniformly in log odds space. Moreover, approach is flexible enough that generalize
7

---

beyond accuracy (a well known result) to cost-sensitive metrics, we demonstrate with DCA Log Score and Weighted Accuracy Log Score (See Appendix for details). This, then, is measure of miscalibration that can be compared directly for its effects in the world.
Moreover, Gneiting and Raftery [0077 showed that the unconstrained log score be decomposed linearly into components of calibration and sharpness. As Vickers and Holland [2021] has argued, the need weigh failures of calibration against failures of sharpness is perpetual problem in the deployment of machine learning models in a medical context; one that measuring both AUC-ROC and ECE separately cannot resolve.
The Bounded DCA log score can easily be decomposed into a sum of miscalibration loss and sharpness, simply by recalibrating version of the model on the test set (denoted s*). The sharpness of the original and recalibrated models is the same, and is measured by:
PPPMN (D, 5, T, c)] = E|PAMNB(D) (D, s8" T, c)
The miscalibration loss is the difference between the original and recalibrated models given by:
E|PAMNB miscalibration (D, 8, T, c)] E,PAMNB(D,
c)] IE|PAMB((D, s", c)].
See Appendix L.3 for an example with real clinical data. Although the sharpness difference measured by clipped cross-entropy is weighted differently than that measured by AUC-ROC, the quantities are typically correlated.
5 Empirical Example: Subgroup Decomposition for eICU Mortality
We illustrate our framework by analyzing subgroup performance of the APACHE IV mortality prediction model in the publicly available eICU database [Goldberger al., 2000 Pollard al.. 2018 Johnsson et al. [021]. The dataset contains electronic health records from ICUS across the US, including vital signs, laboratory values, diagnoses, and pre-computed APACHE severity scores [Zimmerman et al. 2006). We use the model's existing in-hospital mortality predictions rather than retraining, enabling fully reproducible results with minimal computation. Further details on patient demographics are provided in Table 12 Our analysis focuses on two subgroup comparisons, by race and by sex, to demonstrate how the proposed metrics enable interpretable decomposition of model performance. Specifically, they (1) permit principled trade-offs between calibration and discrimination, (2) remain robust prevalence differences, and (3) support restriction clinically meaningful prevalence ranges.
AUC-ROC vs Bounded DCA Log Score Both conventional and proposed calibration measures indicate that APACHE IV is better calibrated for African-Amann American patients but more discriminative for Caucasian patients (Table [ Unlike AUC-ROC and ECE, whose combination ill-defined, the Bounded DCA Log Score integrates calibration and discrimination on common probabilistic
scale, yielding a single interpretable measure of overall decision quality. These results can be
Metric
Caucasian
African-American
1 Bounded DCA Log Score (calibration-only)
0.999
0.927
4 ECE
0.829
0.889
1 Bounded DCA Log Score (discrimination-onlly)
0.957
0.973
t AUC-ROC
0.868
0.907
t Bounded DCA Log Score
0.956
0.900
+ ECE & t AUC-ROC
?
Table 1: Calibration and Discrimination for Caucasian and African-American Patients
decomposed into calibration and sharpness components (see Appendix 1.3 for visualization). The decomposition confirms that most of the performance gap arises from calibration differences rather than discrimination.
Accuracy vs Bounded DCA Log Score We evaluate subgroup performance across prevalences by plotting the accuracy curve and summarizing it with the Bounded DCA Log Score (Figure I).
8

---

Accuracy at any given prevalence (solid lines) is consistently higher for Caucasian patients than patients, indicating better discrimination across the full prevalence range. The Bounded DCA Log Score (horizontal bars) averages these accuracy curves over prevalences, providing a single scalar that correctly preserves the same ranking. At the actual label prevalence (circles), however, accuracy is higher for African-Ameriea patients. That is, the apparent advantage reverses when comparing at a single empirical prevalence rather than across the full range. This difference highlights how the gap in accuracy at fixed prevalence reflects label distribution, whereas the gap in average accuracy reflects model mechanism. See Appendix I.2 for more details.
for
Metric
Caucasian
AaiiaanAAmerian
Accuracy Curve I %
0.990
0.978
Accuracy Curve@ 55%
0.953
0.896
Accuracy Curve 10%
0.912
0.793
Bounded DCA Log Score
0.956
0,900
830
Accuracy
0.922
0.961
5.0 AETo Araddan
Accuracy varying prevalences.
(a) Accuracy vs Prevalence curves and Bounded DCA Log Score (bars)
Figure I: Comparison of subgroup performance using accuracy and the Bounded DCA Log Score.
Cross Entropy vs Bounded DCA Log Score At low prevalences (those most clinically relevant) APACHE IV achieves higher accuracy for female than for male patients (Table 2h) At balanced or high prevalences, accuracy slightly favors males, but these regions contribute disproportionately to cross-entropy, which therefore ranks male performance higher overall. By emphasizing plausible prevalence ranges, the Bounded DCA Log Score (Table corrects this bias, ranking female patients higher. The corresponding securacy-prevaleee curves appear Appendix I.4
Metric
Male
Female
1 Accuracy@ @ %
0.989
0.990
Metric
Male
Female
1 Accuracy @ 5%
0.948
0.954
Brier
0.062
0.062
Accuracy @ 25%
0.818
0.814
1 Cross-Entropy / NLL
0.214
0.218
1 Accuracy 75%
0.846
0.832
1 Bounded DCA Log Score
0.917
0.926
Accuracy @95%
0.953
0.953
Accuracy @ 99%
0.990
0.990
(b) Scoring rules comparison.
(a) Accuracy across prevalences.
Figure 2: Comparison of APACHE IV performance for male and female patients using accuracy across prevalences (a) and proper scoring rules (b).
Summary These analyses show that ranking metrics such as AUC-ROC fail to capture overall usefulness on their own, and cannot meaningfully combined with separate calibration metrics. Accuracy is similarly limited by its mixture of prevalence differences and model mechanism differ- ences. Meanwhile, unbounded scoring rules such as cross-entropy and Brier score put too much weight clinically unrealistic prevalences, echoing carlier critiques Assel et al. [2017] In contrast, the Bounded DCA Log Score provides an interpretable, unified evaluation across subgroups and prevalence ranges. Confidence intervals reported in Appendix [L5] given that mortality prevalence and African African-Ameriaan American representation are both about 10% in the public subset, these results should be viewed illustrative rather than conclusive.
6 Discussion
The prevailing paradigm for evaluating medical MI decision-support systems often misaligns with evidence-based medicine and beneficence by overlooking real-world cost structures, disease preva- lences, and calibration nuances. While label shift correction and cost-sensitive learning are well

---

developed fields, evaluation methods to adapt to them are less widespread: for example, recent review of 173 papers using cost-sensitive learning in medicine found less than 10% of papers ac- tually used existing cost-sensitive metrics [Araf et al., 2024). Evaluating sharpness and calibration separately, with incommensurable metries, remains the standard practice. We wish to emphasize that, to our knowledge, this is the first framework to combine these three dimensions into unified, practically deployable metric, directly addressing the persistent challenge in clinical ML of actionable model selection under label shift and asymmetric costs.
Implications for Practice The key insight is that eliciting decision context precedes evaluation. Assessing model utility requires understanding which tradcoff's matter and which populations will be served. At the same time, practitioners need tools to guide this dialogue. Visualizing accuracy across prevalences (Figure lap provides a concrete basis for conversation: "Here is how the model performs at 2% versus 10 % prevalence and plausible prevalence bounds are elicited, our framework converts these into cost-weighted average performance score, identifying the model expected to be most useful across deployments. This bridges the gap between theoretical desiderata and the operational realities of medical practice.
what range matters for our setting?" Once relative costs
Limitations and Future Work While our framework offers unified and interpretable approach to model evaluation, several theoretical and practical challenges remain.
Cost Uncertainty. Our analysis assumes known cost ratios. Naively integrating DCA log score over cost ratios introduces dilogarithmic expressions that are analytically opaque and limit practical interpretability and scalability. Future work could explore tractable approximations surrogate alternatives, that preserve sensitivity to
objectives, such as piecewise-linens or cost uncertainty while enabling smoother optimization and interpretive clarity.
Sampling Variability under Label Shift. In settings with symmetric misclassification costs, bootstrap resampling binomial confidence intervals suffice for uncertainty estimation. However, under asymmetric costs, especially with population label shift, the evaluation metrics become sensitive to multinomial fluctuations in both the score distribution and the cost-weighted outcome prevalence. This introduces more variance and potentially estimation bias. Quantifying and stabilizing this variability, possibly through Bayesian poststratification or domain-adaptio resampling methods, remains a significant challenge.
Adaptive Base Rate Estimation. Our framework presumes class prevalence is uncertain during model evaluation but fully known once the model is used in deployment. In practice, these rates may be uncertain or drift over time due to changing patient populations, care protocols, or screening policies. Jointly estimating prevalence and calibrating predictions in such regimes introduces second-order uncertainty. Future work could explore integrated approaches that dynamically re-estimate base rates (e.g. via semi-supervised learning or risk-set modeling) while propagating this uncertainty into threshold selection and cost evaluation.
Asymmetric Cost Parameterization. We adopt a general framework for asymmetric cost modeling, but the semantics of varying cost ratios remain under- theorizee Distinct parameterizations, such as utility ratios, slope-based decision thresholds, or expected loss curves, embed different assumptions about clinical values and tradeoffs. A systematic comparative study of these parameterizations could yield more robust and mnatitioner-interpreaabl guidelines for aligning model evaluation with domain-specific norms.
Conclusion By combining decision-theoretie tools, causal framing, and clinically grounded metrics of calibration, this work moves toward evaluation methodologies that are both conceptually principled and actionable in real-world medical settings. Continued advances will require deeper integration of uncertainty quantification, model adaptivity, and domain-informed cost modeling.
7 Societal Impact
This paper concerned with evaluating binary classifiers with due consideration of the decision context. Any incremental improvement to the process of machine learning deployment can aid the malicious actor as well as the righteous. However, we believe that the harm caused by MI. models that are poorly matched to their deployment conditions is the primary area of concern here.
10

---

Acknowledgements
GF and AW are generously supported by the MIT Jameel Clinic in collaboration with Massachusetts General Brigham Hospital and Simons Foundation Collaboration #733792 Algorithmic Fairness.
A.H.S. acknowledges support from the National Science Foundation Graduate Research Fellowship Program under Grant #1938052. Any opinions, findings, and conclusions or recommendations expressed this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
References
N. Adams and D. Hand. Comparing classifiers when the misallocation costs uncertain. Par- tern Recognition, 32(7):113--117, 1999. ISSN 0031-3203. doi: hupp://doi.orr/10..n
$0031--9203098300554XXX URL http://wwwww.sceecccocccc..cccca p:1/$0001122399005447
A. Angstrom. On the effectivity of weather warnings. Nordisk Statistisk Tidskrift, 1:394-408, 1922.
I. Araf, A. Idri, and I. Chairi. Cost-sensitive learning for imbalanced medical data: review: Artificial Intelligence Review, 57(4):80, 2024. doi: 00007//100622-23-1066622 URL https //doi.org//il 10.
10652-8
D. Arnett, Blumenthal, M. Albert, B. Buroker, Z. Goldberger, E. Hahn, C. D. Himmelfarb, A. Khera, D. Lloyd-Jones, J. W. McEvoy, E. D. Michos, M. D. Miedema, D. Muñoz, C. Smith, S. S. Virani, K. A. Williams, Yeboah, and B. Ziaeian. 2019 acc/aha guideline on the primary prevention of cardiovascular disease: A report of the amer- ican college of cardiology/amerriann heart association task force on clinical practice guide- lines. Circulation, 140(11) (00(11):e596-e644 2019. doi: 10. 10.11611 161/CIIRRRR.. 000000000000000000000000000 URL https: pttpss:/vaii h/tps.//wwww... www.oralll.....ls.coomm
/161/CIR. COO....0.0..00000000FF50
M. Assel, D. D. Sjoberg, and A. J. Vickers. The brier score does not evaluate the clinical utility of diagnostic tests or prediction models. Diagnostic and Prognostic Research, 1(1):19, 2017. doi: 10. 1186141111111 152-1117-00-----
URL https:
M. Ayer, H. D. Brunk, G. M. Ewing, W. Reid, and E. Silverman. An empirical distribution function for sampling with incomplete information. The Annals of Mathematical Statistics, 26(4):641-647, 1955. ISSN 00034851. http:://////w....
org/ sttaaeeea/eeeeee sttble/22663777 2236377
K. Azizzadenesheli, A. Liu, F. Yang. and A. Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations, 2019. URL https
//openreview wwwtteeuut/neeeeee 21t.
D. Bamber. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Jourmal of Mathematical P Psychology, 12(4):387-415, 1975. ISSN 0022-2496. http::/wwww.scinceeeeeeee.... com/
doi: http:///ooooog////o/oo00002222/0 actancc/arrttoee//11///222277000
S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A the- ory of learning from different domains. Machine Learning, 79(1):151-175. 2010. doi: 109440009--11-12222 URL https: org 1007 000000///09900000900 009-5-2------------
H. Blockeel, K. Kersting, S. Nijssen, and F. Železný, editors. Area under the Precision-Recaa Curve: Point Estimates and Confidence Intervals, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-44999-33
A. Bradley, The use of the area under the roe curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1 145-1 159, 1997. ISSN 0031-3203.
G. W. Brier. Verification of forecasts expressed terms of probability. Monthly Weather Review, 78:
1-3, 1950. htt //api semanticscholar CorpusID:1229000

---

A. M. Carrington, D. G. Manuel, P. W. Fieguth, T. Ramsay, V. Osmani, B. Wernly, C. Bennett, S. Hawken, Magwood. Y. Sheikh, M. McInnes, and A. Holzinger. Deep roc analysis and auc as balanced average accuracy, for improved classifier selection, audit and explanation. IEEE Trans Puttern Anal Mach Intell, 45(1):329 -411, Jan 2023. ISSN 1939-3539 (Flectronic); 0098-5589 (Linking). doi: 10.1
S. Corbett- Davies, E. Pierson, A. Feller, S. Goel, and A. Huq. Algorithmic decision making and the cost of fairness. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD *17, pages 797-806, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874.
doi: 10. 10.1145533999983330999999
URL https: //w.i org/ 10. 1145/ 30////99999933 3098095
S. Cruz Rivera, X. Liu, A.- Chan, A. K. Denniston, M. J. Calvert, H. Ashrafian, A. L. Beam, G. S. Collins, Darzi, J. J. Deeks, M. K. ElZarrad, C. Espinoza, A. Esteva, L. Faes, L. Ferrante di Ruffano, J. Fletcher, R. Golub, H. Harvey, C. Haug. C. Holmes, A. Jonas, P. A. Keane, C. J. Kelly, A. Y. Lee, C. Lee, E. Manna, J. Matcham, M. McCradden, D. Moher, Monteiro, C. Mulrow, L. Bakden-RRanee D. Paltoo, M. B. Panico, G. Price, S. Rowley, R. Savage, R. Sarkar, S. J. Vollmer, and C. Yau. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the spirit-ai extension. The Lancet Digital Health, 2(10)) 2025/04/29 2020. doi: 00.0016/22589-550022000300 30219-3
URL http::///i..ogg//.... http:///....11111101000001001 (20)
J. Davis and M. Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pages 233-240, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1 1145/ 1143844. 143874. /tt.a///....ttg////////////////0 11t////...1//1148444444111111111 1143874
B. de Finetti. La prévision ses lois logiques, ses sources subjectives. Annales de l'institut Henri Poincaré, 7(1):1-68, 1937.
B. de Finetti. Foresight: Its Logical Laws, Its Subjective Sources, pages 134-174. Springer New York, New York, NY, 1992. ISBN 178-1--6122-0919-5 doi: URL /tdtps://doi.oooo
T. Dimitriadis, T. Gneiting, A. I. Jordan, and P. Vogel. Evaluating probabilistic classifiers: The triptych. International Journal of Forecasting, 40(3): 101-1122, 2024. ISSN 0169-2070. doi: https://
10t////0001////.foccaaaatttooto 022.022.000007 Sttp:tt:::1 /www sciencedirect. wwt.eococccceeeeeeeeeeee com/science)
atticllepp1//0066607703009999
Domingos. Metacost: a general method for making classifiers cost-sensitive. Proceedings of the Fifth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '99, pages 155-164, New York, NY, USA, 1999. Association for Computing Machinery. ISBN 1581131437. doi: 11.114//31212.31222 URL 312220
C. Drummond and R. C. Holte. Explicitly representing expected cost: an alternative to roc represen- tation. In Proceedings of the Sixth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '00, pages 198-207, New York, NY, USA, 2000, Asso- ciation for Computing Machinery. ISBN 1581 132336.. doi: 10. 10.1145/347090.344136 URL
https://doi. org/10 10. 1011/114749999 347126
C. Drummond and R. C. Holte. Cost curves: An improved method for visualizing classifier per- formance. Machine Learning, 65(1):95-130, 2006. doi: 10. 10.10077/110944-00--88888 URL
https://doi org/10. 10w110000/1190 006-8894-000999999 8199-5
C. Duchi and H. Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3): 1406, 2021. doi: 10.121420-A052004, URL
https://doi.. htap::://doioor/1000 org/10. 10. 1014/20 A0S2004
W. Ehm, Gneiting, Jordan, and F. Krüger. Of quantiles and expectiles: Consistent scoring functions, choquet representations and forecast rankings. Journal of the Royal Statistical Society
Series B: Statistical Methodology, 7863):005-56,, 05 2016. ISSN 1369-7412, doi: 10.11 1/rssb.
12154. ht https::////// //t.///oiii/i org/ http://0..1111rsse www.:/isiiss..e.e 12154
12

---

C. Elkan. The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence Volume IJCAT'OL, pages 973-978, San Francisco, CA. USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608122
T. Fawcett and A. Niculescu-Mizil. Pav and the roc convex hull. Machine Learning. 68 (1):97-106, 2007. doi: 810994-007-5011-0
URL https://dor www..oorg..... 1007/
L. Ferrer. Analysis and comparison of classification metrics, 09 2022.
L. Ferrer and D. Ramos. Evaluating posterior probabilities: Decision theory, proper scoring rules, and calibration. arXiv preprint arXiv:2448..2288 2024.
Flach, Hernández- and C. Ferri. coherent interpretation of auc as measure of aggregated classification performance. In Proceedings of the 28th International Conference on International Conference on Machine Learning. ICML' 11, pages 657-664, Madison, WI, USA, 2011. Omnipress. ISBN 978 77814503061999 95.
Futami and M. Fujisawa. Information- generalization analysis for expected calibration error, 2025. URL h 15709
Y. Ganin, Ustinova, H. Ajakan, P. Germain, H. Larochelle, Laviolette, M. March, and Lempit- sky. Domain-ad aamaain-ddvesariiin versarial training of neural networks. Journal of Machine Learning Research, 17 (59):1-35, 2016. /ttt:///mm wtggppapersvv1/115--11
html
S. Garg, Y. Wu, S. Balakrishnan, and Z. Lipton. A unified view of label shift estimation. In Larochelle, M. Ranzato, Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3290-3300. Curran Associates, Inc., 2020. URL https:: www.iiinnnnnne.ssseeeeeeeeee neurips cc/ wwwppppr.....././ec...eeeeee...e wwwllffleesseeeeee /appr//0220/fille pdf
Gneiting and E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, (02(47))3359-778 2007. doi:
URL http::ttt///ee /ttp://0...1///11111000000000000
A. Goldberger, L. Amaral, L. Glass, J. Hausdorff, P. C. Ivanov, R. Mark, J. E. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. Physiobank, physiotoolkit, and physionet: Components of new research resource for complex physiologic signals. Circulation, 101(23)ee15-<222, 2000. [Online].
J. Good. Rational decisions. Journal of the Royal Statistical Society, Series B (Methodological) 14 (1):00-1144, 1952. ISSN 00359246. Uttthtpp://www
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pages 1321-1330 JMLR.org, 2017.
D. Hand and C. Anagnostopoulos. better beta for the measure of classification perfor- mance. Pattern Recognition Letters, 40:41-46. 2014. ISSN 0167-8655. doi: https://di..or// 10.1016//.paaree..a 2013. URL/https: htttps::/www..cieee http///www.ciinneirreeee..tt com/sccience/artickee pi1/8016788551304496
D. Hand. Measuring classifier performance: a coherent alternative to the area under the roc
curve. Machine Learning, 770)):03-1233, 2009. doi: 10. 10.10077/1109944-09-5111911 URL https
//oii (ociioorg11...00000 1091///099000//119999999999 0/00
5119-5
D. J. Hand and C. Anagnostoppoulos. Notes on the h-measure of classifier performance. Advances in Data Analysis and Classification,, 1741)119--124 2023. doi: 10:1007771111334000-004933333 URL https: //doi orgo 10 101100//111144022100099999999900 00100000-33
Hanley and McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29-36. 1982. ISSN 0033-8419.
13

---

U. Hebert-Johnson, M. Kim, O. Reingoldd and G. Rothblum. Multicalibration: Calibration for the ittpraattiinnal--dennniiiill masses. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 1939 1948. PMLR, 10-15 Jul 2018. URL htttttts///prreeddnngg mlr wtw.gggs.......ss.s.... press/ hwweeberr-joohsoooe
html
J. Heckman. Shadow prices, market wages, and labor supply. Econometrica, 42(4) 1974. ISSN 00129682, 14680262. URL
orgg/stable/1913337
J. J. Heckman. Sample selection bias as specification error. Econometrica, 47(1):153-161, 1979. ISSN 00129682, 14680262. URL
org/staabee/1112322
J. Hernández-Orallo, P. Flach, and C. Ferri. Brier curves: a new cost-based visualisation of classifier performance. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML' 11, pages 585-592, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
Hemaandez-Orallo, P. Flach, and C. Ferri. Threshold choice methods: the missing link. 12 2011.
J. Hernández-Orallo, P. Flach, and C. Ferri. A unified view of performance metrics: translating threshold choice into expected classification loss. J. Mach. Learn. Res., 13(1):2813-2869, 10 2012.
Hernández-Orallo, P. Flach, and C. Ferri. Roe curves in cost space. Machine Learning. 93(1)711-11, 2013. doi: 10.10077s110994-133333333 URL http:/:///oooooo11....01.. 110994-013-5328-1
J. Huang and C. Ling. Using auc and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17:299-310, 2005. doi: 10. 1,..109KDD.20005.500
A. Johnson, Pollard, O. Badawi, and Raffa, eicu collaborative research database demo (version 2.0.1). PhysioNet, 2021. doi: 10. 10.13022/4mxkkna84, URL
13026/
4mxk na84
N. Kallus and A. Zhou. The fairness of risk scores beyond classification: bipartite ranking and the xAUC metric. Curran Associates Inc., Red Hook, NY, USA, 2019.
E. M. Keen. Measures and averaging methods used in performance testing of indexing systems. Technical report, The College of Aeronautics. Cranfield, England, 1966. URL wxg/ressources/musevt
Available in the SIGIR Museum resources collection.
E. M. Keen. Evaluation parameters. Scientific Report ISR-13, Department of Computer Science, Cornell University, Ithaca, New York, 1968. Information Storage and Retrieval: Scientific Report No. ISR-13 13 to the National Science Foundation.
J. Kleinberg, S. Mullainathan, and M. Raghavan. Inherent trade-offs in the fair determination of risk scores. C. H. Papadimitriou, editor, 8th Innovations in Theoretical Computer Science Conference (ITCS 2017), volume 67 of Leibniz International Proceedings in Informaties (LIPlcs), pages 43:1- Leibniz-
43:23. Schloss Dugstuhl-Leibmiz Leibmiz-Zeetrumm Zentrum fuer Informatik, 2017. ISBN 978-3-95977-009-3. doi: htt.http: 10.4230W.PKcs./ICS...077 http.//pppp///poo..ggggataaala.a httee//pppppooolleeee/////////// 8156
M. Kull, T. S. Filho, and P. Flach. Beta calibration: well-founded and easily implemented improvement on logistic calibration for binary classifiers. In A. Singh and Zhu, editors, Pro- ceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 623-631. PMLR, 20-22 Apr 2017. URL htttps://proceeddings mlr greess/v54/kul111 httm html
Z. Lipton, Y.-.. Wang, and A. Smola. Detecting and correcting for label shift with black box predictors. 02 2018. doi: 00448500/rxx..11802.0399
X. Liu, S. Cruz Rivera, D. Moher, M. J. Calvert, A. K. Denniston, A.-W. Chan, A. Darzi, C. Holmes, C. Yau, Ashrafian, J. Deeks, L. Ferrante di Ruffano, L.. Faes, P. A. Keane, S. J. Vollmer, A. Lee, A. Jonas, A. Esteva, A. L. Beam, M. B. Panico, C. S. Lee, C. Haug, C. J. Kelly, C. Mulrow,
14

---

C. Espinoza, J. Fletcher, D. Paltoo, E. Manna, G. Price, G. S. Collins, H. Harvey, J. Matcham, Monteiro, M. K. ElZarrad. L. Oakden-Rayner, M. McCradden, P. A. Keane, R. Savage. R. Golub, R. Sarkar, S. Rowley, T. SPIRIT-AI, C.-A. W. Group, SPIRIT-AT, C.-A. S. Group, SPIRIT-AI, and C.-A. C. Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the consort-ai extension. Nature Medicine, 1617)1136413374 2020. doi: 10.1038/s41591 10008//455110000-1100 10330000--3
URL https::/ddoiii... org/ 10. 100.11000//111110000000 1034-x
M. Long, Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 97-105, Lille, France, 07-09 Jul 2015. PMLR. URL htt.ttps:: httt:::://prcceedingge .mmiiiinnnmmm........mmmm...mm.. wtt.eedddigg.....pppsss///////oo http///vv//ooo51515/ html
//
J. McCarthy, Measures of the value of information. Proceedings of the Nationall Academy of Sciences, 42(9))633-653 1956. doi: 10. 11100//ppa..42..... URL http::ttttss/w.wtteee hwtpp//:://wwwww......mmmm www.:/ppp...o...ooooooo/oooo org/doi/ abs/
10. 1..1077/ppass 42. 6564
D. K. McClish. Analyzing portion of the roc curve. Med Decis Making, 9()): 1989. ISSN 0272-989X (Print); 0272-989X (Linking). doi: 10.1177027789XX89900033
D. K. McClish. Evaluation of the accuracy of medical tests region around the optimal point. Academic Radiology, 19112) 2025/05/05 2012. doi: acr.22...cra.22200...0 URL
https://doi org/ 10. 1016/j acra. acr.. 200...9..00.00000
M. B. McDermott, H. Zhang, L. H. Hansen, G. Angelotti, and J. Gallifant. closer look AUROC and AUPRC under class imbalance. In The Thirty- Anmual Conference on Neural Information Processing Systems, 2024. URL https: http:://ppeeeevveeee .ntt/forrumm31d4$33t/448888e
E. Meehl and A. Rosen. Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52(3):194-216, 1955. doi: 10. 10370037/000
C. E. Metz. Basic principles of roc analysis. Semin Nucl Med, 8(4):283-98, Oct 1978. doi: 00.10066s0000-9988888000000
C. E. Metz. Roc methodology in radiologic imaging. Invest Radiol, 21(9)) Sep 1986. ISSN 0020-9996 (Print); 0020-9996 (Linking). doi: 10.1097/00004424-- 1a000004404999900000000000000000 198609000-00009
C. B. Metz. Some practical issues of experimental design and data analysis in radiological roc studies. Invest Radiol, 24(3):234-245 Mar 1989, ISSN 0020-9996 (Print); 0020-9996 (Linking). doi: 10.1097/00000444- 198903000-00012.
J. G. Moreno- Torres, T. Raeder, R. Alaiz-Rodríguez. N. V. Chawla, and F. Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45(1):521-530, 2012. ISSN 0031-3203. doi: scieetee//rricee//////0013111111
patcog
URL
wwwp::wwwwwwwwwwewwettc..ceoeeee com/
K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representa- tion. In Proceedings of the 30th International Conference on International Conference on Machine Learning Volume 28, ICML' 13, pages 1-10--18. JMLR.org, 2013.
A. H. Murphy. A note on the utility of probabilistic predictions and the probability score in the cost-loss ratio decision situation. Journal of Applied Meteorology and Climatol- ogy, 5(4):534 537, 1966. doi: 10. 755777/5522000000006660005555500
0450_1950001 2. URL htttts:///oournallll awetsac httpa////aaaaataaer//e/////////a
005 110544_anotuo_2___co
xm1
A. H. Murphy. The value of climatological, categorical and probabilistic forecasts in the cost-loss ratio situation. Monthly Weather Review, 105(7):803- 816, 1977. doi: 10. (0.11777/5220040441177
URL htttstts///oouraallss ametsoc org/vile:/oournass 11tp1110__000_toccaa220..00000.
xml
A. H. Murphy and R. L. Winkler. A general framework for forecast verification. Monthly Weather Review, 1 115(7):1330 1338, 1987. doi: 10117770155000000097771555553454
CO:2. Uttp http:///rrr...aaaatttttterrrr//a htt.p////..poto.....ccc////////a 1987_115_1330_agff
15

---

J. Nixon, M. Dusenberry, G. Jerfel. T. Nguyen. J. Liu, L. Zhang, and D. Tran. Measuring calibration in deep learning, 2020. URL htttttt////xxiis org/ arg/abl abs/ wtw.//999..0... 01685
M. Pakdaman Nacini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence. 29(1), Feb. 2015. doi: wttp9//aii.2991..990. URL https://ojs. aaai. org/ index, php/ AAAI article/view/9602
G. Pauker and Kassirer. Therapeutic decision making: A cost-benefit analysis. New England Journal of Medicine, 293(5))229-234, 1975. doi: 10. 101110//EENM11977733299000000 URL wttp:///w.............../////1/.
https: //www
J. Pearl. Causality: models, reasoning, and inference. Cambridge University Press, Cambridge [u.a.], repr. with corrections edition, 2001. ISBN 0521773628.
Peterson, W. Wesley and T. G. Birdsall. The theory of signal detectability. Michigan. Univer- sity. Department of Electrical Engineering. Electronic Defense Group. Technical report; no. 13. Engineering Research Institute, Ann Arbor, 1953.
J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. 1999. URL https://ppp http://aap.emmmtiiies....ms. semanticscholas org/CorpusID:eo....ooc : 56563878
T. Pollard, A. Johnson, J. Raffa, L. A. Celi, O. Badawi, and R. Mark. eicu collaborative research database (version 2.0). PhysioNer, 2019. doi: 10. 10.13026/2WMMIR URL http:::t///// 13026/ 13026/C2WM13
T. J. Pollard, A. E. W. Johnson, J. D. Raffa, L. A. Celi, R. G. Mark, and O. Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific Data, 2018. doi: 10. 20.10088dtta..218..11 URL http: wtt.http:////// //dx://dx.o doi. org/ /011000.0 1038/
sdata. 2018. 178
J. Provost and T. Fawcett. Analysis and visualization of classifier performance: imprecise class and cost distributions. Knowledge Discovery and Data Mining, 1997. URL https://api. htw.aappsssmmaanicccccccoooeoooe
Comparison under
org/ wwtg/oorgu.seeeeeoo :::://ou:::
F. P. Ramsey. Truth and probability. In R. B. Braithwaite, editor, The Foundations of Mathematics and other Logical Essays, chapter pages 156- 198. McMaster University Archive for the History of Economic Thought, 1926. URL httts::/ repec. org Stre/RePEc::10 hay:: hetcha! ramsey1926
R. Roelofs, N. Cain, J. Shlens. and M. C. Mozer. Mitigating bias in calibration error estimation. In G. Camps-Var Valls, F. J. R. Ruiz, and I. Valera, editors, Proceedings of The 25th International Conference Artificial Intelligence and Staristics, volume 151 of Proceedings of Machine Learning Research, pages 4036-4054, PMLR, 28-30 Mar 2022. URL BRLLhhtp::///roo httpps://proceediings
mlr. wttppprss/v//eelloolsppooo
html
M. Saerens, P. Latinne, and C. Decaestecker. Adjusting the outputs of a classifier to new priori probabilities: simple procedure. Neural Comput, 14(1):21-41, Jan 2002. ISSN 0899-7667 (Print); 7899-7667 (Linking). doi: 10. 10.1162209976660752244444
Sagawa*, P. W. Koh*, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks, In International Conference on Learning Representations, 2020. URL httpps://openreeiiet net/ forum?id-ryxouJrFvs
L. Savage. Elicitation of personal probabilities and expectations. Journal the American Statistical Association, (6(366)))66)) 66(336):783-80, 1971. ISSN 01621459, 1537274X. URL http::/wwwrt jstor org/stable//284222
M. J. Schervish. A general method for comparing probability assessors. The Annals of Statistics. 1854)1856-1189, 1989. ISSN 00905364, 21 21688966. URL http://wt jstor. wwtssttorrrstteeeeeeeeeeee stable/ 2241668
16

---

Y. Shen. Loss functions for binary classification and class probability estimation. PhD thesis, 2005. URL
theses/ Copy-
fues-functions
right Database copyright ProQuest LLCC; ProQuest does not claim copyright in the individual
underlying works; Last updated 2023-03-03.
E. H. Shuford, A. Albert, and H. Edward Massengill, Admissible probability measurement preasurementprocedueess Psychometrika, 125-141,, 1966. doi: 10. N0.1007/BFO2288503.. httpsss:t////o...o 1007/BF02289503
K. A. Spackman. Signal detection theory: valuable tools for evaluating inductive learning. In Proceedings of the Sixth International Workshop on Machine Learning, pages 160-163, San Francisco, CA, USA, 1989. Morgan Kaufmann Publishers Inc. ISBN 1558600361.
W. Stevens. Thirteen ways of looking at blackbird. In Harmonium. Alfred A. Knopf, New York, 1923. Originally published in Others: An Authology of the New Verse, 1917.
E. W. Steyerberg and A. J. Vickers. Decision curve analysis: discussion. Med Decis Making, 28(1): 146-149, 2008. ISSN 0272-989X (Print); 0272-989X (Linking). doi: 10.117702729899071177755
A. Subbaswamy, Schulam, and S. Saria. Preventing failures due to dataset shift: Learning predictive models that transport. 2018. URL http:://axix: wwtp/bo/////////eeeeeee htsp/bbb/11..... htpp//812.0499 cite 1tt::1102..597 In Proceedings of the 22nd International Conference on Artificial
Intelligence and Statistics (AISTATS), 2019, Previously presented at the NeurlPS 2018 Causal Learning Workshop.
M. Sugiyama, N. Rubens, and K.-R. Müller. conditional expectation approach to model selection and active learning under covariate shift. In Dataset Shift in Machine Learning. The MIT Press, 12 2008. ISBN 9780262255103, doi: 10.7551/mitpress/9999 0tttppress/721..03.0000 .003.0012. URL https:///oo:ii org/ 10. 10.77651/mitprres//799...
J. Swets and T. Birdsall. The human of information-inn Decision-making in signal detection and recognition situations involving multiple alternatives. IRE Transactions on Information Theory, (3):138-165, 1956. doi: 10.1 10.1109/TTT.1T 1999/TT..1956.055555 1056799.
W. P. Tanner, J. A. Swets, and H. W. Welch. new theory of visual detection. Technical Report UMR3825, University of Michigan, 1953. URL https:://dht handle. net/2027.42/7889 Engineering Technical Report.
J. M. C. Thompson and G. W. Brier. The economic utility of weather forecasts. Monthly Weather Re- view, 83:249-253. 253, 1955. (tRLhhtts://ppp
commmmticsccoolaa. www.choaa...g/ooopraaaaaaaeeaeae ortp/Oorpus1:::122117333 122117332
D. G. Turakhia. Thirteen ways of looking: a theoretical inquiry in computational creative thinking. Master's thesis, Massachusetts Institute of Technology, Cambridge, MA, 2017. URL http:
//hdl, handle..... net.neee11111 1/113918 S.M. Thesis, Department of Architecture and Department of Electrical Engineering and Computer Science.
U.S. Centers for Disease Control and Prevention. National overview of STIs in 2023. CDC, November htt.a//sssatttsssssscaaaaaaaaaaa ctttmmarrytttllll
12 2024. URL htttps://www..r cdc. saaddggoo//tiissttsstisss http:/pp////o.c.....//////s//sss
Vaicenavicius, Widmann, C. Andersson, Lindsten, Roll, and Schön. Evaluating model calibration in classification. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Twenty- International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 4467. PMLR, 16-18 Apr 2019. URL ltttps://prooceedings mlr. press/v88) http://8//vvainncce hatae//////aiaiiiaaacaiiineeeee html
A. Vickers and F. B. Elkin. Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making, 26(6)566--77, 2006. doi: 10. 0011177/0279899X6665336 URL
https::/doi. http::///ooiooog1/1/1 10. wttp:::.r..........11.11111111/0
PMID: 17099194.
A. Vickers and F. Holland. Decision curve analysis to evaluate the clinical benefit of prediction models. The Spine Journal, 21 2110):1143311648 2021. ISSN 1529-9430. doi: https://doi.org/
2.10166).pinnee.220100..... URL http:////wwwsccieeeerrree..oooo pi1/S1559943021001122
17

---

A. J. Vickers and S. Woo. Decision curve analysis in the evaluation of radiology research. European Radiology, 32(9):5777-5789, 2022. doi: 00.10007/00033--02--866666 URL https://doi org/10. 00//10..007/80030000222222200000 00000//800300002-0888880
A. J. Vickers, B. van Calster, and E. W. Steyerberg. A simple, step-by-step guide to interpreting
decision eurve analysis. Diagnostic and Prognostic Research, 3(1):18, 2019, doi: 10.1186/ 541512-019-0064-7. URL http::ttttppoooooo 1ttp://..0000111111//////4111111
D. Widmann, F. Lindsten, and D. Zachariah. Calibration tests in multi-class classification: A uni- fying framework. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-But E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL /tppatttps//pproeeeeinggg meuriin cc/ coc/paper_file patger_fflee/pppppe 2pppr/201/// Fil1://3688000222222226666666663
pdf
X.-H. Zhou, N. Obuchowski, and D. McClish. Statistical Methods in Diagnostic Medicine, Second Edition. 01 2002. ISBN 97804701 83144. doi: 10.1002/97804709905514
K. Zhu, Y. Zheng, and K. C. G. Chan. Weighted brier score an overall summary measure for risk prediction models with clinical utility consideration, 2024. BRLL::tt:///arr wttp::/ww.iv.///////o./o /bb// 2408.01626
E. Zimmerman, A. A. Kramer, S. McNair, and F. M. Malila, Acute physiology and chronic health evaluation (apache) iv: hospital mortality assessment for today's critically ill patients. Crit Care Med, 129--1127 May 2006. ISSN 0090-3493 (Print); 0090-3493 (Linking). doi:
A Related Work
Recent literature emphasizes that scoring rules in Al should meaningfully reflect the objectives of deployment contexts, rather than relying standard metrics that can lead to suboptimal misleading outcomes Cruz Rivera et al. 2020), Liu et al. [2020).
Decision Theory. Decision theory has roots in gambling and actuarial sciences but was formally structured by foundational works such as Ramsey (9226] and Finetti 1937 1992]. Within medical
decision- maaiii,, a prominent recent strand has been Decision Curve Analysis (DCA), developed by Vickers and Elkin| (2006). |Vickers al. [2019), a decision-theonee
framework. However. DCA has avoided measuring the area under the decision curve (Steyerberg and Vickers 2008), preferring instead to eschew mathematical evaluation when neither classifier dominates. This body of work critically examines widely-used metrics such as the Area Under the Curve (AUC) (Vickers and Holland, 202T Vickers and Woo 2022) and the Brier score [Assel et al. 2017]. questioning their clinical utility and advocating for metrics directly connected to decision- analytic value.
Proper Scoring Rules. The literature on proper scoring rules began with Brier [1950], quickly enriched by contributions from Good [952] and McCarthy [1956]. A critical ad advancement was the integral representation of proper scoring rules by Shulord al. [1666), explicitly connecting scoring rules with
utility via Savage 1971]. This was followed by the comprehensive characterization provided by |Schervish 1989 who demonstrated that strietly proper scoring rules can be represented as mixtures of cost-weighted errors. The formalism was further elucidated by Shen 2005 through the lens of Bregman divergences and by Gneiting and Raftery [2007) convex combinations of cost-sensitive metrics. Independently. Hand [2009) rediscovered proper scoring rules, reframed as H-measures, in the context of mixtures over cost- weightedd errors. He used this framing to show that the AUC-ROC of a calibrated classifier corresponds to a mixture of cost-weighted errors under particular (and undesirable) distribution over cost ratios.
The idea of generalizing from cost to a cost proportion that depends also on class balance has been repeatedly independently proposed in the setting with known analytic distributions of scores Drummond and Holte [2006). Hernández- Orallo et al. [2012]. Hand and Anagnostopoulos [2023] introduced the idea of double integral over cost and balance, but their work does not explore the semantics of the resulting joint distribution nor provide guidance on how the double integral should be computed.
These foundations have also motivated context-sensitive adaptations. Recent attempts have focused on obtaining a central estimate and fitting Beta distribution around it Hand 2209)] Hand and
18

---

Anagnostopoulos [014], Zhu et al. [2024]. Unfortunately, the dispersion of Beta distribution remains unintuitive to most medical (and perhaps even machine learning) practitioners. Indeed. Zhu et al. [2024] does not provide procedure to set the pseudocount + 3. and |Hand and
Anagnostopoulos [2014] do not quantify uncertainty, but suggest always using as "a sensible default
value".
Calibration Techniques. The Pool Adjacent Violators Algorithm (PAVA), introduced by Ayer et al. [1955]. remains a foundational calibration technique, equivalent to computing the convex hull of the curve [Fawcett and Niculescu-Mizil 2007]. A distinct parametric calibration approach based on logistic regression was popularized by Platt [1999], subsequently refined for slope-only calibration
by Guo et al. [017]. . An intereept-only version aligns closely with simple score adjustments (Saerens
et al. 2002]. while broader generalizations are explored in Kull et al. (2017). More recently, the
calibration literature shifted towards semi-supervised contexts, utilizing unlabeled data to enhance
calibration quality [Lipton et al. 2018 Azizzadenesheli et al.) 2019 Garg et al. 2020].
Despite extensive critiques, for example highlighting that the widely-adopted Expected Calibration Error (ECE) [Pakdaman Naeini et al. 20155 is not a proper scoring rule [Vaicenavicius et al. 2019
Widmann et al. [2019], and statistical issues with binned calibration metries (of which ECE is an
example) Nixon et al. 2020). Futami and Fujisawa 2025 Roelot's et al. [0022]
Ferrer 2022 this
metric remains popular in practice.
Parallel to predictive accuracy, calibration has also emerged as a fairness metric. This perspective, however, has become contentious since calibration was shown to be fundamentally incompatible with other fairness criteria |Kleinberg et al.) 2017]. spurring the development of "multicalibration" approaches that ensure calibration across numerous demographic subgroups (Hebert-Johnson et al., 2018).
Label Shift. Label shift techniques are a particularly useful hyponym of calibration techniques.
While the concept of shifting class prevalences without altering the underlying conditional distribution
Meehl of features is longstanding Meehl and Rosen [1955). Heckman 1974 1979], Saerens et al. [2002).
formal treatments and systematic causal characterizations arose from Moreno- Torres et al. 2012
Earlier explorations of covariate shift (Sugiyama et al., 20088 motivated broader field of research
aimed at developing invariant representations robust to distribution shifts. These efforts encompass methods based on richer causal assumptions (Subbaswamy et al., 2018], invariant representation learning [Ben- et al. 2010. Muandet et al. [2013] and distributionally robust optimization
[ongg al. 2015
Ganin et al. [016 (Sagwaa* et al. 2020 Duchi and Namkoong [021].
AUC-ROC & AUC-PR. The Receiver Operating Characteristic (ROC) curve emerged within signal
detection theory (Peterson and Birdsall 1953 Tanner et al. [953]. later becoming central in radiology and clinical diagnostics where the convention solidified around measuring performance via the Area Under the Curve (AUC) [Metz, 1978 Hanley and McNeil 1982]. Use of AUC to aggregate over multiple thresholds was explored by Spackman [1999, Bradley [1997]. and Huang and Ling (2055) with subsequent critiques noting widespread interpretability issues [Carrington et al. 2023]] Hand [2009] showed how the AUC of calibrated classifiers relates to average accuracy across thresholds, while Feernández-Oralle et al. 2012 described alternative interpretations via uniform distributions of predicted score, or uniform distributions of desired positive fractions (see Appendix G for more details).
Recently, there has been increased scrutiny of AUC-ROC, particularly regarding its lack of calibration and poor decomposability across subgroups (Kallus and Zhou 2019). Precision and Recall metrics originated in information retrieval, with Mean Average Precision (MAP) or the Area Under the Precision- Recall Curve (AUC-PR) formalized by Keen [1966 1968.. While more recent trends in information retrieval have favored metrics such as Precision @ K. Recall@K, and Discounted Cumulative Gain (DCG), Davis and Goadrich (2006) popularized AUC-PR for classifier evaluation, particularly in contexts with imbalanced data. Despite well-documented critiques crriques-including that
AUC-PR poorly estimates MAP [Blockeel et al.) 2013] and lacks clear theoretical justification McDermott et al. 2024], its use persists, particularly in medical and biomedical contexts.
Cost-Sensitive Learning. Cost-sensitive evaluation, historically formalized through Cost/Loss frameworks [Angstrom| 1922 Murphy 1966), was independently introduced in clinical decision- making as early as Pauker and Kassirer 1975]. The modern foundation of cost-sensitive learning emerged prominently the machine learning literature in the 1980s, notably via the seminal work on MetaCost by Domingos
1999 and the canonical overview by Elkan 2001]. Extending these
19

---

frameworks to multi-class settings poses challenges due to the quadratic complexity of pairwise misclassification costs.
value as function of decision thresholds date back to Thompson and Brier [955]]. with subsequent development by Murphy 1977]. Murphy and Winkler [197], who linked visualizations explicitly to scoring rule theory. Later rediscoveries within machine learning were articulated by (Adams and Hand [1999], and independently by Drummond and Holte [2000] [006]. More recently. these visualizations were generalized to include uncalibrated models [Hemándezz-Oralo: et al. [201 and formally named Murphy Diagrams by Ehm et al. al.[ (2066), with further implementation guidance provided by Dimitriadis et al. 2024].
Visualization. Visualization techniques to illustrate economic or
B Calibration
The weather forecasting literature focuses on what are known as strictly proper scoring rules: those metries that have the property that a forecaster is correctly incentivized to report their actual beliefs about the probability of the event. At first glance this seems bit distant from binary classifier evaluation. After all, action is generally binary; we really want the weather report to tell us whether to take an umbrella, not to give decimal places of precision on the long run frequency with which would rain.
B.1 Asymmetric Cost
However, a calibrated, thresholded binary classifier has an immensely useful property: we know how to change the threshold to trade off false positives for false negatives if the class balance or the cost ratio changes.
(y,, 77)
y=1 (Syphilis)
y=0 (No Syphiliss
Let define the value function of particular true and predicted label pair as Vy, 8)
y=1 (Treat)
True Positive
False Positive
G=0 (Don'ttreat)
False Negative
True Negative
Table 2: Value function
The optimality condition for choosing a threshold requires that the first derivative of the expected value zero. This is equivalent to saying that the expected utility of assigning points exactly at the threshold to either class should be the same:
E =.gr
Vly, 0) -.y:((=)=
E
V(y, 1)
V(O, 1) V(O,0) 0)
Ply =|s(z)= =
(V(0, V(O, 0)) + (V(1, 1) V(1, 0))
As a result, we generally call the quantity on the right c and use it to describe the asymmetry of the error costs.
Solving this for 7 requires us to at least implicitly estimate 5(T) Ply = 1|s(x) = r). If we add the constraint of monotonicity to ä(T), then this problem is known as isotonic regression, and there are well-known algorithms for solving it. Assuming the existence of a good estimator 5(T) for this quantity, then ((x()) is of course calibrated estimator for P(y = 1|z). Using the same classifier at varying cost asymmetries requires that the classifer be, at minimum, implicitly calibrated; isotonic regression is in fact how such classifiers are calibrated.
is, of course, possible to develop an estimator calibrated only at s(x) c; for any point higher or lower, ordinal comparison alone is enough to make decision. A classifier optimized in this fashion may be wildly unreliable at other values of sax:) and our calibration may simply give us two scores: higher than and lower than c. If so, the condition of calibration is almost trivially satisfied: it only requires a binary predicted label and statistics for how often the classifier is correct in either case (PPV and NPV).
20

---

B.2 Label Shift
In the weather forecasting literature, it is explicitly understood that Dx - Y, which is to say that rather than today's s atmospheric conditions being emanations of tomorrow's 's decision of whether rain or not, the evolution on physical principles of today's conditions leads to tomorrow's weather. As such, the idea of label shift is incoherent. The study of changes in classifier performance when P(y) changes in this setting known as covariate shift; this out of scope for this paper.
The machine learning evaluation literature does acknowledge links between label shift and calibration. However, the setting is more abstract, with CDFs taken given, and varying thresholds interpreted as response to varying class balances without clear enumeration of assumptions.
B.2.1 Notation
Lemma B.1 (Importance Sampling & distance). Consider the standard importance sampling weights to move from the training (To) to the deployment (T) distribution. Label shift always holds when reweighting data by class because after we stratify by class, we do not change the distribution
within the class.
P (I, y =1/
(TO - T;yy)
Poo (I. y = y²)
Pr (aly 3/)
Pr 3)) PrO (U = 89
Pro (rly = 3¹)
by label shift
1/2 P (y = y²) 1/2
P2 8¹)
- + 1/2.8/) 21 y/)
(2(T 0) if'' 1
W(no 1/2;8')
22(1 7) if 0
W(To 1/2;8') 2 [11 T) - vl
Definition B.2 (Odds Multiplication).
ab
4
ab + (1 - a)(1 - b)
Proposition B.3 (Inverse).
2 b
b (1 a) &
Proposition B.4 (Jacobian).
da
d(a 3 b)
2 b
@ b (a b) 8 b)
a a)
Proposition B.5 (One minus distributes over odds multiplication).
1 (a 8 = =1
Proof.
ab
() b)
+
ab + (1
0) (1
b)
ab + (1 - () b)
@ a) b) =1
(1 (1 b) = - 0
21

---

Proposition B.6 (Logit Odds Multiplication is Additive).
= (a) + (b)
Proof.
ab+ a--aaa( a) (1-b)
log
log
+ log
(1 a] -b)
1 a
1
ab+ (1- a1
Proposition B.7 (Log Odds Interval Invariance).
Proof.
(1 c @ b) - o c a)
o (1 c) + ob] lo (1 c) t [a)
l (1 c) - aaao (1 c) + [o (b) - -a [a)]
(b) [a)]
B.3 Prior-Adjustment
With this notation, working with conditional probabilities is straightforward:
Ply 11s(x)) Dx)
collect this
P(s(x)l = D. Ply = 1)Dm) P(s(r)ly = 0, Dx) P(y O
P(s(x)ly = 1r P 1D
and
and this
P(s(z)ly 1, P(s(x)ly = 1, DT) + P(s(x)ly = 0, DT)
&P(y = 1(D
label shift
P(s(z)lv = 1)
PP 1(D,))
P(s(z))y = 1) + P(s(x)ly 0)
invariant to Drr
P(s(x))y 1)
P(y = 11s(x), Dx) @ P(y = D/D
P(s(x)ly 1) + P(s(I)
invariant to Dr
= P(y = 1|s(x), D mo
P(y = D)
P(y 1[s(s), Dx) P(y 1|s(x), D Doo))
P(y D P(y 1
P( 1|s(x), mo))
@ (1 TO) & n
Here, the propagation of errors is straightforward: the log odds error will be the same size under both distributions, although of course the errors in probability space may larger or smaller. Thus we can specify the best choice of prior-adjusted score:
B.4 Prior-Adjusted, Cost- weighte Threshold
Combining these two, find that the best choice of decision threshold is
(1 no) X s(x) >
We will thus often refer to the induced optimal classifier x(π & s1/2(x), c).
22

---

C Derivation of Set-based Metrics
We start with the most popular family of evaluation methods, which are based on accuracy but include cost-sensitive generalizations. These only require a binary classifier, which we define as a function k(x)) X 1).
They differ along two axes: the way they factor in the cost of errors, and the way they factor in the class balance of the dataset.
Empirical
Balanced
Prior-Adjusted Maximum
Accuracy
Accuracy
BA
PAMA
Weighted Accuracy
WA
BWA
PAMWA
Net Benefit
NB
BNB
PAMNB
Table 3: Taxonomy of set-based evaluation metrics. Each row represents a different approach to handling error costs, and each column represents different approach to handling class balance. Note that when balanced, the second and third rows are equivalent.
C.1 Accuracy
ViJa 9
y=1 (Syphilis)
y=0 (No Syphilis)
Definition C.I (Accuracy). The accuracy of a thresholded binary classifier s(x, )) is given by:
8=1 (Treat)
Accuracy (Dro T, = Σ a.yED
Vijalu, ((((),
f=0 (Do'' I treat)
1
Table Value function for Accuracy
This is impractically neutral with regard to cost in that Vy 1
is not function of y. which corresponds to the contingency table in Table 4 is practical but neither neutral nor flexible with regard to distribution shift in the sense that it implicitly assumes: H(Dx) 5(D,r D Droo).
The simplest way make this more neutral to evaluate on balanced dataset, which we denote as D./2- Mechanically, we draw from this dataset using importance sampling, if we assume Drr + X and therefore P(X Dx)) = P(X[Y).
Definition C.2 (Balanced Accuracy). The bal- anced accuracy of thresholded binary classifier <(0,7 T) is given by:
BA(D no: 8, T)
V y
y=1 (Syhillis)
y=O Syphilis)
Σ
((sx),
Ā=1 (Treat)
2no
5=0 (Don' oton't treat)
Σ
W(no (e.))i/s(u.nss())
2(1 ((-
a.yED m
Table 5: Value function for Balanced Accuracy
Σ v(())))())
=
E.yEDro
If more flexibility is desired, at the expense of neutrality, it is necessary to evaluate at an arbitrary class balance. Moreover, evaluating how well classifier performs at one specific threshold less useful than understanding how the best threshold performs at specific class balance.
23

---

Definition C.3 (Prior- Adjjut-d Maximum Ac- curacy). The prior-adjusted maximum accuracy given scoring function and a threshold with a class balance is given by:
V 17
y=1 (Sypiilis)
==0 Syphilis)
PAMA(D xx;8, 8,7)
3=1 (Treat
Σ B,VED,
Vi ((s(x),)))
50
(Don't treat)
1 -Tp
Σ B.yEDro
W(x V// K(T & 71)(1)),7)
Table 6: Value function for Shifted Accuracy
Σ m.yEDro m
V(y,n( @ (1/((x))))
C.2 Weighted Accuracy
This problem is further complicated by the need to realistically confront asymmetric costs. Consider the syphilis testing case: unnecessary treatment is 10 to 100 times less costly than missed detection. We will 1/30 representative value for exposition, the exact mechanics of syphilis testing are not central to this work.
First, we consider the balanced case, which is more mathematically tractable.
Definition C.4 (Balanced Weighted Accuracy). The balanced weighted accuracy of a score func- tion with threshold is given by:
V 7
y=1 (Syphilis)
==0 (No Syphilis)
BWA(D nay S,
9=1 (Treat)
1-c To
0
Σ Dyy
(1 - (1-c)ac VVpp(,
treat)
Σ
W(T + -1-. x(s()), Table Value function for Balanced Weighted
r.yEDro
Accuracy
Σ F.yED Tg
v(()x(s(z).c))
The minimum possible value of this expression is clearly 0 V 9) = 0 for all y. The maximum also clear:
Σ
W(To 1/22))((122 ag)lo c)- +c=1
a.ye D
The obvious combination of the two weighting terms not correct, however.
W(no + W12/2 + -- w/ - T: a(1 - e) + T)e #
R.yEDmo
The most intuitive approach involves rescaling the value of the true and false positives to be in the 1:30 ratio and then normalizing such that the maximum possible value remains 1 regardless of class balance. This is known as Weighted Aceuracy. This procedure is the worst possible value and the best is generally known in the forecast evaluation literature as a skill score, but in the medical decisionmaking literature this particular metrie generally called Weighted Accuracy.
normalizing the metric so that
24

---

Definition C.5 (Weighted Accuracy). The weighted accuracy of thresholded binary clas- sifier vix,, 7) is given by:
Viy
(Syphilis)
(No Syphiliss)
WA(D no: T, En.yEDm. (1 - c))0 -
g=1 (Treat)
[1 (--))o-c1--
VVaa (v, ((s()),TT)
y=U (Don't treat)
ExyEDro (1 -cjcct VV/,(V..3) y)
Table 8: Value function for Weighted Accuracy
V(y,n(s(z),1)) T))
B.yEDro
C.3 Net Benefit
However, this makes comparisons across different class balances less meaningful, since as the class balance varies, the normalizing factor changes. As result, the effective value of true positive changes. One common approach from the Decision Curve Analysis literature instead normalize the true positive to and then rescale the false positive to keep the right ratio. The baseline in the DCA literature always predict the negative class, whereas the weighted accuracy literature uses baseline of always predicting the wrong class. Since this is equivalent up to constants, we will modify the parameterization of Net Benefit to make more directly comparable.
Definition C.6 (Net Benefit). The net benefit of scoring function with threshold r is given by:
V(y, 77
y= 0 (No Syphilis)
(Syphilis)
NB mos 8, c)
ō=1 (Treat)
1- (
1-v
Σ
5=0 (Don' 't treat)
0
I-c
DoDEDmo
Table 9: Value function for Net Benefit
Σ V(y, x(s()),)))
=
Z.DEDmo
The disadvantage of this approach is that it's unintuitive that the net benefit of perfect classifier is not reliably and instead depends on the class balance. The advantage that when comparing at different class balances, the value of true positive and a true negative remain fixed, so measurements are directly compared on the same seale.
C.4 Prior-Adjusted Maximum Cost-Weighted Metrics
We can combine the prior-adjusted maximum value approach with the weightee metrics to get two new metrics that make sense to compare across different class balances.
25

---

Definition C.7 (Prior-Adjusted Maximum Weighted Ac- curacy). The rrjor-adjsttdd maximum weighted accuracy given a scoring function s and a threshold with class balance x is given by:
PAMWA (D n. 8, T, c)
[ c)
E c) VV/a v)
(1-c)x
2 (I (--oot+el
Σ [1 c)t| - "V/' 0)
cll x1-
2[1 Fo)
Diya
Σ (1 c)m|p([(1 (V/s
Table 10: Value function for Prior-
Dip2
Adjusted Maximum Weighted Accu- racy
Σ [1 - c)a(1 mo)] - x) (1, 1
Duo
Σ [1 c)x(1 To) [o)] x >) -
Dmp
= Σ V(y, n(T @ 01/((1),
Dzo
Proposition C.8 (PAMA Equivalence).
PAMWA TD, 8,
PAMA 1-087, 8,
Proof.
PAMWA(D 8, T.
E(1 c)uc -V (V,)2
Σ [[L c)t|[(((1 - +)] 2 1-11111 V1aa (3, Ø)
Di/2
Σ [1 c)t|[(((1 +)] 1 V (y.
Di/2
Σ V4 1)
Di-
Σ V1,(4.3) v)
D Di-e0r
PAMA(DI 1- 8)
We combine the ideas of Net Benefit (Definition C.6) adapted from the concepts used in the Deci- sion Curve Analysis literature (Vickers and Elkin 2006), with Prior-Adjusted Maximized metrics (Definition C.3) used in various settings, but originally adapted from base-rate adjustments in the psychometric literature [Meehl and Rosen [955] to define new metric.
Definition C.9 (Prior-Adjusted Maximum Net Benefit). The prior-adjusted maximum net bene- fit of a scoring function with a threshold with a class balance is given by:
V ;)
y=1 (Syphilis)
==0 (No Syphilis)
5-1 (Teaa)
0
Zo
PAMNB D x. S. T. c)
8=0 (Don' treat)
Σ
--ر Visa(y. D)
I-c I To
D.
Table 11: Value function for Prior-Adjusted Max- imum Net Benefit
Σ
V(y, 8 81/((x),
Dro
26

---

We focus on the second because although the semantics of a single value are more confusing (since the perfect classifier normalized the values different class balances are commensurable.
C.5
Conclusion
If we are willing to accept the causal diagram Dr + Y X, then we have tools available in different parts the literature to broaden Vy, x: (I, T)) to capture asymmetric costs and move from
(D))) 5(D Dmo) (D) 8(D DTr) for any given
D Cost-sensitive Error Averaged Across Class Balances
core obstacle the use of proper scoring rules like the Brier Score evaluate classifier performance under label shift in applied work is the ubiquity of asymmetric costs. Traditional derivations of the equivalence between Mean Squared Error and Average Accuracy over a range of class balances have taken the CDF of the positive and negative classes as given. These do not easily generalize to world of asymmetric costs. This appendix develops rigorous mathematical framework for analyzing cost-wer cost-weighted binary classifier performance averaged across range of class balances. The average of accuracy represents an integral over class balances of sum over data of an e ordering between the class balance and the model score each data point. The broad outline of the approach is as follows:
Σ (s(x)) n. y)
(a)
Dr
importance sampling
J
W(T + J: v) (s(x), n, y)dr
(n)
Dxo
importance weight is
Σ E' (TT (s(x), v)d
(m)
Daya
swap
(TT v) (s(a),, v) dr
D1y2 (m)
intersect
Σ (s))aa) Diy2
C' y) di
antidifferentiate
Σ (s((x) v) do Diya
This flexible mathematical framework transforms challenging integration problem into tractable calculation using our core lemma. The key innovations (i.e., reframing importance weights and expressing classification correctness as set membership) provide deeper insight than the traditional bbrraat integration-by. -pratt approach while generalizing beyond simple accuracy to various cost-sensitive losses. In the sections that follow, we demonstrate how this unified approach yields practical formulas for robust decision decision-making under class distribution uncertainty and asymmetric costs.
27

---

D.1 Preliminaries
See Appendix B.2.1 for some of the notation used in this appendix.
Definition D.1 (Adjusted Score).
Se (z)
x
s1,((2)
This represents the score optimally adjusted for the change in the prior probability. See Appendix B.2 for more details. Definition D.2 (The set of 1 for which gives the correct y)).
(). - [| - y, 81)2(a)] if y
(s))(z), - y)
if y =
For proof that these are the right conditions, see Lemma D.3
Lemma D.3 (x is the right set).
E x(2, 3)
y a( @ s1/(2), 7)
Proof. We start by expressing the thresholding condition terms of 1 π.
π 3 (1 - x s(x) N
(1 7) 2 (1 mo) s(x) 2
81/((2) 2 -
- E 00, (1)
Now we re-express the negation of this:
2 81/2 (x) ž 7
x Φ 00 [1:]
x [s/2(x)) 1)
We then compare to the label.
π 29 (1)2(x) 2 if y =
π 8 $1/2(x)
< r if =
1 E [1 - y, (1)(
ify
x E (x), v]
if = 0
- E x(
Definition D.4 (Clipping operator). Also called projection or restriction other contexts.
clip (z) A max(a, min 2))
[a.b]
Lemma D.5 (Intersecting X with a closed interval). This equality holds almost everywhere, because when the intersection is empty, the set the right will be of measure zero. Integrals of bounded functions over these two sets will give the same results.
la, bj 0 x(x, v)
[clip(1 v), dlip(s/=(x)) if 1
[a.8]
[0.6]
(clip(ss/=xx))
clip v)1 if = 0
(a.b)
[a,6]
28

---

D.2
Main Lemma
Lemma D.6 (Average over Class Balance Sum over Data of Cost- Weighted Correctness Condition).
(b a) a~UUnform(a.)) (a.))~D,
C( v) 1 (1=N(=80))2t)).))
clp (1-v) [1-b,1-o]
2(p g)C(1 p: v) dp
(zy)) Jp p= [1-b1-n]
clip (12/((1)
Proof.
b
E (r
C(T ( (1) di
E
C(T; v) 1(1 10-
Jaza (r,
W( + n: T;1) v) ((--
==a D (D.
W(To + 1/2: v) 2 - т) - yl ((::)) 10 1
(Drul (D.
==a
6 f 7=a
Σ
W( 1/2; v)
2 x) - y1lC(r; v) ((==re((as)dd
(Drol
(2.0)ED=a
We will now focus only on the inner integral.
2((1 x) y]C((r;)) v) 1(-mex(ew))
2/p - y|C(1 - p; p. ((pE((x.)))
p=1-b
2/p - y|C(1 pı y) dp
by case analysis on y
clip (1-y)
[1-61-0a
2(p 8)C(1 p. v) dp
p= clip (01)((2))
I| [--b--ol
29

---

Combining the two parts, we get the result.
a)
E
C(T:y) 1 (1=n(r00)2()).))
Ih L-intoormmaa b) (2.y)~>a
C(Tiy) 1 (1=6( du
D (x.
W + 1/2:8)
2/(1 л) - y](((T; y) 1a de
iDool
(z,y)eDTo
==a
clip (1-9) [1-biinl]
Σ
W(To + 3/2:1)
2/p 5/C(1 ps v) dp
mo (a,
p= clip (01/a(1)) D-b,1-al
clip (1-y)
E (z.0)=
[1b,1-n]
2(p g)C(1 p) dp
p= clip (x)/
[1-b,1-1]
D.3 Example Use
Theorem D.7 (Bounded Brier Score).
(b a)
E PAMA n. T) [a
27
E
clip (1 - 8) - - clip (s))(()) - y
(r,y)>Dayaa
[1-b,1-a)
[1-b,1-a)
30

---

Proof. Straightforward application of Lemma D.6 with C(;; v) 1.
(b a) E PAMA(D,, 8, 7) x
C(T: v) 1 (V=n(r00)p(t).1)
by Lemma D.6
clip (1-2)
E
[1-
2(p () v) dp
(x.1)~
Diy2 p= clip (01/(10))
1-6,1-0)
clip (1-2)
=
E
|1-b,--|
2(p - y)
(a.))~
Diya
clip
(01/((0))
(1-611-0
by antidifferentiation
clip (1-1)
[(p - y) p= clip (x))
27
clip (8)a(z)) -3)
E (Dyp)>DD/a
clip (1 8) -
11-b
11-bll-a
E Log Scores with Asymmetric Costs
Theorem E.1 (Bounded Log Score).
(0-1 (b) - (a))
(n)raoo (x)) (()) PAMA(D, 8,
E
log
clip (1 8) (1 y) - log
clip (8)z(2)) - (1 - -11
(k.y)) DDbya
1 1-b,1-a]
[1-b,1-a]
31

---

Proof. Use Lemma D.6 with C(T:8)
1
*(1-x))
[a_ [r-1(b) - oa)
E (+)~/aa (a), 166]
PAMA 8, r)
[b al
E
C(T; u) (1)11p111 (1=N(T80))(0)))))
Now () Dr
clip (1-b) ul
EF
2(p y)c(1
p: v) dp
Lemma D.6
(a.y)-pppp
p= [1- b.
clip (#1)((x))
clip (1-9) bi ul
dp
E
2(p y)
(a.z) 1/2 p= clip (=1/((x))
p)p
clip (1-#)
2dp
E
|| b1-b,-ul
case analysis
(a.y)~ 11/2 p= [1
clip (x1) (1 - p)
2
E In y
clip (1 8)
In
y
clip s1/(xx
[1-b1-o]
[1-b,l-a]
E.1 Weighted Accuracy
Refer to Definition C
Theorem E.2 (PAMWA Log Score).
( oa)
PAMWA(D,: 8, c.
o (a)
= - y
E
clip
(1 - y) - In
clip
(s1)((z))
(x,y)~D1/1
1001-b.c81-1 oa]
32

---

Proof. The high level idea is we replace a PAMWA term with PAMA term by Proposition C.8 and then do change of variables, and apply Theorem E.
(o-1() (a)]
E 1(1)~/a (a)a 166]
PAMWA(DPr R, 8, 8,
- o [a)]
(A-A(D1-
(by Proposition C.8
o (T)) (a)
now we do change of variables T' (1 c) 8 n
-o-
(6) (6)
0- 'a)( 1(11)~10 ((1-))00) ((1-6)86)
E
PAMA D 8,
jo (1 2 6) o (1 x a)]
PAMA x/; 8) (1((0))0 (a))) ((1-c)86)
(by Proposition B.7
now by Theorem
2 E Dryz
In 1 [1-1-c)001-111---33
clip
(1 - y) In 1 (1-1-c))b1-(---06 (012(1²)
clip
2
E
In 1
clip (1 y) In 1 y
clip
(hhs) (2))((r))
(x.y) ~ Daf2
081-b,co1-a)
kcs1-b,e3--a|
E.2
Net Benefit
Refer to Definition C.9
Theorem E.3 (PAMNB Log Score).
10 (o-(6) (a)
E [1)())) (4)
PAMNB 8, T,
2 E
In 1
clip
(1 y)
In
1
clip (s (8(1))((z)) 1-b,1-a
1 (x.
D c)
11 aa]
33

---

Proof.
(b)-
E
PAMNB D 8, T,
1(aiao (a),o 16]]
r (b)
PAMNB &, do do (T)
(x)==)))) (a)
dm
PAMNB D mi
TT(
PAMNB (D 8, T, C.
(0
E
[m
tT1
T)
1-y
(V=n(T00)p(())c)
(6
E
E
Xr [0,
2.
T(1 T)
c--((1 c)
(b
E
E
x~
D.
(1
(1 (0=N(1001-088001111111
now we use Lemma D.6
clip (1-9)
1 cl-v(1 c)y²
4,1
2(p - 8) (1 ppp
dp
(e.y)-Dipp 1/2
p= clip bl
(1-08e17(())
2
E
(1 - c)u
f
clip (1-0)
- dp
1 - (a.y) Diya
clip ((1 (11-0) (1- -1pp
116-1-0
and we use importance sampling
2
clip (1-v) (1bb1_a)
p dp (1- p)p
E
1 (a.z)) ~ Di-
clip (+1-
[ibbi-al
and by case analysis on y
2
clip (1-1) b.)
1
E
clip (5 (x)) y/ dp 1-b,i-a
1 (0.N)
D
p=
2
E
In 1 y
clip (1 y)
In 1 y
clip (z))
1
11b,1-a|
11-b1--a
F DCA Brier Score
Although we believe the uniform log-odds for prevalence better fits clinical decision-makinn scenarios, we provide a derivation of the equivalent formulation using uniform prevalence for completeness.
Theorem F.1 (PAMNB Brier Score).
bb a]
E Tn
PAMNB(D c)
2
1
E
clip (1 y) 1-b,1-a]
y
clip (s1-c(z)) 1-b,1-o|
2(1 c)
(w.y)~ 1-c
34

---

Proof. Recall the definition of the adjusted score: (x) - x @ sya(aa).
And recall s(x)
C = (1
s(x) > 7
(b a) [a,bb
PAMNB DT 8, T,C)
1-y
(b
E
1 (v=n(180))()))0))
R. DT
1 C
c--1(1
c)y
(b
E
1
The b
T.
D
(1 - c)
now we use Lemma D.6
clip (1-w)
2(p y) c1- - c)u
E
[1-6.1-0]
dp
(a.y)~
D1y2 p= [1-b1--l
(1 (1-0891/((1)
c
clip (1-1)
c' c)u
(1-b,1-a)
2(p - y) dp
- c (11))~D112
clip (i1 (1100)831/(00)
D-b1-al
and we use importance sampling
clip (1-9)
E (a.))-Di-c
|| [1-b.--1)
2(p y) dp
2
clip (#1 (81)
1.-6.1-a
by antidifferentiation
clip (1-2)
E
(p y)
1-
2(1 - c) (w.g))~D...
clip (x))) b
27
clip clip (1 v) -y 11-0,1-o)
E
y
clip (81-o(x)) - -V) 11-,1-o)
2(1 - c) (i
G 13 ways of looking at an AUC-ROC
For largely contingent historical reasons, medical informaticists have long reported results either in terms of specificity and sensitivity (if they want fixed threshold), in terms of the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). What exactly does this mean? There are 12 standard interpretations of the AUC-ROC, and none of them fit very well. We add a thirteenth which is more relevant but still inadequate for our purposes.
1. 0.5 when the classifier random, and 1.0 when the classifier is perfect. Carrington et al.) [2023 reports this as the most common interpretation, bit tongue in cheek. It is, fortunately, also the current authors' experience that this is the most commonly given interpretation in practice.
2. The alternative forced choice accuracy rate Swets and Birdsall (9566].
This only makes sense in the original psychometric setting where experimenter in fact guarantees that there one positive and one negative case Carrington et al. 2023).
3. A rescaled version of the Mann- WainnWy U statistic Bamber 1975 Hanley and McNeil
1982]. This is actually the same as the statement above, but sounds more impressive.
35

---

Note that AUC-ROC is never reported as a p-value based on this statistic, which suggests that the interpretation not practically very useful.
4. A rescaled version of the Kendall's correlation coefficient Heenandez-Orraloo et al. 2013
It is technically true that the AUC-ROC is a pairwise permutation distance between the ideal ranking and the actual ranking. But there are only 2 ranks! This makes the exercise meaningless.
5. An average of precision (though not " "Average Precision" which refers to something else) There is an occasional attempt to rescue the paradigm by arguing that AUC-ROC shows an average of TP = K x Precision& K over range of K. The trouble is twofold:
This gives a uniform average over all possible values of K. In quantity constrained setting where we' we're forced pick out only K items to give positive labels (imagine only 10 doses of penicillin in the freezer) we generally have very small K.
Few binary classification problems are actually intended for explicit quantity con- straints, so this doesn't fit well.
6. An average of power over a range of sizes (in the Neyman-Pearsee sense)
This one very popular with practitioners and virtually absent from the literature, aside from McClish 1989,, which was later criticized by McClish [2012). Carrington et al. [2023). The trouble is that in the Keyman-Pearson paradigm we're re meant to pick a power, and then find out what the size of the test is. Even if reverse this interpretation as average size over a range of powers, the power in the NP paradigm isn't 't an empirical quantity we might observe distribution over. We're meant to pick one.
7. It is the area under curve if FPR is plotted against TPR. Hanley and McNeil 1982]
8. Average accuracy on the positive class across a uniform distribution of accuracy on the negative class, vice versa Metz 1986 1989 Zhou et al. (2002). This is actually the same as the statement above, but more useful-soundie and slightly less mysterious since it doesn't 't the words "False Positive Rate" or "True Positive Rate".
9. Given two thresholds distribution of accuracy on the negative class between those two thresholds, plus the average accuracy the negative class across uniform distribution of accuracy the positive class between those two thresholds, weighted by the class balance between those two thresholds Carrington al. [2023). This is a bit more useful, but it's not clear why we're using the exact class balance between the thresholds.
b, the average accuracy on the positive class across uniform
10. An average of accuracy as we set the threshold at each data point, leaving operating conditions the same FFlcch et al. [2011). This interpretation is the only one from this set of authors that directly addresses the problem that data is empirical and discrete.
11. An average of weightei error over range of cost ratios. Hand (2009) shows that if a score is calibrated, then the AUC-ROC is an average of the cost- nost-weihhted error over a range of prevalences. The trouble is that:
Calibration is a really important property of score! Without it, we can only make top-K decisions. Even then we have to assume there are more than K real positives.
The range of costs is function of model scores on the training data. But the risk that a given individual has syphilis not a good estimator of how much less harmful unnecessary penicillin is than untreated tertiary syphilis.
Because the costs are function of the model, if two people train models the same data, they will get different costs, and the average accuracies will be incommensurable.
12. An average of weighted cost, for skew 2 = c @ (1 Tτ. Hernández-Orallo et al. [2013) proposes this interpretation, although again the costs are set arbitrarily and distinctly by each model, and classcconditional distributions are assumed to be continuous and fully known. Furthermore, it is not clear what it means to integrate over values of @ (1 л².
One possible interpretation is that we set e = (1 - л)) and then calculate a line integral along a particular curve in cost / prevalence space, by way of estimating the area integral over the entire space.
13. An average of accuracy under label shift, where the distribution of positive class prevalences
is derived by sampling from the model scores the training data. This is syntactically
36

---

similar, but specifically derived in the case of the sampling problems that arise from label shift (see Theorem H..). While this is the most relevant definition for our problem, it shares the issues of the distribution being unrelated to the class balance of the deployment and incommensurable across models.
H AUC-ROC as Average Accuracy under Label Shift (symmetry, no precision)
See Appendix for enumeration of alternative interpretations of the AUC-ROC, idea inspired by Turakhia [2017], Stevens] 1923]. Here we focus on the interpretation as an average of shifted accuracy, when the class balance distribution is derived by taking minus the score from samples drawn
from the training data, assuming that the classifier is calibrated, so that Pyy = s (x) = T) =
for all [0, 1].
As discussed Section 3 this interpretation shows a clear meaning for AUC-ROC in our context, but also shows its shortcomings.
Our key innovation focus on using importance sampling directly address label shift. We begin with definition. which we then reexpress in terms of balanced classes. Then we use importance sampling to reexpress components of the AUC-ROC in terms of true positives or true negatives. Finally, thanks to the intrinsic class symmetry of the AUC-ROC, we combine the results.
Definition H.1 (AUC-ROC as Average Data).
AUC-ROC D no, 8) A
Σ
- y
V/ D (((0))200)) + 1/21 (m'
Σ
D
To
(D.V)ED
We now show that this is equivalent to drawing from balsnca distribution, known property of the AUC-ROC, but one which will simplify our analysis.
Lemma H.2 (Importance Weighting).
y V/
1 To ((((0)20(E) + 1/21 (s(a'===()
D 2
(x.y)
E
(1 - y) [1(0002200)) + 1/21(002)=0001)
(a.))Di// v/ \/D22
In the continuous case, the 1/21 (s()))=(())) term disappears. In the discrete case, the AUC-ROC is neither exactly equal to the average of true positives nor the average of true negatives, but instead to the average of the two.
Lemma H.3 (Average over Label Shifts of True Positive Rate).
E
E (w' WED
(1 - v> ((((r))><()) + ((s(x))=((2))
1 y/ o [[00e120]]
E (2.0) (e
tEs
(E''D')
(((())==) (e's)))DA
E
(1- [1002)22]]
(2,))E (ep/1
y/ 1(((2)21)
re: De)(D1/2) (a'
By reversing the order of summation, we can also express a related quantity as an average of true negatives, instead.
37

---

Lemma H.4 (Average over Label Shifts of True Negative Rate).
E (x,y)EDa ())
- - -1-' ((o(e))aa(a)
1/11 [)(o(00022000)) (2)1)
E
(0t ( v/
(1 y) [
B
E
E
rea(Dryal (xy))
111 v) [
E (E (((()))=))
E (2)))))D)/2
(Es(Daya
(1 - n) (s(x)<()
rEx(Drp) (29))
Finally, we combine these three results to show that the AUC-ROC is an average of accuracy under label shift.
Theorem H.5 (AUC-ROC As Average Accuracy).
2AUC-ROC(S)
E eno(Di/2)
PAMA 1-6, 1/2)
Proof.
2AUC-ROC(D mp: 8)
E
E
(1 y) 2 16(0))200)) + (((o))=((a))
( (e) 1/D1/
= =
E
(1 vlr (100)220))) + (((()))((a))
(m (z) ()ppa
E (1 viv/ [Moce)sace)]
(Da))Di Di (o' (e
E V/ (2()2) + E
(1 1) 10(2)40) (a.))D,
Diyal t~i
E
(Accuracy(2)) (DI,) s(x), [)]
t~a
Diyzl
(Accuracy(D1--
3 s(x), t = 1/2)]
Iws
Diyal
E
(OMA(DI-,s(I),77 1/2)
t~a
Diyal
I eICU Subgroup Analysis
L.1 Patient Demographics
38

---

Table 12: Baseline Characteristics and Outcomes by Sex and Race/Ethnicity
Characteristic
Overall
Female
Male
African- American
Caucasian
2520
1008
1508
231
2010
Age (years), Mean (SD)
63.3 (17.8)
64.7 (18.2)
62.3 (17.4)
56.0 (16.8)
64.5 (17.5)
APACHE IV, Mean (SD)
52.0 (25.3)
54.3 (25.3)
50.5 (25.1)
50.4 (27.7)
52.0 (24.8)
ICU LOS (days), Median
1.5
1.5
1.4
Hospital LOS (days), Median
4.4
4.8
4.3
5.2
4.3
Hospital Mortality, N (%)
212 (8.4%)
87 (8.6%)
124 (8.2%)
9 (3.9%)
173 (8.6%)
ICU Mortality, (%)
126 (5.0%)
(4.2%)
(5.5%)
(3.0%)
(5.0%)
1.2 Mechanism Shift vs Label Shift
To separate which parts of the difference in log score are due to different class balance vs different mechanism of prediction, we can rely on the label shift assumption (within subgroups) and use the do calculus of Pearl [0011]
D
D
4
>
Y
do Y X
(1)
(2)
Figure 3: Causal diagrams: (1) shows differences in performance are based both label shift and differences in mechanism between subgroups (2) if we can intervene on Y to hold it constant, then we can measure differences in performance based purely on differing performance the model between subgroups. As covered in the main body of the text, under label shift, we can simulate changes in Y using importance weighting and prior-adjusted maximal accuracy.
For brevity, all our expectations over in this section will assume the distribution (Tt) Uniform(o" (Thiue) a² (Taraagge) )
The causal impact after intervention Y is given by the clipped cross entropy:
AD
E PAMNBB orange, x 8, e) PAMNB (D blue, 8, c)
EICU
A
pur
0.96
We can now additively decompose the difference in Accuracy:
0.94
AD = AD- + AP-,
Ap
0.90
0.88
0.80
Positive Class Fraction
Which allows us calculate the label shift effect:
Ap-y E
PAAMNB ( orange, vr
PAMNB orange, 8, s,c)
+ PAMNB (D blue, 8, c) PAMNB D blue 77 8, c0)
39

---

We can think of these two terms as the difference between actual accuracy and average accuracy for the two separate subgroups.
1.3 Calibration Sharpness
We can also decompose the difference in accuracy into calibration loss and sharpness.
ATOAAL As + Ac
It's ss easy to calculate the overall gap.
ATOTAL -E PAMNB D orange, 8, PAMNB D blue, 8, c)
The sharpness gap is also straightforward, since we can simply recalibrate the model on the subgroup of the evaluation set (which we' call s*), and measure the loss of the recalibrated model.
A
E PAMNB(D orange, Sorango c) PAMNB bl blue C)
EICU
Figure 4: Accuracy Prevalence curves for subgroup- recalibrated models (dashed lines) plotted on the same y-axis as AUC (solid lines)
0.905
0.900
The difference sharpness for clipped cross- cross-entropy is not ex- actly the same the difference in sharpness measured by the AUC-ROC because, as mentioned earlier, the AUC-ROC weights different thresholds differently. However, the two are fairly corre- lated. Sharpness across the full range of prevalences is somewhat higher for African Americans (orange).
0,875
0.850
Caucasian
becalibrated)
0.825
African American
recalibratedi
0.01
0.10
Postive fraction
This remaining gap is the calibration gap.
Ac = E PAMNB (D Poongee,ry 8, c) - PAMNB orange, 8 orange c)
+ PAMNB (D Diluc Siine c)
PAMNB Dolue .r c)
Figure 5: Accuracy vs Prevalence curves for original and recali- brated models
EICU
We can measure the miscalibration of the model by the differ- ence accuracy between the original and recalibrated versions. We plot the accuracy of the original model (solid lines) and accuracy of a perfectly recalibrated model (dashed lines). Miscalibration for Caucasian patients (gap between blue lines) essentially zero, while miscalibration for African American patients (gap between orange lines) is quite large.
0.98
Accuracy
0.34
0.90
Ceucesian
African American
The solid lines combine both the sharpness and miscalibration losses onto single axis. There is no equivalent way to plot ECE on this y-axis.
0.86
African American
trecalibrotedl
0.01
0.10
Postive Class Fraction
40

---

1,4
The Importance of Bounding the Prevalance Range
BICU
1.00
Accuracy vs. prevalence curves by gender. Accuracy clinically relevant (low) prevalences higher for female patients (blue) than for male patients (green).
Accuracy
0.90
However, at roughly balanced or high prevalences-ranges that are not clinically meaningful trend reverses slightly, with higher accuracy for males. Traditional scor- ing rules such as cross-entropy overweight these high- prevalence regions, reporting a better overall score for male patients despite inferior clinical performance at realistic prevalences. By contrast, the Bounded DCA Log Score correctly emphasizes clinically plausible ranges and thus ranks the model higher for female patients.
0.85
0.80
Male
Female
0.10
Positive Class Fraction
Figure Accuracy vs Prevalence curves
by gender
1,5 Confidence Intervals
As is often the case, it's easy to take this too seriously. Since the prevalence of mortality is low, and the fraction of black patients is already low well, it turns out there's tremendous amount of sampling error in the public subset. When we draw the confidence intervals, we see that we can't really conclude anything.
EICU
EICU
1.0
0.9
0.9
Accuracy
0.8
Accuracy
0.8
0.6
0.5
0.5
0.01
0.10
0.01
0.10
Positive Class Fraction
Positive Class Fraction
Figure 7: There's 's significant gap between the performance on the public dataset for black and white patients, but it's well within the 95% confidence interval. By comparison, when we use the full dataset, we see that the quality of the models is actually quite similar, and any difference in clinical utility is largely due the difference in prevalence.
I.6 Conclusion
While AUC-ROC and Accuracy each handle some aspects of performance, we show an example with well known public data where both give the same misleading intuition about subgroup performance differences. What makes the clipped cross entropy approach useful in this setting is that its linearity makes easy to decompose different effects, and its integral across scenarios allows us to catch effects that can't 't be seen at any one class balance.
41

---

It is also important that in each of these decomposition plots the two effects we were trying to decompose were measured the same units, so don't have to ask ourselves which one is more important. This generalizes to Net Benefit and Weighted Accuracy decompositions as well.
J
Metric Failure Mode Comparison
Failure Mode Cost-Sensitive Evaluation
Metric
Domain Adaptation
Decomposability
Notes
Accuracy
Does not handle subgroup preva- lence shifts.
Assumes symmetric costs (FP FN)
Does not reflect cal- ibration. Very little information dis- crimination. Pure discrimination, calibration.
Simpson's paradox effects when sub- group prevalences differ.
AUC-ROC
Invariant to preva- lence.
Range of costs cho- sen by model, not
Can't combine with (e.g.) ECE.
clinicians.
Expected Cali-
Domain-spceiffe adaptation unclear.
No cost sensitivity.
Pure calibration, discrimination.
Can't onn' combine with (c.g.) AUC-ROC.
bration Error (ECE) Brier Score
Unrealistic preva- lence ranges (e.g.. half of weight above 50% mortal- ity).
Cannot combine asymmetric costs with imbalanced classes.
Decomposes into calibration and discrimination.
Misleading at clini-
cal prevalences.
Cross-Entropy /NLL
Unrealistic preva- Ience ranges (e-g., half of weight above 50% mortal- ity). Requires known Requires prevalence.
Cannot combine asymmetric costs with imbalanced classes.
Decomposes into calibration and discrimination.
Misleading at clini- cally relevant preva- lences.
Net Benefit
known
No decomposition Requires knowledge.
costs.
into calibration and discrimination.
Table 13: Summary of metric failures under different evaluation scenarios.
42

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose that scoring functions for clinical purposes should account for heterogeneous label shift, asymmetric enror costs, and calibration. We show that the clipped cross entropy a natural way to do this, which allows us to specify clinically relevant ranges of the class balance in intuitive way.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations section.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Clarifying the assumptions of previous work regarding label shift is central motivation for the theorems stated in this paper.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper proposes the central contribution, but we do include the code used to generate the data needed to generate figures on the subset, but cannot include code used generate corresponding figures in the appendix for the full dataset.
evaluation method. The experimental result is the figures. We include
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the code used to generate the figures. We cannot include the EICU dataset itself, but that is all that is missing to run the code.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: There is no model training in the paper; we only evaluate the existing APACHE IV model using the EICU dataset.
7. Experiment statistical significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We show error bars in the figures, which show that the disparities being analyzed are likely due to small sample size.
43

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: This is an evaluation paper; all experiments ran in under a second on laptop several years old.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics http://////.aaa....ooo/o///////a
Answer: [Yes]
Justification: See social impact section.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Foundational research. Better binary classifiers can be used for good or evil, but we believe the first order effect is allowing practitioners in clinical setting to better evaluate their models.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Paper does not create large pretrained models, nor scrape data from the Internet.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Only dataset used is EICU, cited at Goldberger et al. (2000). Pollard et al. [2018 2019), Johnson al. [2021] The license is Open Data Commons Open Database License v1.0.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Paper does not release new assets.
14. Crowdsourcing and research with human subjeets
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Paper does not involve crowdsourcing or research with human subjects. 15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
44

---

Answer: [NA]
Justification: Paper does not involve crowdsourcing or research with human subjects.
16. Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Research was not LLM based.
45


Reviews:
[{"review_id": "ewReSFoMHs", "reviewer": "Reviewer_75b4", "summary": "This study presents a new method for measuring the performance of ML predictive models for clinical applications. The method emphasizes measuring performance at deployment and aims to keep the method simple and straightforward (for calculation and interpretation).", "questions": "Is Theorem E.1 a proof for 4.1? What's the exact novelty in the formulation, is it only the clipping process or something beyond?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The study targets an important and critical area in ML for healthcare. \n- Extensive background and context are provided prior to presenting the proposed method (mostly Theorem 4.1). \n- The study also aims to maintain a pragmatic approach by making the proposed way easier to interpret.\n\n**Weaknesses**\n- The study starts with a rather unorthodox introduction, and right off the bat presents “three priorities for clinical purposes.” These are then treated as the foundation of all that follows. While these three are sound and reasonable, they are not an exhaustive list encompassing all issues in this area, as the text seems to imply.\n- While the study targets a very critical and timely issue, the main pitch is hardly news to the practitioners in the field. As the study acknowledges, many have pushed for going beyond accuracy and AUROC, yet those measures remain mainstream. Even using simple calibration or net benefit calculations are less frequently seen. Now, one (including this reviewer) may argue that part of the reason for the current situation is that most studies similar to the present work remain very high-level and skip the practical implications and use cases in favor of the theoretical foundations. The current work may appeal to a limited Nuerips audience, but expanding the real-world scenarios (as presented in section 5) is critical to demonstrate the true value of the method.", "was_revised": true, "final_justification": "The author's response addresses (and acknowledges) some of my comments.", "rebuttal": "# Introduction: New Analysis and Major Commitments for Revision\n\nWe thank the reviewers for thoughtfully highlighting the value of more explicit comparisons of our proposed metric to popular alternatives on real data.  To that end, we have extended the analysis from Section 5 on the performance of the Apache IV mortality prediction algorithm in different subgroups of the publicly available eICU dataset, and will include the following tables in the revised draft.\n\n| Metric | Caucasian | African-American |\n|--------|-----------|------------------|\n| Bounded DCA Log Score (calibration-only) | **0.999** | 0.927 |\n| ECE | **0.829** | 0.889 |\n| Bounded DCA Log Score (discrimination-only) | 0.957 | **0.973** |\n| AUC-ROC | 0.868 | **0.907** |\n| Bounded DCA Log Score | **0.956** | 0.900 |\n| ECE & AUC-ROC | ? | ? |\n\n*Table 1: Calibration and Discrimination for Caucasian and African-American Patients*\n\nBoth commonly used and proposed calibration measures agree that the model has better calibration for African-American patients than Caucasian, but worse discrimination.\nHowever, the calibration and discrimination components for Bounded DCA Log Score can be linearly combined, while there is no principled way to combine AUC-ROC with ECE to reach an overall judgment.\n\n| Metric | Caucasian | African-American |\n|--------|-----------|------------------|\n| Accuracy Curve@1% Prevalence | **0.990** | 0.978 |\n| Accuracy Curve@5% Prevalence | **0.953** | 0.896 |\n| Accuracy Curve@10% Prevalence | **0.912** | 0.793 |\n| Bounded DCA Log Score | **0.956** | 0.900 |\n| Accuracy | 0.922 | **0.961** |\n\n*Table 2: Accuracy for Caucasian and African-American patients at varying prevalences*\n\nThe most powerful analysis technique here is to graph accuracy at each prevalence and compare the two entire curves visually. As the table suggests, Apache IV on the public eICU data performs better for Caucasian patients than for African-American patients at any given prevalence.\nComparing two Bounded DCA Log Scores averages the heights of these curves and the scalar summary preserves their ranking.\nThe empirical accuracy for each subgroup is measured at different prevalences.\n\n| Metric | Male | Female |\n|------------|------|--------|\n| Accuracy@1% | 0.989 | 0.990 |\n| Accuracy@5% | 0.948 | 0.954 |\n| Accuracy@25% | 0.818 | 0.814 |\n| Accuracy@75% | 0.846 | 0.832 |\n| Accuracy@95% | 0.953 | 0.953 |\n| Accuracy@99% | 0.990 | 0.990 |\n\n*Table 3: Accuracy for Male and Female Patients*\n\nApache IV performs better for female patients than male patients at low mortality prevalences, worse for roughly balanced mortality, and about the same at high mortality prevalences. However, only the low prevalence ranges are clinically relevant.\n\n| Metric | Male | Female |\n|--------|------|--------|\n| Brier | 0.062 | 0.062 |\n| Cross-Entropy / NLL | 0.214 | 0.218 |\n| Bounded DCA Log Score | 0.917 | 0.926 |\n\n*Table 4: Scoring Rules for Male and Female Patients*\n\nAlthough unbounded Brier score and cross-entropy are related to integrals of net cost, and are therefore related to net benefit, they give misleading answers because they put too much weight at high, clinically unrealistic prevalences.\n\n\n\n# Specific Reviewer Questions and Replies\n\nWe thank the reviewer for their thoughtful, thorough review.  Their comments brought out many of the issues that got squeezed out by page limits, and we appreciate the feedback and the opportunity to bring back important context.\n\n## While the study targets a very critical and timely issue, the main pitch is hardly news to the practitioners in the field.\n\n> As the study acknowledges, many have pushed for going beyond accuracy and AUROC, yet those measures remain mainstream. Even using simple calibration or net benefit calculations are less frequently seen. Now, one (including this reviewer) may argue that part of the reason for the current situation is that most studies similar to the present work remain very high-level and skip the practical implications and use cases in favor of the theoretical foundations. The current work may appeal to a limited Nuerips audience, but expanding the real-world scenarios (as presented in section 5) is critical to demonstrate the true value of the method.\n\nIt is correct that adoption is the key challenge.\nThe math is necessary here because it shows that measuring net benefit and calibration together with one good metric can be easier and more powerful than measuring them separately with two metrics.\nHowever, the main way we intend to move the needle is by providing practical advantages:\n1) having a single summary of calibration and discrimination such that the user can simply pick the largest value (less complex than AUC-ROC + ECE, as shown in Table 1)\n2) restricting to the domain of clinical relevance (more relevant than Brier score or cross-entropy, as shown in Table 3 and 4)\n3) separating accuracy differences from prevalence differences (see comparison to Accuracy in Table 2)\n4) allowing uncertainty about exact prevalence (does not require precise knowledge like Net Benefit)\n5) computational simplicity: while our approach is not computationally trivial, it is nonetheless based on clipping the well-known cross-entropy measure.\n\n## Question 1: Is Theorem E.1 a proof for 4.1? What's the exact novelty in the formulation, is it only the clipping process or something beyond?\n\nWe appreciate the reviewer pointing out this error; the proof of 4.1 is in fact Theorem E.3.\n\nThe novelty in the formula itself is applying the clipping process and making it compatible with asymmetric costs.  Combining the two is, in fact, tricky, and it is a major reason that we prefer the log score-based measures over the Brier-based measures.\n\nOur derivation is novel because we are explicit about starting from empirical aggregates of observed data. We do not assume the distributions of positive and negative classes are given.  This clarifies the causal assumptions needed for this to be valid; distribution shift will not always mirror data resampling.\n\n## Conclusion\n\nWe appreciate the reviewer's recognition of the clinical motivations for designing a new scoring rule approach to capture distribution shift under asymmetric costs and their focus on clinical adoption.  Their encouragment has helped us center the importance of crisp examples with clear descriptions using real data, and commit to expanding Section 5 with further empirical comparison of the Bounded DCA Log Score to the most widely adopted metrics, and more emphasis on describing how to use the DCA Log score to resolve issues in subgroup decomposition that are otherwise difficult to tackle."}, {"review_id": "ui8bpKmtFc", "reviewer": "Reviewer_RD77", "summary": "This work focuses on the mismatch between model evaluation and clinical priorities, specially calibration and robustness to distributional shifts. This work characterizes where existing methods, like accuracy and AUROC, fail, and propose an evaluation framework for selecting calibrated thresholded classifiers that can handle bounded distributional shift and domain-specific cost asymmetries. To do so, the authors propose a variant of the cross-entropy score using clipping that averages cost-weighted performance over real ranges of distributional label shift.", "questions": "1. How do brier score and expected calibration error compare to the proposed evaluation metric towards the goal of measuring calibration and providing an interpretable metric real-world deployment? \n2. How can we apply the new evaluation metric to real clinical data, and how can the results be interpreted in a meaningful way? \n3. How would evaluation in real clinical data differ between existing measures of calibration and the proposed clipped log metric? \n4. Can the brier score be used in a similar context to the log cross entropy, as the brier score is also a proper scoring rule that can be decomposed into sharpness and calibration?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The work approaches an important problem and correctly identifies that many metrics that we use today (i.e., accuracy and AUROC) are incompatible with clinical needs. \n2. A solution which allows clinical input is useful, as domain experts provide a better opportunity to know what type of label/distributional shift is expected. \n3. The clipped cross-entropy loss is a nice extension of a commonly used loss, and the provides a nice interpretation with respect to net-benefit over multiple class prevalences. \n4. The authors do a good job surveying past work in limitations of brier score and expected calibration error, which are often used as measures of calibration in the clinical space but can often not be extended to asymmetric costs or ranges of different prevalences. \n\n**Weaknesses** \n\n1. The findings of accuracy and AUROC with respect to misalignment with some clinical objectives is not particularly novel and is well known in the clinical space. This is especially true in the very related space of survival analysis, where recent work has moved beyond simply reporting the C-index (an analog of AUROC) and focused more on calibration. Hence, many of the results showing the limitations of AUROC/accuracy are not particularly interesting findings when placed in the broader context of the clinical space. Instead, the paper should focus more time showing why existing measures of calibration in the clinical space (i.e., NLL, brier score, expected calibration error (ECE)) are not sufficient. \n2. Similar to the point above, the experimental results do not provide significant findings for this paper. As noted, accuracy and AUROC are already well-known in their limitations for calibration and insensitivity to class balance and label shift. It would be more useful to provide an apples to apples comparison of existing calibration metrics in the clinical space (brier score, ECE) with the proposed loss to show how theoretical guarantees manifest and how interpretations differ. The current analysis does not help in showing how the proposed evaluation metric can be applied nor how it should be interpreted in real clinical scenarios.", "was_revised": true, "final_justification": "Author response helped clarify major reasons for rejection, impact of proposed metric is still of concern.", "rebuttal": "# Introduction: New Analysis and Major Commitments for Revision\n\nWe thank the reviewers for thoughtfully highlighting the value of more explicit comparisons of our proposed metric to popular alternatives on real data.  To that end, we have extended the analysis from Section 5 on the performance of the Apache IV mortality prediction algorithm in different subgroups of the publicly available eICU dataset, and will include the following tables in the revised draft.\n\n| Metric | Caucasian | African-American |\n|--------|-----------|------------------|\n| Bounded DCA Log Score (calibration-only) | **0.999** | 0.927 |\n| ECE | **0.829** | 0.889 |\n| Bounded DCA Log Score (discrimination-only) | 0.957 | **0.973** |\n| AUC-ROC | 0.868 | **0.907** |\n| Bounded DCA Log Score | **0.956** | 0.900 |\n| ECE & AUC-ROC | ? | ? |\n\n*Table 1: Calibration and Discrimination for Caucasian and African-American Patients*\n\nBoth commonly used and proposed calibration measures agree that the model has better calibration for African-American patients than Caucasian, but worse discrimination.\nHowever, the calibration and discrimination components for Bounded DCA Log Score can be linearly combined, while there is no principled way to combine AUC-ROC with ECE to reach an overall judgment.\n\n| Metric | Caucasian | African-American |\n|--------|-----------|------------------|\n| Accuracy Curve@1% Prevalence | **0.990** | 0.978 |\n| Accuracy Curve@5% Prevalence | **0.953** | 0.896 |\n| Accuracy Curve@10% Prevalence | **0.912** | 0.793 |\n| Bounded DCA Log Score | **0.956** | 0.900 |\n| Accuracy | 0.922 | **0.961** |\n\n*Table 2: Accuracy for Caucasian and African-American patients at varying prevalences*\n\nThe most powerful analysis technique here is to graph accuracy at each prevalence and compare the two entire curves visually. As the table suggests, Apache IV on the public eICU data performs better for Caucasian patients than for African-American patients at any given prevalence.\nComparing two Bounded DCA Log Scores averages the heights of these curves and the scalar summary preserves their ranking.\nThe empirical accuracy for each subgroup is measured at different prevalences.\n\n| Metric | Male | Female |\n|------------|------|--------|\n| Accuracy@1% | 0.989 | 0.990 |\n| Accuracy@5% | 0.948 | 0.954 |\n| Accuracy@25% | 0.818 | 0.814 |\n| Accuracy@75% | 0.846 | 0.832 |\n| Accuracy@95% | 0.953 | 0.953 |\n| Accuracy@99% | 0.990 | 0.990 |\n\n*Table 3: Accuracy for Male and Female Patients*\n\nApache IV performs better for female patients than male patients at low mortality prevalences, worse for roughly balanced mortality, and about the same at high mortality prevalences. However, only the low prevalence ranges are clinically relevant.\n\n| Metric | Male | Female |\n|--------|------|--------|\n| Brier | 0.062 | 0.062 |\n| Cross-Entropy / NLL | 0.214 | 0.218 |\n| Bounded DCA Log Score | 0.917 | 0.926 |\n\n*Table 4: Scoring Rules for Male and Female Patients*\n\nAlthough unbounded Brier score and cross-entropy are related to integrals of net cost, and are therefore related to net benefit, they give misleading answers because they put too much weight at high, clinically unrealistic prevalences.\n\n\n\n# Specific Reviewer Questions and Replies\n\nWe thank the reviewer for their helpful comments.\nWe appreciate that the reviewer agrees that this is an important problem, that clipped cross-entropy is a nice extension, and that allowing clinician input is useful.\n\n## Question 1: How do brier score and expected calibration error compare to the proposed evaluation metric towards the goal of measuring calibration and providing an interpretable metric real-world deployment?\n\n### Expected Calibration Error\nECE has flaws as an estimator of calibration, and there is no principled way to combine it with measures of discrimination (e.g. AUC-ROC minus ECE doesn't mean anything).\n\n1. **Binning causes statistical difficulties**\n\nThere is an intrinsic bias-variance tradeoff with binning [1]:\n- if the bins are small, the variance of the average label is high, and this biases the estimate up.\nEven with thousands of samples and perfect calibration, [2] shows this can produce large overestimates of miscalibration.\n- if the bins are large, miscalibrations can cancel out, biasing the estimate down.\n[3] dramatizes this by showing that in the limiting case of 1 bin, guessing 0% or 100% randomly at the population base rate gives ECE of 0.\n\n2. **ECE doesn't combine with measures of discrimination**\n\nIt is commonly recommended to report both ECE and AUC-ROC which, as mentioned in Section 3, provides a pure measure of discrimination with no calibration component.\nUnfortunately, there is no principled way to combine the two numbers to make a joint decision.\nFor example, subtracting ECE from AUC-ROC has no interpretation, nor can ECE be combined with other discrimination measures.\nTable 1 shows an example where this means we cannot decide which subgroup has better performance.\nThe calibration and discrimination components of Bounded DCA log score can, in contrast, be easily combined.\n\n## Brier Score\n\nComparisons that use a metric ought to indicate when the model does better on one population than another on average across the clinically relevant range of prevalences.\nThis requires us to restrict the range of prevalences, which Brier score does not do.\nOur contribution is to:\na) show when and why scoring rules can be interpreted as distributions of net benefit over varying prevalences.\nb) show how to restrict the range of prevalences using a simple analytic formula.\nc) show why the clipping the log score is more useful than clipping the Brier score.\n\n1. **Why Clipping?**\n\nThe two concrete differences between Brier score and our metric are that our metric introduces bounds on the prevalence and that it also allows for asymmetric costs.\nClincians typically do not know actual deployment prevalence at the time they choose a model, but they are often able to place useful constraints on prevalence, such as \"less than half.\" See Table 3 and Table 4 for a real clinical illustration where this is important.\n\n2. **Why DCA Log Score and not DCA Brier score?**\n\nThis overlaps with Question 4, so we will address it in greater detail there.\n\n# Question 4: Can the brier score be used in a similar context to the log cross entropy, as the brier score is also a proper scoring rule that can be decomposed into sharpness and calibration?\n\nYes, the Brier score can also be clipped, but we show that cross-entropy makes more sense in this context.\n\n1. Cross-entropy reflects what we mean by an uncertainty interval that covers orders of magnitude, as we discuss in section 4.2 (lines 290 to 295).\n\nTo give a concrete example, the range of syphilis prevalence in US states is between 2 in 1,000 and 2 in 100,000, depending on the state.\nWe believe that the typical practitioner would expect the median of this interval to be 2 in 10,000.\nBrier score, by contrast, favors prevalences close to 1/2, so the median of the interval will be 1 in 1,000.\n\n2. The formulation of cross-entropy is readily adapted to a restricted range of prevalences under asymmetric costs.\n As we show in Theorem D.7, the Brier score can be adapted to restrict the range of prevalences if symmetric costs are assumed.\nAdding asymmetric costs produces an unwieldy expression, reducing intepretability.  We will include it in the appendix for completeness.\n\n# Question 2: How can we apply the new evaluation metric to real clinical data, and how can the results be interpreted in a meaningful way?\n\nWe appreciate the reviewer prompting us to clarify how we interpret ranking differences among metrics for real clinical data.\nWe have applied the evaluation metric to real mortality predictions from real ICU data [4] in Section 5 and added new analyses in Tables 1 to 4 of this rebuttal.\nWhen it is possible to a detailed analysis, we think visualizing accuracy as a function of prevalence provides intuition for why metrics differ.\nHowever, we believe the mark of a good scalar metric is that we can simply compare magnitudes; in the example in Section 5, the DCA log score is simply the average accuracy across the clinically relevant range of prevalences, and the examples show that comparing magnitudes is valid.\n\n# Question 3: How would evaluation in real clinical data differ between existing measures of calibration and the proposed clipped log metric?\n\nWe thank the reviewer for prompting us to explain why the bounded DCA log score gives different results on the ICU data from existing metrics.\nThe introduction provides concrete examples:\n\n- When one subgroup has higher calibration and another has higher discrimination, it is possible to answer the question \"which is doing better?\" by a linear combination of the scores.\n- When subgroups have different prevalence, it is still possible to identify *where* the classifier is performing better.\n- Evaluation can be restricted to clinically meaningful ranges of prevalence.\n\n## Conclusion\n\nWe appreciate the reviewer's recognition of the clinical motivations for designing a new scoring rule approach to capture distribution shift under asymmetric costs.\nSimilarly, we are grateful for the encouragement to expand our empirical validation to show real world examples in which bounded DCA log score outperforms the most widely used metrics (i.e. ECE, AUC-ROC, Accuracy, Brier score and cross-entropy / NLL).\n\n\n[1] Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., & Tran, D. (2019, June). Measuring calibration in deep learning. CVPR Workshops.\n\n[2] Roelofs, R., Cain, N., Shlens, J., & Mozer, M. C. (2022, March). Mitigating bias in calibration error estimation. AISTATS (Vol. 151, pp. 4036-4054).\n\n[3] Ferrer, L., & Ramos, D. (2024). Evaluating posterior probabilities: Decision theory, proper scoring rules, and calibration. arXiv:2408.02841.\n\n[4] Johnson, A., Pollard, T., Badawi, O., & Raffa, J. (2021). eICU collaborative research database demo (version 2.0.1). PhysioNet."}, {"review_id": "fjIjBlgzRU", "reviewer": "Reviewer_SEhd", "summary": "This paper is a theoretical paper motivated by clinical problems. In particular, it proposed an evaluation metric for binary classifiers in clinical decision support, explicitly addressing three key clinical concerns: calibration, robustness to label shift, and asymmetric error costs. Leveraging the Schervish representation of proper scoring rules, the authors derive a cost-sensitive, clipped variant of cross-entropy—termed the DCA log score—that averages performance over a bounded range of plausible class prevalences. A subgroup analysis using the eICU dataset illustrates how traditional metrics may obscure disparities that the proposed framework reveals.", "questions": "1. Comparative Evaluation: Have you conducted empirical comparisons between your proposed DCA log score and existing cost-sensitive or calibration-aware metrics such as expected calibration error (ECE)? It would be helpful to understand how your metric performs in model selection or ranking tasks relative to these established alternatives.\n\n2. Figure Clarity (Section 5): The figure in Section 5 is difficult to interpret. Specifically, Subfigure 1 includes two labels marked “B,” and Subfigure 2 contains two “A”s and two “B”s. The current caption does not adequately explain what each label refers to.", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. High Relevance: The paper addresses an important and underexplored challenge in medical ML—namely, that commonly used metrics (e.g., accuracy and AUC-ROC) often fail to align with key clinical priorities such as calibration under distribution shift and sensitivity to asymmetric error costs. \n2. Strong Theoretical Foundation: The authors build on the Schervish representation to derive a bounded variant of the DCA log score. This derivation effectively bridges calibration and cost-sensitive thresholding under uncertainty, yielding a theoretically grounded and interpretable evaluation framework.\n\nWeaknesses:\n1. Limited Novelty Relative to Prior Work: While the clinical framing is timely and well-motivated, much of the theoretical basis—including proper scoring rules, label shift correction, and cost-sensitive evaluation—has been explored in prior literature. The main contribution lies in adapting and synthesizing these ideas for the clinical evaluation context.\n2. Narrow Empirical Scope: The empirical analysis is limited to a single illustrative case. While the example is helpful, more extensive benchmarking across multiple datasets and models would provide stronger evidence for the practical advantages of the proposed metric over standard alternatives.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns.", "rebuttal": "# Introduction: New Analysis and Major Commitments for Revision\n\nWe thank the reviewers for thoughtfully highlighting the value of more explicit comparisons of our proposed metric to popular alternatives on real data.  To that end, we have extended the analysis from Section 5 on the performance of the Apache IV mortality prediction algorithm in different subgroups of the publicly available eICU dataset, and will include the following tables in the revised draft.\n\n| Metric | Caucasian | African-American |\n|--------|-----------|------------------|\n| Bounded DCA Log Score (calibration-only) | **0.999** | 0.927 |\n| ECE | **0.829** | 0.889 |\n| Bounded DCA Log Score (discrimination-only) | 0.957 | **0.973** |\n| AUC-ROC | 0.868 | **0.907** |\n| Bounded DCA Log Score | **0.956** | 0.900 |\n| ECE & AUC-ROC | ? | ? |\n\n*Table 1: Calibration and Discrimination for Caucasian and African-American Patients*\n\nBoth commonly used and proposed calibration measures agree that the model has better calibration for African-American patients than Caucasian, but worse discrimination.\nHowever, the calibration and discrimination components for Bounded DCA Log Score can be linearly combined, while there is no principled way to combine AUC-ROC with ECE to reach an overall judgment.\n\n| Metric | Caucasian | African-American |\n|--------|-----------|------------------|\n| Accuracy Curve@1% Prevalence | **0.990** | 0.978 |\n| Accuracy Curve@5% Prevalence | **0.953** | 0.896 |\n| Accuracy Curve@10% Prevalence | **0.912** | 0.793 |\n| Bounded DCA Log Score | **0.956** | 0.900 |\n| Accuracy | 0.922 | **0.961** |\n\n*Table 2: Accuracy for Caucasian and African-American patients at varying prevalences*\n\nThe most powerful analysis technique here is to graph accuracy at each prevalence and compare the two entire curves visually. As the table suggests, Apache IV on the public eICU data performs better for Caucasian patients than for African-American patients at any given prevalence.\nComparing two Bounded DCA Log Scores averages the heights of these curves and the scalar summary preserves their ranking.\nThe empirical accuracy for each subgroup is measured at different prevalences.\n\n| Metric | Male | Female |\n|------------|------|--------|\n| Accuracy@1% | 0.989 | 0.990 |\n| Accuracy@5% | 0.948 | 0.954 |\n| Accuracy@25% | 0.818 | 0.814 |\n| Accuracy@75% | 0.846 | 0.832 |\n| Accuracy@95% | 0.953 | 0.953 |\n| Accuracy@99% | 0.990 | 0.990 |\n\n*Table 3: Accuracy for Male and Female Patients*\n\nApache IV performs better for female patients than male patients at low mortality prevalences, worse for roughly balanced mortality, and about the same at high mortality prevalences. However, only the low prevalence ranges are clinically relevant.\n\n| Metric | Male | Female |\n|--------|------|--------|\n| Brier | 0.062 | 0.062 |\n| Cross-Entropy / NLL | 0.214 | 0.218 |\n| Bounded DCA Log Score | 0.917 | 0.926 |\n\n*Table 4: Scoring Rules for Male and Female Patients*\n\nAlthough unbounded Brier score and cross-entropy are related to integrals of net cost, and are therefore related to net benefit, they give misleading answers because they put too much weight at high, clinically unrealistic prevalences.\n\n\n\n# Specific Reviewer Questions and Replies\n\n## Limited Novelty Relative to Prior Work:\n> While the clinical framing is timely and well-motivated, much of the theoretical basis—including proper scoring rules, label shift correction, and cost-sensitive evaluation—has been explored in prior literature.\nThe main contribution lies in adapting and synthesizing these ideas for the clinical evaluation context.\n\nWe appreciate the reviewer's recognition of this work's progress towards synthesizing proper scoring rules, evaluating classifiers under unknown label shift, and providing frameworks for cost-sensitive evaluation.\nWhile label shift correction and cost-sensitive learning are well developed fields, evaluation methods to adapt to them are less widespread; for example, a recent review of 173 papers using cost-sensitive learning in medicine found less than 10\\% of papers actually used existing cost-sensitive metrics [1].\n\nWe wish to emphasize that, to our knowledge, this is the first framework to combine these three dimensions into a unified, practically deployable metric, directly addressing the persistent challenge in clinical ML of actionable model selection under label shift and asymmetric costs.\n\n| Metric | Domain Adaptation Failure | Cost-Sensitive Evaluation Failure | Decomposability Failure | Notes |\n|--------|---------------------------|-----------------------------------|-------------------------|-------|\n| **Accuracy** | Does not handle subgroup prevalence shifts. | Assumes symmetric costs (FP = FN) | Does not reflect calibration. Very little information on discrimination. | Simpson's paradox effects when subgroup prevalences differ |\n| **AUC-ROC** | Invariant to prevalence | Range of costs chosen by model, not clinicians | Pure discrimination, no calibration | Can't combine with (e.g.) ECE |\n| **Expected Calibration Error (ECE)** | Domain-specific adaptation unclear | No cost sensitivity | Pure calibration, no discrimination | Can't combine with (e.g.) AUC-ROC |\n| **Brier Score** | Unrealistic prevalence ranges (e.g. half of weight above 50% mortality)| Cannot combine asymmetric costs with imbalanced classes | Decomposes into calibration and discrimination | Misleading at clinical prevalences |\n| **Cross-Entropy / NLL** | Unrealistic prevalence ranges (e.g. half of weight above 50% mortality) | Cannot combine asymmetric costs with imbalanced classes | Decomposes into calibration and discrimination | Misleading at clinically relevant prevalences |\n| **Net Benefit** | Requires known prevalence | Requires known costs | No decomposition into calibration and discrimination | Requires exact knowledge |\n\nWe will clarify the practical implications of this synthesis in the revision; we will also expand Section 5 to include the additional empirical tables and guided figure walkthroughs outlined in the introduction.\n\n## Narrow Empirical Scope:\n> The empirical analysis is limited to a single illustrative case.\nWhile the example is helpful, more extensive benchmarking across multiple datasets and models would provide stronger evidence for the practical advantages of the proposed metric over standard alternatives.\n\nWe have added new analyses of the Apache IV mortality prediction (Tables 1-4 above).\nThis real clinical dataset on the paradigmatic task of ICU mortality serves to illustrate a typical medical task with low but subgroup-heterogeneous prevalence.\nBy carefully studying this dataset, we were able to find subtle but important effects, such as the Simpson's paradox-style effect of subgroup prevalence on measured subgroup accuracy and  Apache IV's stronger performance for women at clinically relevant prevalences.\nImportantly, these effects were masked by worse performance at high prevalence, and we show that our metrics captured those.\nThe new analysis finds further examples where existing metrics make the wrong decision.\nWe commit to including tables and clear descriptions of these analyses in the next version.\n\n## Comparative Evaluation:\n> Have you conducted empirical comparisons between your proposed DCA log score and existing cost-sensitive or calibration-aware metrics such as expected calibration error (ECE)? It would be helpful to understand how your metric performs in model selection or ranking tasks relative to these established alternatives.\n\nWe thank the reviewer for suggesting these improvements\n\n-  We've added ECE in the new analysis (Table 1, above); this demonstrates a clear clinical example where the comparison in ECE gives one answer, but AUC-ROC gives a different one.\nThe critical advantage of DCA log score is that the calibration and discrimination components can be linearly combined, producing a single scalar score to guide decisions, where ECE cannot be combined with (e.g.) AUC-ROC.\n- Table 2 shows a real clinical example where the classifier performs better on one subgroup than another at any relevant prevalence, but measuring accuracy gives the wrong answer.\nThis occurs because prevalences in the evaluation data are different from prevalences in the training data.\n- Table 3 and 4 show a real clinical example where one subgroup has better performance at clinically relevant prevalences, but the effect reverses at unrealistically high prevalences.\nThis leads the unbounded Brier score and cross-entropy to give the wrong results.\n\n## Figure Clarity (Section 5):\n> The figure in Section 5 is difficult to interpret.\nSpecifically, Subfigure 1 includes two labels marked “B,” and Subfigure 2 contains two “A”s and two “B”s.\nThe current caption does not adequately explain what each label refers to.\n\nWe appreciate the reviewer pointing out this error -- we will use unique labels in the next revision, rather than repeating letters in different colors.\nFurther, we will revise the caption to more clearly walk through the quantities represented by various vertical intervals on the graph.\n\n## Conclusion\n\nWe appreciate the reviewer's recognition of the clinical motivations for designing a new scoring rule approach to capture distribution shift under asymmetric costs.\nTheir encouragment has helped us commit to expanding Section 5 with further empirical comparison of the Bounded DCA Log Score to the most widely adopted metrics, focusing on the narrative description of where and how the metric captures different aspects of model performance, and thereby justifying its use.\n\n[1] Araf, I., Idri, A., & Chairi, I. (2024, March). Cost-sensitive learning for imbalanced medical data: A review. Artificial Intelligence Review, 57(4), 80."}, {"review_id": "MtGjOwc1hK", "reviewer": "Reviewer_syGE", "summary": "Widely used performance scores like accuracy and AUC-ROC often fail to adequately capture key clinical priorities. The authors propose an evaluation framework that explicitly incorporates calibrated thresholded classifiers, leveraging uncertainty for in-class prevalences and domain-specific cost asymmetries. They adapt a model originally developed for weather prediction, based on Schervish representation. They extend this framework to accommodate label shift and asymmetric costs, and to average cost-sensitive metrics over a bounded range of class balances.", "questions": "### Section 5 - application to subgroup decomposition\n- This section lacks a brief description of the eICU dataset beyond the task of predicting in-hospital mortality using APACHE IV scores. I recommend including contextual information about the clinical setting, the types of clinical variables available, what the APACHE IV scores represent, the number of patients, patient demographics, the class distribution, etc.\n- It is unclear how the evaluation metrics were obtained, as the authors do not describe the model training process. Key experimental details are missing.\n- Typically, AUC-ROC curves plot the true positive rate (TPR) on the y-axis and the false positive rate (FPR) on the x-axis. However, the authors use “Accuracy” on the y-axis and “Positive Class Fraction” on the x-axis in their figures. This choice should be clarified. Additionally, the authors state that “(C) African American patients (orange) have noticeably better AUC-ROC than Caucasian patients (blue),” but the figures appear to show the opposite.\n- The analysis of the eICU dataset is very brief and would benefit from further elaboration.\n\n**Discussion**: It is unclear how the proposed method would perform in real-world clinical settings, especially when the class distribution at deployment is unknown. Based on lines 369–371, it appears that prior knowledge of prevalence is required before deployment, which is typically not available in practice.\n\n**Code Availability**: The authors provide the code in the Supplementary Material, but it would be more practical and reproducible to host it in an anonymized GitHub repository. Additionally, the code lacks documentation, such as a README file, which makes it harder to understand and use. However, the code itself is clean and well structured.\n\nMinor comments:\n- Punctuation at the end of paragraph header is missing in some cases, and consistently in the Appendix.\n- Figures in page 8 are missing numbers, and they should be referenced in the manuscript.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and structured. However, I find confusing that there is a \"Related work\" section, but background/related work spans both Sections 3 and 4.\n- The methodology is well formulated, with justifications appropriately referenced and detailed in the appendix.\n- The discussion section is well organized, clearly presenting limitations and outlining directions for future work.\n\nWeaknesses:\n- Section 5 lacks important experimental details, including dataset descriptions and model training procedures. The discussion of the results is very brief.\n- The proposed method appears to be designed for binary classification. However, in medical contexts, this may be too limiting. Risk stratification often requires multiple categories (e.g., low, medium, high) to prioritize treatment. Is the framework restricted to binary classification, or could it be extended to multi-class settings?\n- The method assumes that prevalence is known prior to deployment, which is often not the case in real-world applications.", "was_revised": true, "final_justification": "I am satisfied with the authors’ responses regarding binary classification, prevalence, and their extended evaluation.\n\nMost of my concerns have been addressed, and after considering the other reviewers' comments along with the authors’ rebuttal, I have decided to increase my score.", "rebuttal": "# Introduction: New Analysis and Major Commitments for Revision\n\nWe thank the reviewers for thoughtfully highlighting the value of more explicit comparisons of our proposed metric to popular alternatives on real data.  To that end, we have extended the analysis from Section 5 on the performance of the Apache IV mortality prediction algorithm in different subgroups of the publicly available eICU dataset, and will include the following tables in the revised draft.\n\n| Metric | Caucasian | African-American |\n|--------|-----------|------------------|\n| Bounded DCA Log Score (calibration-only) | **0.999** | 0.927 |\n| ECE | **0.829** | 0.889 |\n| Bounded DCA Log Score (discrimination-only) | 0.957 | **0.973** |\n| AUC-ROC | 0.868 | **0.907** |\n| Bounded DCA Log Score | **0.956** | 0.900 |\n| ECE & AUC-ROC | ? | ? |\n\n*Table 1: Calibration and Discrimination for Caucasian and African-American Patients*\n\nBoth commonly used and proposed calibration measures agree that the model has better calibration for African-American patients than Caucasian, but worse discrimination.\nHowever, the calibration and discrimination components for Bounded DCA Log Score can be linearly combined, while there is no principled way to combine AUC-ROC with ECE to reach an overall judgment.\n\n| Metric | Caucasian | African-American |\n|--------|-----------|------------------|\n| Accuracy Curve@1% Prevalence | **0.990** | 0.978 |\n| Accuracy Curve@5% Prevalence | **0.953** | 0.896 |\n| Accuracy Curve@10% Prevalence | **0.912** | 0.793 |\n| Bounded DCA Log Score | **0.956** | 0.900 |\n| Accuracy | 0.922 | **0.961** |\n\n*Table 2: Accuracy for Caucasian and African-American patients at varying prevalences*\n\nThe most powerful analysis technique here is to graph accuracy at each prevalence and compare the two entire curves visually. As the table suggests, Apache IV on the public eICU data performs better for Caucasian patients than for African-American patients at any given prevalence.\nComparing two Bounded DCA Log Scores averages the heights of these curves and the scalar summary preserves their ranking.\nThe empirical accuracy for each subgroup is measured at different prevalences.\n\n| Metric | Male | Female |\n|------------|------|--------|\n| Accuracy@1% | 0.989 | 0.990 |\n| Accuracy@5% | 0.948 | 0.954 |\n| Accuracy@25% | 0.818 | 0.814 |\n| Accuracy@75% | 0.846 | 0.832 |\n| Accuracy@95% | 0.953 | 0.953 |\n| Accuracy@99% | 0.990 | 0.990 |\n\n*Table 3: Accuracy for Male and Female Patients*\n\nApache IV performs better for female patients than male patients at low mortality prevalences, worse for roughly balanced mortality, and about the same at high mortality prevalences. However, only the low prevalence ranges are clinically relevant.\n\n| Metric | Male | Female |\n|--------|------|--------|\n| Brier | 0.062 | 0.062 |\n| Cross-Entropy / NLL | 0.214 | 0.218 |\n| Bounded DCA Log Score | 0.917 | 0.926 |\n\n*Table 4: Scoring Rules for Male and Female Patients*\n\nAlthough unbounded Brier score and cross-entropy are related to integrals of net cost, and are therefore related to net benefit, they give misleading answers because they put too much weight at high, clinically unrealistic prevalences.\n\n\n\n# Specific Reviewer Questions and Replies\n\n## Question 0. The proposed method appears to be designed for binary classification. However, in medical contexts, this may be too limiting. Risk stratification often requires multiple categories (e.g., low, medium, high) to prioritize treatment. Is the framework restricted to binary classification, or could it be extended to multi-class settings?\n\nThe framework we present is only valid in binary classification settings.\nThe current binary scoring rule literature relies on properties of extremal convex functions which do not generalize to higher dimensions [4].\nIf labels are discrete with total ordering (e.g. low, medium, high), Brier and log scores can be extended using ranked probability scores [1].\nIf labels form a metric space, ideas like Hyvarinen's rule [2] and continuous ranked probability score [3] apply.\nWe are studying other approaches.\n\n## Question 1. This section lacks a brief description of the eICU dataset beyond the task of predicting in-hospital mortality using APACHE IV scores.\n\n> I recommend including contextual information about the clinical setting, the types of clinical variables available, what the APACHE IV scores represent, the number of patients, patient demographics, the class distribution, etc.\n\nWe will include more detail on APACHE IV [5], the eICU data [6] [7], and descriptive statistics for both datasets in the next draft, as mentioned in our new analysis section.\n\n## Question 2. It is unclear how the evaluation metrics were obtained, as the authors do not describe the model training process. Key experimental details are missing.\n\nTo make reproducibility simple, we did not train new models.\nInstead, we used the Apache IV mortality predictions recorded in the eICU dataset itself; these are available for direct download from the PhysioNet website (links are not allowed in rebuttals this year).\n\n## Question 3. Figure label issues\n\n> Typically, AUC-ROC curves plot the true positive rate (TPR) on the y-axis and the false positive rate (FPR) on the x-axis.\nHowever, the authors use \"Accuracy\" on the y-axis and \"Positive Class Fraction\" on the x-axis in their figures.\nThis choice should be clarified.\nAdditionally, the authors state that \"(C) African American patients (orange) have noticeably better AUC-ROC than Caucasian patients (blue),\" but the figures appear to show the opposite.\n\nAs we will indicate more clearly in the revision, these are not ROC plots.\nWe will also more explicitly explain what is being shown.\n\n## Question 4. The analysis of the eICU dataset is very brief and would benefit from further elaboration.\n\nWe will expand the analysis, including the four tables mentioned in the introduction and providing clearer exposition of the figures.\n\n## Question 5 (Discussion).\n\n> It is unclear how the proposed method would perform in real-world clinical settings, especially when the class distribution at deployment is unknown.\nBased on lines 369–371, it appears that prior knowledge of prevalence is required before deployment, which is typically not available in practice.\n\nBounded DCA log score is intended for use when prevalence can be bounded *but not precisely specified.*\nIn the limit where bounds coincide, it becomes net benefit.\nHowever, the dearth of exact prevalence knowledge appears to be why net benefit-based approaches have not been more widely adopted.\nWe show that non-clipped log score and Brier score are equivalent to integrating over the full interval between 0 and 1.\nHowever, in clinical settings, loose bounds are often available.\nFor example, in this dataset, we can bound ICU patients' mortality between one in 100 and one in 10. As we show in Tables 3 and 4, those bounds are sufficient to improve classifier evaluation clinical utility.\n\n## Question 6 (Code Availability).\n\n> The authors provide the code in the Supplementary Material, but it would be more practical and reproducible to host it in an anonymized GitHub repository. Additionally, the code lacks documentation, such as a README file, which makes it harder to understand and use. However, the code itself is clean and well structured.\n\nWe appreciate the reader's advice.\nWe will add a README file with explanations of the code's APIs, and host the code on GitHub for the camera-ready version.\n\n## Punctuation at the end of paragraph header is missing in some cases, and consistently in the Appendix.\n\nWe appreciate the correction; we missed the inconsistent punctuation in \\paragraph{} tags entirely in our pre-submission reviews.\n\n## Figures in page 8 are missing numbers, and they should be referenced in the manuscript.\n\nWe are grateful for the reviewer's attention to detail, particularly their highlighting of specific issues in the formatting of Section 5.\nAs mentioned in the introduction, we will revise Section 5 in our next draft, including adding the tables, simplifying the diagrams, and better describing their contents.\n\n## Conclusion\n\nWe appreciate the reviewer's recognition of the clinical motivations for designing a new scoring rule approach to capture distribution shift under asymmetric costs.\nThey also encouraged us to clarify that our approach relies on being able to bound prevalence at evaluation time, and they suggested that we expand and clarify our empirical validation with more real world examples in which bounded DCA log score outperforms the most widely used metrics.\nWe thank the reviewer for these suggestions, which we have implemented as described above.\n\n\n[1] Epstein, E. S. (1969). A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology and Climatology, 8(6), 985-987.\n\n[2] Hyvärinen, A. (2005). Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6, 695-709.\n\n[3] Matheson, J. E., & Winkler, R. L. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087-1096.\n\n[4] Johansen, S. (1974). The extremal convex functions. Mathematica Scandinavica, 34(1), 61-68.\n\n[5] Zimmerman, J. E., Kramer, A. A., McNair, D. S., & Malila, F. M. (2006, May). Acute physiology and chronic health evaluation (APACHE) IV: Hospital mortality assessment for today's critically ill patients. Critical Care Medicine, 34(5), 1297-1310.\n\n[6] Pollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018). The eICU collaborative research database, a freely available multi-center database for critical care research. Scientific Data, 5, 180178.\n\n[7] Johnson, A., Pollard, T., Badawi, O., & Raffa, J. (2021). eICU collaborative research database demo (version 2.0.1). PhysioNet."}]

Author rebuttal:

Meta-review: **(a) Paper summary**  
The paper introduces a bounded DCA log score (clipped cross-entropy) that combines calibration and discrimination into one interpretable metric, supports asymmetric costs, and averages results over clinician-specified prevalence ranges, building on proper scoring rules.

**(b) Strengths**  
The method is principled, practical, and easy to compute, while addressing calibration under shift and asymmetric costs. The authors added subgroup analyses on eICU data with comparisons to AUC-ROC, ECE, Brier, and NLL, and showed useful accuracy-vs-prevalence diagnostics.

**(c) Weaknesses**  
The novelty is mostly in combining and adapting existing ideas, and the experiments are still somewhat limited. The work is restricted to binary outcomes, and there is an ongoing debate about using one combined metric versus reporting separate scores, plus some minor clarity issues.

**(d) Reasons for decision**  
Even with modest novelty and limited scope, the paper provides a clear, theory-based evaluation method that matches clinical needs and gives an actionable scalar for model selection. The added experiments, clarified proofs, and commitments for code and clearer presentation address the main concerns, so I recommend acceptance.

**(e) Discussion period**  
Reviewers questioned novelty, empirical breadth, clarity of data and figures, prevalence assumptions, and the use of a single metric. The authors clarified proofs, expanded analyses with more tables and comparisons, explained prevalence bounds, and promised code release and figure fixes. Most reviewers raised their scores, so I consider the issues resolved enough for acceptance.
Decision: Accept (poster)