Title: Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs
Venue: NeurIPS 2025
Accepted: yes
Keywords: binary classification

calibration

label shift

cost-sensitive evaluation

proper scoring rules
Abstract: Machine learning-based decision support systems are increasingly deployed in clinical settings, where probabilistic scoring functions are used to inform and prioritize patient management decisions.
However, widely used scoring rules, such as accuracy and AUC-ROC, fail to adequately reflect key clinical priorities, including calibration, robustness to distributional shifts, and sensitivity to asymmetric error costs.
In this work, we propose a principled yet practical evaluation framework for selecting calibrated thresholded classifiers that explicitly accounts for uncertainty in class prevalences and domain-specific cost asymmetries.
Building on the theory of proper scoring rules, particularly the Schervish representation, we derive an adjusted variant of cross-entropy (log score)  that averages cost-weighted performance over clinically relevant ranges of class balance.
The resulting evaluation is simple to apply, sensitive to clinical deployment conditions, and designed to prioritize models that are both calibrated and robust to real-world variations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWidely used performance scores like accuracy and AUC-ROC often fail to adequately capture key clinical priorities. The authors propose an evaluation framework that explicitly incorporates calibrated thresholded classifiers, leveraging uncertainty for in-class prevalences and domain-specific cost asymmetries. They adapt a model originally developed for weather prediction, based on Schervish representation. They extend this framework to accommodate label shift and asymmetric costs, and to average cost-sensitive metrics over a bounded range of class balances.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well written and structured. However, I find confusing that there is a \"Related work\" section, but background/related work spans both Sections 3 and 4.\n- The methodology is well formulated, with justifications appropriately referenced and detailed in the appendix.\n- The discussion section is well organized, clearly presenting limitations and outlining directions for future work.\n\nWeaknesses:\n- Section 5 lacks important experimental details, including dataset descriptions and model training procedures. The discussion of the results is very brief.\n- The proposed method appears to be designed for binary classification. However, in medical contexts, this may be too limiting. Risk stratification often requires multiple categories (e.g., low, medium, high) to prioritize treatment. Is the framework restricted to binary classification, or could it be extended to multi-class settings?\n- The method assumes that prevalence is known prior to deployment, which is often not the case in real-world applications.\n\nQUESTIONS:\n### Section 5 - application to subgroup decomposition\n- This section lacks a brief description of the eICU dataset beyond the task of predicting in-hospital mortality using APACHE IV scores. I recommend including contextual information about the clinical setting, the types of clinical variables available, what the APACHE IV scores represent, the number of patients, patient demographics, the class distribution, etc.\n- It is unclear how the evaluation metrics were obtained, as the authors do not describe the model training process. Key experimental details are missing.\n- Typically, AUC-ROC curves plot the true positive rate (TPR) on the y-axis and the false positive rate (FPR) on the x-axis. However, the authors use “Accuracy” on the y-axis and “Positive Class Fraction” on the x-axis in their figures. This choice should be clarified. Additionally, the authors state that “(C) African American patients (orange) have noticeably better AUC-ROC than Caucasian patients (blue),” but the figures appear to show the opposite.\n- The analysis of the eICU dataset is very brief and would benefit from further elaboration.\n\n**Discussion**: It is unclear how the proposed method would perform in real-world clinical settings, especially when the class distribution at deployment is unknown. Based on lines 369–371, it appears that prior knowledge of prevalence is required before deployment, which is typically not available in practice.\n\n**Code Availability**: The authors provide the code in the Supplementary Material, but it would be more practical and reproducible to host it in an anonymized GitHub repository. Additionally, the code lacks documentation, such as a README file, which makes it harder to understand and use. However, the code itself is clean and well structured.\n\nMinor comments:\n- Punctuation at the end of paragraph header is missing in some cases, and consistently in the Appendix.\n- Figures in page 8 are missing numbers, and they should be referenced in the manuscript.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is a theoretical paper motivated by clinical problems. In particular, it proposed an evaluation metric for binary classifiers in clinical decision support, explicitly addressing three key clinical concerns: calibration, robustness to label shift, and asymmetric error costs. Leveraging the Schervish representation of proper scoring rules, the authors derive a cost-sensitive, clipped variant of cross-entropy—termed the DCA log score—that averages performance over a bounded range of plausible class prevalences. A subgroup analysis using the eICU dataset illustrates how traditional metrics may obscure disparities that the proposed framework reveals.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. High Relevance: The paper addresses an important and underexplored challenge in medical ML—namely, that commonly used metrics (e.g., accuracy and AUC-ROC) often fail to align with key clinical priorities such as calibration under distribution shift and sensitivity to asymmetric error costs. \n2. Strong Theoretical Foundation: The authors build on the Schervish representation to derive a bounded variant of the DCA log score. This derivation effectively bridges calibration and cost-sensitive thresholding under uncertainty, yielding a theoretically grounded and interpretable evaluation framework.\n\nWeaknesses:\n1. Limited Novelty Relative to Prior Work: While the clinical framing is timely and well-motivated, much of the theoretical basis—including proper scoring rules, label shift correction, and cost-sensitive evaluation—has been explored in prior literature. The main contribution lies in adapting and synthesizing these ideas for the clinical evaluation context.\n2. Narrow Empirical Scope: The empirical analysis is limited to a single illustrative case. While the example is helpful, more extensive benchmarking across multiple datasets and models would provide stronger evidence for the practical advantages of the proposed metric over standard alternatives.\n\nQUESTIONS:\n1. Comparative Evaluation: Have you conducted empirical comparisons between your proposed DCA log score and existing cost-sensitive or calibration-aware metrics such as expected calibration error (ECE)? It would be helpful to understand how your metric performs in model selection or ranking tasks relative to these established alternatives.\n\n2. Figure Clarity (Section 5): The figure in Section 5 is difficult to interpret. Specifically, Subfigure 1 includes two labels marked “B,” and Subfigure 2 contains two “A”s and two “B”s. The current caption does not adequately explain what each label refers to.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the mismatch between model evaluation and clinical priorities, specially calibration and robustness to distributional shifts. This work characterizes where existing methods, like accuracy and AUROC, fail, and propose an evaluation framework for selecting calibrated thresholded classifiers that can handle bounded distributional shift and domain-specific cost asymmetries. To do so, the authors propose a variant of the cross-entropy score using clipping that averages cost-weighted performance over real ranges of distributional label shift.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The work approaches an important problem and correctly identifies that many metrics that we use today (i.e., accuracy and AUROC) are incompatible with clinical needs. \n2. A solution which allows clinical input is useful, as domain experts provide a better opportunity to know what type of label/distributional shift is expected. \n3. The clipped cross-entropy loss is a nice extension of a commonly used loss, and the provides a nice interpretation with respect to net-benefit over multiple class prevalences. \n4. The authors do a good job surveying past work in limitations of brier score and expected calibration error, which are often used as measures of calibration in the clinical space but can often not be extended to asymmetric costs or ranges of different prevalences. \n\n**Weaknesses** \n\n1. The findings of accuracy and AUROC with respect to misalignment with some clinical objectives is not particularly novel and is well known in the clinical space. This is especially true in the very related space of survival analysis, where recent work has moved beyond simply reporting the C-index (an analog of AUROC) and focused more on calibration. Hence, many of the results showing the limitations of AUROC/accuracy are not particularly interesting findings when placed in the broader context of the clinical space. Instead, the paper should focus more time showing why existing measures of calibration in the clinical space (i.e., NLL, brier score, expected calibration error (ECE)) are not sufficient. \n2. Similar to the point above, the experimental results do not provide significant findings for this paper. As noted, accuracy and AUROC are already well-known in their limitations for calibration and insensitivity to class balance and label shift. It would be more useful to provide an apples to apples comparison of existing calibration metrics in the clinical space (brier score, ECE) with the proposed loss to show how theoretical guarantees manifest and how interpretations differ. The current analysis does not help in showing how the proposed evaluation metric can be applied nor how it should be interpreted in real clinical scenarios.\n\nQUESTIONS:\n1. How do brier score and expected calibration error compare to the proposed evaluation metric towards the goal of measuring calibration and providing an interpretable metric real-world deployment? \n2. How can we apply the new evaluation metric to real clinical data, and how can the results be interpreted in a meaningful way? \n3. How would evaluation in real clinical data differ between existing measures of calibration and the proposed clipped log metric? \n4. Can the brier score be used in a similar context to the log cross entropy, as the brier score is also a proper scoring rule that can be decomposed into sharpness and calibration?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study presents a new method for measuring the performance of ML predictive models for clinical applications. The method emphasizes measuring performance at deployment and aims to keep the method simple and straightforward (for calculation and interpretation).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The study targets an important and critical area in ML for healthcare. \n- Extensive background and context are provided prior to presenting the proposed method (mostly Theorem 4.1). \n- The study also aims to maintain a pragmatic approach by making the proposed way easier to interpret.\n\n**Weaknesses**\n- The study starts with a rather unorthodox introduction, and right off the bat presents “three priorities for clinical purposes.” These are then treated as the foundation of all that follows. While these three are sound and reasonable, they are not an exhaustive list encompassing all issues in this area, as the text seems to imply.\n- While the study targets a very critical and timely issue, the main pitch is hardly news to the practitioners in the field. As the study acknowledges, many have pushed for going beyond accuracy and AUROC, yet those measures remain mainstream. Even using simple calibration or net benefit calculations are less frequently seen. Now, one (including this reviewer) may argue that part of the reason for the current situation is that most studies similar to the present work remain very high-level and skip the practical implications and use cases in favor of the theoretical foundations. The current work may appeal to a limited Nuerips audience, but expanding the real-world scenarios (as presented in section 5) is critical to demonstrate the true value of the method.\n\nQUESTIONS:\nIs Theorem E.1 a proof for 4.1? What's the exact novelty in the formulation, is it only the clipping process or something beyond?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}