Title: Estimating Model Performance Under Covariate Shift Without Labels
Venue: NeurIPS 2025
Accepted: yes
Keywords: unsupervised performance estimation

confidence-based performance estimation

OOD error prediction

machine learning

model monitoring

probabilistic methods

covariate shift
Abstract: After deployment, machine learning models often experience performance degradation due to shifts in data distribution. It is challenging to assess post-deployment performance accurately when labels are missing or delayed. Existing proxy methods, such as data drift detection, fail to measure the effects of these shifts adequately. To address this, we introduce a new method for evaluating binary classification models on unlabeled tabular data that accurately estimates model performance under covariate shift and call it Probabilistic Adaptive Performance Estimation (PAPE). It can be applied to any performance metric defined with elements of the confusion matrix. Crucially, PAPE operates independently of the original model, relying only on its predictions and probability estimates, and does not need any assumptions about the nature of covariate shift, learning directly from data instead. We tested PAPE using over 900 dataset-model combinations from US census data, assessing its performance against several benchmarks through various metrics. Our findings show that PAPE outperforms other methodologies, making it a superior choice for estimating the performance of binary classification models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Probabilistic Adaptive Performance Estimator (PAPE), a novel method to estimate binary classification metrics defined using the confusion matrix, by extending the Confidence-based Performance Estimation (CBPE) in the following two ways:\n(i) The proposed method does not need a perfectly calibrated model.\n(ii) The proposed method can adapt to covariate shift.\n\nFirst, the authors consider the relaxed notion of calibration\ncalled $\\alpha$-approximately multicalibration, and they show that if the likelihood ratio belongs to the class of test functions used in the measure of $\\alpha$-approximately multicalibration, the measure maintains the inequality in Eq. (2) before and after covariate shift (Theorem 3.1).\nFurthermore, Theorem 3.2 shows that the use of the $\\alpha$-calibrated score in estimating the metric as in Eq. (4) only suffers bias up to $\\alpha$.\nThen, the paper presents a method that corrects the bias due to covariate shift using importance sampling when estimating the function $c$.\nFinally, the experiment demonstrates the usefulness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper studies an important problem in a challenging scenario commonly encountered, covariate shift.\n- The extension to this setup and the removal of the requirement of a perfectly calibrated model seem relevant and useful.\n- The proposed method outperforms the other methods in the experiment.\n\n### Weaknesses\n- The message of the paper is a bit confusing to me. The paper claims that it works on any metrics defined using the confusion matrix, but Definition 3.1, Definition 3.2, Theorem 3.1, and Theorem 3.2 all only concern very specific cases.\n- There is no analysis of estimation errors, despite this claim \"We provide theoretical guarantees for the estimation quality of PAPE and derive bounds for its estimation error under approximate calibration.\" Theorem 3.2 is probably related to the bias, but apart from that, it does not provide an analysis on the estimation error.\n- In particular, the likelihood ratio estimation being seemingly the statistically hardest part, it may be important to study its impact of the error on the quality of the estimation of the performance metric. However, this error is assumed to be zero in the presented analyses.\n- I found some of the descriptions unclear, especially for the entire proposed algorithm, composable metrics, and the IW method. I do not know if the reader would be able to reproduce the experiments.\n- There is no (intuitive) explanation on why we can expect the dataset to satisfy the covariate shift assumption. It would be nice if there were simulations in which we know that the assumption holds.\n- I could not find which part of the paper addresses this matter: \"Furthermore, PAPE models the full distribution of the estimated metric, which can be leveraged to produce valid confidence intervals for the estimates\".\n\nQUESTIONS:\n- Is the dataset expected to satisfy the covariate shift assumption?\n- Which part of the paper does this correspond to?: \"Furthermore, PAPE models the full distribution of the estimated metric, which can be leveraged to produce valid confidence intervals for the estimates\".\n- l.42, \"PAPE is provably robust against covariate shift\": what is the meaning of robustness here?\n\nLIMITATIONS:\n- The work assumes that the image of $f$ is a countable set.\n- The work is limited to binary classification..", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

comments: SUMMARY:
This paper proposed PAPE, a method to estimate the performance of a binary classifier under covariate shift. Their method relies on first calibrating the model for the target distribution (using the likelihood ratio) and then using the calibrated classifiers to estimate performance  on the target distribution.

STRENGTHS AND WEAKNESSES:
Strength
- This paper proposes an effective method for performance estimation under covariate shift when no labels for the target distribution. 

Weaknesses
- It's well know that things break when you're trying to estimate a likelihood ratio for high-dimensional x. How robust is PAPE to an increase in dimensionality? My guess is that it is more robust than IW but less than CBPE.
- It's not clear why PAPE is better then other method that rely on the likelihood ratio estimation. For example, why is PAPE supposed to be better than IW? 
- The method is simple and require few changes from previous approaches (this could be seen as a strength if we have a good response for the previous point).

QUESTIONS:
- Why do you need the image of the classifier f to be countable?
- Is it hard to extend this idea to multi class classification tasks? Maybe putting a hint on the paper could be useful for practitioners?
- I think some references are missing, e.g., this paper https://arxiv.org/pdf/2212.02742.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a method, PAPE, for evaluating binary classification models on unlabeled tabular data that estimates model performance under covariate shift, which can be applied to any performance metric defined with elements of the confusion matrix. It tests PAPE using a dataset from US census data and shows it outperforms other methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper provides theoretical guarantees for the method and derives bounds for its estimation error under approximate calibration. They also use experiments to show PAPE outperforms previous methods across metrics.\n\n**Weaknesses**\n\n- The paper only considered one dataset, which is the US census data. Although different combinations of the data are provided, the single source raises questions about the generalizability.\n- Theoretical gap: The theorem claims that the method is applicable to composable metrics, and for metrics that are not composable, the bound is not guaranteed to hold. Although the empirical results show that PAPE can be used for metrics like F1, this finding isn’t supported theoretically, especially considering the paper’s claim that the method can be applied to \"any classification metric defined with elements of the confusion matrix\". This statement lacks sufficient support. Besides, only Accuracy, AUROC, and F1 metrics are considered in the evaluation, the claim of “any” might be overstated, especially since the theorem only applies to composable metrics.\n- The covariate shift assumption (ps(y|x) = pt(y|x)) is highly restrictive. In real-world deployment, we can encounter concept drift alongside covariate shift. How can practitioners verify that ps(y|x) = pt(y|x) in real applications? What happens when this assumption is violated?\n- The proposed method is closely related to the previous method CBPE. A clearer illustration of CBPE as preliminary, as well as highlighting where the proposed new method is different from it, would be helpful in the method section (section 3).\n- There are some typos in the paper, like line 188: “composable metric with with” → “composable metric with”.\n\nQUESTIONS:\nPlease see the weaknesses above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

comments: SUMMARY:
This work proposes an evaluation of binary classifier performance under conditions of covariance shift. The suggested probabilistic approach builds on a previously established method (CBPE). It extracts data from the shifted distribution and fits a calibration mapping to this distribution. One of the benefits of this approach is its applicability to any model that uses a confusion matrix as an evaluation metric. The method has been tested on standard datasets and compared with existing techniques.

STRENGTHS AND WEAKNESSES:
he method has been tested on standard datasets and compared with existing techniques. The study is clearly presented, with thorough mathematical explanations, and demonstrates improvements over prior work. There are a few suggestions to further refine the work

1) A discussion on how the proposed work is different from CBPE must be presented in detail. This will allow the reader to ascertain the novelty of the work

2) What could be the possible reason for a minor improvement in F1 as compared to accuracy and AUROC (Fig 2)?  A discussion could be added.

QUESTIONS:
Please see comments above

LIMITATIONS:
Limitations of the proposed approach is presented
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02