Estimating Model Performance Under Covariate Shift Without Labels
Authors: Jakub Białek
Juhani Kivimäki
Wojtek Kuberski
Nikolaos Perrakis
Venue: NeurIPS 2025 poster ()
Keywords: unsupervised performance estimation
confidence-based performance estimation
OOD error prediction
machine learning
model monitoring
probabilistic methods
covariate shift
Primary area: general_machine_learning
TL;DR: The paper proposes an algorithm for estimating the performance of classification models under covariate shift and shows that it outperforms existing methods on multiple real-world datasets.

Abstract: After deployment, machine learning models often experience performance degradation due to shifts in data distribution. It is challenging to assess post-deployment performance accurately when labels are missing or delayed. Existing proxy methods, such as data drift detection, fail to measure the effects of these shifts adequately. To address this, we introduce a new method for evaluating binary classification models on unlabeled tabular data that accurately estimates model performance under covariate shift and call it Probabilistic Adaptive Performance Estimation (PAPE). It can be applied to any performance metric defined with elements of the confusion matrix. Crucially, PAPE operates independently of the original model, relying only on its predictions and probability estimates, and does not need any assumptions about the nature of covariate shift, learning directly from data instead. We tested PAPE using over 900 dataset-model combinations from US census data, assessing its performance against several benchmarks through various metrics. Our findings show that PAPE outperforms other methodologies, making it a superior choice for estimating the performance of binary classification models.

Estimating Model Performance Under Covariate Shift
Without Labels
Jakub Bialek* NannyML NV Belgium
Juhani Kivimäki" University of Helsinki Finland
Wojtek Kuberski NannyML NV Belgium
j jakub@nanymll.omm com
juhani kivimak chelsinki... .f
wojtek@nammmll.com com
Nikolaos Perrakis"
NannyML NV Belgium
nikos@nannyml.comm com
Abstract
After deployment, machine learning models often experience performance degra- dation due to shifts in data distribution. It is challenging to assess post-deployment performance accurately when labels are missing or delayed. Existing proxy meth- ods, such data drift detection, fail measure the effects of these shifts adequately. To address this, we introduce a new method for evaluating binary classification models unlabeled tabular data that accurately estimates model performance under covariate shift and call it Probabilistic Adaptive Performance Estimation (PAPE). It can be applied any performance metric defined with elements of the confusion matrix. Crucially, PAPE operates independently of the original model, relying only on predictions and probability estimates, and does not need any assumptions about the nature of covariate shift, learning directly from data instead. We tested PAPE using over 900 dataset-model combinations from the US census data, assessing its performance against several benchmarks through various met- rics. Our findings show that PAPE outperforms other methodologies, making a superior choice for estimating the performance of binary classification models.
Introduction
The final step in the machine learning (ML) model development process is to evaluate the model on unseen dataset, known fest Successful performance this dataset, indicating potential business value, typically leads to the model's deployment to production. The model's performance on the production data expected to match the performance measured on the assumption often fails due to data distribution shifts that lead to model performance degradation 2 Therefore, ongoing monitoring of the model's 's performance essential ensure it continues to meet expected outcomes. However, measuring real-time performance is challenging in many scenarios because true labels are either unavailable or significantly delayed.
[1] Yet. this
In scenarios where labels are unavailable, performance is commonly estimated using proxy methods that primarily monitor changes in the input data distribution 3 However, such changes in input data often have only minimal effect on the model's actual performance [4], and even when there is a notable effect, might not harmful [5) Additionally, the methods used measure changes in the input distribution typically provide estimates in units that do not correspond to performance metrics.
'These authors contributed equally to this work
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Thus, at best, input data monitoring can offer some qualitative understanding of the performance stability. When distribution shift does occur, these methods do not provide any insight into the direction or magnitude of the impact on performance. The true extent of performance changes only becomes apparent once the labels become available, which may sometimes never happen. Until then, the model might incur significant financial losses [6].
In recent years, a new approach of unsupervised accuracy estimation 7 has produced several methods to estimate model performance directly without access to labels. A notable shortcoming of these methods has been that they focus solely on estimating the aceuracy of a given classifier model. However, accuracy is often not the most appropriate metric for assessing model performance, which has recently motivated the shift of focus to unsupervised performance estimation 88, where estimators should be applicable to any classification metric, not just accuracy. In this paper, we extend this line of work and present Probabilistic Adaptive Performance Estimator (PAPE), novel method for estimating any classification metric that can be defined using the elements of the confusion matrix. PAPE is provably robust against covariate shift. Furthermore, PAPE models the full distribution of the estimated metric, which can be leveraged to produce valid confidence intervals for the estimates.
With these estimates, the impact of changes in model performance on business operations can be quantified, allowing informed decision-making about corrective adaptations. These may include altering how model predictions are utilized in downstream processes, determining if model retraining is required, or whether some fail-safe mechanism should trigger.
The main contributions of this paper are the following:
We introduce PAPE, a novel method for estimating model performance under covariate shift. PAPE can be applied to any classification metric that can be defined with elements of the confusion matrix. Importantly, PAPE autonomously learns from the data without any user input regarding the nature of the covariate shift.
We provide theoretical guarantees for the estimation quality of PAPE and derive bounds for its estimation error under approximate calibration for composable metrics.
We demonstrate PAPE's effectiveness empirically. Our experiments show that PAPE signifi- cantly outperforms all previous methods across all metrics evaluated.
2
Related Work
In recent years, there has been a surge of new suggested methods seeking to solve the unsupervised accuracy estimation problem. Importance Weighting (IW) [9] is a simple but powerful method used in model adaptation [10] that can also used to estimate model performance. IW calculates a likelihood ratio of observing test set input data in production and uses this ratio to estimate performance for production data as a weighted metric calculated for the test set data. An extended version of IW [[T] incorporates knowledge about the differences between test and production data distributions and their impact on performance, but requires the user to specify this information a priori.
A set of methods is based on training an ensemble of models and comparing their predictions 12 13). Other approaches requiring additional training include Contrastive Automated Model Evaluation (CAME) [14], where the model training objective is augmented with contrastive learning component, and reverse testing (RT) [5), which uses the monitored model predictions as labels train a reverse model on production data. The reverse model's 's performance on the test dataset is then assumed to indicate the monitored model's performance on the production data. Whilst these approaches offer promising results, they cannot be applied off the shelf but require additions or alterations in training the model being estimated, and often come with a significant computational overhead.
Another set of methods measures distributional distances between the test and production data distributions [I6] 17 18 19] and either tries to identify the change in performance directly based on this distance or by training light-weight regressor model that is used to map the distance into a change performance. The main challenge of these methods is in translating the measured distance into a meaningful performance estimate, either requiring the creation of multiple synthetic shifted data distributions to train the regressor [16] or resorting to quantifying the shift in performance in terms of some secondary metric such correlation [18].

---

In this work, our main interest lies in methods that utilize the model's confidence in estimating its performance. Average Confidence (AC) [ was originally suggested for Out-of- cuttof-Distributtonn (OoD) detection but has since become the de facto baseline of confidence-bane
estimators [21]. Difference of Confidence (DoC) [16] uses the difference between confidence scores from test to production data and fits regression model that maps this difference to accuracy. Average T'hresholded Confidence (ATC) [22] learns a threshold on confidence scores and estimates accuracy as the fraction of predic- tions that exceed this threshold. Confidence Optimal Transport (COT) [9 measures prediction error as Wasserstein distance between the test set's ett' label distribution and the production set's pseudo-label distribution.
key limitation of all of the above methods that they are developed to estimate only classification accuracy, whereas other classification metrics are often more appropriate and descriptive of model performance. For example, in situations with severe class imbalance, dummy model, which always predicts the majority class, can achieve deceptively high accuracy 2 Recently, a new approach of Confidence-ball based Performance Estimation (CBPE) has been proposed to address this shortcoming of previous estimators.
Since PAPE is essentially an extension of the CBPE method, we describe its key properties here (see also Appendix [A). In CBPE, confidence scores produced by the model for a sample of predictions are used to estimate the elements of the confusion matrix. Each element is treated as random variable following Poisson binomial distribution whose full probability distribution is derived using the confidence scores as parameters. Using the expected values of the distributions as point estimates for the elements of the confusion matrix, these estimates have been shown to be unbiased and consistent under perfect calibration 8 Having estimated the elements of the confusion matrix, one can take any classification metric that can be defined in terms of these elements and derive the distribution of the metric based on the distributions the elements using an appropriate algorithm (8). Again, the expected value of this distribution can then be used as point estimate for that metric. Additionally, one can produce valid confidence intervals for these metrics from the derived probability distributions (8)
Although CBPE a theoretically sound approach to solving the unsupervised performance estimation problem, its main downside is its reliance on model calibration, which is known to deteriorate under certain distributional shifts 24] This phenomenon undermines the usability of CBPE and all other confidence based estimators in way that is currently not well understood. There have even been conflicting reports on whether calibration in confidence-basee estimators is useful or not [16] 22]. In this work, we offer remedy to this problem by augmenting the CBPE approach to make its calibration more robust against distributional shifts, resulting in PAPE, which we will describe in the following section.
3 Methodology
In this section, we present PAPE, a new algorithm for estimating model performance under covariate shift. We begin by describing the setting Section 3.1 Then, in Section we define a-approximate
calibration along with some theoretical insights. Finally, In Section 3.3 we describe the PAPE method
and give a bound for its estimation error for certain metries.
3.1 Unsupervised Performance Estimation Under Covariate Shift
Suppose we have trained a probabilistic binary classifier, where f: X + [0, 1] outputs a confidence score for the positive class and g [] + {0, 1} maps these scores to binary predictions. The classifier trained using data from some source distribution D, = (x y). where we have access to labels. We further assume that the image of ff is countable set R(f)), where each value v E R(f) defines a levelset {x E X f(x) = u}. We seek to estimate our model's performance in potentially shifted target distribution D, Pt (æ, v), where we have access only samples from (xx).
no assumptions about the nature of the shift are made, the unsupervised performance estima- tion task is impossible 225 19 In this work, we make the most commonly used covariate shift assumption (26), where the shift from the source to target distribution can be attributed solely to changes in the marginal distribution of the covariates. That is, since any distribution can be factor-
3

---

ized as p(a. y) = (x))) we assume that the tabel-assigning conditional distribution remains unchanged P. (g) Pr ()) while the marginal distribution of covariates shifts p. p.(x) # (x()).
3.2 Approximate Confidence Calibration
As mentioned, a key shortcoming with confidence-based estimators has been the dependence on small calibration error, which f is not guaranteed to achieve out-of-the- The simplest way to fix this is to use a small amount of validation data from the source distribution to train a post- hoc calibration mapping c [0. 1] + [0, 1] to minimize the expected absolute calibration error Ep, [IP (Y = c((())) c((()]]] Unfortunately, even if calibration error eliminated in the source distribution, it typically remanifests in the target distribution under covariate shift (24
Previous research has provided theoretical guarantees for confidence based performance estimation in the ideal situation of perfect calibration (8) In this work, we generalize this scope to approximate calibration. Recently, in Al-fairness research, researchers have borrowed ideas from the field of differential privacy 17 which has led to the notion of a-approximate calibration defined as follows:
Definition 3.1. Assume a model f - [0. 1] with countable set of values R(J), and that 2 O. We say that f is a-approximately calibrated in D. if:
K(J
- Po(f(x)= = v) Boly f(x)) =0]- -7-00 Sa. vER())
(1)
Note that by setting a = O, we get the commonly used definition of perfect calibration 28 (marginal- ized over all possible values f can take) as a special case. Calibration in this sense is a marginal guarantee, since the left-hand side of the Inequality ① is an average over the whole distribution. In fairness research, it has become apparent that such guarantees should also apply conditionally on some properties of the instance, such as group membership. Otherwise, a model might be well- calibrated overall, but yield high bias for some (possibly protected) minority groups This has led to the notion of multicalibration, which we will define in its general form.
Definition 3.2. Assume J is model f X + [0, 1] with countable set of values R()), H. is collection of functions h >> R. and a 2 0. We say that f is a-approximately multicalibrated in 'D with respect to H. if vh E H:
Σ PD(f(x) = u)) BolM()) v) f(x)) = v) Sa. RER())
KJJ LU,,D,H H)
(2)
Here, the functions were originally indicator functions for group membership for groups of interest, but this was later generalized to any real-valued functions to allow for weighted memberships [29]. Recently, H was used as a hypothesis space for density ratio estimation (DRE) models, which are
used to approximate the true likelihood ratios w.-r(x) p.(w)
p.(a)) under subpopulation shift (30]. For
this setting, we have the following theorem:
Theorem 3.1. Assume that ps (z)[z) pe (y(()))) and that f a-approximately multicalibrated in D. with respect to H. If ws E H. then (f, D.) S a.
In addition to providing proof of this theorem in Appendix B. will also give upper bound for the calibration error in D. when W,-+t & H and some h E H is used to approximate w,-+t instead (in Appendix B.3 Multicalibration in this setting is used to anticipate potential changes in the marginal distribution p(z) a priori and to approximately adapt to all such changes simultaneously. Although algorithms for achieving multicalibration exist, they are computationally demanding (27). This problem becomes increasingly difficult with the size of H. (29], which can be infinite when considering all possible ways distribution can shift. PAPE circumvents this problem by extracting data from an actual shifted distribution and fitting a calibration mapping to exactly that distribution, essentially becoming multicalibrated with respect an H that has only member. We will describe the details of this process next.
3.3 Probabilistic Adaptive Performance Estimation (PAPE)
In this section, we introduce PAPE and explain how it extends CBPE by addressing CBPE's main limitation: maintaining calibration under covariate shift. Let Ý xxf(X)) denote the binary

---

prediction and m 3 x 3 - IR be a binary classification performance metric with some unknown expected value (...D.))
ED, [m(Y, Y) in distribution D,, where we don't 't have access to labels. However, we do have access to labels in some source distribution D.- Assume that f is already trained with data from D.. We start by collecting (i.i.d)) random samples {(X; {² ~ pa y) and {X(]m 5 11 ~ Pe (r) and training a DRE model from a hypothesis space of binary classifiers H. _ {h| h X + [)] defined by the learning algorithm of user's 's choice as follows.
First, we assign indicator labels 2 0 to all X² and z] = 1 to all x,. Next, we concatenate the features X [x :x] t(, (where i' the dimensionality of X') and their corresponding indicator labels 2n = [2"; R7.t", Then, we use (Xx*, as training data for the DRE model that learns to discriminate between samples from the source and target distributions
described in [31, chapter 2.7.5]. Once the best-fit DRE model h* E H is found, we can use it to estimate the probabilities of observing instances a; from the source distribution within the target distribution, formally P(z; = 1 X² = x;)) = h² (æ.)). Finally, we can approximate Wa-st with
R2 h (xx)))
The 1 10 (x:))
and train a weighted calibration mapping e by fitting it to ((f(x)), ) using (x;)) as weights. Once the calibrated scores (x))) are available, they be used to derive performance estimates with CBPE without access to labels from the target distribution [8]
for this purpose, as they enforce monotonicity, keeping the ranking of the scores and the resulting binary predictions unchanged. However, with PAPE, calibration mapping since the calibrated scores are used solely for performance estimation purposes and don't I affect the binary predictions in any way.
As side note, one would typically choose dedicated calibration mapping 32
33
can any regression model of choice as the
If o f is perfectly calibrated in Dr, the theoretical guarantees from CBPE carry over, and PAPE estimates are guaranteed to be unbiased and consistent. However, since perfect calibration unattainable in any real-life situation, let us next explore the relation between calibration error and PAPE estimation error in limited setting: Some performance metrics, such as accuracy and precision, can calculated as a mean of observation-level values. We call such metrics composable. For any composable metric m, the PAPE estimate can be written as
fgg (x) [c(f(x))m (9. 1) + (1 c(f(x))) m O)]
In practice, this expectation is approximated with the sample mean. For any composable metric, the estimation error of PAPE is bounded by the calibration error as described by the following theorem, which we will prove in Appendix [B.2]
Theorem 3.2. Let e f be a-calibrated in D,. Also, assume that f has countable image set R(f), m is a composable metric with 0 & m(0. y) S 1, and that in is its PAPE estimate. Then,
|m{sg.DD)) (o S a.
For metrics that are not composable, such as the F1 score, the above bound is not guaranteed to hold. However, empirical experiments show that the PAPE estimates superior to any other unsupervised performance estimation method for these metrics as well. We will present these findings next.
Experimental Evaluation
This section deseribes our experimental setting, where we evaluated and compared the proposed method against existing benchmarks. We describe the datasets we used in Section The experimen- tal setup, along with the ML models whose performance was estimated, is described in Section 4.2] We provide practical implementation details of the evaluated benchmarks in Section 4.3 (see also Appendix A). In Section 4.4 we propose novel evaluation metrics suitable for aggregating over multi- ple dissimilar evaluation cases and present the results of our experiments in terms of these metrics. We ran additional experiments with data from the recently published TableShift benchmark (35) and explored the effect of sample size on the estimators. These experiments are described in Appendix D
5

---

4.1 Datasets
The datasets we used to evaluate the method come from Folktables [36] Folktables uses US census data and preprocesses it to create set of binary classification problems. We used the following tasks: ACSIncome, ACSPublicCoveraage ACSMobility, ACSEmployment, ACSTravelTime. For each of the five prediction tasks listed above, we fetched Folktables data for all 50 US states spanning four consecutive years (2015-2018). This gave us 250 datasets.
4.2 Experimental Setup
We started by separating the first-year data (2015) in each fetched dataset and used it training period. For each resulting training data set, we fitted five commonly used binary classification algorithms: Logistic Regression, Neural Network Model (37]. Random Forest [38]. XGBoost [39),
and LGBM [40] with default parameters. We used these models to predict labels on the remaining
part the datasets. We recorded both binary predictions and confidence scores for the positive class. This resulted in I 1,250 dataset-model pairs, which we call evaluation cases.
The of the data for each production (2017, 2018). Reference data was used to fit the performance estimation methods with model imputs, model outputs, and actual labels. Production data was further split into data chunks of 2,000 observations each, maintaining the order of the observations. The realized performance of the monitored model was recorded for each data chunk based on the monitored model's s outputs and actual labels. The performance of the monitored model for the same chunks was then estimated based on the monitored model's inputs and outputs.
further divided into two periods reference (the year 2016) and
For each production data chunk, we compared the realized performance and the estimated performance for each performance estimation method. We filtered out evaluation cases with fewer than 6,000 observations (3 chunks) in the reference data set and cases where the performance of the monitored model on the reference data was worse than random (that is, with an AUROC lower than 0.5). After this filtering, we ended up with 959 evaluation cases, containing 36,557 production data chunks (evaluation points) for each evaluated method We measured the estimation performance of each method with three metrics: Accuracy, F1 score, and AUROC. Figure [ shows an example of an AUROC estimation result.
0.92
0.90
0.88
AUROC
0.86
Realized auroc of Monitored Model (LGBM)
Estimated auroc TEST SET performance
Estimated auroc PAPE
Reference data 2016
0.80
Production data 2017,
2018
50
100
150
200
250
300
Data chunk number
Figure 1:
Estimation of AUROC for ACSIncome data (California) and LGBM as the monitored model. The black line the realized AUROC the monitored model for each data chunk. The red line is the AUROC estimated with PAPE. The brown dashed line is the TEST SET performance.
4.3 Benchmarks
Below, we describe the benchmarks that were evaluated against PAPE and their implementation as it requires user input on the nature of the covariate shift. details. We omit MANDOLINE [I]]
We used single 11th Gen Intel 17- 1800H 2.30GHz machine; computation took 120 hours.

---

We implement PAPE as described in Section 3.3 We use an LGBM 40 Classifier as the DRE model, and an LGBM Regressor for the calibration mapping, both with default hyperparameters. We assume that the reference data originates from the source distribution D, and that each production data chunk originates from some target distribution D;, which is potentially different for each chunk.
PAPE
TEST SET performance For each evaluation data chunk from production data, the performance estimated by this benchmark equals the performance calculated on reference data (and typically, the test set is chosen the reference set). It is our baseline benchmark, representing the initial assumption under which the model's performance on the production data is constant and equal the performance calculated on the test set with reference data.
Confidence-baseed Performance Estimation (CBPE) CВPE is a simpler version of PAPE. We train calibration mapping with PAPE, using reference data from D., but we do not use likelihood ratios in training the mapping (it equivalent to PAPE with wa-sz fixed to for all observations). This results in classifier that has a small calibration enror with the reference data, but is not guaranteed to be well-calibrated with the production data from Dz.
Average Threshold Contidence (ATC) With ATC 2dd take the raw scores provided by the monitored model f()) and apply the maximum confidence function to them, denoting it as MC:
f(x) 2 0.5 MC (f(x)) = 11- otherwise
[(x))
(5)
Then we use it to learn a threshold on reference data such that the fraction of observations above the threshold is equal to the performance metric value calculated on reference data. When inferring, we apply MC to the evaluation data chunk and calculate the fraction of observations above the learned threshold. The estimated metric is equal to the calculated fraction.
Difference of Confidence (DoC) DoC [16] assumes that the performance change is proportional to the change in the mean of maximum confidence ( To learn this relationship, we fit a Linear Regression model on the difference of confidence between the shifted and the reference data input, and the difference in performance between these two as the target. In order to get datasets with synthetic distribution shifts, we perform multiple random resamplings of the reference dataset.
Confidence Optimal Transport (COT) In binary classification, COT finds the optimal transport plan from the scores f(X)) with {X() (xr)" to the labels vf with {uf) (v()", where is the chunk size. The cost of this transportation plan is used as an estimate the classification error of f in D. We estimate only accuracy with this method (see Appendix A for details).
Modified Reverse Testing (RT-mod) Reverse Testing methods train a reverse model on production data with the monitored model inputs features and monitored model predictions as targets. This reverse model then used to make predictions on reference inputs, and its performance evaluated with the reference labels. The reverse model's performance on reference data is a proxy for the monitored model's performance on the production data. We modify the method by additionally fitting Linear Regression to relate the reverse model performance change with the monitored model performance (just like in DoC).
Importance Weighting (IW) For IW, we first estimate density ratios w.-st between reference and production data with the DRE classifier, the same as we use for PAPE. Then, we use them as weights to estimate the weighted performance metric of interest using reference data.
4.4 Evaluation
Performance estimation is regression problem, which motivates the use of evaluation metrics such as the Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE). However, aggregating MAE/RMSE over multiple models and chunks of data from different datasets leads to skewed and uninterpretable results. Large MAE/RMSE for an evaluation case where the model's performance has a large variance might still be satisfactory. On the other hand, even small changes (in the absolute scale) in performance might be significant in cases where the model's performance is very stable.
7

---

To account for this scaling issue, we used normalized versions of MAE and RMSE, where we scaled absolute/squared errors by the standard error (SE) calculated for each evaluation case. We first measured SE: as the standard deviation of the realized performance sampling distribution on the reference data. We did this by repeatedly sampling 2,000 observations (the size of the evaluation data chunk) at random from the reference data with replacement. Then, we calculated the realized performance metric on each sample. We repeated this 500 times and calculated the standard deviation of the obtained per-sample metrics the standard error (SE). Then, for each evaluation case, divided the MAE and RMSE by SE and squared SE, respectively, resulting in normalized mean absolute error (NMAB) and normalized root mean squared error (NRMSE), defined formally
k
maj ma
NMAE
Ei
Ci
SE;
i=
j=l
(mij maj)
NRMSE
Σ
SE
C5
where is the evaluation index ranging from to k, is the index of the production data chunk ranging from 1 to (number of production chunks in a case i), and myj and mijj are respectively realized and estimated performance for case and chunk j. SE; is the standard error of the i-th evaluation case. The results are shown in Table
Accuracy
AUROC
F1
NMAE
NRMSE
NMAE
NRMSE
NMAE
NRMSE
TEST SET
1.62
2.88
1.45
2.30
2.53
8.23
RT-mod
2.31
5,41
1.85
4.29
2.06
4.74
COT
2.10
4.31
ATC
1.58
2.79
1.90
3.52
2.97
9.06
DOC
1.13
1.80
1.37
2.75
2.56
8.52
CBPE
1.08
1.75
1.07
1.68
1.03
2.12
IW
1.04
1.40
1.06
1,56
1.07
1.74
PAPE
0.97
1.28
0.99
1.45
0.90
1.34
Table 1: NMAE and NRMSE of the evaluated methods for each estimated metric.
PAPE, together with CBPE and IW, shows strong improvement the TEST SET baseline. PAPE significantly outperforms the second-bes method for each estimation and evaluation metric (paired Wilcoxon signed-rank test p 0.001). PAPE result NMAE equal to 0.97 for accuracy estimation means that the estimation is, on average, less than 1 SE away from the realized performance. For intuition, the NMAE of PAPE for the evaluation case shown in Figure equal 1.11. The TEST SET performance NMAE for accuracy is 1.68, indicating that accuracy changes significantly in the production data chunks. If the performance was stable with only random normal fluctuations, the NMAE of the TEST SET performance would be around 0.8 The changes for Fj are much stronger, as shown by the increased NMAE of TEST SET 2.49. This explains why performance estimation methods provide the strongest improvement compared to the TEST SET performance for this metric. Additionally, we calculated NMAE and NRMSE for each monitored model type PAPE consistently outperformed IW for each of the monitored models and each estimation and evaluation metric.
We also analyzed each performance estimation algorithm's accuracy across different levels of real- ized performance changes. For relatively small realized performance changes (within +2SH), the TEST SET performance baseline is sufficiently accurate. Any useful performance estimator should outperform this baseline at least when the realized performance changes are significant (not within +22E).. We sorted the data by the magnitude of the performance change and calculated the rolling NMAE for 2SE-wide intervals. The results are shown in Figure
That is because the average absolute deviation for normal distribution is 23 0.8 of its standard deviation.

---

Accuracy
AUROC
Edtiation Gror INKAET
COT DOC IN PAPE
DOC
DOG
10
of churres bucket
N PAPE Rt-mod
HAPE
modd
RT-mmd
5
101
10
36
3-
10
Chonge Accuracy (K 1563
Change
Chonge
Figure 2: Estimation errors (NMAE) of estimated metric vs. realized absolute change as SE for all estimators. The x-axis indicates the center of the data bucket for example, value indicates a bucket that contains data chunks for which the absolute performance change was between O SE. The left y-axis shows NMAE of the evaluated method for the data bucket. The right y-axis shows the number of data chunks in each bucket on a logarithmic scale as depicted by the grey dashed line.
For all the estimated metrics, PAPE shows the best estimation quality in nearly the whole evaluated region. gets significantly better than other methods when the change realized performance is large. IW and CBPE show comparable performance, with IW being more accurate in most buckets. The RT and COT methods produce estimation errors, which render them unusable for all metrics used in the experiment. The DOC and ATC methods somewhat competitive when estimating accuracy (for which they were designed) and when the change in realized accuracy is small enough. For the other metrics, the estimation errors are too high for practical monitoring purposes.
Discussion
PAPE is perhaps best understood as an improvement over the CBPE method [8] PAPE retains all of the benefits of CBPE, such as applicability to multiple metrics instead of just accuracy, but also addresses the problem of deteriorating calibration under covariate shift, which is the major shortcoming of CBPE. Our experimental findings show that PAPE consistently outperforms CBPE in all experiments. Alternatively, PAPE can be viewed as an in-between solution, trying to achieve the best of both worlds of IW and multicalibration approaches.
Multicalibration requires post-processing, where the original model is iteratively updated through an auditing process (29] This might result in predictions different from those of the original model. Also, the hypothesis space H needs to be large enough to contain Ws-st, or at least good approximations of them, for any potential shifts. Unfortunately, especially in -igh-dimmnnionall settings, this might be infeasible or at least increase the computational burden significantly. In contrast, PAPE focuses time, learning the density ratios required directly from the data. Thus,
on one target distribution PAPE does not need to compromise performance by trying to deal with multiple distributions at the same time. PAPE is also non-invasive, making no alterations to the monitored model, which makes it more suitable for monitoring purposes. In fact, the model being monitored does not need to be calibrated at all, since PAPE provides calibration on the fly.
On the other hand, IW is known to suffer from high variance estimates when there a significant discrepancy between the source and target distributions [41] Since the labels used with IW either 0 or 1, it is also susceptible to large sampling errors, especially in small sample settings that are typical in model monitoring. Since PAPE uses soft scores from the interval [0, 1]. tends to smooth out these sampling effects. We provide a more comprehensive comparative analysis of the variances of PAPE and IW in Appendix
5.1 Limitations
PAPE will not work under concept shift, that is, ps(laa)) + (y(aa)). Also, when operating covariate shift setting, the data may shift outside the support of the source distribution, which means

---

that there is region S S X, where for all a E S we have Pr(æ) > 0 and (x(x) = 0. Weighted calibration for instances from such regions is not possible as the weights w,-r(2)) are not defined.
PAPE hinges on calibration performance within the target distribution Dr, which in turn depends on good enough density ratio estimates. Both be hard to achieve if not enough labeled reference data is available to train the DRE model and the calibration mapping. This data demand increases with the dimensionality of the covariates. As such, PAPE most performant with now-dimensional data.
5.2 Future Work
Although we have described PAPE as a method for estimating the performance of binary classifiers, extending it to multiclass classifiers is straightforward. For instance, in the case of macro-averaged metrics- where performance is computed separately for each class in a one-vs-all manner and then averaged PAPE can be applied directly by estimating the per-class performance and averaging the results, In fact, PAPE can be used to monitor the performance of any model that produces confidence scores addition its predictions. We leave the details of this for future work.
One challenge with PAPE or any other method relying on density-ratio estimation that estimating these ratios becomes increasingly hard with -igheiinnsiinal data. Examining the sample complexity requirements and relating those to the estimation quality of PAPE is an interesting and important research question to be addressed by later research.
By using AUROC as an estimated metric in our experiments, we expanded the scope CBPE to previously unestimated metric. We explain in Appendix SoaadixxAhooo how this was done precisely. Contrary CBPE, where a full distribution for each metric estimated, our approach with AUROC results only in approximation of the expected value of the metric. We intend to examine metrics such as AUROC and AUPR further to provide a way to estimate the full distributions of these metrics, which require calculations over multiple thresholds.
6 Conclusion
We introduced PAPE, an innovative approach for accurately estimating the performance of binary classification models when faced with covariate shift. We examined its theoretical properties and provided a bound for its estimation error for composable metrics under approximate calibration. We performed rigorous testing for PAPE using real-world datasets drawn from US Census data, introducing novel evaluation metrics essential for aggregating results over multiple datasets and model monitoring scenarios. We analyzed over 900 model-dataset pairs and generated more than 36,000 data chunks for thorough evaluation. The results demonstrated that PAPE significantly outperforms existing methods across all metrics assessed.
Acknowledgments and Disclosure of Funding
This work partly supported by local authorities ("Business Finland") under grant agreement 23004 ELFMo of the ITEA4 programme, which funded one of the authors. The remaining research was conducted at NannyML-
venture-backed open-source software company focused on post- deployment data science solutions which has received public R&D funding from Flanders Innovation & Entrepremeurship (VLAIO) under project number HBC.2022.0846.
10

---

References
[1] Michael Lones. How to avoid machine learning pitfalls: a guide for academic researchers. arXiv preprint arXiv:2108.02497, 2021.
[2] Daniel Vela, Andrew Sharp, Richart Zhang, Trang Nguyen, An Hoang, and Oleg S. Pianykh. Temporal quality degradation models. Scientific Reports, 12(1 1654), 2022.
[3] Janis Klaise, Arnaud Van Looveren, Clive Cox, Giovanni Vacanti, and Alexandru Coca. Moni- toring and explainability of models in production. arXiv preprint arXiv:2007.06299 2020.
[4] Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[5] Tom Ginsberg, Zhongyuan Liang, and Rahul G Krishnan. A learning based hypothesis test for harmful covariate shift. arXiv preprint arXiv:2212.02742, 2022.
[6] Samson Tan, Araz Taeihagh, and Kathy Baxter. The risks of machine learning systems. arXiv preprint 2:Xiv:22044098522 2022.
[7] Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems, 19800-1444 2021.
[8] Juhani Kivimäki, Jakub Biatek, Wojtek Kuberski, and Jukka K. Nurminen. Performance estimation in binary classification using calibrated confidence. arXiv preprint arXiv:2505.05295, 2025.
[9] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[10] Nan Lu, Tianyi Zhang, Tongtong Fang, Takeshi Teshima, and Masashi Sugiyama. Rethinking importance weighting for transfer learning. In Federated and Transfer Learning, pages 185-231. Springer, 2022.
[11] Mayee Chen, Karan Goel, Nimit Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Re. Mandoline: Model evaluation under distribution shift. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1617-1629. PMLR, 18 24 Jul 2021.
[12] Christina Back. Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on-tuen Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274-19988 19289, 2022.
[13] Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and Zico Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations, 2022.
[14] Ru Peng, Qiuyang Duan, Haobo Wang, Jiachen Ma, Yanbo Jiang, Yongjun Tu, Xiu Jiang, and Junbo Zhao. Came: Contrastive automated model evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20121-20133 October 2023.
[15] Wei Fan and Ian Davidson. Reverse testing: An efficient framework to select amongst classifiers under sample selection bias. In Proceedings the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, page 147-156, New York, NY, USA, 2006. Association for Computing Machinery.
[16] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Pre- dicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 1134-1144, October 2021.
[17] Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15069-15078, June 2021.

---

[18] Weijian Deng, Yumin Suh, Stephen Gould, and Liang Zheng. Confidence and dispersity speak: Characterising prediction matrix for unsupervised accuracy estimation. arXiv preprint. arXiv:2302.01094, 2023.
[19] Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ketong Chen, Zhenlin Wang. Soheil Kolouri, Simon Stepputtis, Joseph Campbell, and Katia Sycara. Characterizing out-of-distribution error via optimal transport. Advances in Neural Information Processing Systems, 76:17602-17622, 2023.
[20] Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples neural networks. International Conference on Leurning Representations, 2017.
[21] Juhani Kivimäki, Jakub Biatek, Jukka K. Nurminen, and Wojtek Kuberski. Confidence-based estimators for predictive performance in model monitoring. Journal of Artificial Intelligence Research, 82:209-240, 2025.
[22] Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribtiion performance. International Conference Leurning Representations, 2022.
[23] Mohamed Bekkar, Hassiba Kheliouane Djemaa, and Taklit Akrouf Alitouche. Evaluation measures for models assessment over imbalanced data sets. Journal of Information Engineering and Applicationn, 3(10), 2013.
[24] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dil- lon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? eval- uating predictive uncertainty under dataset shift. In H. Wallach, Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[25] Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 129-136. JMLR Workshop and Conference Proceedings, 2010.
[26] Jose G. Moreno- Torres, Troy Raeder, Rocío Maiz-Rodríguezz Nitesh V. Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45(1):521 2012.
[27] Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the saappuutttionally-ddnffffii Machine Learning, pages 1939-1948. PMLR, 2018.
masses. In International Conference on
[28] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML' 17, pages 1321-1330. JMLR.org. 2017.
[29] Ira Allobus-Harriss Declan Harrison, Michael Kearns, Aaron Roth, and Jessica Sorrell, Multical- ibration as boosting for regression. In International Conference on Machine Learning, pages 1 11459-11492. PMLR, 2023.
[30] Michael P. Kim, Christoph Kern, Shafi Goldwasser, Frauke Kreuter, and Omer Reingold. Universal adaptability: Target-independent inference that competes with propensity scoring. Proceedings of the National Academy of Sciences, 1 119(4), 01 2022.
[31] Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023.
[32] John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10:61-74, 03 1999.
[33] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML. '01, pages 609-616, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc
12

---

[34] Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Aarti Singh and Jerry 7hu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 623-631. PMLR, 20-22 Apr 2017.
[35] Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 36, 2024. Folktables library is MIT licensed.
[36] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021. Folktables library is MIT licensed.
[37] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. CoRR, abs/2106. 1959, 2021.
[38] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 112222-28830 2011.
[39] Tianqi Chen and Carlos Guestrin. abs/1 100.00044,, 2016.
Xgboost: A scalable tree boosting system. CoRR,
[40] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146-3154 2017.
[41] Fengpei Li, Henry Lam, and Siddharth Prusty. Robust importance weighting for covariate shift. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statisties, volume 108 of Proceedings of Machine Learning Research, pages 352-362. PMLR, 26-28 Aug 2020.
13

---

A Implementation Details
In this section, first explain how F1 and AUROC were estimated for PAPE (and CBPE) in the experiments in Section Then, we describe our implementation for estimating the same metrics with ATC and DOC, since these methods were not originally intended for estimating any other metrics besides accuracy.
A.1 Estimating F and AUROC with PAPE
We omitted confidence intervals in experiments to relieve the computational burden and because only CBPE and PAPE are capable of providing these intervals. Thus, with CBPE and PAPE, we resorted to deriving only point estimates. Let recall equation 4
fco P.) (x)) (=(f()))) (ú. 1) (1 - c(f(x))) m (g. 0)]
One can use the sample mean to approximate this expectation and use the approximation in turn to estimate the performance of composable metrics, such that can be calculated
mean of observation- level values, like accuracy. For other metrics, we start by estimating elements of the confusion matrix. Assume have sample of instances and that there are n4+ positive and n negative predictions in the sample. Then, as explained in [8), the elements of the confusion matrix can be estimated with:
1 TP
Σ
(((x))
n
FP 1
- (1-c(((z)))
n
TN 1
cf(x))) (1 (1-9:)
n-
TN 1
(=(e((()))) (1 ii)
With the estimated elements of the confusion matrix, one can estimate F1 with:
2.
F
(6)
TP + FN + n
we treat the value of F1 as a random variable XF,, the approximation given in 6 converges to E[X₁, rapidly when grows. This expectation, in turn, is an unbiased and consistent estimator of FL under perfect calibration. g
To estimate AUROC, the model's binary predictions ý are not needed. We use the scores f(x) returned by the classifier to define a new thresholding function h.:
11, f()) 00 f(z) <
hy (f(z),))
For each R(f) and for each æ; in the sample, elements of the confusion matrix as explained above. Then, using these estimates, we can approximate true positive rates (TPR) and false positive rates (FPR)
h (f and estimate the
TP.
TPR TP., FN.
TN.
FPR TN W.+FP. PP.
Finally, we can calculate estimate for AUROC with the approximated TPR and FPR. In our experiments, we tried several different calibration mappings and settled on LGBM, since it gave the best overall performance.
14

---

A.2 Estimating F and AUROC with Other Estimators
Since COT [9 is essentially estimating the 0/1 classification error, there no clear way of how could be used to estimate metrics other than accuracy. For this reason, we left it out of experiments where F1 and AUROC were estimated. Although ATC [22] and DOC [I6] were also originally de- signed for estimating only accuracy, we used the following procedures to expand them for estimating other metrics as well.
When estimating the F1 score and AUROC with ATC, we applied the same logic as with accuracy. For accuracy, ATC finds a confidence threshold such that the fraction of instances from the reference data set with predicted confidence score above the threshold matches model accuracy in the reference set. With production data, the fraction of confidence scores above the same confidence threshold is taken as the estimated accuracy for the production data. When estimating the F1 and AUROC, we similarly found confidence thresholds matching these metrics with reference data and used the fraction of production data instances with confidence scores above those thresholds as estimates.
With DOC, we simulated distribution shifts by resampling reference data, We collected the F1 and AUROCs and fitted a linear regression model with the difference of average confidence between the reference and simulated sets the single feature and the metric of interest as the target (just as with accuracy). We did not apply calibration to the confidence scores for either the DOC ATC methods. Our reasoning behind this is that with DOC, the linear regression model is indifferent with respect to any rescaling of the confidence scores. On the other hand, with ATC, we found that the reported differences between the calibrated and uncalibrated versions of ATC with binary classification in [21] using the models we included in our experiments were so minimal that we chose include only the variant that had better reported overall performance, in this case, the uncalibrated version.
did with accuracy.
B Proofs
In this section, we provide proofs for the theorems presented in the main paper. We begin with Theorem 3.
B.1 Proof of Theorem 3.1
To improve readability, we start by proving two lemmas, which we can then leverage in the main proof First, we make use of the following connection between the expectations of the source D. (x. y) and target D; pt (x, y) distributions:
Lemma B.1. Assume pa (y((a)) pe(y)æ) and fix any S S X. Then, for any integrable function F:XY y + R, we have:
Po. E (us,o F(x,y) x S Po, (or E S)DD, (F( S).
Proof.
PD. E ()Ep,,wa.r Fxx, 8) S]
(z(x)) ()))(())
Bptiya [F v)] dx
(x (a
(x)) (Ep(y [F w) dx
J
(x)
E
Ep(yle) [F(,, w)] dæ
Pp, (x E ES) F((, v) S].
Now, we can leverage Lemma B.I to prove the following:
'Acknowledgement: Our derivations follow closely those of Prof. Aaron Roth on his "Learning with Conditional Guarantees" course, which he teaches the University of Pennsylvania.
15

---

Lemma B.2, Assume Pe (y [+) pıly)>), Suppose f is a- a-aaproximately multicalibrated in D, with
respect H. Then, f is also a-approximately multicalibrated in Dr with respect Hz-st. where
h(x:)
Ho-t
h(x) E
(x)
Since f a-approximately multicalibrated in D, with respect H. for every h E H. have:
Proof.
a k(f.D..h)
E
PD. (f((f()) = Bolb(x) f(x) = =
vER(S)
= Σ
(f.(f()) = v) ED, [1(x)) 0) f(x) = =u
vER(S)
h(x))
E .ER(S)
(fxx) = RD, W (--)(ar)
(g))( (y/ - u)
f(x) =
h(x))
Σ
(f(x) - ED.
(y v) f(x)
(
vER())
h K [i
W/s+t/
where Lemma B.1 is applied each of the terms
(o,()(x)) v) ED, w h(xr)
(y - v) [(x) =
using S = (x: f(x) = 4) and F(x, = we sc() (U v).
h(a))
We are now ready to prove Theorem 3.1 which we will restate here:
Theorem 3.1 Assume that ps (ylæ)) Pr and that f is a-approximately multicalibrated in D, with respect to H. If w.-. H. then K(f, D.) S a.
Proof. Since we assumed that wa-s E H. we can choose h = Wa-+s and apply Lemma B.2 to get
xx( fDD
h
K
w
h(xx)
Σ
(f(x) uV)ED. LED.
(y v) f(x) = W. w.-re())
veRlf)
E (f.(f(x) = v) Ep. [(--2)
-2) 1)) f(x)) = u]
vER())
K()
B.2 Proof of Theorem 3.2
In this section, we will offer the proof of Theorem 3.2 which we restate below:
Theorem 3.2 Let f be a-calibrated in D.. Also, assume that has countable image set R( m is a composable metric with 0 S m(g. y) < 1, and that in is its PAPE estimate. Then,
[M(..g.P.)
-mpoo.....)) S
16

---

Proof. We start by decomposing the calibration error as a sum of terms related to each levelset {a E X: f(z) v). For each v, we assume that calibration error a, remains after applying the calibration mapping c. Formally,
Ep,ly f(z) = =j = Pp. - f(x) = u] = (v) + aa.
For any fixed u and deterministic c, clearly Ep, Ep,(((u) f(x) = 2] c(v), which yields us
ED, v [xx) = = =c(u) + Oh
Ep, w (ax o] -c(v) 0h
ED, y (()) [(x) = v] = Cty-
The total calibration error is then:
Σ (f(())((()) = u) c(t)f((( u| .ER(S)
E (f(x) c) layl a. VER(F)
Furthermore, notice that since the levelsets for X: ff(x) et form a partition of X, for any integrable function : [0, 1] - R we have pp)(() [F(b) = EveR(n) (f(()) = F()) Thus,
we can write:
D.) = ED, Im (g(f(x)). v)]
Σ VER(S)
Pm(( (f(x) = E.. (m)o..m(((((()).))) f(x)) =u]
Ppr(an (f(x)) v) (m( - f(x) o) +
Σ vER(F)
[f) -2 = =))
-haaa 0) (1
(c(v) + aa.)))
vER(S)
Σ = vER(S)
= [|00::/0000000000000000000000000 +
= =) (m) (a
Σ vER(S)
+ E (f(x) = = ( m((())))))) eER())
= =m{fa..
which we can further manipulate as m(fggD.) - ..f...)))
= () - (7)
vER(S)
For any metric m with 0 S m(0. v) S 1, we have [m(g(v),1) m(g()),0) <
(m((m()))), 1) m((vv), 0)) S Jayl:
Thus, if we take the absolute values of both sides of Equation 7 get
Σ Pp)(a) (f(x) (a) (m)
[m(...P..) -......))
VER()
Σ (f(x)) 1) 1... (m(((e)))) m)(g()))))
vER())
S Σ ()( v)laol vEn(S)
S a,
completing the proof.
17

---

B.3 Estimation Error Under Imperfect Weights
Our proof of Theorem 3.1 assumes that we have access to exact density ratios. Here, we will produce an upper bound for the calibration error when we resort approximatng the weights with some function h E H. To quantify the approximation error, we make use of the following definition:
Definition B.1. Assume that (z) Pr(y)a)). For a function
E. we write
c(h, wg-1) = (m) w--+(
Similarty, for any subset S S X, we write:
<h,, S,, S) (p. w.-r(<)] a E..
Now, we can prove the following lemma:
Lemma B.3. Assume (x(lx) p(ly)z) and fix any S S X. Then, for any integrable functions
F:Xx) x 1-,, and h X + IR, we have:
PD. (x E 5) (b,(I(()) F(æ, v) x E S
> Pr, (x E S) Ed, va) x E S] Pr, (ae E S): E(h, S).,,
Proof. We can Lemma B.1 to write
Pd. (w E S) ED, [h(x) F( x E E - PD, (x S) ED, (F(x) x S]
8] (a S)ED, (w) x S]] (x 9) ED. (h)()) (1, (x)) F(z,y) a S
=P e S)DD,, [1(x)) F(
(p ES) max
(Fx,,)) (p. [1(())) ( x S
(x.))E(X.))
s Po, E S) c(h, Wg-1, S),
since clearly max (x.)))((X))) inequality to write
|F(x, w)1 < Then, we reverse the direction and use the reverse triangle
PD. (x E S) e(h, Wa-r, S)
2 Pb. (x E S). ED, (x(x) F(x, y) x E 5] (x ED, (F(,,)) x E S]]
2 Pr. (x E S) Ep, [F(x,,,,))) æ E S]] - Po. (a E S) Ep,(h(x)) F(x, v) x E S]|.
from which the statement follows by simply rearranging the terms.
This lemma can be used to prove the following theorem, which is relaxed version of Theorem and gives an upper bound for the calibration error with approximate likelihood ratios:
3.1
Theorem B.4. Assume that Pa (a(y) = pe(y)ae and that f is a-approximately multicalibrated in D. with respect to H. Then,
K(J,D.) S a + min (h, w.-+)). hCH
Proof. Let ht arg min e(h,, w.++e) and S, {a E : [(x) = f} so that the collection NEH
{S.lven()) forms a partition of X. Thus, by the law of total probability,
Σ Pp.(f(x) = v) c(hi, we-ee S.) = c(hi, (.)-t))
VER())
18

---

Since f is a-approximately calibrated in D3 with respect to H, we have: a 2 ((, Dz,h") h²)
Σ
(o,( - ED, [1((x)(y - v) [(xr) =
vER(S)
Σ PER(F)
Po.(f(x) =v) LED, (B- - f(x) =0]
Σ .ER(F)
PD. (f(x)) = =) e(h", S..:::
= K(f, D.) - e(h" w)+1).
Here, Lemma B.3 is applied to each of the terms
(b.x)()) v) ED, (x)x)) f(x) u
with F(x,3) y) = V.
C IW and PAPE Variance Comparison
In this section, compare the variances of IW and PAPE when used estimate the accuracy of a model that is trained with data from some source distribution pa (x. y) in some target distribution pe(ar,2) y). We operate under the covariate shift assumption of ps py(yya (y] (o (z). We assume that the scores S = f(X) produced by the classifier are perfectly calibrated in ps (2x y) (but not necessarily
in pe (ar y)). That is, if (X, Y) ps (s y), then P(Y =1 = 8)
vs E [0, 1]. In addition,
we assume that there is some discriminator function g [0, + {0, 1] mapping the scores to binary
predictions Y = g(S).
C.1 Estimators
We start by describing the estimators in this setting and the notation used.
C.1.1 Probabilistic Adaptive Performance Estimation
The CBPE 8
estimator for accuracy from a sample X2, X.) pe(x)) is defined as
X Naceuracy Xeorrect
(8)
where Xeorrect follows a Poisson binomial distribution with parameters Z; defined as
S₁, ==1
2, = 1 - S; Y=0.
(9)
Let X'' + [0, 1] denote the mapping sXX,) Z, Under perfect calibration, using Ep, [Xaceurocy) as point estimate yields an unbiased and consistent estimator for accuracy [8]
1 TI
ACCEBPE
Z
(10)
However, the assumption of perfect calibration in the source distribution Ps Ps(x, (x. y) does not extend to the target distribution (r(x, y). which typically results in a biased estimator in any target distribution. PAPE allows us to train weighted calibrator : [0, 1] - [0, 1] using the same (exact) density ratios w(x) as IW (explained in the next section), ensuring that f = co f is perfectly calibrated in Pe(x, y). Now we can define
1 3 2
AccpAPE =
(11)
where the difference to AcceurE is that the scores ST used to define Z. originate from f instead of f. Similarly, we let s. : X [0, 1] denote the mapping s" (X,) = z." Under the assumption of exact density ratios w(x), AccPAPE is unbiased and consistent in the target distribution.
19

---

C.1.2 Importance Weighting
The empirical importance-weighter (IW) estimator from sample X2, Xn) ~ pr(x)) is defined as
Acew
Ia (x;),
(12)
where the indicator I; is defined as
1 1, = Y
(13)
0. Y; + Y
with y; = D1 ~ paly x)", and for each X; = Di
Pr(2)))
w( w(x;)
(14)
(x4)
the density ratio relating the marginal source and target distributions, Ps px( and (<(a:) respectively
(with ps (x)) > 0 whenever (=(x) > 0). Under the assumption perfect calibration (in the source distribution), have Bernoulli(Z)). The empirical importance-wette weighted estimator (under exact density ratios) is known to be unbiased and consistent in the target distribution.
C.2 Variance Comparison
If we have access to exact density ratios, both PAPE and IW are unbiased and consistent estimators for accuracy. Then, the choice between the two should depend only on their sample efficiencies. That is, which estimator has the smallest variance? On the other hand, if density ratios are not exact, is a reasonable assumption (by the principle of insufficient reason) that both estimators would suffer roughly equally on average.
By using Pt as shorthand for (x():, the per-observation variance of PAPE is, by definition E2 Ep. 12 = Ep, (s (x) 1] Ep. [s" (x))².
Varp, - =
(15)
Let us take a look at the per-observation terms of the IW estimator in a similar fashion. We start by denoting W Twe(X where recall that 7 is Bernoulli variable with parameter Z s(X). Conditioning on fixed X = æ, we have Bernoull((((z)) and the weight w(x) is constant. Using P. as a shorthand for Y Ps (v æ). the conditional mean and variance of W are
Ep, [ æ]] Ep, [/ 10 ai Ep, [I æ) w())s(())
(16)
Varp. (W) x)) Varp, (x)) x)) wxx)² Varp. (I x)) w(x)² s(x)).
(17)
Next, we can use the law of total variance to get
Varp, (W)- = EpM [Var,, (W x)] - Var,, (E I æ))
Ep, (w(x) (x)) s(x))] + Var, (w(x)s(x))
Ep, [p) s(x))) Ep, [(w()))(<)))]] Ep, [u(())(()))²
=E,,((2))'(((z)) - s(x)²) + = Ep. [=(=)s(<))]
Ep. =(z) (x)) Ep. [=(=)s(x))
Ep, (w(x) (x=) - Ep, (=)))
Now, we can compare the derived variances and state that the per-observation variance of the PAPE estimator is less than equal to that of the IW estimator if and only if
Ep, (x=)² - E,, 1s [=)] < (w))
Ep, [=))))(<))]
(18)
'In expectations such as E,, (x)). use as dummy variable representing a specific realization of X. Thus, any reference to in these expressions to be understood a placeholder that integrated out. 'Here we again revert to slightly abused notation for compactness. In fact, we should write the total variance as (x))) (vla)) However, since we consider the source distribution to be fixed, we omit from the left-hand side for readability.
20

---

Given that the value of the expression on the right-hand side depends heavily on the interplay between w(x) and s(x). and we don't 't know the exact relation between s" (a)) and s(x). it is impossible to verify whether Inequality (18) is true or not a priori. Here, we look at only one interesting special case, that being P. (=(x)) = Pt (+(a), which leads to constant density ratio w((r) = In this setting, the
calibrator e is the identity mapping so that f = f", which means that s(x) = s (²()) pointwise. With
these insights, criterion (8) can be written as
[p, - Ep, [s x []2 - Ep [s([]]]]] Ep [s(x)]
(19)
Ep, [s()) = Ep [sx)]]
(20)
Because 0 S s(x) S 1, it follows that
s(x)² S s(x)
Bp, [s(x) < Ep, (s()),
which means that the special case criterion (20) is always satisfied. Thus, under no shift, the per- observation variance of PAPE is always at most that of the IW estimator, making the former more sample efficient. also gives reason to believe that this is likely the case when the shift is small, and hence the weights w(x) are close to one and (*) R s(x: pointwise. Having said that, it is possible to envision situations where the IW estimator has lower per-observation variance.
Up to this point, we have looked only at the per-observation variances. However, it is straightforward to justify this approach follows. We can write the variances of our estimators as variances of the means of n mutually independent observations as
n
n
1 Varp. (Zm)
Var Var(ACCPAPE) Varp.
w
Varp. (2!")
n
n
n1
n
n Varp, (W;)
1
Var Var(ACC1w) Varps
Σ
W
Varp, (W),
n
n²
n
which means that in comparing the variances of the two estimators, it suffices to compare the per-observational variances Varp, (Z*) and
Finally, we suggest a pragmatic empirical approximation for criterion (18), which can be used as heuristic guide in choosing which estimator use for given (i.i.d.) sample pe(z)". This approximation replaces the expectations with empirical means, leading to
Xn) ~
2
2
(X.)2
Σ (X.)
w (xx))
Σ (x.))(
If this inequality is true, one would favor PAPE over IW, and conversely so it is false. Although this heuristic not guaranteed to result in better estimates in every case, for any sufficiently large n, should result in better estimates on average. If (for whatever reason) one has to choose the estimator before observing any data, a rational agent would make the (uninformative) assumption about the weights, assigning wxx) = 1, which by criterion (20) would lead them to choose PAPE.
D Additional Experiments
D.1 Sample Size Effect
The experiments described in Section 4 were run for an arbitrarily chosen evaluation data chunk size (2000 observations). In this section, we describe an ablation study, where we investigated the quality of performance estimates for different chunk sizes. We split production data into chunks of the following sizes: 100, 200, 500, 1000, 2000, and 5000. For each consecutive sample and each chunk size, the first instance of the data chunk was advanced 1,000 observations so that the first instances of each chunk for different chunk sizes were aligned. This was done to make the results between different chunk sizes comparable and to minimize the effect of changes in the underlying data. This meant that for chunk sizes of less than 1,000, not all data was used, and for chunk sizes larger than 1,000, there was some overlap between consecutive samples.
21

---

MAE of AUROC extimetion (log scale)
0.06
ATC
0.05
CBPE
0.04
DOC
0.03
0.02
RT-mod TEST_SE SET
0.01
0.005
5000
Chunk Size (log scale)
Figure 3:
Effect of sample size on mean absolute error AUROC estimation. Calculated for sample sizes of 100, 200, 500, 1000, 2000, and 5000, data from California, for the prediction task: employment.
Due to computational complexity, we ran the experiment only for one evaluation case. We selected the biggest data set (California), the prediction task for which performance changes significantly (ACSEmployment), the default choice algorithm for the classification task on tabular data LGBM,, and the default metric AUROC. Since the results come from single evaluation case, we used typical regression metric mean absolute error (MAE). Figure 3 shows that all methods give less accurate AUROC estimates for smaller chunk sizes. This is expected as the random noise effects are more significant for small samples. For the evaluated case, this has the strongest impact on IW accuracy of estimation.
D.2 TableShift Experiments
In addition main experiments with US census data, tested PAPE and our other benchmark methods with tabular data from the recently published TableShift benchmark [35]. Due to the defective functionality of the TableShift API, benchmark. Thus, we resorted to using a subset datasets we were able to retrieve, namely: ASSISTments, College Scorecard, Diabetes, Food Stamps, Hospital Readmission, Hypertension, Income, and Unemployment There is some overlap between these datasets and the datasets used in Folktables [36]. The main differences are that the datasets in TableShift come from wider range of providers, the datasets are generally smaller, and the datasets are preprocessed differently, in particular, to create distributional shifts between the In-Domain (ID) and Out-of-Domain (Ool) portions. These shifts do not necessarily conform to our covariate shift assumption, giving us an interesting opportunity experiment how PAPE performs relative other estimators in undefined types of shift scenarios.
unable extract all 15 datasets contained within the
Accuracy NMAE
AUROC
F1
NRMSE
NMAE
NRMSE
NMAE
NRMSE
TEST SET
2.11
2.92
1.03
1.82
1.01
1.49
RT-mod
2.22
3,05
1.39
2.15
1.19
1.69
COT
2.80
3.49
ATC
1.94
3.20
1.32
2.06
1.50
2.11
DOC
1.74
2.54
0.94
1.67
1.03
1.45
CBPE
1.79
2.46
0.97
1.65
0.77
1.32
IW
1.59
2.06
0.88
1.13
0.75
0.95
PAPE
1.52
1.98
0.83
1.06
0.67
0.86
Table NMAE and NRMSE of the evaluated performance estimation methods for each estimated metric.
These datasets described in detail
22

---

We used the same evaluation framework as with our main experiment in Section with the same benchmarks, implementation details, and evaluation metrics. However, the data preprocessing was handled a bit differently. We used the "training" portion of the datasets for training our models, the "validation" portion (both for IDD and OoD) to train our estimators, and the "test" portion (both for and OoD) for evaluation. We concatenated the ID and OoD sets because for some datasets, the Ool portions were very small, sometimes less than 2,000 instances, which we used as our chunk size. The results shown in Table align with our main experiments, showing that PAPE is superior to other estimators for metrics tested.
D.3 Experiments With Synthetic Data
Since all the other experiments thus far have been performed real-world data, where cannot fully control the nature of the datset shift, we conducted further experiments with synthetic data created ensure that only covariate shift is present. We start by describing the data generation process.
D.3.1 Data Creation
We created a spherically symmetric distribution of inputs centered at the origin of R with di- mensionality d 20. More specifically, generated feature vectors by first drawing Gaussian directions E 22 with independent N 1) entries and normalizing each vector unit length, = ||,// thereby ensuring isotropy on the unit sphere. Radial distances R, were then sampled independently from a then derived as X; R. Ug, and only those with radius R; 0.5 were retained. This procedure yielded a base pool of 100,000 feature instances with uniform angular structure and controlled radial distribution.
distribution with a 0.15. Each observed feature instance was
Next, labels for each feature instance were assigned based on smooth, monotonic relationship between the features and the probability of the positive class. For each observed feature instance æ, the radius r(z) = | l/2 was calculated and the conditional probability of the positive class was set to P(y = 1 2) distribution. The base dataset was randomly partitioned into training (80,000) and reference (20,000) subsets, and LightGBM classifier with default parameters was trained the d 20 features using samples from the training set.
r(x). The true labels y were then sampled from the corresponding Bernoulli
The distribution of the resulting distances in the training dataset is visualized in Figure 4 We see that most instances reside near the origin and are thus likely to bear the positive label. Instances farther from the origin get increasingly rarer. This results in an unbalanced dataset, where positive labels are far more common, with the fraction positive labels in the training dataset being roughly 88%.
Train data radius distribution
1500
Count
1000
500
Distance from center
Figure 4: The distribution of distances from the origin in the training set for the synthetic data.
23

---

D.4 Experiment with Gradually Increasing Covariate Shift
In this experiment, we simulated covariate shift by repeatedly sampling production chunks of size 2,000 while gradually increasing a threshold for the radius samples include within the production data chunk. We let the threshold increase from 0 to 0.4 in increments of 0.025. For each resulting threshold t, we conducted 1,000 trials, where in each trial we sampled chunk of data while enforcing r(æ) > t. Increasing the threshold is used to simulate gradually increasing covariate shift. This makes predicting the right label by the monitored classifier increasingly difficult for two reasons. First, when the input data distribution shifts further away from the center, the label entropy increases since P(y= = x) is equal to in the center and 0.5 at the hypersphere with r(z) = 0.5. Second, the regions further away from the center are less dense in the reference distribution (see Figure + Thus, we expect the performance of the classifier to decrease with increasing shift magnitude, and we would like our estimators to be able catch that.
After the 1,000 trials for each threshold, we averaged both the true model performances and the estimated performances for accuracy, F1 score, and AUROC. The results are plotted in Figure 5
The effect of shift degree on the estimator performance for all methods
Accuracy
F1
AUROC
0.8
extimetes
CBPE PAPE
CBPE
CBPE
IW
06
PAPE
0.4
PAPE
RT-rod
RT-mod
0.2
RT-mod
0.2
DOC
DOC
DOC
true
true
true
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0,2
0.3
0.4
shift magnitude
shift magnitude
magnituda
Figure 5: Estimator performance of each method for all three metrics with gradually increasing covariate shift. Shift magnitude corresponds threshold
We that both ATC and RT-mod generating estimates that unusable, decided drop them from further analysis and focus on the rest. For the remaining methods, we plot the Mean Absolute Error (MAE) between the estimated and true metric values. The results where we see that PAPE is consistently able to generate estimates with low error even for the most intense shifts for all three metries and has the best overall performance of all the methods tested.
in Figure 6
The Mean Absolute Error over increasing covariate shift
Accuracy
AUROC
CBPE
PAPE
0.10
IW
010
NY
0.10
I
0.08
DOC
0.08
DOC
0.08
DOC
MAE
COT
0.06
0.06
0.06
0.04
0.04
0.02
0.02
0.02
0.00
0.00
0.00
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
shift magnitude
shift magnituue
shift magnituce
Figure 6: The Mean Absolute Errors of the estimators for all three metrics with gradually increasing covariate shift.
24

---

D.5 Experiment with Different Chunk Sizes
In this experiment, fix the threshold to 0.25 and vary the chunk size to see how it affects the performance of the estimators. We test on chunk sizes of [100, 200, 400, 800, 1600, 3200], again, performing 1,000 trials for each chunk size. As in the previous experiment with increasing covariate shift, we noted that ATC and RT-mod performed so poorly that they were left out of the comparison. For the other estimators, we show the Mean Absolute Error (MAE) between the estimated and true metric values in Figure 7 Unsurprisingly, all methods provide better estimates with increasing chunk size, with PAPE and IW still improving while other estimators start flat out. Overall, PAPE shows the best performance.
The Mean Absolute Error with different chunk sizes
Accuracy
F1
AUROC
CBPE
CBPE
CBFE
0.07
PAPE
PAPE
PAPE
0.06
0.06
0.06
0.05
0.05
0.05
0.04
0.04
0.04
0.03
0.03
0.03
0.02
0.02
0.02
0.01
0.01
0.01
0.00
100
200
400
800
1600
3200
100
200
400
800
1600
3200
100
200
400
800
1600
5200
chunk
chunk size
chunk size
Figure 7: The Mean Absolute Errors for each estimator for all three metrics over different chunk sizes.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims made in the abstract and introduction describe the main characteristics of the proposed method and state that overall it has given more accurate performance estimates on the datasets used for testing compared to other methods. All the claims are true and do not overgeneralize.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper contains a dedicated section (Section 5.1 Limitations), where the theoretical and practical limitations of the proposed method are discussed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We present two theorems the main paper. Formal proofs for these theorems are provided in Appendix [ along with three additional lemmas and one additional theorem. The lemmas are needed to prove the theorems. All lemmas and theorems are explicitly numbered and properly cross-referenced, and all required assumptions are clearly and exhaustively expressed.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
The proposed method is explained in detail in Section The details of the
conducted experiments are given in 4 More specifically, we describe the (open access) datasets in Section [. Data preprocessing and model training explained in Section 4.2
The implementation details of the benchmark methods are described concisely in Section 4.3
and in more detail in Appendix A Finally, the evaluation methodology clearly described
in 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
27

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: We make our code available as a public github repository
com/pape-ressarcch cappcpppprr.... The code together with the paper enables reproducing all the experiments and their results described in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.cc/pitt wttppccccpppppciiia...ca.ocopppa http:///ggidee//dccuuubbbiiiee
hat////ooeubiieeee serbbbisssionn assionnoicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The description of the proposed method (Sections 3 and 4.3) other evaluated
benchmark methods (Section 4.3). and experimental setup (Sections 4.1 and 4.2) given in the main paper and Appendix A The low-level details of the method implementations can be found in the code provide in our public repository.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
28

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Statistical significance is provided for the experimental results supporting the main claim of the paper. is provided in section[4.4] and reproducible with the code we supply via our public repository. We did not produce error bars for our plotted graphs due to readability issues and the computational complexity would have required.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The description of resources and computation time for the described calcula- tions are provided in footnotes of the relevant sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We use public, anonymised datasets. To our best knowledge, there are no harmful consequences of our research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper contributes to foundational research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
To our best judgement, the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
30

---

Answer: [Yes]
Justification: We properly cite all sources of existing assets with their licenses. All standard py-data stack libraries we use (like numpy, matplotlib etc.) are not necessarily cited, but they are listed in the requirements. txt in our public code repository.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersaippetteeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces a novel algorithm, and the code repository linked by the authors contains its implementation. This code is released under the CC BY-NC-SA 4.0 license, which stated the LICENSE file the repository.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
31

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs in any way.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hutte://heeurpssccoooorrrrreer/p for what should or should not be described.
32


Reviews:
[{"review_id": "LIGvqsIJIo", "reviewer": "Reviewer_Yf3X", "summary": "This work proposes an evaluation of binary classifier performance under conditions of covariance shift. The suggested probabilistic approach builds on a previously established method (CBPE). It extracts data from the shifted distribution and fits a calibration mapping to this distribution. One of the benefits of this approach is its applicability to any model that uses a confusion matrix as an evaluation metric. The method has been tested on standard datasets and compared with existing techniques.", "questions": "Please see comments above", "limitations": "Limitations of the proposed approach is presented", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "he method has been tested on standard datasets and compared with existing techniques. The study is clearly presented, with thorough mathematical explanations, and demonstrates improvements over prior work. There are a few suggestions to further refine the work\n\n1) A discussion on how the proposed work is different from CBPE must be presented in detail. This will allow the reader to ascertain the novelty of the work\n\n2) What could be the possible reason for a minor improvement in F1 as compared to accuracy and AUROC (Fig 2)?  A discussion could be added.", "was_revised": true, "final_justification": "I have gone through the response, and would stand by my previous score. The work is indeed interesting but a more thorough discussion is required.", "rebuttal": "Dear Reviewer,\n\nWe appreciate your review. You’ll find our replies directly beneath every comment you have made. If any issue is still unresolved or our reply needs clarification, we’re happy to address it in the discussion period.\n\n1. *A discussion on how the proposed work is different from CBPE must be presented in detail. This will allow the reader to ascertain the novelty of the work*\n\n    **Reply:** \n    We would definitely make this more explicit in the revised version, as this was indicated in another review as well. In **3.3 Methodology** we could first outline CBPE as a preliminary and then introduce components that got us into PAPE. To intuitively explain this here: CBPE calibrates classifier's scores on the entire *reference/source* data, using classifier's uncalibrated scores and labels, treating each observation equally. PAPE performs weighted calibration on the *reference/source* data with the weights obtained using DRE (Density Ratio Estimator). Intuitively, these weights are higher for the *reference/source* data observations, which inputs are *similar* to the *production/target* observations. So the weighted/adaptive calibration is the crucial improvement over CBPE.\n    \n\n2. *What could be the possible reason for a minor improvement in F1 as compared to accuracy and AUROC (Fig 2)? A discussion could be added.*\n\n    **Reply:** \n    Indeed, the *gap* between PAPE and the method with the second lowest error (which is IW or CBPE, depending on the true performance change range) is the lowest compared to other metrics. Our intuition, based on familiarity with those estimation methods is the following:\n    - The main advantage of PAPE over CBPE is the adaptivity of calibration process to the distribution shifts. So, given that assumptions for both methods (covariate shift, common support) are fulfilled, small improvement of PAPE over CBPE indicates that data distribution shift happened in a way that does not brake calibration **in the way that it makes it harmfull for F1 estimation**. Depending on the metric that is estimated using the calibrated scores, some types of miscalibration will cause large estimation errors while other types of miscalibration may cancel out. \n    \n    - The main advantage of PAPE over IW is that PAPE is typically more sample efficient, resulting in lower variance estimates. We discuss this question in detail in **Appendix C, IW and PAPE Variance Comparison**. Intuitively, this is because PAPE is using (soft) calibrated scores from the interval [0,1] to estimate the metric, while IW uses (hard) labels (0 or 1). Using these soft scores tends to have a smoothing effect that alleviates the high variance estimates IW is known to suffer from, especially in high-dimensional settings. Our empirical experiments align with this assesment. This effect can be larger or smaller, depending on data distributions, sample sizes, and finally - the estimated metric.\n    \n    Analyzing what is the relationship between different miscalibration patterns and their effect on the estimated metric (CBPE vs PAPE) or the impact on the advantage of using probabilities vs. *reference/source* data labels (IW vs PAPE) could be a topic of another research paper."}, {"review_id": "S7UFcGBNV0", "reviewer": "Reviewer_FPvQ", "summary": "This paper proposed a method, PAPE, for evaluating binary classification models on unlabeled tabular data that estimates model performance under covariate shift, which can be applied to any performance metric defined with elements of the confusion matrix. It tests PAPE using a dataset from US census data and shows it outperforms other methods.", "questions": "Please see the weaknesses above", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper provides theoretical guarantees for the method and derives bounds for its estimation error under approximate calibration. They also use experiments to show PAPE outperforms previous methods across metrics.\n\n**Weaknesses**\n\n- The paper only considered one dataset, which is the US census data. Although different combinations of the data are provided, the single source raises questions about the generalizability.\n- Theoretical gap: The theorem claims that the method is applicable to composable metrics, and for metrics that are not composable, the bound is not guaranteed to hold. Although the empirical results show that PAPE can be used for metrics like F1, this finding isn’t supported theoretically, especially considering the paper’s claim that the method can be applied to \"any classification metric defined with elements of the confusion matrix\". This statement lacks sufficient support. Besides, only Accuracy, AUROC, and F1 metrics are considered in the evaluation, the claim of “any” might be overstated, especially since the theorem only applies to composable metrics.\n- The covariate shift assumption (ps(y|x) = pt(y|x)) is highly restrictive. In real-world deployment, we can encounter concept drift alongside covariate shift. How can practitioners verify that ps(y|x) = pt(y|x) in real applications? What happens when this assumption is violated?\n- The proposed method is closely related to the previous method CBPE. A clearer illustration of CBPE as preliminary, as well as highlighting where the proposed new method is different from it, would be helpful in the method section (section 3).\n- There are some typos in the paper, like line 188: “composable metric with with” → “composable metric with”.", "was_revised": true, "final_justification": "Most of my concerns have been addressed and I am willing to raise my score. I think the paper could still benefit from clearer explanations, such as a more detailed illustration of CBPE, clarification on whether the covariate assumption holds in the dataset (as reviewer fcbb pointed out) etc.", "rebuttal": "Dear Reviewer,\n\nThank you for your insightful comments. We’ve placed our response directly under each point below. If anything remains unclear or if we missed something, just let us know and we’ll be happy to clarify.\n\n\n1. *The paper only considered one dataset, which is the US census data. Although different combinations of the data are provided, the single source raises questions about the generalizability.*\n\n    **Reply:** \n    Folktables datasets [A] that we use (based on US Census data collection) is a NeurIPS-accepted set of dataset-benchmarks for studying geographical and temporal distributional shifts in tabular data. We use 5 different tasks, each of them having different set of features (yet many features appear in all of tasks, like *age*). Then, we collect this data for 50 different states, which gives 250 datasets for which some features are the same but their distributions are different. Finally, we fit 5 different model architectures on each of the task-state combinations - each model type performs differently so it makes it a separate task for performance estimation methods. So, it is true that it is a single source of data, but we argue that this is not one dataset.\n        \n     Additionally, in appendix **D Additional Experiments** we show results for 8 extra datasets (all that we were able to download) from TableShift - a NeurIPS-accepted benchmark with datasets for distribution shifts in tabular data [B]. These are: ASSISTments, College Scorecard, Diabetes, Food Stamps, Hospital Readmission, Hypertension, Income, and Unemployment. The last 2 are also US Census datasets. We made a decision to not add those results to the main body of the paper. This is because we could have either aggregated them together with the US census datasets - but then they would have been outweighed by the amount of the US Census data we used, or to have a separate section and result table for TableShift experiments - but there was not enough space for that. We would be grateful for a suggestion on which of the two approaches above you would find beneficial for the quality of the paper, and would be happy to add it in the revised version.\n \n\n2. *Theoretical gap: The theorem claims that the method is applicable to composable metrics, and for metrics that are not composable, the bound is not guaranteed to hold. Although the empirical results show that PAPE can be used for metrics like F1, this finding isn’t supported theoretically, especially considering the paper’s claim that the method can be applied to \"any classification metric defined with elements of the confusion matrix\". This statement lacks sufficient support. Besides, only Accuracy, AUROC, and F1 metrics are considered in the evaluation, the claim of “any” might be overstated, especially since the theorem only applies to composable metrics.*\n\n    **Reply:** \n    Two concerns are mixed here. First, as we claim in the paper, PAPE can be applied to any binary classification metric that can be defined with the elements of the confusion matrix. This is a matter of procedure, so the claim holds. \n    \n    The second thing is to assess the quality of the estimates. That is, can theoretical guarantees or empirical evidence be given to show that using PAPE results in estimation errors that are acceptable?\n    \n    We provide both theoretical and empirical analysis. First, in Theorem 3.2, we show that for any composable metric, the expected estimation error of PAPE is bounded by the calibration error. It is true that this result covers only specific (composable) metrics. It is to be noted, though, that most unsupervised estimation methods included in our benchmarks focus on experimental results and usually come with no theoretical guarantees at all. So even though the Theorem 3.2 is limited to specific metrics we still consider it a major contribution, given the current state of research in this area. Second, our empirical results show that PAPE yields the most accurate performance estimates in binary classification with tabular data, for 3 metrics: Accuracy (composable), F1 and AUROC (both non-composable, so not bounded by Theorem 3.2). Although our evaluations are restricted to these three metrics, considering the sheer amount of metrics we would have had to evaluate, had we tried to address them all, this would not have been feasible within the limited scope of the paper. \n    \n    Rereading our article with focus on that part of the review, we realized where the confusion may come from. While listing the contributions, we should write that \"We provide theoretical guarantees for the estimation quality of PAPE and derive bounds for its estimation error under approximate calibration **for composable metrics**. We will be happy to update the revised version.\n\n\n\n3. *The covariate shift assumption $(p_s(y|x) = p_t(y|x))$ is highly restrictive. In real-world deployment, we can encounter concept drift alongside covariate shift. How can practitioners verify that $p_s(y|x) = p_t(y|x)$ in real applications? What happens when this assumption is violated?*\n\n    **Reply:** \n    There are some things that practitioners can do to determine what type of shift they are dealing with (pure covariate shift or concept shift as well). First of all, they should evaluate how likely concept shift is in their use case. In some areas, concept shift is highly unlikely, when for example using Machine Learning to model physics. Physics does not change. In other cases it is highly likely - for example in adversarial use cases, like fraud detection. Then, if concept shift can't be ruled out, they could try to determine its existence and magnitude by for example getting proxy labels or subset of labels (if getting all the labels is expensive) and verify if the concept remains unchanged. Still, in all those use cases a robust method of estimating performance under covariate shift is crucial component of the full performance monitoring solution.\n\n    When the assumption of pure covariate shift is violated and there is no knowledge about concept shift, literally anything can happen with the true performance. As an example, imagine that PAPE estimates a drop in performance. Intuitively, it means that the input data has shifted to a *region* in the input distribution where the model performs worse (compared to the whole *source* distribution). If there is no concept shift (and other assumptions are met), then PAPE estimates are accurate. If there is concept shift, two things can happen. Concept can change in a way that makes the model perform better in that *region* - so that final performance will be higher than the PAPE estimate, as concept shift mitigates the effect of covariate shift. The opposite is more likely though - typically concept shift makes the performance of the model worse, so that the true performance is even lower than estimated by PAPE.\n\n\n4. *The proposed method is closely related to the previous method CBPE. A clearer illustration of CBPE \nas preliminary, as well as highlighting where the proposed new method is different from it, would be helpful in the method section (section 3).*\n\n    **Reply:** \n    We would definitely make this more explicit in the revised version, as this was indicated in another review as well. In **3.3 Methodology** we could first outline CBPE as a preliminary and then introduce components that got us into PAPE. To intuitively explain this here: CBPE calibrates classifier's scores on the entire *reference/source* data, using classifier's uncalibrated scores and labels, treating each observation equally. PAPE performs weighted calibration on the *reference/source* data with the weights obtained using DRE (Density Ratio Estimator). Intuitively, these weights are higher for the *reference/source* data observations, which inputs are *similar* to the *production/target* observations. So the weighted/adaptive calibration is the crucial improvement over CBPE.\n\n\n5. *There are some typos in the paper, like line 188: “composable metric with with” → “composable metric with”.*\n\n    **Reply:** \n    Thank you for spotting this. We will fix it in the revised version.\n\n[A] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets\n    for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.\n    Folktables library is MIT licensed\n\n[B] Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular\n    data with tableshift. Advances in Neural Information Processing Systems, 36, 2024."}, {"review_id": "Ql4X4hLg79", "reviewer": "Reviewer_WaR3", "summary": "This paper proposed PAPE, a method to estimate the performance of a binary classifier under covariate shift. Their method relies on first calibrating the model for the target distribution (using the likelihood ratio) and then using the calibrated classifiers to estimate performance  on the target distribution.", "questions": "- Why do you need the image of the classifier f to be countable?\n- Is it hard to extend this idea to multi class classification tasks? Maybe putting a hint on the paper could be useful for practitioners?\n- I think some references are missing, e.g., this paper https://arxiv.org/pdf/2212.02742.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n- This paper proposes an effective method for performance estimation under covariate shift when no labels for the target distribution. \n\nWeaknesses\n- It's well know that things break when you're trying to estimate a likelihood ratio for high-dimensional x. How robust is PAPE to an increase in dimensionality? My guess is that it is more robust than IW but less than CBPE.\n- It's not clear why PAPE is better then other method that rely on the likelihood ratio estimation. For example, why is PAPE supposed to be better than IW? \n- The method is simple and require few changes from previous approaches (this could be seen as a strength if we have a good response for the previous point).", "was_revised": true, "final_justification": "I am satisfied with the authors responses. Then I decide to increase my score.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful feedback. Below, each of your comments is followed by our response. If any answer is unclear, or if we have misunderstood or overlooked a point, please let us know and we’ll be happy to clarify.\n\n1. *It's well know that things break when you're trying to estimate a likelihood ratio for high-dimensional x. How robust is PAPE to an increase in dimensionality? My guess is that it is more robust than IW but less than CBPE.*\n\n    **Reply:** \n    Indeed. The difficulty of estimating the likelihood ratios depends on dimensionality and the joint distribution of the data. Moreover, with higher dimensionality it is more likely that the *common support* assumption will get violated (that is, there will be regions in the input space for which ${p_t(x)>0}$ but ${p_s(x)=0}$). \n\n    Our intuition is similar. For example, with increasing dimensionality, the likelihood estimator may assign a lot of zero weights to the *source* observations. This means that these observations will not be included in the process of fitting calibrator (PAPE) or weighted metric calculation (IW), making these operations more susceptible to noise. This should affect IW more as it works on *source* observation labels (0 and 1), compared to PAPE that operates on *target* data calibrated scores (this is related to the next question). Whether it will affect PAPE more than CBPE is not clear to us. For sure, if lots of weights are fixed to 0 then the weighted calibration process will result in a poor calibrator. But, intuitively, with a simple fix of not allowing the weights to be zero, this might still result in a calibrator which is at the *worst case* (when all weights are close to zero) the same as for CBPE. It is only when the likelihood estimator picks up a wrong signal and as an effect assigns higher weights to those of *source* observations to which it shouldn't, CBPE ends up better than PAPE.\n\n\n2. *It's not clear why PAPE is better then other method that rely on the likelihood ratio estimation. For example, why is PAPE supposed to be better than IW?*\n\n    **Reply:**\n    Our analyses do not guarantee that PAPE would be superior to IW in all cases. In fact, under an ideal setting, both can be shown to be unbiased and consistent estimators of model performance. The question then becomes, which of the two estimators has lower variance and is thus more sample efficient?\n    \n    We discuss this question in detail in **Appendix C, IW and PAPE Variance Comparison**. In short, we conclude that while it is possible for IW to have lower variance in some corner cases, PAPE is typically more sample efficient. Intuitively, this is because PAPE is using (soft) calibrated scores from the interval $[0,1]$ to estimate the metric, while IW uses (hard) labels (0 or 1). Using these soft scores tends to have a smoothing effect that alleviates the high variance estimates IW is known to suffer from, especially in high-dimensional settings. Our empirical experiments align with this assessment.     \n\n \n3. *The method is simple and require few changes from previous approaches (this could be seen as a strength if we have a good response for the previous point).*\n\n    **Reply:**\n    We also believe that this is the case.\n\n4. *Why do you need the image of the classifier f to be countable?*\n\n    **Reply:**\n    This assumption is always at least implicitly made when dealing with calibration. Otherwise, the conditioning event $f(x)=v $ would not make sense, because the probability of the continuous variable taking the value $v$ is zero for any $v \\in [0,1]$. This condition is easy to enforce in practice, for example, by rounding. When rounded to 3 decimals, the interval $[0,1]$ is divided into 1001 possible (discrete) values. Returning classifier's scores up to 3 decimals is more than enough for most cases, but if needed, one could go further and round to 4, 5 decimals etc. So this assumption is not limiting at all.\n\n\n5. *Is it hard to extend this idea to multi class classification tasks? Maybe putting a hint on the paper could be useful for practitioners?*\n\n    **Reply:**\n    It is actually straightforward for the typical, macro-averaged metrics (like accuracy, precision, recall, F1), but just like other researchers, we focused first on binary classification. There are two steps that must be adjusted to multiclass classification.\n\n    First - weighted calibration. Existing methods for multiclass classifier calibration can use weights as well (for example one-vs-rest Platt Scaling or calibrators-regressors just like in PAPE, or Temperature Scaling). Likelihood ratio estimation remains unchanged as it only works on inputs. \n    Second - turning probabilities into metrics (the CBPE part). This step depends on the metric of interest. For accuracy it is simple: one selects the class that was predicted (based on predictions of the classifier) and uses calibrated confidence score for this class as an estimate of accuracy. For other typical metrics (precision, F1, recall etc.) the difficulty depends on the way of averaging (macro/micro). Macro averaging is the most straightforward as it just requires repeating the same operation as in binary over all classes. Micro-averaging is more complex and we haven't explored it yet. \n\n6. *I think some references are missing, e.g., this paper https://arxiv.org/pdf/2212.02742.*\n\n    **Reply:**\n    We will update the related work section in the revised version. We just read this with great interest; it is a simple and clever idea. We did not come across this paper probably because we were looking for methods that directly estimate the value of performance metric. It seems that Detectron and PAPE could be made complementary. Using the notation of the authors - there could be covariate shifts towards subregion of P that is well-covered by training data, so CDCs will not disagree, but are still harmful - in the sense that performance of the model in those subregions of P is lower than for the whole distribution P. Detectron will not indicate it as harmful (with our current intuition about it), but PAPE will. On the other hand, there could be covariate shifts outside of P (or rather: outside of R), where the *common support* assumption for PAPE is violated and hence one cannot trust the estimates, and these should be well indicated by Detectron."}, {"review_id": "EtIFUEsJwI", "reviewer": "Reviewer_Fcbb", "summary": "This paper proposes the Probabilistic Adaptive Performance Estimator (PAPE), a novel method to estimate binary classification metrics defined using the confusion matrix, by extending the Confidence-based Performance Estimation (CBPE) in the following two ways:\n(i) The proposed method does not need a perfectly calibrated model.\n(ii) The proposed method can adapt to covariate shift.\n\nFirst, the authors consider the relaxed notion of calibration\ncalled $\\alpha$-approximately multicalibration, and they show that if the likelihood ratio belongs to the class of test functions used in the measure of $\\alpha$-approximately multicalibration, the measure maintains the inequality in Eq. (2) before and after covariate shift (Theorem 3.1).\nFurthermore, Theorem 3.2 shows that the use of the $\\alpha$-calibrated score in estimating the metric as in Eq. (4) only suffers bias up to $\\alpha$.\nThen, the paper presents a method that corrects the bias due to covariate shift using importance sampling when estimating the function $c$.\nFinally, the experiment demonstrates the usefulness of the proposed method.", "questions": "- Is the dataset expected to satisfy the covariate shift assumption?\n- Which part of the paper does this correspond to?: \"Furthermore, PAPE models the full distribution of the estimated metric, which can be leveraged to produce valid confidence intervals for the estimates\".\n- l.42, \"PAPE is provably robust against covariate shift\": what is the meaning of robustness here?", "limitations": "- The work assumes that the image of $f$ is a countable set.\n- The work is limited to binary classification..", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The paper studies an important problem in a challenging scenario commonly encountered, covariate shift.\n- The extension to this setup and the removal of the requirement of a perfectly calibrated model seem relevant and useful.\n- The proposed method outperforms the other methods in the experiment.\n\n### Weaknesses\n- The message of the paper is a bit confusing to me. The paper claims that it works on any metrics defined using the confusion matrix, but Definition 3.1, Definition 3.2, Theorem 3.1, and Theorem 3.2 all only concern very specific cases.\n- There is no analysis of estimation errors, despite this claim \"We provide theoretical guarantees for the estimation quality of PAPE and derive bounds for its estimation error under approximate calibration.\" Theorem 3.2 is probably related to the bias, but apart from that, it does not provide an analysis on the estimation error.\n- In particular, the likelihood ratio estimation being seemingly the statistically hardest part, it may be important to study its impact of the error on the quality of the estimation of the performance metric. However, this error is assumed to be zero in the presented analyses.\n- I found some of the descriptions unclear, especially for the entire proposed algorithm, composable metrics, and the IW method. I do not know if the reader would be able to reproduce the experiments.\n- There is no (intuitive) explanation on why we can expect the dataset to satisfy the covariate shift assumption. It would be nice if there were simulations in which we know that the assumption holds.\n- I could not find which part of the paper addresses this matter: \"Furthermore, PAPE models the full distribution of the estimated metric, which can be leveraged to produce valid confidence intervals for the estimates\".", "was_revised": true, "final_justification": "I thank the authors for the useful discussion. They addressed some of my misunderstandings and concerns. I believe that I now understand the paper better than I did when I wrote the first review, and I decided to write an overall summary of my evaluation below.\n\n### My final recommendation: 3 (Borderline reject; the same as my original score)\n- The novelty and contribution might be marginal relative to the paper of CBPE [7].\n-   As far as I can tell, the novelty of this paper is in extending CBPE to the case of covariate shift, which seems to be an important problem.\n-   The solution of modifying the calibration mapping estimation by likelihood ratio weighing is a simple solution that seems to work, which is a good point.\n-   On the other hand, the rest of the contributions, the theory and the empirical studies are underwhelming in my opinion (please see the following sections).\n\n### Theoretical guarantees are of limited relevance\n\n-  About Section 3.2 and Theorem 3.1\n\n    Section 3.2 (Approximate Confidence Calibration), with Theorem 3.1 about $\\alpha$-approximately multicalibration, seems to be a separate result that does not have a connection to the rest of the paper including the proposed method.\n    For this reason, Theorem 3.1 appears less relevant. Also, this structure of the paper makes the reading experience less smooth.\n\n-  [Not satisfactorily addressed] No study on the error coming from the likelihood ratio estimation, which is the bottleneck step of the proposed method.\n\n    The authors comment on this, \"we derive a bound for the estimation error under imperfect weights in Appendix B.3.\"\n    The result (Theorem B.4 in Appendix B.3) is an extension of Theorem 3.1 and quantifies the sensitivity (or robustness) of the metric $K(f, D_t)$ with respect to the misspecification of $\\mathcal{H}$ for the likelihood ratio.\n    \n    It is of quite limited interest for the following reasons.\n    \n    -   It only addresses $K(f, D_t)$ but not other metrics.\n    -   We usually do not know $\\min_{h \\in \\mathcal{H}} \\epsilon(h, w_{s\\to t})$.\n    -   We cannot exactly achieve $\\min_{h \\in \\mathcal{H}} \\epsilon(h, w_{s\\to t})$ by training with finite data. We should analyze the generalization error.\n    -   It does not provide any guarantee on the estimation error of the proposed method.\n    \n- In connection with these points, the authors also write \"if density ratios are not exact, it is a reasonable assumption (by the principle of insufficient reason) that both estimators would suffer roughly equally on average\" (l. 981).\n    However, this is not a reasonable assumption but a strong claim that needs a proof.\n\n-  [Not satisfactorily addressed] Limited setup\n\n    This study focuses on a somewhat underwhelming, simplified setup, where\n    \n    -   $f$ has a countable image\n    -   and $y$ is binary (binary classification)\n    \n    Most studies that appear in NeurIPS handle multi-class classification with continuous variables, unless there are some intractability issues. However, this paper does not provide good reasons to justify this simplification. This brings up the question about this study's significance for the standards of this conference.\n    \n    -   Theorem 3.2 is a property inherited from CBPE [7], on which the proposed method is based. However, this work had to make an additional assumption (composable metrics) for the guarantee.\n\n### Results are overstated\n\n-  [Not satisfactorily addressed] ll.42-43: \"Furthermore, PAPE models the full distribution of the estimated metric, which can be leveraged to produce valid confidence intervals for the estimates.\"\n\n    The paper does not provide a proof. The authors mentioned that it will be similar to CBPE.\n    However, it is not convincing because the proposed estimate involves density ratio estimation, for which even asymptotic normality will not hold unless we know the parametric model for the density ratio (but it is rarely the case in practice).\n\n-  [Addressed] Misleading claim: \"It can be applied to any performance metric defined with elements of the confusion matrix.\"\n\n    The theoretical gurantee (Theorem 3.2) only holds for composable metrics.\n    The authors promise that they will adjust the claim to something like \"we provide theoretical guarantees for the estimation quality of PAPE and derive bounds for its estimation error under approximate calibration for composable metrics.\"\n    A similar adjustment should also be applied to other parts of the paper such as the abstract.\n\n### About experiments\n\n-  [Partially addressed] The covariate shift assumption of the datasets in the experiments \n\n    The authors pointed out that Appendix D had additional experiments for which the dataset shift conditions were exactly known. They also suggest adding more controled datasets to ensure the assumption. They did not perform the additional experiments during the discussion, but I understand it is not an easy thing to do within this short time.\n\n-  Tables 1 and 2 report aggregated results \"over multiple models and chunks of data from different datasets\" (See ll.272-278)\n\n    I find it unusual that the paper chose this way of reporting results. I misunderstood these tables at first because of this.\n    The authors themselves also mention that aggregation \"leads to skewed and uninterpretable results.\" It would be more informative if the paper reported the results separately.\n\n### [Partially addressed] Unclear description of the proposed method\n\nMy initial concern around this issue was more with the reproducibility. The authors provide the complete code for the implementation and experiments, which perfectly address this concern.\n\nNevertheless, I still think that the proposed algorithm should be described more clearly.\n\nFor instance, the description about the calibration mapping estimation in the methodology section is the following part:\n\n-   ll.171-172: \"Once these weights are available, we can use them to train a weighted calibration mapping $c$ by fitting it to $\\\\{(f(X_i^s), Y_i^s)\\\\}\\_{i=1}^{n_s}$.\"\n\nThere can be many ways to \"fit\" a mapping to a dataset. There may be a hypothesis class of the mappings unexplained in this part as well.\nThe authors explain that the details are in the experiment section, but the reader would expect that the methodology is explained in the methodology section. The methodology becomes harder to follow when mixed with implementation or experimental details. I am not sure if saying \"an LGBM Regressor\" was used is sufficient either. It may be more important to explain in what way a regressor was applied with the data.\n\n-   l.167: \"we fit the DRE model on $(X^{\\mathrm{st}}, z^{\\mathrm{st}})$ as described in [30, chapter 2.7.5]\"\n\nIt is great that the paper gives references, but solely relying on references to describe the proposed method like this makes the paper less self-contained and less comprehensible. Moreover, some readers night not have the access to the book.\n\n### Minor clarity issues\n\nFor clarity issues, several things such as \"composable metrics\" and the notation $P_{\\mathcal{D}}$ and $\\mathbb{E}_{\\mathcal{D}}$ are not clearly defined. It is not immediately clear either which varaibles are considered as random variables.", "rebuttal": "Dear Reviewer,\n\nThank you for the detailed review. Your remarks appear below, each followed by our answer. Should any response need more detail, or if we have misinterpreted a point, please let us know.\n\n1. *The message of the paper is a bit confusing to me. The paper claims that it works on any metrics defined using the confusion matrix, but Definition 3.1, Definition 3.2, Theorem 3.1, and Theorem 3.2 all only concern very specific cases.*\n\n**Reply:** Two things are conflated here. First, as we claim in the paper, PAPE can be applied to any binary classification metric that can be defined with the elements of the confusion matrix. This is a matter of procedure.\n\nThe second thing is to assess the quality of the estimates. That is, can theoretical guarantees or empirical evidence be given to show that using PAPE results in estimation errors that are acceptable?\n\nWe provide both theoretical and empirical analysis. First, in Theorem 3.2, we show that for any composable metric, the expected estimation error of PAPE is bounded by the calibration error. It is true that this result covers only specific (composable) metrics. Second, our empirical results show that PAPE yields the most accurate performance estimates in binary classification with tabular data, for 3 metrics: Accuracy (composable), F1 and AUROC (both non-composable, so not bounded by Theorem 3.2).\n\nRereading our article with focus on that, we realized where the confusion may come from. We should write that \"We provide theoretical guarantees for the estimation quality of PAPE and derive bounds for its estimation error under approximate calibration **for composable metrics**\". We will update this in the revised version.\n\n2. *There is no analysis of estimation errors, despite this claim \"We provide theoretical guarantees for the estimation quality of PAPE and derive bounds for its estimation error under approximate calibration.\" Theorem 3.2 is probably related to the bias, but apart from that, it does not provide an analysis on the estimation error.*\n\n**Reply:** Theorem 3.2 bounds the estimation error in expectation for composable metrics, so it indeed measures bias. As we wrote in the answer to the previous concern - we will make the claim cited in that point more specific in the revised version, by adding \"for composable metrics\" at the end. Estimation error variance analysis is not provided. Given the current state of research in this area, where most unsupervised estimation methods included in our benchmarks focus on experimental results and usually come with no theoretical guarantees at all, we still consider the theoretical analysis that we provided a major contribution.\n\n3. *In particular, the likelihood ratio estimation being seemingly the statistically hardest part, it may be important to study its impact of the error on the quality of the estimation of the performance metric. However, this error is assumed to be zero in the presented analyses.*\n\n**Reply:**\n\nWhile in the main part of the paper we only deal with exact likelihood ratios, we derive a bound for the estimation error under imperfect weights in **Appendix B.3**, as stated on lines 152-154 in the main part of the paper. \n\n4. *I found some of the descriptions unclear, especially for the entire proposed algorithm, composable metrics, and the IW method. I do not know if the reader would be able to reproduce the experiments.*\n\n**Reply:** We are committed to improving these descriptions and would love to hear your feedback on what is unclear precisely in your opinion. When it comes to reproducibility, we attach the full repository (L610-611) that allows to reproduce the experiments. Self-contained class that implements IW can be specifically found in the repository in pape_r/methods/Importance_Weighting.py. Other methods are included there as well.\n\n5. *There is no (intuitive) explanation on why we can expect the dataset to satisfy the covariate shift assumption. It would be nice if there were simulations in which we know that the assumption holds.*\n\n**Reply:** We deliberately chose not to simulate the shifts as we were afraid of the concerns that we introduced the shifts in ways that favor our methods. Folktables (based on US Census data) [A] is a NIPS-accepted benchamark for studying temporal data distribution shifts. There are signals that indicate these datasets are relevant. First, we see that the base benchmark (that assumes that performance is constant - named TEST-SET) performs *bad* - which means that performance changes significantly. We comment on that in the paper, saying that TEST SET NMAE would be around 0.8 if there where only random normal fluctuations in the performance metric, while it was 1.68 for accuracy, 1.45 for AUROC and 2.53 for F1. Then, the question is whether the performance changes can be attributed to covariate shift? For example, the drop in performance that one can see in Figure 1. at around chunk number 190 can be attributed to the fact, that at this time US Census performed large surveys at universities and dormitories (this can be noticed while analyzing other variables - describing age, education level, current place of living etc.) This leads to covariate shift (a *subregion* of input space got overrepresented). Moreover, performance estimation methods, which are designed to work with covariate shift only, perform better than benchmark TEST-SET benchmark. This also suggest that covariate shift is responsible for non-neglible proportion of performance change. \n\nAdditionally, in appendix **D Additional Experiments** we show results for 8 extra datasets (all that we were able to download) from TableShift [B] - a NIPS-accepted benchmarking datasets for distribution shifts in tabular data [A]. These datasets have simulated shifts introduced by the authors, typically the Out-of-Domain data contains a category in one of the categorical variables that was not available in the training data. \n\nIn the revised version, we could perform experiments with simulated data where we control some variable and introduce pure covariate shift (just like TableShift) or we could add discussion like the one above. Please tell us what do you prefer. \n\n6. *I could not find which part of the paper addresses this matter: \"Furthermore, PAPE models the full distribution of the estimated metric, which can be leveraged to produce valid confidence intervals for the estimates\".*\n\n**Reply:** As an extension of the CBPE method [C], PAPE inherits this property directly from CBPE, where each metric is treated as a random variable whose full distribution is then derived. \n\n7. *Is the dataset expected to satisfy the covariate shift assumption?*\n\n**Reply:** See 5.\n\n8. *Which part of the paper does this correspond to?: \"Furthermore, PAPE models the full distribution of the estimated metric, which can be leveraged to produce valid confidence intervals for the estimates\".*\n\n**Reply:** See 6.\n\n9. *l.42, \"PAPE is provably robust against covariate shift\": what is the meaning of robustness here?*\n\n**Reply:** A problem with earlier confidence-based methods such as ATC or CBPE has been the effect of deteriorating calibration under covariate shift [D]. It is a known issue in fairness research that simple calibration on the entire *source* population may give uncalibrated predictions for the subgroups/regions of the input space. We cannot calibrate wrt. to all the potential groups that we may observe in the future *target* data, but once we observe the distribution of *target* inputs, we can adapt to it. That is what PAPE does. PAPE adapts calibration to the new input distribution (by using likelihood ratios learnt with DRE) which ensures that it remains calibrated under *target* input distribution (assuming it is a covariate shift). \n\n10. *The work assumes that the image of is a countable set.*\n\n**Reply:** This assumption is always at least implicitly made when dealing with calibration and it is not limiting at all. Otherwise, the conditioning event $f(x)=v $ would not make sense, because the probability of the continuous variable taking the value $v$ is zero for any $v\\in [0,1]$. This condition is easy to enforce in practice, for example, by rounding. When rounded to 3 decimals, the interval $[0,1]$ is divided into 1001 possible values. Returning classifier's scores up to 3 decimals is more than enough for most cases, but if needed, one could go further and round to 4, 5 etc.\n\n11. *The work is limited to binary classification.*\n\n**Reply:** The proposed method can be extended to multiclass classification in a straightforward way. We are not sure yet about multilabel cases.\n\n12.  *conc(., .) is not defined.* \n\n**Reply:** It is an operation of concatenation, which is introduced in the sentence \"Next, we concatenate the features $X_{st} = conc(X_s, X_t)$\". Should we simply write $X_{st} = concatenate(X_s, X_t)$ or $X_{st} = stack(X_s, X_t)$ to make it clearer?\n\n\n13. *l.292, \"paired Wilcoxon signed-rank test p = 0.0\": is p the p-value? Is it exactly zero?*\n\n**Reply:** Thank you for spotting this. We will update with $p<0.001$.\n\n[A] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets\nfor fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.\nFolktables library is MIT licensed.\n\n[B] Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 36, 2024.\n\n[C] Juhani Kivimäki, Jakub Białek, Wojtek Kuberski, and Jukka K. Nurminen. Performance\nestimation in binary classification using calibrated confidence. arXiv preprint arXiv:2505.05295, 2025.\n\n[D] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems 32 (2019)."}]

Author rebuttal:

Meta-review: **(a) Paper summary**  
The paper introduces PAPE, a method to estimate classifier performance under covariate shift without labels by calibrating source predictions with density-ratio weights. It works for confusion-matrix metrics in general, has theoretical guarantees for composable metrics, and shows strong results against CBPE and IW on Folktables and TableShift datasets.

**(b) Strengths**  
Tackles an important monitoring problem with a simple and practical extension over CBPE using target-aware calibration. Provides useful theory, clear variance advantages over IW, and broad empirical validation across many dataset–model pairs.

**(c) Weaknesses**  
Scope claims were initially too broad since theory covers only composable metrics, and the method relies on the covariate shift assumption with limited analysis under violations. Variance and density-ratio errors are not fully studied, multiclass is not demonstrated, and there are some clarity and reporting issues; one reviewer found novelty modest.

**(d) Reasons for decision**  
The method’s practical relevance, consistent empirical improvements, and simple implementation outweigh these limitations. Author commitments to clarify scope, distinguish PAPE more clearly from CBPE, fix presentation issues, and state limitations make the remaining concerns acceptable.

**(e) Discussion period**
Reviewers questioned novelty, scope of theory, covariate-shift validity, variance/error analysis, multiclass, and reporting. Authors agreed to clarify some explanations in the method section, emphasize the composability assumption of theoretical claims, add a reference and do several other minor fixes. With one accept and two borderline accepts after rebuttal against one borderline reject, the strong empirical results and author clarifications support acceptance.
Decision: Accept (poster)