Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Authors: Angéline Pouget
Mohammad Yaghini
Stephan Rabanser
Nicolas Papernot
Venue: ICML 2025 oral (2025)
Keywords: suitability
reliability
robustness
classifier
unlabeled data
Primary area: social_aspects->everything_else
TL;DR: The suitability filter detects performance degradation in machine learning models by comparing accuracy on unlabeled user data and labeled test data.

Abstract: Deploying machine learning models in safety-critical domains poses a key challenge: ensuring reliable model performance on downstream user data without access to ground truth labels for direct validation. We propose the _suitability filter_, a novel framework designed to detect performance deterioration by utilizing _suitability signals_—model output features that are sensitive to covariate shifts and indicative of potential prediction errors. The suitability filter evaluates whether classifier accuracy on unlabeled user data shows significant degradation compared to the accuracy measured on the labeled test dataset. Specifically, it ensures that this degradation does not exceed a pre-specified margin, which represents the maximum acceptable drop in accuracy. To achieve reliable performance evaluation, we aggregate suitability signals for both test and user data and compare these empirical distributions using statistical hypothesis testing, thus providing insights into decision uncertainty. Our modular method adapts to various models and domains. Empirical evaluations across different classification tasks demonstrate that the suitability filter reliably detects performance deviations due to covariate shift. This enables proactive mitigation of potential failures in high-stakes applications.

Suitability Filter: A Statistical Framework
for Classifier Evaluation in World Deployment Settings
Angéline Pouget
Mohammad Yaghini? Stephan Rabanser"
Nicolas Papernot
Abstract
Suitabiiity Filter
4.0
Deploying machine learning models in safety- critical domains poses key challenge: ensur- ing reliable model performance on downstream user data without access to ground truth labels for direct validation. We propose the suitabil- ity filter, a novel framework designed detect performance deterioration by utilizing suitabil- ity signals-model output features that sitive to covariate shifts and indicative of poten- tial prediction errors. The suitability filter evalu- ates whether classifier accuracy on unlabeled user data shows significant degradation compared to the accuracy measured on the labeled test dataset. Specifically, ensures that this degradation does not exceed pre-specified margin, which repre- sents the maximum acceptable drop in accuracy. To achieve reliable performance evaluation, we aggregate suitability signals for both test and user data and compare these empirical distributions using statistical hypothesis testing, thus providing insights into decision uncertainty. Our modular method adapts various models and domains. Empirical evaluations across different classifica- tion tasks demonstrate that the suitability filter reliably detects performance deviations due to co- variate shift. This enables proactive mitigation of potential failures in high-stakes applications.
Bata
Estimated Probability Denatty
3.5
SUITABLI
U'SSITTBLE
2.5
2.0
1.5
1.0
0.5
0.0
Per-Sample Prediction Correcmnss Probabiltty
Figure I. A model M is suitable for use on D,. if its accuracy does not fall below the accuracy Drest by more than pre- defined margin m. The suitability filter calculates per-sample pre- diction correctness prohabilities for both test and datasets and compares the two distributions through statistical non inferrior testing. The dashed vertical lines represent mean values of the distributions corresponding the estimated accuracies.
distributions differ, performance can drop and cause harm. For example, credit risk models trained on limited historical data may fail in new contexts, disproportionately harming underserved communities through unfair denials or higher interest rates (Kozodoi et al., 2022). Ideally, deployed pre- dictions could be compared directly to ground truth for real- time performance monitoring. However, ground truth may be unavailable (e.g., limited expert labeling (Culverhouse et al., 2003)), unobservable (e.g., counterfactual outcomes in healthcare (Tal, 2023)), or only available much later (e.g., re- cidivism in law enforcement (Travaini et al., 2022)), thereby causing significant monitoring challenges in deployment.
1. Introduction
Machine learning (ML.) models often operate in dynamic, uncertain environments. After model is tested on a holdout set, satisfactory evaluation result typically leads to produc- tion deployment. However, test and deployment covariate
In this work, we tackle the challenge determining whether classification accuracy on unlabeled user data degrades sig- nificantly compared to labeled holdout dataset- issue not directly addressed by existing methods. Our approach combines insights from distribution shift detection, unsuper- vised accuracy estimation, selective prediction, and dataset inference into a novel performance deterioration detector.
'ETH Zurich, work performed while interning at the Univer- sity of Toronto and the Vector Institute University of Toronto and Vector Institute. Correspondence Angéline Pouget <ange- line, nouget gmail.com >
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Central to our solution is the suitability filter, an auxil-

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
iary function J. X' (SUITABLE, INCONCLUSIVEJ;
tecting model performance deterioration during deploy- ment. Our filters detect covariate shift via unlaheled representative dataset provided by the model user.
Given an unlabeled user dataset Da Dargee sampled from the target deployment domain, and a labeled test dataset Dieat ~ Doorce sampled from the original training domain, the filter assesses whether classifier accuracy on Du falls below that on Dest by more than a predefined margin m. Our work proposes both (i) a framework for the suitability filter; as well as (ii) a perffornrn default instantiation of the filter using are broadly applicable across classifiers, independent of the model architecture or the training algorithm.
2. We propose a statistical testing framework to build suit- ability filters that aggregate various signals and output a suitability decision. Leveraging formal hypothesis test- ing, our approach enables control of the false positive rate via a user-interpretable significance level.
suitability signals that
3. We theoretically analyze the end-to end-to-end false positive rates of our suitability filters and provide sufficient condi- tions for bounded false positive rates. We then consider practical relaxation of this condition, and suggest an adjustment to the prediction margin that maintains our end-to-end bounded false error guarantees.
To arrive at its decision, the suitability filter relies on suit- ability signals (model output features such as maximum logit/softmax or predictive entropy). These signals are sensi- tive to covariate shifts and can indicate potential prediction errors. In particular, we design per-sample prediction cor- rectness estimator leveraging these suitability signals. This allows us to assess consistency in the model's predictive behavior on both D., and Drest by aggregating sample-level suitability signals. As a result, we are able to detect subtle shifts indicative of changes in model performance. As il- lustrated in Figure we then compare the means of these distributions (i.e., the estimated accuracies) arrive a suitability decision. Our decisions rely on statistical test- ing to assess whether the estimated difference in means is significant, thus offering a measure of predictive uncertainty.
4. We demonstrate the practical applicability of suitability filters across 29k experiments on realistic data shift sce- narios from the WILDS benchmark. On MMoW-WILDS, for example, we are able to detect performance deterio- ration of more than 3% with 100% accuracy as can be seen in Figure 4.
2. Related Work
Our work builds on insights from distribution shifts, accu- racy estimation, selective prediction, and dataset inference.
To ensure the reliability of suitability decisions, we study the statistical guarantees for the suitability filter. Specifically, we identify the theoretical conditions that ensure a bounded false positive rate for end-to-end suitability decisions. We also consider the practical scenarios where such condition may not hold and provide a relaxation of this theoretical condition. This allows model providers to ensure reliability of suitability decisions in spite of theoretical limitations.
Distribution Shift Detection. Distribution shift detec- tion methods aim to identify changes between training and deployment data distributions Muuñonero-Candee et al., 2022), generally requiring access to ground truth labels. Early research emphasizes detecting shifts in high- dimensional data using approaches such statistical testing on model confidence distributions (Rabanser et al., 2019) or leveraging model ensembles (Ovadia et al., 2019; Arpit et al., 2022). Recent efforts increasingly prioritize inter- preting shifts (Kulinski & Inouye, 2023; Koh et al., 2021; Gulrajani & Lopez-Paz, 2021) and mitigating their impact on model performance (Cha et al., 2022: Zhou et al., 2021; Wiles al., 2021; Zhou al., 2022; Wang et al., 2022). Some works argue that while small shifts are unavoidable, the focus should be on harmful shifts that lead to signifi- cant performance degradation (Podkopaev & Ramdas, 2021; Ginsberg et al., 2022). These approaches aim to detect covariate shifts and subsequently assess their impact on per- formance. To do so, they rely on ground truth labels or model ensembles to evaluate harmfulness. This assumption makes these techniques unsuitable for our setting where we aim to detect performance degradation without label access.
Building on these theoretical insights, we empirically show that the filter consistently detects performance deviations arising from various covariate shifts, including temporal, ge- ographical, and subpopulation shifts. Specifically, we assess the effectiveness of our approach using real-worte datasets from the WILDS benchmark (Koh et al., 2021). These in- clude FMOW-WILDS for land use classification (Christie et al., 2018), Civi Commentss LILS for text toxicity classification (Borkan et al., 2019), and RxRx1 -WILDS for genetic perturbation classification (Taylor et al., 2019). Furthermore, we explore how accuracy differences between user and test datasets impact the filter's sensitivity, analyze calibration techniques to control false positives, and con- duct ablations on suitability signals, sample sizes, margins, significance levels, and classifier options.
In summary, our key contributions are the following:
Unsupervised Accuracy Estimation. Unsupervised ac- curacy estimation, also known as AutoEval (Automatic
We introduce suitability filters as a principled way of de-
2

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Model Evaluation (Deng & Zheng, 2021)), aims esti- mate model's classification accuracy (a continuous met- ric) unseen data without relying on ground truth labels. Early approaches in this field primarily centered on model confidence, calculated as the maximum value of the soft- max output applied to the classifier's logits, and related metrics which we demonstrate to be valuable suitability signals (Hendrycks & Gimpel, 2016; Garg et al., 2022: Kivimäki et al., 2024; Bialek et al., 2024; Guillory al., 2021; Lu et al., 2023; Wang et al., 2023; Hendrycks & Di- etterich, 2018; Deng et al., 2023). Our work differs from these approaches in three key ways: we focus on reliably detecting performance deterioration (a binary decision) in relation to a labeled test dataset using statistical testing.
such a model M on labeled data sampled from a source dis- tribution Dourre over domain X. Specifically, the provider usually partitions the data into two disjoint subsets: train- ing dataset Duaann ~ Dsuure.. which is used to optimize the parameters of M and a test dataset Drest ~ 'D Dasurcee which is reserved to evaluate the performance of M on unseen data.
To ensure an unbiased evaluation of model performance, these two datasets are disjoint, i.c., Draain n Dist = 0.
Model User. A model user interested in deploying model M on their data provides unlabeled, representative data sample D. Parget. In most scenarios of practical interest, Durget differs from Dsouree. i.e., D Durget * Doource. Note that if D,, were to be drawn from the same distribution as Diest, the model's performance on both datasets would be identical in expectation, eliminating the need for the suitability filter. The user might be third party looking to use the model, or the model provider and user could be the same party.
Selective Classification. Selective classification tech- niques aim to detect and reject inputs a model would likely misclassify, while maintaining high coverage and accepting as many samples as possible (Chow, 1957; El-Yaniv al., 2010). In contrast to selective classification, we do not re- ject or accept individual input data samples. Instead, we leverage sample-level signals and aggregate them to provide a statistically grounded suitability decision for the entire dataset. Initial selective classification methods for neural networks base the rejection mechanism on the model pre- diction confidence (Hendrycks& & Gimpel, 2016; Geifman & El-Yaniv, 2017), signal that we also leverage in our work.
Suitability Filter. The suitability filter assesses whether the performance of classifier M on unlabeled user data D, degrades relative to the known performance on the labeled test dataset Diea. In our work, we focus on model accuracy as the performance metric. We define suitability as follows:
Definition 3.1 (Suitability). Given a classifier M : X + V, a test data sample Dust ~ D a user data sample Du ~ Durres, and a performance deviation margin m E IR, we define M as suitable for use on D. if and only if the estimated accuracy of M on D., deviates at most by m from the accuracy on Drest- Formally:
Dataset Inference. Our approach is inspired by dataset inference (Maini et al., 2021), technique used to deter- mine whether a model was trained Similarly to dataset inference, we compare suitability dis- tributions between two different data samples through sta- tistical hypothesis testing. However, in contrast to dataset inference, we focus only on evaluation and aim to detect possible performance deterioration, essentially reversing the null and alternative hypotheses. Moreover, dataset inference relies on representative data from both sample domains- the original source and the deployed target domain to train a confidence regressor. Instead, we assume that label access is only available in data sampled from the source domain.
particular dataset.
1
Σ ( ZED
Dul
(1)
Σ
T(M(x)= = y) m.
Drest
( Dest
Here, 11{}} is the indicator function and ((x) represents an oracle that provides the true label y for any input (the ground truth label is unavailable for samples E D.))
Definition 3.2 (Suitability Filter). Given model M, test dataset Drest user dataset Du. performance metric g and performance margin m as in Definition 3.1, we define a suitability filter to be a function fa : X - (SUITABLEE INCONCLUSIVE) that outputs SUITABLE if and only if M is suitable for use on D,, according to Definition 3.1 with high probability and INCONCLUSIVE otherwise.
3. Problem Formulation
Our suitability filter framework distinguishes between the model provider, who trains and tests the classifier the source distribution, and the model user, who applies the model to (possibly distributionally shifted) target data.
4. Method
Model Provider. Let y' = (1,...,,k) k) denote the label space, representing the set of all possible output labels for a classification problem with i classes. We define our predic- tor as a model M : X + y mapping inputs from a covariate space X' to classification decisions. A model provider trains
The suitability filter is introduced as statistical hypothesis test designed to assess if the performance of a model on user data D., deviates from its performance on a test dataset Diest by more than a specified margin m. By aggregating
3

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
DDessiyy
Statistical Hypothesis Test
Prediction correctness prohubility estimator
Classifier a y
Da r Darper
Density
INCONCLUSIVE p a
C: R' - [0.
722
Prediction comeciness prohubility estimator C: R" [0. 1]
Density
SUITABLE upsa
Classlier M x + y
Drea Danss
Pe
Ho Horga
Parrs
Classifier trained on Dinn: : + y
Signal in E
Logit s(;; M)
1
D4 Drasse
mf 0.5
PP
Classitier A is suitable for deployment on D..
Signal e R
Per-sample prediction correctness probability ostimator trained on Da: C: + [0, 1).
Aoo(M, D.) Acc(Mr Drze) m
bolds with high probubility.
Drnn ~o DesIne
Signal
Aggregated into Pe data points.
Figure 2. Schematic overview of the suitability filter. The suitability filter assesses whether model performance on user sample D,
deviates from its performance the test dataset Dies. This is achieved by combining different suitability signals {si,....e Sa s., } to estimate per-sample prediction correctness and comparing the distribution of these estimates between the two datasets using a statistical test.
a diverse set of suitability signals predictive of classifier correctness, the test compares predicted accuracy between Du and Dist using a non-inferiority test to ensure the mean performance difference does not exceed the performance margin m (Wellek, 2002; Walker & Nowacki, 2). We present an schematic overview of our approach in Figure 2.
train the parameters of the prediction correctness estimator. This dataset is essential because it enables the suitability filter to learn the relationship between different signals and classifier prediction correctness. D&r has to be separate from both Dumin and Drest avoid overfitting to these samples.
For each sample E Dst, the selected signals {si...., ss). which are functions of both the sample and the model M, are evaluated, normalized, and aggregated into single feature vector s(x; M) = [s] (z; M), $2;;;
4.1. Suitability Signals
The first step in constructing the suitability filter is to select set of signals .......,.5. } that are predictive of per-sample classifier prediction correctness. These signals are inher- ently dependent on the model M and capture information about its predictions and confidence levels. As discussed in Section 2, a variety of signals have been proposed in the literature on unsupervised accuracy estimation, selective classification, and uncertainty quantification. Such signals include but are not limited to the maximum logit/softmax scores, the energy of the logits, or the predictive entropy. The exact signals used in this work have been selected to ensure the broad applicability of the suitability filter across diverse settings as outlined in more detail in our experiments (Section 5) and in Appendix A.4.2 and A.4.3. We note that any signal that can be computed for an individual sample be incorpo-
M)] E R.S. The suitability filter framework leverages this feature vector s(z; M) to predict whether the model M correctly classifies the input x. This is achieved by training a pre- diction correctness classifier C RS + 1), that es- timates the per-sample probability of prediction correct- ness (I() on the hold-out dataset Dat. In particular, we want to minimize the binary cross-entropy loss between the true correctness label c = (x) = y) and Pe(x)) for each (a, y) E Dzt We instantiate C as a logistic regres- sor' which models the prediction correctness probability (x() [w M) + b), where (2) ite is the sigmoid function.
We can then leverage C' to estimate per-sample prediction correctness for user data samples x E D, (since calculating Pe(x) does not require ground truth label access) as well as the test data D Next, we discuss steps to verify and ensure that C' generalizes effectively to D,, Diarget-
and is predictive of prediction correctness rated into our framework, allowing for flexible extension based on the specific task, dataset, or model M.
4.2. Per-Sample Prediction Correctness Estimator
To ensure that the mean estimated probabil- ity of prediction correctness directly reflects accuracy, Pe (1))
Calibration.
To learn per-sample prediction correctness estimator, we require the model provider to have separate, labeled hold- out dataset D&t D sourco. While ultimately the goal is to assess performance on the unlabeled D,, ~ Daureet provided by the user, the hold-out dataset D&r serves as a proxy to
Note that while more flexible model classes can be used for the correctness estimator C, we did not find any empirical evi- dence that they provide consistent performance improvement over logistic regression (see Appendix A.4.4 for details).
4

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
must be well-calibrated for both samples from Paouree and Dunges- However, absent specific assumptions about the dif- ferences between Ssouree and Purget, achieving this desired calibration is impossible in practice (David et al., 2010).
Here, po(2))) represents the estimated probability of predic- tion correctness for each sample 2::-
Hypothesis Setup. We define the true means of the esti- mated prediction correctness probabilities for data drawn from the source and target distributions follows:
One reasonable assumption under which such calibration issues can be mitigated if the potential target distributions consist of subpopulations of the source distribution. In credit scoring, for instance, the target distribution may include subpopulations S, such minority groups individuals with limited credit histories, who are underrepresented in the training data. In such scenarios, multicalibration techniques can ensure that C' provides accurate predictions for every subpopulation S E C, thereby improving reliability across all possible Durget (Hébert-JJhnson et al., 2018). Here, C denotes a collection of computationally identifiable subsets of the support of D source and i ~ S is sample drawn from Soouree conditioned on membership in S.
Psooucee ==
PagneDDuee [Pc(2))
(4)
Parget ==
Dazge [Pc(a)]
(5)
The primary goal of the non-inferiority test is to compare the true mean predicted correctness between the two distribu- tions and determine whether Huanget is not lower than Mlource by more than a pre-specified margin m. This is formally expressed as the following hypothesis testing setup:
Ho Marget
<
JSsource 171
(6)
H1 Murget 2
Msourcee 1
(7)
When no assumptions about Daoaree and Dearett can be made, achieving reliable calibration is challenging. Calibrating C' on D&r (c.g.. using Platt's 's method (Platt et al., 1999) or temperature scaling (Guo et al., 2017)) ensures that the classification correctness estimator C' provides reliable esti- mates of the probability that model M correctly classifies samples in Dat ~ Deouree. While, in theory, this calibra- tion extends to Duett ~ "Dsuuree. we generally cannot assume calibration on D2 ~ 'Dtarget- Our approach to addressing this issue combines ongoing quality assurance checks with appropriate margin adjustments and will be discussed in more detail in Section 4.4.
The null hypothesis Ho posits that the estimated perfor- mance on the user dataset is worse than on the test dataset by more than the margin m. The alternative hypothesis H₃ asserts that the estimated performance on the user dataset is either better than, equivalent or not worse than that on the test dataset within the allowed margin m. We conduct the statistical non-inferiority test using one-sided Welch's ss t-test (see Appendix A.1.I).
4.4. Suitability Decision
Finally, the decision on the suitability of the model for the user dataset is based on the outcome of this non-inferiority test. If the test indicates non-inferiority, we conclude that the model's performance on D,, is acceptable and we output SUITABLE. If the test fails to reject the null hypothesis, the model is either unsuitable for the user dataset or the number of samples provided was insufficient to determine suitability and hence we return INCONCLUSIVE. To ensure the reliability of these suitability decisions, we next discuss statistical guarantees and the conditions under which they hold for the end-to-end suitability decision.
4.3. Non-Inferiyryty Testing
Non-inferiority testing is statistical method used to as- sess whether the performance of a new treatment, model, or method is not significantly worse than reference con- trol by more than pre-specified margin m (Wellek, 2002; Walker & Nowacki, 2011). Unlike other statistical tests, which typically test for a difference between distributions, this test aims to confirm that the new method is not inferior by more than a margin of m. Consequently, the null hypoth- esis that the method is inferior, in contrast to the usual null hypothesis of no difference.
Statistical Guarantees. To account for miscalibration er- rors, we define 8-calibration as follows:
Correctness Distributions. If the per-sample prediction correctness estimator C is well-calibrated, the mean of the estimated prediction correctness probabilities across dataset approximates the accuracy of the model M on that dataset. Formally, let DeaD and Pe De] denote the vectors of estimated prediction correctness probabilities for the test dataset Dust and the user dataset Da, respectively:
Definition 4.1 (6-Calibration). Let Px(x:) denote the esti- mated probability of prediction correctness for a sample x with predicted label M (x) and true label y. Assuming that (e(x) has a well-defined probability density function (v( over [o, 1], we say p=(x) is 6-calibrated if
[M = y pe(x) = u) = =-+c(u)
(8)
Pel = ( ( )) E [0, PPul
(2)
Vv E 1] with calibration error fo ((v))c(p) du = 8 for
Pe [D[D.]= = (a(x1)...... [0, PD.1
(3)
S 88 <1
5

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Under the assumption of testing two independent and nor- mally distributed samples, the non-inferiority test ensures controlled false positive rate (FPR), bounding the probabil- ity of incorrectly concluding non- -on-ineeri.rity.
Margin adjustment mn'
m
Aest A..
Suitable user data Da
Unsuitable user data Da
Esimated Accuracy from C
D
0.8
0.8
Theorem 4.2 (Non- Inn--orrri Test Guarantee). Let Psource
secutacy degerna
and Prarge represent the true mean prediction correctness for the source and target distributions, respectively. Assuming that these samples are independent and normally distributed, non- non-inferiority test based Welch's 's t-test at significance level or guarantees that the probability of rejecting the null hypothesis Ho : Prarge <
perfect accarayy
A
Au
0.6
0.6
171
171
0.4 0.1
0.4
0.6
0.8
0.4
0.6
0.8
Msouree - m (i.e., concluding Hurget 2 Psource m) when Hy is true is controlled at az
Ground Truth Accurncy
Ground Truth Accuracy
P(Reject Ho Ho is true) Sa.
(9)
Figure 3. Margin adjustment under accuracy estimation error. In each panel, the solid gray line is the perfect-calibration diagonal, the dashed black/gray lines mark the original margin and its corrected value m respectively. The blue/orange arrows indicate the estimation errors on the test set (Atea) ) and user data (A.)). respectively, In the left panel, the user data D., is deemed suitable; in the right panel it is deemed unsuitable.
where m is the non-inferiority margin (Lehmann et al., 1986; Wellek, 2002).
The following results extend this guarantee to the end-to- end suitability filter under 5-calibration for the correctness estimator C with respect to both Doaucee and Durrget. All expectations and probabilities are over samples (I, v) ~ X' x )' unless specified otherwise.
perfect calibration in practice is rare. In most real-world deployments, accurately determining the calibration errors &onncee and especially durget can be difficult. Consequently, adjusting the margin as proposed in Corollary 4.4 may be challenging. To address this, we draw inspiration from best practices in quality assurance and propose that the model owner periodically collects a small labeled dataset, D.. from potential user of the system. Given access the test dataset Dust the model owner can compute both the esti- mated accuracies (ftu. Pres)) as approximated by C, as well as ground truth accuracies (Acca. AcCres). This enables an empirical evaluation of accuracy estimation errors, A,, and Atest which correspond to òrarget and dsources respectively:
Lemma 4.3 (Expectation of Correctness). Under 8- calibration as defined in Definition 4.1, the deviation be- tween the true probability of prediction correctness and the estimation by classifer C' is given by:
E(po(z) (( (x) y) ===6d 8.
(10)
Proof in Appendix A.1.2.
We use Lemma 4.3 to derive the end-to-end guarantee for the suitability filter.
Corollary 4.4 (Bounded False Positive Rate for Suitability Filter under 8- d-Calibration). Given a prediction correctness estimator C that is 8-calibrated on both the source and target distributions with 8 Bouuce and Storget, respectively, let us define m' = m + Sgowree - Surget and conduct a non- inferiority test with Ho Huaget < Msouree - m'. The proba- bility of incorrectly rejecting Ho (i.e., returning SUITABLE) when the model accuracy on Durrett is lower than on D Domunce by more than a margin m is upper bounded by the signifi- cance level a. Proof in Appendix A.1.3.
N
1 N
Σ po(x.) - I(M(s)) = =
(11)
Following the margin adjustment in Corollary 4.4, the up- dated margin is:
m m Arest A..
(12)
The intuition behind this adjustment is that the decisions output by the suitability filter reflect the expected ground truth suitability decisions even in the presence of prediction errors as can also be seen in Figure Regular recalibration and careful margin tuning ensure that C continues to provide reliable estimates, even in the presence of distribution shifts or evolving deployment conditions.
The following remark details the limits of these guarantees. Remark 4.5 (Impossibility of Bounded False Positive Rate without 8-Calibration). If the calibration deviations dsource and bunget are not provided or are not much smaller than 1, it is not possible to choose m' according to Corollary 4.4. Hence, without Ó-calibration, no guarantees on the false positive rate of the suitability filter can be provided.
5. Experimental Evaluation
Practical Considerations. Under perfect calibration, the calibration errors vanish, i.c., &soucc = òurget = O, eliminat- ing the need for any margin correction. However, achieving
To evaluate the performance of our proposed suitability filter, we conduct a series of experiments with different datasets,
6

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Table Evaluating detection performance of the proposed suitability filter on FMOW-WILDS, RxRx1-WILDS and CivilComeents for m = 0 with both ID and OOD user data. We report the area under the curve for ROC and PR (capturing the tradeoff's at various significance thresholds), as well as accuracy and the true fälse positive rate a 0.05, We also report 95% confidence intervals based on 3 models M trained on the same Dann with different random seeds.
DATASET
Acc
FPR
ROC
PR
WMLWSWELDS ID FMoW-WILDN LDS OOD
8188 + 3.1%
0.027 0.033
0.969 0.023
0.967 0.029
91.9 + 2.5%
0.018 + 0.017
0.965 + 0.016
0.891 0.035
RxRx1 WILLDS ID RxRxI-WILDN OOD
100.0 1 0.0%
0.000
0.000
1.000 0.000
1.000 0.000
97.5 = 7.2%
0.031 + 0.088
0.997 + 0.006
0.989 - 0.024
CIVILCOMENTT -WELLEENNNS-ISSSSSSSEEES
93.3 + 5.3%
0.002 0.007
0.997 0.008
0.971 0.067
model architectures, and naturally occurring distribution shift types from the WILDS benchmark (Koh et al., 2021).
tives. Additionally, we report PR AUC, accuracy, and false positive rate the common a 0.05 threshold.
5.1. General Evaluation Setup
5.2. Suitability Signals
We evaluate the suitability filter on FMON-WII.DS (Christie et al., 2018), CivilComment -WILD (Borkan et al., 2019) and RxRx1-WIII dataset, we follow the recommended training paradigm to train model M using empirical risk minimization and the pre-defined Derain ~ Dsource- We then further split the provided in-distribution (ID) and out-of- distibibitt (OOD) validation and test splits into folds as detailed in Appendix A.2.2 (16 ID and 30 OOD folds for TMOW-WILDS, 4 ID and 8 OOD folds for RxRx1 -WIDDS and 16 ID folds for werilComments-wlie
We use the following suitability signals in our instantiation of the suitability filter (more details in Appendix A.2.1):
(Taylor et al., 2019). For each
conf.max: Maximum confidence from softmax. conf std: Standard deviation of softmax outputs, indicating confidence variability.
con conf.entropy: Entropy of the softmax outputs, measuring prediction uncertainty.
conf_ratio: Ratio of top two class probabilities. top_k_conf
Sum of the top 10% class proba- bilities, indicating concentration of probability mass. logit t mean: Mean of the logits, representing the overall output magnitude.
We conduct two types of experiments: first, each ID fold is used as the user dataset (D.), and the remaining ID data is split into 15 subsets, used as Drest and Dar. This yields 16x15x14 exper- iments for FMOW-WILDS, 4x 1x14514 for I and 16x1 5x14 for CivilComments- WILDS. Second, each OOD fold is used Da, and the ID data is split into 15 subsets, used for Dies and Dat This yields 30x15x14 experi- ments for FMOW-WILDS and 8x15x14 for RXRXI-WILDS.
logi t mmax: Maximum logit value, corresponding to the highest predicted class.
1 logit_std. Standard deviation of logits, showing the spread of model outputs.
loggi..dif..ttoeee...ooooo Difference between the top two logits, indicating confidence in distinguishing classes.
We define the binary suitability ground truth as Ac, D.) 2 Acc(M, Dusa) - m. While statistical guarantees are discussed under margin adjustments in Sec- tion 4.4, achieving the necessary calibration error estimates in practice is challenging. In particular, obtaining a reliable approximation for ourge requires access to small labeled user dataset D,,, which may not always be available. More- over, even if D., is collected, its representativeness of the true deployment distribution Durget is uncertain, introduc- ing potential biases in the accuracy estimation error A.. To account for this in our experiments, we set m' m for the non-inferiority test, effectively using the predefined margin without additional corrections. We discuss this in more detail in Appendix 1. We evaluate suitability de- cisions by computing the ROC AUC across significance levels, capturing the trade-off between true and false posi-
loss: Cross-entropy loss w.r.t. the predicted class. margin loss: Difference in cross-entropy loss be- tween the predicted class and next best class.
energy: Logits energy. computed as the negative log-sum-exponential, measuring model certainty.
5.3. Results
As our work introduces a novel problem setting with no ex- isting baselines for direct comparison, the primary objective of the following is to provide an intuition for the conditions under which our approach works effectively, its limitations, and the factors influencing its performance.
Table summarizes the performance of the proposed suit- ability filter across three benchmark datasets from the WILDS collection: FMON-WILDS, RxRx1 -WILDS, and
7

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
SUITABLE Decisions Across Varying Accuracy Differences
atives are observed in the range [0%, 3%). reflecting scenar- ios where the empirical evidence provided by D, and Dres is insufficient to reject the inferiority null hypothesis at the chosen significance level a = 0.05. However, for accuracy difference buckets exceeding 3%, the percentage of SUIT- ABLE decisions consistenlly exceeds 80% and increases to 100% above 6% of accuracy difference, demonstrating the robustness of the approach in scenarios with sufficiently large accuracy differences. Additional experiments, results, and interpretations can be found in Appendix A.4.
SUU SINCOEIEER 7541
1.09
Pklae Foottives
80
True Pusitlune
100
40% det
off
cit
28% the
4 toe
ont 1 2% si
29
3% t
to the
5% 6%
6% 7%
to
5%%
AccM mب)
Accle
Figure 4. Sensitivity of suitability decisions to accuracy dif- ferences between user and test data on FMOW-WILDS. The plot, summarizing results from nearly 29k individual experiments, shows the pereentage of SUITABLE decisions for a = 0.05 and m 0 across various accuracy difference bins. We combine both ID and OOD suitability filter experiments based on 3 models trained with different random seeds.
6. Discussion
Conclusion. We introduce the suitability filter, novel framework for evaluating whether model performance on unlabeled downstream data in real-world deployment set- tings deteriorates compared to its performance on test data, We present an instantiation for classification accuracy that leverages statistical hypothesis testing. We provide theo- retical guarantees on the false positive rate of suitability decisions and propose a margin adjustment strategy to ac- count for calibration errors. Through extensive experiments on real-world datasets from the WILDS benchmark, we demonstrate the effectiveness of suitability filters across di- verse covariate shifts. Our findings highlight the potential of suitability filters as a practical tool for model monitoring, enabling more reliable and interpretable deployment deci- sions in dynamic environments. Suitability filters provide an effective way to expose model capabilities and limitations and thus enable auditable service level agreements (SLAs).
Civi 1 W.Coomentts s WI.LDS Although ID and OOD results cannot be directly compared due to the differing numbers of ground truth positives and negatives, interesting trends still emerge. On FMO--WILD for example, we observe higher accuracy and a lower FPR at the 5% significance level for OOD user data, while ROC AUC and PR AUC are higher for ID user data. This discrepancy may stem from class imbalance: across OOD experiments, we have nearly three times many true negatives as true positives, making it easier to achieve high accuracy despite it generally being harder to maintain discriminative performance in an OOD setting, The latter is also confirmed for RxRx1-it WILDS, where we see decreased performance on OOD user data compared to ID user data. Another noteworthy observation the high overall performance on RxRx1-WIILD reason for this is that we observe large differences in model performance on RxRx1- WWILDS depending the fold con- sidered, as can be seen in Table 3 (Appendix). This variation helps the suitability filter detect performance deterioration more easily, as larger performance differences enhance its ability to identify changes.
Possible Extensions. The suitability filter framework's modularity makes it adaptable to various contexts. For fair- ness assessments, for instance, ensuring comparable accu- racy across groups can be achieved by substituting the non- inferiority test with an equivalence test. (Wellek, 2002) to evaluate if performance differences fall within predefined margin. If the goal extends beyond snapshot evaluations to continuous monitoring, this can be achieved by applying multiple hypothesis testing corrections to the p-values. Sim- ilarly, the framework can support sequential testing, where a decision is made iteratively: a user provides an initial sample, and more data can be requested if no conclusion reached, using methods such as O' Brierren--- fieeen-leemigg (O Brien & Fleming, 1979) or Pocock (Pocock, 2013) for control- ling eтror rates. For a more detailed discussion of these extensions, we refer the interested reader to Appendix A.3.
The
This sensitivity of suitability decisions to differences in ac- curacy between the user and test datasets is also illustrated in Figure 4 on MoW--WILDS for = O. The ideal relation- ship would be a step function, where SUITABLE decisions occur only when user dataset accuracy exceeds test accuracy. However, achieving this requires a perfect estimate of accu- racy on D.. which is impossible without ground truth labels. In practice. we observe that the slope of the suitability deci- sion curve is flatter than the ideal step function. There are few erroneous SUITABLE decisions when the accuracy difference is below 0%, indicating occasional false positives. However, for differences < -3% (indicating a performance deterioration of at least 3%, this is the case for 8.4k experi- ments out of nearly 29k in total), our proposed suitability filter achieves 100% accuracy. Additionally, some false neg-
Limitations. Our method is designed to detect accuracy degradations due to covariate shifts and does not address other types of distribution shift, such as label shift. This is due to the assumption that we generally only have access to unlabeled data from the target distribution Darget- Future work could extend this by incorporating information from a
8

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
(potentially small) number of labeled samples from the tar- get distribution. Moreover, our current approach is limited to classification due to the choice of signals. Though our set of suitability signals, designed to be applicable across data types, model architectures and training paradigms, provides a useful baseline, choosing signals tailored to the specific deployment setting would likely improve suitability filter performance. While our framework is general and could be used with different performance metries, our current instanti- ation and experimental evaluation are limited to accuracy. It thus focuses on scenarios where average-case performance is the primary concern and does not address safety-critical applications where ensuring good performance on a per- instance (or worst-case) basis is often crucial. Lastly, it should also be noted that one of the key underlying assump- tions of our framework is non-adversarial behavior from both model providers and users, who are expected to pro- vide representative data. This assumption is justified by the user's goal of identifying a suitable model for their task, but it implies vulnerability to deliberate adversarial manipula- tion designed to bypass the filter.
model evaluation, this work has the potential enhance trustworthiness in automated systems especially in safety- critical deployment contexts. However, there are ethical considerations to note. The methodology assumes access to well-calibrated prediction correctness estimators, which might not hold in all scenarios, potentially leading to in- correct evaluations. Additionally, while the framework adaptable, improper parameter choices or misinterpretations of results could exacerbate existing biases in datasets or models. Careful application and thorough understanding of the framework are critical to mitigating these risks. Future societal consequences of this work include its potential to improve fairness by enabling consistent performance eval- uation across diverse subpopulations. However, misuse or overreliance on such automated evaluation frameworks without human oversight could have adverse effects. We encourage practitioners complement this framework with domain expertise and ethical considerations during deploy- ment. This paper aims to advance the field of Machine Learning by providing tools for model evaluation in dy- namic deployment contexts. While we believe the societal implications are largely positive, we acknowledge the impor- tance of responsibly applying this methodology to prevent unintended harm.
Code Availability
The source code for the suitability filter framework and the experiments presented in this paper is pub- liely available on GitHub at https:/ ggithub.ccom cleverhans- 1ab/ sttbsuiitabiiitt..
References
Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting per- formance in domain generalization. Advances in Neural Information Processing Systems, 35:8265-888 8277, 2022.
Acknowledgements
We thank Anvith Thudi, Mike Menart, David Glukhov, and other members of the Cleverhans group for their feedback on this work. We would like to acknowledge our spon- sors, who support our research with financial and in-kind contributions: Apple, CIFAR through the Canada CIFAR AI Chair, Meta, Microsoft, NSERC through the Discovery Grant and an Alliance Grant with ServiceNow and DRDC, the Ontario Early Researcher Award, the Schmidt Sciences foundation through the AI2050 Early Career Fellow pro- gram. Resources used in preparing this research were pro- vided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
Back, C., Jiang Y., Raghunathan, A., and Kolter, J. Z. Agreement-on -oeee-t-----ee--eeee Predicting the performance of neural networks under distribution shift. Advances in Neu- ral Information Processing Systems, 35:19274-19289, 2022.
Benjamini, Y. and Hochberg, Y. Controlling the false dis- covery rate: practical and powerful approach to multiple testing. Journal of the Royal statistical society: series (Methodological), 57(1) -00, 1995.
Bialek, J., Kuberski, W., Perrakis, N., and Bifet, A. Esti- mating model performance under covariate shift without labels. 2024.
Impact Statement
Borkan, D.,, Dixon, L.. Sorensen, J., Thain, N., and Vasser- man, L. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
This work proposes a suitability filter framework for eval- uating machine learning models in real-world deployment settings, specifically designed to detect performance degra- dations caused by covariate shifts. The primary goal is to improve the robustness and fairness of machine learning models by offering tools to assess their suitability with- out requiring ground-truth labels. By facilitating reliable
Cha, J., Lee, K., Park, S., and Chun, S. Domain gener- alization by
regularization with pre-

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Scttings
trained models. In European conference on compuler vision, pp. 440-457. Springer, 2022.
EI- El-yaniv, R. et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11 (5), 2010.
Chen, J., Liu, F., Avei, B., Wu, X., Liang, Y., and Jha, S. Detecting errors and estimating accuracy unlabeled data with self-training ensembles. Advances in Neural In- formation Processing Systems, 34:14980-14999
Elsahar, H. and Gallé, M. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP- IJCNLP), pp. 2163-21723 2019.
20.
Chen, M., Goel, K., Sohoni, N. S., Poms, F., Fatahalian, K., and Ré, C. Mandoline: Model evaluation under dis- tribution shift. In International conference on machine learning, pp. 1617-1629. PMLR, 2021b.
Fan, W. and Davidson, I. Reverse testing: an efficient framework to select amongst classifiers under sample selection bias. Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 147-156, 2006.
Chow, C.-K. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, (4):247-22) 254, 1957.
Christie, G., Fendley, N., Wilson, J., and Mukherjee, R. Functional map of the world. In Proceedings of the IEEE Conference Compuler Vision and Puttern Recognition, pp. 6172-6180, 2018.
Feng, J., Sondhi, A., Perry, J., and Simon, N. Selective prediction-set models with coverage rate guarantees. Bio- metrics, 79(2):81 1-82,, 2023.
Chuang. C.-Y., Torralba, A., and Jegelka, S. Estimating gen- eralization under distribution shifts via domain-invarian representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1984-1994, 2020.
Gal, Y. and Ghahramani, Z. Dropout as a bayesian approx- imation: Representing model uncertainty in deep learn- ing. In international conference on machine learning. pp. 1050-1059, PMLR, 2016.
Culverhouse, P. F. Williams, R., Reguera, B., Herry, V., and González-Gil, S. Do experts make mistakes? a comparison of human and machine indentification of di- noftagellates. Marine ecology progress series, 247:17-25, 2003.
Gangrade, A., Kag, A., and Saligrama, V. Selective classi- fication via meddided prediction. In International Con- ference on Artificial Intelligence and Statistics, pp. 2179- 2187. PMLR, 2021.
Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi, H. Leveraging unlabeled data pre- dict out-of-distribution performance. arXiv preprint arXiv:2201.0423: 2022.
David, S. B., Lu, I., Luu, T., and Pál, D. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelli- gence and Statistics, pp. 129-136. JMLR Workshop and Conference Proceedings. 2010.
Geifman, Y. and El- Vaniv,, R. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
Deng, W. and Zheng, L. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15069-15078, 2021.
Geifman, Y. and El- Yaniv, Selectivenet: A deep neural network with an integrated reject option. In International conference machine learning, pp. 2151-151120 1159.. PMLR, 2019.
Deng, W., Gould, S., and Zheng. L. What does rotation prediction tell us about classifier accuracy under varying testing environments? In International Conference on Machine Learning, pp.
Geifman, Y., Uziel, G., and El-Yaniv, R. Bias-reduced un- certainty estimation for deep neural classifiers. Inter- national Conference on Learning Representations, 2019.
PMLR, 2021.
Deng, W. Suh, Y., Gould, S., and Zheng, L. Confidence and dispersity speak: Characterizing prediction matrix for unsupervised accuracy estimation. In International Con- ference on Machine Learning, pp. 1658-76744 PMLR, 2023.
Ginsberg, T., Liang, Z., and Krishnan, R. G. learning based hypothesis test for harmful covariate shift. arXiv 2022.
preprint arXiv:2212.088
Donmez, P., Lebanon, G., and Balasubramanian, K. Unsu- pervised supervised learning i: Estimating classification and regression errors without labels. Journal of Machine Learning Research, 11(4), 2010.
Ginsberg, T., Liang, Z., and Krishnan, R. G. A learning based hypothesis test for harmful covariate shift. The Eleventh International Conference on Learning Repre- sentations, 2023.
10

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Guan, L. and Yuan, X. Instance segmentation model evalua- tion and rapid deployment for autonomous driving using domain differences. IEEE Transactions on Intelligent Transportation Systems, 24(4):4050-400 4059, 2023.
Jiang, Y., Nagarajan, V., Baek, C., and Kolter, J. Z. As-
sessing generalization of sgd via disagreement. arXiv preprint (tfi::2006.139999 2021.
Kingma, D. P. Adam: A method for stochastic optimization.
Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., and Schmidt, L. Predicting with confidence on unseen distri- butions. Proceedings of the IEEE/CVF international conference on compuler vision, pp. 1134-1144, 2021.
arXiv preprint arXiv:1412.6980. 2014.
Kivimäki, J., Bialek, J., Nurminen, J. K., and Kuberski, W. Confidence-base estimators for predictive performance in model monitoring. arXiv preprint urXiv:2407.08649 2024.
Gulrajani, I. and Lopez-Pazz D. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L.,, Gao, I,, et al. Wilds: A benchmark of in-the- wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Kozodoi, N., Jacob, J., and Lessmann, S. Fairness in credit scoring: Assessment, implementation and profit implica- tions. European Journal of Operational Research, 297 10:,003-1094, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kulinski, S. and Inouye, D. Towards explaining distri- bution shifts. In International Conference on Machine Learning, pp. 17931-17952. PMLR, 2023.
Hébert-Johnson, U., Kim, M., Reingold, O., and Rothblum, G. Multicalibration: Calibration for the (computationally- identifiable) masses. In International Conference on Ma- chine Learning, pp. 1939-1948, PMLR, 2018
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. In International Conference on Learning Represen- tations, 2018.
Lehmann, L., Romano, J. P., and Casella, G. Testing statistical hypotheses, volume 3. Springer, 1986.
Li, L., Wang, H., Zha, L., Huang, Q., Wu, S., Chen, G., and Zhao, Learning data-driven policy network for pre- training automated feature engineering. In The Eleventh International Conference Learning Representations, 2023.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and nut-of-distribution examples in neural networks. arXiv preprint 1rX:0:1610002136, 2016.
Hu, Q., Guo, Y., Xie, X., Cordy, M., Papadakis, M., Ma, L.., and Le Traon, Y. Aries: Efficient testing of deep neural networks via labeling free accuracy estimation. In 2023 IEEFACC 45th International Conference on Software Engineering (ICSE), pp. 1776-1787. IEEE, 2023.
Liu, Z., Wang, Z., Liang, P. P., Salakhutdinov, R. R., Morency, L.-P., and Ueda, M. Deep gamblers: Learn- ing to abstain with portfolio theory. Advances in Neural Information Processing Systems, 32, 2019.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Loshchilov, I., Hutter, F., et al. Fixing weight decay regu- larization in adam. arXiv preprint arXiv: 171.::1111.010011 5, 2017.
Lu, Y., Wang, Z., Zhai, R., Kolouri, S., Campbell, J., and Sycara, K. Predicting out-of-distribution er- ror with confidence optimal transport. arXiv preprint arXiv:2302.05018, 2023.
Huang, L., Zhang, C., and Zhang, H. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems, 33:19365-19376, 2020.
Jaffe, A., Nadler, B., and Kluger, Y. Estimating the ac- curacies of multiple classifiers without labeled data. In Artificial Intelligence and Statistics, pp. 407-415. PMLR, 2015.
Madani, O., Pennock, D., and Flake, G. Co-validation: Using model disagreement on unlabeled data to validate classification algorithms. Advances in neural information processing systems, 17, 2004.

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Maggio, S., Bouvier, V., and Dreyfus-Schmidt, L. Per- formance prediction under dataset shift. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 146662444 IEEE, 2022.
Rabanser, S., Günnemann, S., and Lipton, Z. Failing loudly:
An empirical study of methods for detecting dataset shift. Advances in Neural Information Processing Systems, 32, 2019.
Maini, P., Yaghini, M., and Papernot, N. Dataset inference: Ownership resolution in machine learning. arXiv preprinl arXiv:2104.10706, 2021.
Rabanser, S., Thudi, A., Hamidich, K., Dziedzic, A., and Papernot, N. Selective classification via neural network training dynamics. arXiv preprint arXiv:2205.13532, 2022.
Miao, S., Zheng, L., Liu, J., and Jin, H. K-means clustering based feature consistency alignment for label-free model evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3299- 3307, 2023.
Redyuk, S., Schelter, S., Rukat, T., Markl, V., and Biess- mann, F. Learning to validate the predictions of black box machine learning models on unseen data. In Proceedings Data Analytics,
of the Workshop on Human-In-the- pp. 1-4, 2019.
O' Brien, P. C. and Fleming, T. R. A multiple testing proce- dure for clinical trials. Biometrics, pp. 549-556, 1979.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211-252, 2015.
Ovadia, Y., Fertig, E.. Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshninarayanan, B., and Snock, J. Can you trust your model's uncertainty? evalu- ating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Sanh, V. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint urXiv:1910.n 108, 2019.
Peng, R., Duan, Q.. Wang, H., Ma, J., Jiang, Y., Tu, Y., Jiang, X., and Zhao, J. Came: Contrastive automated model evaluation. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pp. 20121-20132, 2023.
Schelter, S., Rukat, T., and Bießmann, F. Learning to val- idate the predictions of black box classifiers on unseen data. In Proceedings of the 2020 ACM SIGMOD Inter- national Conference on Management of Data, pp. 1289 1299, 2020.
Peng, R., Zou, H., Wang, H., Zeng, Y., Huang, Z., and Zhao, J. Energy-baseee automated model evaluation. arXiv preprint arXiv:2401. 12689, 2024.
Sun, X., Hou, Y., Li, H., and Zheng, L. Label-free model evaluation with semi-structured dataset representations. arXiv preprint arXiv:2112. 1tw::2112.0000 2.0944 2021.
Platanios, E., Poon, H., Mitchell, T. M., and Horvitz, F. J. Estimating accuracy from unlabeled data: A probabilistic logic approach. Advances in neural information process- ing systems, 30, 2017.
Tal, E. Target specification bias, counterfactual prediction, and algorithmic fairness in healthcare. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 312-321, 2023.
Platanios, E. A., Dubey, A., and Mitchell, T. Estimating accuracy from unlabeled data: A bayesian approach. In International Conference on Machine Learning, pp. 1416- 1425. PMLR, 2016.
Taylor, J., Earnshaw, B., Mabey, B., Victors, M., and Yosin- ski, J. Rxrx1: An image set for cellular morphological variation across many experimental batches. In Interna- tional Conference on Learning Representations (ICLR), volume 22, pp. 23, 2019.
Platt, J. et al. Probabilistic outputs for support vector ma- chines and comparisons to regularized likelihood meth- ods. Advances in large margin classifiers, 10(3):61-74, 1999.
Travaini, G. V., Pacchioni, F., Bellumore, S., Bosia, M., and De Micco, F. Machine learning and criminal jus- tice: A systematic review of advanced methodology for recidivism risk prediction. International journal of en- vironmental research and public health, 19(17):10594, 2022.
Pocock, S. J. Clinical trials: a practical approach. John Wiley& & Sons, 2013.
Podkopaev, A. and Ramdas, A. Tracking the risk of a deployed model and detecting harmful distribution shifts. arXiv preprint arXiv:21 10.06777,, 2021.
Tu, W., Deng, W., Gedeon, T., and Zheng, L. A bag-of- prototypes representation for dataset-level applications. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 2881-2892, 2023.
Quiñonero- Ciaioner--Candel,.. J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D. Dataset shift in machine learning. Mit Press, 2022.
12

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Unterthiner, T., Keysers, D., Gelly, S., Bousquet, O., and Tolstikhin, Predicting neural network accuracy from weights, arXiv preprint arXiv:2002. 2020.
Walker, and Nowacki, A. Understanding equivalence and noninferiority testing. Journal of general internal medicine, 26:192-196, 2011.
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., and Philip, S. Y. Generalizing to unseen domains: A survey on domain generalization. IEEE transactions knowledge and data engineering, 35(8))8052-8072 2022.
Wang, J., Chen, J., and Su, B. Toward auto-evaluation with confidence-based category relation-aware regression. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Welch, L. The generalization of 'student's problem when several different population varlances are involved. Biometrika, 34(1-2):28 35, 1947.
Wellek, S. Testing statistical hypotheses of equivalence. Chapman and Hall/CRC, 2002.
Wiles, O., Gowal, S., Stimberg, F., Alvise-Rebufii S., Ktena, I., Dvijotham, K., and Cemgil, T. A fine-grained analysis on distribution shift. arXiv preprint arXiv:21 11328, 2021.
Xie, R., Wei, H., Feng, L., Cao, Y., and An, B. On the importance of feature separability in predicting out-of- distribution error. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Yu, Y., Yang, Z., Wei, A., Ma, Y., and Steinhardt, J. Pre- dicting out-of-distribution error with the projection norm. In International Conference on Machine Learning, pp. 25721-25746. PMLR, 2022.
Zhou, K., Yang, Y., Qiao, Y., and Xiang, T. Domain gener- alization with mixstyle. In International Conference on Learning Representations, 2021.
Zhou, K., Liu, Z., Qiao, Y., Xiang, T.,, and Loy, C. C. Do- main generalization: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):4396- 4415, 2022.
13

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
A. Appendix
A.I. Statistical Hypothesis Testing
A.1.1. WELCH'S T-TEST
Welch's s t-test is a modification of the standard Student's t-test that adjusts for unequal variances and unequal sample sizes between the two groups (Welch, 1947). The test statistic for the suitability filter Welch's 's t-test is given by:
AD PD t a2 Diea
(13)
D..
In the above, PDD D. EOED. Pe(x) + m is the margin-adjusted sample mean of Pe[Du] and P Pieal ExeDun (x) is the sample mean of PelDesl. de and d are the sample variances of Pa[Du] and Pe Dusll respectively. The test statistic follows t-distribution with degrees of freedom (df) given by:
D
D..
df
(14)
TDaml
TPET
Diexl
Du
The degrees of freedom are dependent on the size of provided user and test samples and are used to determine the appropriate critical value for the t-distribution. Since non-inferiority testing is inherently a one-sided test, after calculating the two-sample t-test statistic, the p-value is divided by 2 to reflect the one-sided nature of the non-inferiority test. This adjusted p-value is then compared to the chosen significance level a to determine whether the null hypothesis Ho can be rejected.
A.1.2. PROOF OF LEMMA 4.3
Proof. In the following, all expectations and probabilities We begin by noting that the predicted correctness probability for given sample x: is denoted as Pc(r). We assume that P=(x) has a well-defined probability density function fe(v) over the interval [0, 1]. This means that the predicted correctness probabilities (r(r)) for samples x are distributed according to fe(v), where E [0, 1] represents the possible values of prediction correctness. We can hence represent the expected value of the correctness probability Pc(x)) under the distribution defined by fe(v) as:
samples (x, y) X' ) unless specified otherwise.
E Elpe(r)) = f'' vff(v)
(15)
The true probability of prediction correctness for model M on sample x with predicted label M (x) and true label y denoted as (1) ) v]. This can be expressed the integral over all possible predicted correctness probabilities weighted by the conditional probability of correctness given (=(x) and the probability density fv).. This decomposition follows from the law of total probability, where the predicted correctness probability pe(x)) serves as an intermediate variable. We can hence write:
F[M(x) - ([M(2) y po(x) v)fa(v) dv.
(16)
Due to the inherent uncertainty and error in the calibration process, the predicted probability (x( is not necessarily equal to the true probability P. (x) = y]. We model this calibration eπror as E(v), which represents the deviation of the predicted correctness from the true correctness for each possible correctness value v. Following Equation 8, we can decompose the true probability prediction correctness as:
[M(M)) = v) w]= [ ufclv)du du ((V)sa()) = E.(pa(>)]+ [ ((v)sa(v) dv.
(17)
The first term represents the expected value the predicted correctness, while the second term represents the expected error introduced by the calibration process.
Now, we make the assumption that the calibration error is equal to some small value 8 with 0 < | < This assumption referred to as ô-calibration defined in Definition 4.1 (CHANGE THIS). Under this assumption, we have that:
e(v)f((v) du = 6. [
(18)
14

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Combining this result with Equation 17, we find for the difference between the true probability and the expected correctness
that:
[x)] (x) yl 6.
(19)
This completes the proof of Lemma 4.3.
A.1.3. PROOF OF COROLLARY 4.4
Proof. For simplicity, let us use the following notation:
Acesuuree = P an ormee [M(x) = O (z)),
(20)
ACCraget P D [M (=)),
(21)
Msource Ex D Descce [
(22)
Mtarget Ex [o(2))]
(23)
Here, (xx) represents oracle that provides the label y for any input x. Let M be model with correctness estimator C' that is d-calibrated on both the source and target distributions. It hence follows from Lemma 4.3 that the predictions output by the correctness estimator C' satisfy:
Msouree
ACCsource
8soucce
and
Ptarget
ACCurget
Otarget-
(24)
We interested in upper-hounding the false positive rate of the end-to-end suitability filter, i.e., the probability of rejecting the null hypothesis Ho at significance level and returning SUITABLE when, in reality:
ACCrrget
ACCsounce m0.
(25)
We can define m' m Osourre
Garget and leverage Equation 24 write: <
AcCrarget ACCsouree
m
Mtarget
òrarget
Msoouree
Ssaurce m71
Mtarget <
Psource m'.
(26)
With this margin m', the corresponding null hypothesis for the non non-inferiority test Hg is:
Ho : Ptarget <
Psource
m'.
(27)
Assuming normaley and independence and applying Theorem 4.2 at significance level it is guaranteed that for the true mean prediction correctness Pssoucee and Puarget of the source and target distributions, the probability of rejecting the null hypothesis Ho Mrarget < Psouree m' (i.c., concluding Mtarget 2 Msource - mm') when Ho is true is controlled at ac:
PReeect Ho Ho is true) S a.
(28)
This ensures that the difference in mean prediction correctness between the source and target distributions is bounded by the margin m' with high probability.
us now derive the implication for the end-to-end suitability filter. By definition:
P(Reeect Ho AcCranget Acc - m) P(Reject Ho 7 AcCraret ACCsnurce m))
TP PP(ACCaaga < ACCsounce m))
(Reject Ho Praggt < Psource m))
(29)
P(MMaaget < Psource - m!)
IP(Reject Ho Ho is true)
- a.
The inequality follows from the guarantees of the non-inferiority test as outlined Equation 28, All other transformations are applications of Bayes' theorem and Equation 26.
15

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Under perfect calibration, target Osuuree 0 and thus no adjustments the performance deviation margin needed achieve bounded false positive rate. Hence, under perfect calibration, the probability of rejecting the null hypothesis for non-inferiority test with margin given that the model accuracy on Durget is lower than source by more than m is upper bounded by the chosen significance level a. If we do incur miscalibration and observe | > 0 or |ōsoureel > 0, we have to adjust the performance deviation margin m accordingly to reflect this. As shown, when choosing mn' == m. + davurve Ourget. the end-to-end suitability filter false positive rate remains bounded. This concludes the proof of Corollary 4.4.
A.2. Additional Experiment Details
A.2.1. SUITABILITY SIGNALS
General Suitability Signals. Let M E MM a classifier mapping inputs x E X' to probabilities over &t classes a²t
y (k). Denote the logits of (x) as E IR* and the softmax outputs as softmax(2), where pt Σ Ej
The following sample-level signals are derived from and p:
Maximum confidence (conf.max):
conf.max
max ie k1..,))
(30)
The maximum predicted probability.
Confidence standard deviation (conf.std):
conf std
Σ Pi p)2 p
Pt
(31)
The standard deviation of the softmax probabilities.
Confidence entropy (conf .onffye)troy):::
k
p. log(p: + +c) conf entropy = Σ
(32)
The Shannon entropy of the predicted probabilities, measuring uncertainty. We add 10-10 for numerical stability.
Confidence ratio (conf.ratio):
P(1)
conf ratio
(33)
P(2) +
The ratio of the highest to the second-highest predicted probabilities, where P(1) and P(2) are the largest and second- largest P., respectively. We add e 10-10 for numerical stability.
Sum of top 10% confidences (top k conf sum):
top.k.conf_sum
Σ Pi: indices top-[0.1k] probabilities
(34)
SEX
The sum of the largest 10% of all predicted probabilities.
Mean logit (logit_mean):
logit.mean
24
(35)
The mean of the logits.
16

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Maximum logit (logit_mane
logit max
max 2/ i6 (e),...)))
(36)
The maximum logit value.
Logit standard deviation (logit_std):
1 k
logit_std=
=
22
2)3,
2
(37)
i=1
The standard deviation of the logits.
Difference between largest logits (logit.diff.topp),
logit.diff.topp = 2(1) - 2(2)
(38)
The difference between the two largest logits, where 2(1) and 2(2) are the largest and second- largest 2,, respectively.
Loss with respect to predicted label (loss):
loss = log(p()) +
(39)
The cross-entropy loss with respect the predicted label, P(1) is the largest p..
Difference in loss between top two classes (margin.loss).
margin log(p(1) + E) + log(P(z) +e)
(40)
The difference in cross-entropy loss between the top two predicted probabilities. We add 10-10 for numerical stability.
Energy (energy):
energy - -log
(41)
The energy function derived from the logits, measuring prediction certainty.
Alternative Suitability Signals. In work, we deliberately rely on suitability signals that avoid assumptions about architecture, training, data domains and applicable any classifier. However, many other signals shown to be indicative of model performance have been proposed in the literature.
In unsupervised accuracy estimation, more recent approaches measure disagreement between predictions by different Chen et al., 2021a; Baek et al., 2022; Jiang
models (Madani et al., 2004; Donmez et al., 2010; Platanios et al., 2016; 2017; et al., 2021; Jaffe et al., 2015; Fan & Davidson, 2006; Yu et al., 2022: Chuang et al., 2020; Ginsberg et al., 2023), rely on manual provision of information about make assumptions on the nature of the distribution shift between training and deployment (Redyuk et al., 2019; Chen al., 2021 b; Elsahar & Gallé, 2019; Guillory al., 2021; Schelter al., 2020; Peng et al., 2024; 2023; Deng & Zheng, 2021), focus specifie input data types (Maggio al., 2022; Deng al., 2021;
Bialek et al., 2024; Sun et al., 2021; Deng & Zheng, 2021; Unterthiner al., 2020; Guan & Yuan, 2023; Li et al., 2023), or analyze classification decision boundaries and feature separability (Hu et al., 2023; Xie et al.. 2024; Tu et al., 2023; Miao et al., 2023). To ensure generality and broad applicability of the suitability filter across diverse settings, these signals are not included in our experimental evaluation. However, signals shown to predict accuracy in these studies could serve as additional suitability signals in scenarios where their specific constraints are met.
Similarly, in selective classification, recent methods enhance the underlying model by augmenting its architecture (Geifman & El- Yaniv, 2019; Lakshminarayananr
al., 2017), employing adapted loss functions during training (Gangrade et al., 2021; Huang et al., 2020; Liu et al., 2019), utilizing more advanced prediction correctness signals, albeit often with increased inference costs (Geifman et al., 2019; Gal & Ghahramani, 2016; Rabanser et al., 2022; Feng et al., 2023). These approaches require modifications to model architecture, training processes, or inference, and are thus not generally applicable. Having said that, while these approaches are not incorporated into our work, they can serve as additional suitability signals in scenarios where these modifications are feasible.
17

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
A.2.2. DATASETS AND MODELS
FMOW-WILDS. The FMow TMON-WILDS dataset contains satellite images taken in different geographical regions and in different years (Christie al., 2018: Koh et al., 2021), thus considering both temporal and geographical shift. The input is an RGB satellite image (resized to 224 x 224 pixels), the label y is one of 62 building or land use categories, and the domain represents the year the image was taken and its geographical region. We train a DenseNet-122 121 model (Huang et al., 2017) pretrained on ImageNet (Russakovsky et al., 2015) and without L2 regularization with empirical risk minimization. We use the Adam optimizer (Kingma, 2014) with an initial learning rate of 10 that decays by 0.96 per epoch, and train fro 50 epochs with early stopping and a batch size of 64. All reported results are averaged over 3 random seeds. Following the standard WILDS training setup, we use 76, 863 images from the years 2002-2013 as training data. We split the remaining ID and OOD splits into 16 different ID folds and 30 different OOD data folds as detailed in Table 2. These folds were chosen with the aim to be as representative possible of shifts likely to occur in practice while still ensuring a sufficient number of samples per fold for statistical testing (at least 666).
RXRx1-WILDS. The RxRx1 -WILSS dataset reflects the disribution shifts induced by batch effects in the context of genetic perturbation classification (Taylor al., 2019; Koh et al., 2021). The input x: is 3-channel image of human cells obtained by fluorescent microscopy (nuclei, endoplasmic reticuli and actin). the label y indicates which of the 1,139 genetic treatments (including no treatment) the cells received, and the domain specifies the batch in which the imaging experiment was run. The images in RxRx1- LWL are the result of executing the same experiment 51 times, each in a different batch of experiments. Each experiment was run in a single cell type, one of: HUVEC (24 experiments), RPE (11 experiments), HepG2 (11 experiments), and U2OS (5 experiments) across sites. The dataset is split by experimental batches into training, validation, and test sets. For all experiments, we fine-tune a ResNet-50 model (He et al., 2016) pretrained on ImageNet (Russakovsky al., 2015), using learning rate of 10-4 and L2-regularization strength of 10-5, The models are trained with the Adam optimizer (Kingma, 2014) and batch size of 75 for 90 epochs, linearly increasing the learning rate for 10 epochs and then decreasing it following a cosine learning rate schedule. Results are reported averaged over 3 random seeds. Following the standard WILDS training setup, we use 40, 612 images from 33 experiments in site 1 as training data. We then split the remaining data by cell type into 4 ID data folds (same experiments as training data but different images but site 2) and OOD data folds (different experiments, combining sites and 2) as detailed in Table 3.
Civil we..Commettsiiine The Civi 1 Comment s SWWI dataset focuses on text toxicity classification across de- mographic identities, aiming to address biases in toxicity classifiers that can spuriously associate toxicity with certain demographic mentions (Borkan et al., 2019; Koh et al., 2021). The input x is a text comment on an online article, and the label y is whether the comment was rated as toxic or not. The domain is represented as an 8-dimensional binary vector, where each component corresponds to the mention of one of the 8 demographic identities: male, female, LGBTQ, Christian, Muslim, other religions, Black, and White. The dataset consists of 450,000 comments, annotated for toxicity and demographic mentions by multiple crowdworkers and randomly split into train, validation and test splits. We hence have no additional OOD data splits (and correspondingly. no OOD data folds) for this dataset. We train a model (Sanh, 2019) with the Adam optimizer (Loshchilov et al., 2017), using learning rate of 10 10-5, batch size of 16, and an L2 regularization strength of 10-2 for epochs with early stopping. All reported results are averaged over 3 random seeds. Following the standard WILDS training setup, we use 269, 038 comments as training data. We split the remaining data into 16 different ID folds as detailed in Table 4. Since the data is generally from the same distribution as our training data but we divide it into folds depending on the sensitive attributes mentioned in each comment, this is an example of the target distribution consisting of subpopulations of the source distribution.
A.3. Possible Extensions
1. EQUIVALENCE TESTING
In equivalence testing, the goal is to assess whether the performance on the target dataset Paages is statistically similar the performance on the source dataset Dunuree within a specified margin m, i.e., we want to test whether the difference between the two means is sufficiently small (Wellek, 2002). This is formalized as the following hypothesis setup:
Ho: : Muaree Msource|
> m
(42)
H1 : Mtarget Msource S m
(43)
18

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Scttings
Table Summary of different ID and OOD data folds for FMow-WI LDS.. For accuracy, we report the and the 95% confidence
interval based on three models trained with different random seeds.
SPLIT
YEAR
REGION
NUM SAMPLES
ACCURACY
ID FOLDS
VAL
2002-2006
A1.
1420
53.99 + 1.32%
ID_VAL
2007-2009
ALI
1430
55.78 + 0.70%
2010
AL.
2459
62.60 + 1.64%
ID VAL
2011
ALL
2874
65.98 1.25%
ID VAL
2012
ALL
3300
64.03 + 0.13%
ID_VLL
ALL
ASIA
2693
62.42 1 1.03%
ID_VAL
ALL
EUROPE
5268
59.88 1 0.56%
IDAVLL
ALL
AMERICAS
3076
63.85 + 2.10%
ID. TEST
2002-2006
ALL
1473
51.39 - 4.54%
ID_TEST
2007-2009
ALL
1423
57.25 + 0.61%
ID_TEST
2010
ALL
2456
61.01 + 0.41%
ID_TEST
2011
ALL
2837
65.03 1.15%
ID.TEST
2012
ALL
3138
62.12 1.36%
ID. TEST
ALL
ASIA
2615
59.39 - 1.94%
ID_TEST
ALL
EUROPE
5150
58.99 + 0.51%
ID_TEST
ALL
AMERICAS
3130
63.05 - 1.39%
OOD FOLDS
VAL
2013
A1.I
3850
60.29 + 1.81%
VAL
2014
ALL
6192
62.44 1 1.48%
VAL
2015
ALL
9873
57.77 1: 1.37%
VAL
ASIA
4121
11 56.30 0.73%
VAL
ALL
EUROPE
7732
63.28 + 1.07%
VAL
ALL
AFRICA
803
50.73 + 1.25%
VAL
ALL
AMERICAS
6562
58.04 2.05%
VAL
ALL
OCEANIA
693
66.38 + 2.51%
VAL
2013
EUROPE
1620
+ 1.11%
VAL
2014
EUROPE
2523
68.05 + 1.94%
VAL
2015
EUROPE
3589
60.82 : 0.73%
VAL
2013
ASLA
813
57.40
VAL
2014
ASIA
1311
5
VAL
2015
ASIA
1997
55.45
VAL
2013
AMERICAS
1168
61.13 + 1.66%
VAL
2014
AMERICAS
1967
60.85+1.228 1.266
VAL
2015
AMERICAS
3427
55.36 1 3.32%
TEST
ALL
15959
55.48 1 1.14%
TEST
2017
ALL
6149
48.64 1 2.13%
TEST
ALL
ASIA
4963
55.67 11: 0.72%
TEST
ALL
EUROPE
5858
5 1.96%
TEST
AFRICA
2593
33.50 3.87%
TEST
ALL
AMERICAS
8024
56.20 1.17%
TEST
ALL
OCEANIA
666
59.56 + 0.43%
TEST
2016
EUROPE
4845
58.42 + 2.68%
2017
EUROPE
1013
46.63 + 1.48%
TEST
2016
ASIA
3216
53.58 1 0.80%
TEST
2017
ASIA
1747
59.53 1 1.42%
TEST
2016
AMERICAS
6165
57.21 1 1.42%
TEST
2017
AMERICAS
1859
52.86 + 1.49%
The null hypothesis Ho asserts that the difference in means between the target and source distributions is greater than the margin m. The alternative hypothesis Hi posits that the means are equivalent, with their difference being smaller than or equal to the margin m. In practice, this is achieved by conducting two one-sided tests (TOST). This involves testing both lower and upper bounds the margin to confirm that the performance difference not meaningfully large in either direction.
19

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Table Summary of different ID and OOD data folds for RxRx L.DS. For accuracy, report the and the 95% confidence
interval based on three models trained with different random seeds.
SPLIT
CELL TYPE
NUM SAMPLES
ACCURACY
ID FOLDS
ID_ TEST
HEPG2
8622
25.39 + 1.43%
ID_ ID_TEST
HUVEC
19671
50.30 + 1.21%
TEST
RPE
8623
23.86 + 1.17%
TEST
U2OS
3696
17.00 + 0.98%
OOD FOLDS
VAL
HEPG2
2462
21.01 +: 1.77%
HUVEC
2464
36.85 + 0.10%
VAL
RPE
2464
16.44 0.96%
VAL
U2OS
2464
2.27
TEST
HEPG2
7388
22.63 1.28%
TEST
HUVEC
17244
39.99 1.13%
TEST
RPE
7360
21.32 0.11%
TEST
U20S
2440
8.96 + 1.78%
Table Summary different ID data folds for Civil Cow.oooonntt--eeommmmmmmm -WILDS For accuracy, report the mean and the 95% confidence interval based on three models trained with different random seeds.
SPLIT
SENSITIVE ATTRIBUTE
NUM SAMPLES
ACCURACY
VAL
MALE
4765
89.31 + 0.22%
VAL
FEMALE
5891
90.09 + 0.68%
VAL
LGBTQ
1457
80.00 = 0.97%
VAL
CHRISTIAN
4550
92.72 = 0.17%
VAL
MUSLIM
2110
81.52 = 1.39%
VAL
OTHER RELIGIONS
986
85.87 + 0.77%
VAL
BLACK
1652
77.85 + 1.11%
WHITE
2867
77.26 + 0..66%
TEST
MALE
14295
88.84 0.16%
TEST
FEMALE
16449
90.02 1 0.18%
TEST
LGBTQ
4426
79.78 1: 0.68%
TEST
CHRISTIAN
13361
92.22 = 0.25%
TEST
MUSLIM
6982
82.65 = 0.58%
TEST
OTHER RELIGIONS
3500
88.18 = 0.58%
TEST
BLACK
4872
78.39 + 0.68%
TEST
WHITE
7969
79.88 + 0.47%
A.3.2. CONTINUOUS MONITORING
In continuous monitoring, the aim is regularly re-evaluate a model is still suitable for given deployment context based on new incoming data samples. When performance is evaluated over time with changing data, the Benjamini-Hochberg (BH) procedure is used to control the false discovery rate (FDR) across multiple tests (Benjamini & Hochberg, 1995). The BH procedure adjusts valuue by considering the number tests performed up to the current point, ensuring that the proportion of false positives remains controlled. This formalized as follows: for each p-value pt, the null hypothesis is rejected p: a, where is the total number of tests and ar is the desired FDR threshold. The rolling window approach further refines this by evaluating significance across fixed window of recent data, smoothing out short-term fluctuations and focusing long-term trends in performance. This approach helps identify true changes in model performance while accounting for variations in individual datasets over time.
A.3.3. SEQUENTIAL TESTING
When testing the same hypothesis sequentially with accumulating data, the O' O'Brien-FHe sieen-Feeming (O Brien Fleming, 1979) and Pocock (Pocock, 2013) methods are used to control the overall false positive rate (type error rate) across multiple tests.
20

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
These methods are designed for sequential testing, where a decision is made at each stage based the data collected so far, and more data can be added if no conclusion is reached. The O' Briien-Fleming method is more conservative early on, requiring stronger evidence to reject the null hypothesis at earlier stages and relaxing this criterion more data becomes available. Specifically, the significance threshold at stage l is adjusted as:
ag = 1 - (l1-a)
(44)
where is the total number of stages and is the desired overall Type error rate. In contrast, the Pocock method applies constant critical value across all stages of testing. For each stage k the significance level remains:
Or
(45)
n
Both methods adjust the significance threshold at each stage to control the family-wise error rate (FWER), ensuring that the probability of making least one Type error remains below a specified threshold, a.
A.4. Additional Results
A.4.1. CALIBRATION
Impact of Calibration on Darget- We visualize the impact of calibration on classifier C and the performance of the end-to-end suitability filter in Figure 5. To this end, we select the ID validation and test splits from Europe as D&r and Duet- respectively. We then plot the actual accuracy versus the mean of the prediction correctness estimated by a classifier C' trained on Dar, with or without additional calibration Du. It should be noted that this is mainly a theoretical experiment, as in practice calibration on D,, is not possible since we do not have access to ground truth information for user data. We observe that the false positive rate of the end-to-end suitability filter is elevated due to miscalibration on the different distribution Parget- Although the relationship between actual accuracy and mean estimated prediction correctness weaker without calibration, these metrics remain highly correlated. Therefore, the increased risk from miscalibration can be mitigated by selecting an appropriate non-inferiority margin m.
Impact of Calibration on Suitability Filter on FMoW
Meant Estimated Prediction Correctness
0.68
SUTTABLE UNSUTTABLE
0.66
Uncaliested
Calibrated
0.64
Accurscy
SUITABLE
0.62
SUTTABLE whith margin
UNSUITABLE
0.54
0.52
0.52
0.54
0.56
0.58
0.60
0.62
0.64
0.66
0.68
Actual Accuracy
Figure 5. Suitability filtering different OOD folds of FMON-WILDS with and without additional calibration on D.. We choose non-inferiority margin of m = 0.05 for this experiment.
Accuracy Estimation Error. In Section 4.4, we propose using the empirical accuracy estimation error A to adjust the margin and mitigate the effects of miscalibration in C. To illustrate this in practice, Figure 6 presents the distribution of A As expected, Aest is centered around zero, indicating
for both test and user data across 6300 experiments on that the estimated accuracy closely matches the ground truth accuracy and there is no clear directional bias. However, A, is frequently positive, indicating that accuracy is often overestimated. This miscalibration can lead to incorrect suitability decisions. While adjusting the performance deterioration margin m, as proposed in Section 4.4, would mitigate this issue,
21

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Distributions Accuracy Estimation Ervor for Correct Incorrect Suttablitty Decisions on FMoW
Correct Decidons
Suttaliiity
700
Probel:iity Densily
600
800
200
100
0
0.00
0.15
0.04
0.00
0.04
User Estimation
Test Accuracy Estimation
Figure 6. Distribution of the empirical accuracy estimation error for both the user and the test data across 6300 experiments on FMow-W LDS. The suitability decisions depicted here have been made for choice of m 0 without margin adjustment due miscalibration and at a significance level of 0.05.
no such adjustment was applied here to highlight the impact of miscalibration on suitability decisions. Notably, suitability decision eтrоrs do not occur for examples with large accuracy estimation errors.
To better understand this phenomenon, Figure explores the relationship between accuracy estimation error A, and the actual performance degradation from test to user data. As performance deteriorates, the accuracy estimation error tends to increase. However, performance degradation grows at faster rate than A.. meaning that the overall impact of A,, on suitability decisions remains limited for m 0. This explains why incorrect suitability decisions are primarily concentrated near the decision boundary rather than in cases with extreme accuracy estimation errors (and, correspondingly, larger performance degradation).
Scatler Plot Accuracy Difference User AEE
0.10
Correct Suitabliiy Decistons
Suitablitty
User Accuracy Estimazin Etror (ARE)
0.08
Trand Line
0.02
0.00
-0.02
0.04
-0.15
-0.05
0.00
Accuracy Difference (User Accuracy Test Accuracy)
Figure 7. Relationship between performance deterioration for model M and the empirical accuracy estimation error A for the data across 6300 experiments on MMCN-WILDS. The suitability decisions depicted here have been made for a choice of m = 0 without margin adjustment due miscalibration and significance level of 0.05. can seen, incorrect suitability decisions are centered around the suitability decision boundary and are not, might be expected, in areas of large empirical accuracy estimation enror A.
By incorporating margin adjustments based on the empirical accuracy estimation error, suitability decisions can be made more robust against calibration errors, ultimately improving the reliability of the suitability filter in deployment.
A.4.2. USING DIFFERNT SIGNAL SUBSETS FOR PREDICTION CORRECTNESS ESTIMATOR
In Table 5, we compare our proposed suitability filter that trains the prediction correctness estimator using various suitability signals to alternatives that rely on only a single signal. As can seen, the suitability filter leveraging all signals
22

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
generally outperforms single-signal alternatives, demonstrating the benefit of integrating diverse signals for robust suitability decisions. However, we find that certain signals, such energy logit max, perform nearly as well on their own. Unsurprisingly, these signals are also identified the most predictive of per-sample prediction correctness for classifier C' (see Appendix A.4.3). Noteworthy outliers in Table 5 include mean and logit_std that have relatively high accuracy but higher FPR and lower ROC and PR AUC than comparable signals. Upon closer examination, we find that prediction correctness classifiers C' trained on only these signals generally have a higher expected calibration error even when tested on in-distribution data as can be seen in Table 6. As demonstrated in Corollary 4.5, proper calibration theoretically crucial for reliable suitability decisions, and this importance evident in practice here. Signals that yield the best-perforning prediction correctness estimators C' (high accuracy and low maximum calibration error) also demonstrate superior performance when applied in the end-to-end suitability filter.
Table 5. Comparing performance of the proposed suitability filter against individual signal based suitability decisions on FMcW-WINE for m with both ID and OOD user data. We report the area under the curve for ROC and PR (capturing the tradcoff' at various significance thresholds), well as accuracy and the true false positive rate at 0.05. We also report 95% confidence intervals based on 3 models M trained on same Dusin with different random seeds.
METHOD
Acc
FPR
ROC
PR
ID USER
DATA
SUITABILITY FILTER
81.8 3.1%%
0.027 + 0.033
0.969 + 0.023
0.967 + 0.029
ENERGY
80.6 - 4.3%
4.3%
0.024 0.040
0.965 1:65:00 0.020
0.962 - 0.033
LOGIT_MAX
80.2 1 4.8%
0.025 0.041
0.965 0.018
0.963 = 0.030
LOGIT_MEAN
80.1 10.4%
0.112
0.113
0.896 = 0.196
LOSIT_DIFF_TOP2
73.5 4.5%
0.008 0.001
0.963 + 0.017
0.963 = 0.017
MARGIN.LOSS
4.5%
0.008 0.001
0.963 + 0.017
0.963 + 0.017
LOGIT_SID
0.170 7
0.855 0.144
0.779 = 0.300
CONF CONF_ENCROPY
0.003 0.012
0.969 1 0.014
0.967 0.012
CONF_SCD
68.8
0.008 0.019
0.963 1 0.020
0.960 1 0.017
TOP_K.CONF_SU
68.2 6.9%
0.005 + 0.015
0.917 - 0.027
0.944 + 0.029
CONF_MA,
67.9 4.4%
0.008 = 0.021
0.957 + 0.025
0.954 - 0.025
LOSS
67.0 = 2.8%
0.008 = 0.021
0.952 + 0.031
0.948 0.035
CONF_RACIO
62.3 4.5%
0.046 = 0.053
0.846 + 0.056
0.826 = 0.016
OOD USER DATA
SUITABILIYY FILTER
91.9+
0.018 + 0.017
+965++00011
0.891 + 0.035
ENERGY
0.794477
0.008 + 0.007
+.00
0.910 0.028
LOGIT MAX
91.9 4.7%
0.008 0.007
0.971
0.910 1 0.030
COMF_ENTROPY
3.1%%
0.011
0.957
0.872 1 0.078
CONF_SID
88.9 4.0%
0.010
0.952
0.854 - 0.108
LOSIT_DIFF_TOP2
88.9
0.005
0.976
0.917 = 0.074
MARGIN.SOSS
2.79
0.005
0.976
0.917 = 0.074
CONF_MAX
4.24%
0.948 0.001
0.842 = 0.121
LOSS
3.4%%
0.911 0.014
0.831 + 0.118
TOP_K.CONF_SU
86.7 + 1.1%
0.026 + 0.057
0.916 0.005
0.773 + 0.097
CONF_ RACIO
83.6 6.1%
0.001 0.005
0.905 0.050
0.711 0.102
LOGIT-MEAN
61.7 20.5%
0.446 0.266
0.845
0.698 = 0.175
LO9IT_8CD
28.3 - 16.3%
0.812 + 0.166
0.324 + 0.256
0.7 = 0.019
A.4.3. SIGNAT. IMPORTANCE FOR THE PREDICTION CORRECTNESS ESTIMATOR
To analyze the importance of individual signals used to estimate prediction correctness, we present the ANOVA results on FMOW-WILDS in Table 7. All signals, except for class prob ratio, show extremely high F-values with corresponding p-values essentially zero, indicating their strong statistical significance in explaining the variance in prediction correctness. The most valuable signals, as indicated by the highest F-values in Table 7, are logit_max: energy, margin and logit t di ff top2. For certain signals, the sign of the logistic regression coefficient matches our expectations, with a higher logit logit_max max value, an increase in the logit difference for the predicted class and the rumner-up (logit di diff.tor diff.top)) or low energy indicating a correct prediction. Interestingly, however, we also observe that for features such as fonf_max, the sign is negative, indicating that lower confidence is indicative of higher likelihood of correct prediction. While this seems counterintuitive at first, should be noted that for a large majority of samples this signal is and is hence heavily
23

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
Table Table showcasing the mean accuracy and calihration metrics (ECE, MCF, RMSCE) for prediction correctness estimators trained on different signals, with 95% confidence intervals. Suitability Filter refers to the classifier C' trained using all available signals. The metries evaluate classifiers' prediction quality and their calibration over random splits the FMOW-WILDS ID train and validation data splits.
SIGNALS
ACCURACY
ECE
MCE
RMSCE
SUITABILITY FILTER
77.5 = 0.3%
0,021 0.007
0.055 + 0.021
0.027 + 0.006
ICGIT_MAX
+027=00011
0.068 0.050
0.033 + 0.020
ENERGY
0.9%
0.029
0.054
0.035 0.017
KARGIN
7.65
0.021 0.003
0.060
0.028 + 0.002
ICGIT
0.6%
0.021 0.003
0.060
0.028 = 0.002
SONF_ENTRON
74.4 0.9%
0.089 1 0.007
0.208 0.055
0.113 + 0.009
CONF-STD
73.2 0.6%
4.102
0.263 0.068
0.134 + 0.013
CCNF-MAN
0.116 0.57
1.200
0.143 + 0.015
ICSS
0.40
0.121 0.012
0.265 0.036
0.148 = 0.022
IOP_K_CONF_SUX
1.87
0.146 0.026
0.177 = 0.030
ICGIT_XEAN
2.33%
0.081 + 0.033
0.268
0.119 = 0.046
CONF_RATTO
61.1 2.9%
0.144 0.008
0.320 0.008
0.187 + 0.001
TGGTT STD
59.8 2.4%
0.132 + 0.070
0.294 + 0.209
0.167 + 0.108
Table ANOVA results showing the signíficance of individual signals predicting model correctness. Signals are ordered by decreasing F-value, which measures the variance explained by each signal relative to the residual variance. We also include the sign of the model's coefficients for each signal, indicating whether a given feature positively or negatively influences the prediction correctness estimate.
SIGNAL
F-VALUE
P-VALUE
REI..
LOGIT-MAX
2090.61
ENERGY
2051.45
0
MARGIN.LOSS
1982.44
LOGIT.DIFF_TOP2
1978.44
CONFLENTROPY
1390.30
CONF_STD
1232.76
CONF MAX
1108.32
LO5S
934.26
LOGIT MEAN
755.29
TOP_X.CONE.SUM
281.76
LOGIT_STD
116.16
9.18
10 10
CONF_RATTO
12.05
5.21
10
concentrated around 0 after normalization. The contribution from this signal is thus mostly relevant in cases where the maximum confidence is below 1 anyways, in which case it seems that higher confidence can be indicative of incorrect predictions.
SHAP (SHapley Additive exPlanations) is a model-agnostic method for interpreting machine learning models by assigning each feature a contribution value to the model's prediction. It calculates Shapley values based on cooperative game theory, ensuring that the contribution of each feature is fairly distributed by considering all possible feature combinations and their impact on the prediction. As can be seen in Figure 8, the signals deemed most predictive of prediction correctness are the same ones as identified by the ANOVA analysis in Table 7.
A.4.4. CHOICE OF MODEL ARCHITECTURE FOR PREDICTION CORRECTNESS ESTIMATOR
In Table 8, we compare the accuracy of different prediction correctness estimators on the FMoN-WILD dataset. We evaluate a range of classifiers, including simple models like Logistic Regression and more complex architectures such as Single-Layer and Two-Layer Neural Networks. We observe that logistic regression performs as well as more complex models, delivering high accuracy and low expected calibration error. While this may seem surprising, it is important to note that the suitability signals are already non-linear transformations of the model's output logits. Since these transformations capture the key relationships needed for our task, using more complex models capable of learning additional non-linen patterns, such as neural networks, does not provide any further benefit.
24

---

Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings
SHAP Summary Plot for Prediction Correctesss Estimator
High
logit max
energy
oon
logit, dift top?
margin Joss
Peature value
conf entropy
logit std
loss
conf std
confraatio
logil, reean
top conf sum
SHAP value (impect model output)
Figure 8. SHAP analysis for the prediction correctness estimator on
Table Table showcasing the mean accuracy and calibration metrics (ECE, MCE, RMSCE) for various classifiers, with 95% confidence intervals. The metrics evaluate the classificrs' prediction quality and their calibration over 3 random splits of the FMoN-WILDS ID train and validation splits.
CLASSIFIER
ACCURACY
ECE
MCE
RMSCE
LOGISTIC REGRESSION
77.2 1.4%
0.022 + 0.004
0.055 + 0.040
0.027 0.007
GRADIENT BOOSTING CLASSIFIER
77.1 1.3%
0.020 0.018
0.062 0.102
0.027
0.030
SINGLE-LAYER NEURAL AL NETWORK
4.2%
0.037 + 0.006
0.086 0.017
0.046 0.006
SUPPORT VECTOR MACHINE
7
0.077 + 0.034
0.232 0.147
0.107 = 0.050
RANDOM FOREST
76.5 0.9%
0.031 0.007
0.067 + 0.068
0.037 = 0.016
TWO-LAYER NEURAL NETWORK
75.7 2.3%
0.040 0.006
0.087 0.012
0.048 0.002
DECISION TREE
70.7 0.9%
0.111 0.021
0.161 0.050
0.123 0.029
25


Reviews:
[{"review_id": "gk7siavx7C", "reviewer": "Reviewer_i1Vd", "summary": "The authors developed a framework to evaluate how well a trained model will perform when deployment for real-world inference.\nThe framework combines ideas from distribution shift detection, selective inference, and interestingly dataset inference.\nA specific instantiation of the framework is presented and evaluated on datasets from the WILDS benchmark collection.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time spent assessing our paper and for their critical and detailed feedback. We particularly appreciate their positive evaluation of our work’s practical applicability, theoretical foundation and experimental design. We provide our answers to the reviewer’s raised concerns below, are looking forward to their response, and hope for a favorable reassessment of our score.\n\n\n**C1: Claim of detecting performance deterioration of more than 3% with 100% accuracy.**\nWe assume that the reviewer refers to key contribution 4 at the end of the introduction section. It should be clarified that this is not a general performance guarantee but an empirical finding that has been validated across 29k experiments on FMoW-WILDS. We have made sure to clearly mention this in an updated revision of the paper and to point interested readers to Figure 4, which visualizes this finding. \n\n\n**C2: Claim of model architecture agnosticism.**\nThe proposed suitability filter framework is inherently compatible with various model architectures because it only requires the model to be a classifier, without making any other assumptions. As mentioned in the introduction, we propose a well-performing default instantiation of the filter. This instantiation uses domain-agnostic suitability signals and can be applied to any classifier that outputs logits, regardless of its underlying architecture. To show this empirically, we evaluate this instantiation on diverse architectures including a DenseNet-121, a ResNet-50 and a DistilBERT-base-uncased model. Additional details about our experiments are provided in Appendix A.2.2. We hope that this addresses the reviewer’s concern and welcome any further questions.\n\n\n**C3: Connection to Gruber et al.** \nWe appreciate the reviewer’s suggestion to consider the decomposition of uncertainty into aleatoric and epistemic factors, as proposed by Gruber et al. However, this approach does not directly align with our primary objectives. Our method does not rely on quantifying or separating out specific sources of uncertainty; rather, it compares predictive performance deviations between the user’s dataset and the model provider’s test set. This viewpoint effectively marginalizes over varying error sources that might lead to a decrease in accuracy, including aleatoric and epistemic uncertainty. Nonetheless, we acknowledge that such a decomposition may be valuable as an alternate instantiation of suitability filters beyond accuracy (i.e., determining suitability based on deviations in uncertainty beyond a margin $m$).\n\n\n**C4: Adversarial examples and the limitations of suitability filters.**\nWe thank the reviewer for their insightful comments regarding deliberate perturbations and adversarial robustness. There seem to be different angles to this concern that we would like to address separately:\n\n1. Deliberate perturbation of examples: The reviewer correctly pointed out that a model user could deliberately perturb their data to be misclassified by the model without these perturbations being detected by the suitability filter. One of the key underlying assumptions of our framework is that both the model provider and the model user are non-adversarial and provide representative samples of their datasets. We believe this is realistic because the model user's goal is to identify the most suitable model for their actual task. Deliberately perturbing their own data samples to pass a suitability check would undermine this goal. We will clarify this assumption in the paper.\n2. Using suitability filters to evaluate adversarial robustness: While we have not confirmed this experimentally, we do think it should be possible to use suitability filters to evaluate adversarial robustness. We will leave it to future work to explore suitability filters where suitability is defined based on other metrics than accuracy.\n3. Effects of distribution shift for adversarially-trained models: Our framework could indeed be used to evaluate whether adversarially trained models exhibit greater robustness to distribution shifts. Since the suitability filter is independent of the training algorithm, it could be used to compare the suitability of two models on the same user data, one of which was trained normally and the other with adversarial training. \n\n\n**C5: Figures to illustrate calibration.**\nWe visualize the effect of miscalibration and motivate the margin adjustment under accuracy estimation error in Figure 3. In Figure 5 in the Appendix, we empirically show the effect of miscalibration on the FMoW-WILDS dataset. We would be happy to include additional figures to illustrate calibration if the reviewer has a specific figure in mind.\n\n\nLastly, we thank the reviewer for pointing out the typos and we will make sure to correct them in future versions of our paper. We hope that we have addressed the reviewer’s concerns and are looking forward to their response.", "claims_and_evidence": "One claim is unwarranted: \"we are able to detect performance deterioration of more than 3% with 100% accuracy\".\nAlso the claim about being agonistic to the model architecture needs to be substantiated. \nOtherwise, the claims are reasonable.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I skimmed over the proofs of Lemma 4.3. and COROLLARY 4.4. There are a few assumptions introduced but they are reasonable.", "experimental_designs_or_analyses": "Reasonable. WILDS is a a great benchmark of in-the-wild distribution shifts.", "relation_to_broader_scientific_literature": "The presented work combines ideas from closely related topics and utilizes them to design a practical framework.", "essential_references_not_discussed": "It would quite helpful to relate the presented solution with the statical view of uncertainty by Gruber et al: \n\"Sources of Uncertainty in Supervised Machine Learning – A Statisticians’ View\"\nhttps://arxiv.org/abs/2305.16703", "other_strengths_and_weaknesses": "This is a neat work that builds on the progress in distribution shift detection to provide a comprehensive and practical solution to make decision on the suitability of an ML model at deployment time.\n\nSome highlights:\n+ careful usage of statistical measures to instantiate the proposed framework and calibrate it against a target dataset.\n+ statical guarantees backed by theoretical analysis.\n+ the authors discuss several practical aspects such as continuous monitoring.\n\nSome recommendations:\n- Provide more details about the model architectures used in the evaluation. Demonstrate quantiatively that you approach is agnostic to the architecture used.\n- Provide examples that the framework do not predict well. It is not hard to introduce delibrate perturbations to a dataset that degrades the performance significantly while passing all the checks you introduce.\n- Consider breaking down uncertainty into aleatoric and epistemic as proposed by Gruber et al  (see reference cited above).\n- Provide figures to illustrate some key concepts such as calibration.", "other_comments_or_suggestions": "I encountered a few typos:\n- degredation => degradation\n- are no limited => not\n- deplyoment", "questions_for_authors": "Can the developed framework be instantiated for evaluating adversarial robustness?\nSimilarly, I would curious to see if adversarially-trained models fare better in handling distribution shifts as measured by your framework.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vi62M4mwxN", "reviewer": "Reviewer_RLS2", "summary": "This paper proposes a new paradigm for detecting whether the performance on unlabeled data processed during deployment, falls by a certain margin below the performance on a held-out dataset sampled from the training distribution but used for evaluation. This is a novel framework for detecting performance deterioration in deployment settings under covariate shift.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time spent assessing our paper, and appreciate their positive evaluation of our work’s relevance, novelty, and soundness.", "claims_and_evidence": "The results indicate that the proposed approach shows good performance on both in-distribution and out-of-distribution experiments across different tasks and datasets.", "methods_and_evaluation_criteria": "This paper evaluates on different tasks and corresponding benchmark datasets. The datasets for classification span land use classification, text toxicity classification, and genetic perturbation classification.", "theoretical_claims": "I have looked at the proofs at a high level, but they are beyond my area of expertise. I think that it would be best if someone with more expertise could kindly check the details.", "experimental_designs_or_analyses": "The paper uses feature groups in the benchmark datasets to understand out-of-distribution performance. This approach is sound as these groups are naturally occurring. The suitability signals seem to cover different metrics informative to the suitability filer.", "relation_to_broader_scientific_literature": "Given the rise in deployed machine-learning models in safety-critical systems, this work is highly relevant.", "essential_references_not_discussed": "The related works section describes different orthogonal ways to do different parts of their approach, and why and how they differ from the proposed approach.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjOK6J5BRq", "reviewer": "Reviewer_kVMa", "summary": "This paper presents a new problem of predicting whether the model's performance on a runtime dataset (called user dataset) has decreased compared to the test dataset. To tackle this problem, they introduce a method called suitability filter, which consists in two main steps: 1. several sample-level performance scores from the literature are computed and fused together with a learned regression model. The output of the regression model should predict correctness of individual samples. 2.  A statistical test is conducted on fused sample signals to predict whether the whole user dataset differs in performance from the test dataset.\n\nThe authors present several theoretical proofs regarding the efficiency of their approach under restrictive assumptions.\n\nThey conduct experiments to test their approach on benchmark datasets from WILDS.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time spent assessing our paper and for their critical and detailed feedback. We particularly appreciate their positive evaluation of our work’s novelty, the framework’s modularity and our experimental evaluation. We provide our answers to the reviewer’s raised concerns below, are looking forward to their response, and hope for a favorable reassessment of our score.\n\n**C1: Practical applicability of theoretical results.**\nAs can be seen in Figure 5, margin adjustments become increasingly important the further away we move from the test set accuracy. On FMoW-WILDS, adjusting a 5% margin with e.g., 500 labeled user samples reduces the false positive rate from unbounded to below 0.03, satisfying a 0.05 significance level. The number of labelled samples available plays a crucial role here: the higher the number of samples, the more accurate the estimates of $\\Delta_u$ and thus, the more precise the margin adjustment that leads to theoretical guarantees.\n\n**C2: Importance of the prediction correctness estimator for the end-to-end suitability filter performance.**\nOur prediction correctness estimator leverages multiple suitability signals—several of which have been independently proposed in the selective classification literature (see Section 2)—yielding performance on par with or better than any single signal for misclassification prediction. The contribution of the statistical test to the end-to-end suitability filter performance is orthogonal to this, as it allows for the aggregation of prediction correctness estimates across data samples. \n\n\n**C3: Clarity of Experimental Setup (Section 5.1).**\nWe thank the reviewer for remarking that Section 5.1 is hard to follow and propose to rewrite the section as follows:\n\"We evaluate the suitability filter on FMoW-WILDS, CivilComments-WILDS and RxRx1-WILDS. For each dataset, we follow the recommended training paradigm to train a model using empirical risk minimization and the pre-defined $D_{\\text{train}}$. We then further split the provided in-distribution (ID) and out-of-distribution (OOD) validation and test splits into folds as detailed in Appendix A.2.2 (16 ID and 30 OOD folds for FMoW-WILDS, 4 ID and 8 OOD folds for RxRx1-WILDS, and 16 ID folds for CivilComments-WILDS). We conduct two types of experiments: first, each ID fold is used as the user dataset ($D_u$​), and the remaining ID data is split into 15 subsets, used as $D_\\text{test}$ and $D_\\text{sf}$​. This yields 16×15×14 experiments for FMoW-WILDS, 4×15×14 for RxRx1-WILDS, and 16×15×14 for CivilComments-WILDS. Second, each OOD fold is used as $D_u$, and the ID data is split into 15 subsets, used for $D_\\text{test}$ and $D_\\text{sf}$​​. This yields 30×15×14 experiments for FMoW-WILDS and 8×15×14 for RxRx1-WILDS.\"\n\n\n**C4: Delineation of applicability scenarios.**\nWe thank the reviewer for highlighting the potential limitations of our suitability monitors in autonomous systems. We agree that for critical applications, focusing on average-case suitability is not sufficient. Instead, ensuring good performance on a per-instance (or worst-case) basis is crucial, a problem setting that is addressed in the selective classification literature. Our work, however, focuses on scenarios where average-case suitability is the primary concern. We will ensure to clarify this in future versions of the paper.\n\n\n**C5: Possibly contradictory statement in Definition 3.2.**\nWe respectfully disagree with the concern that using “if and only if” and “with high probability” in the same sentence is contradictory. As per Definitions 3.1 and 3.2, \"if and only if\" defines the deterministic rule for outputting “SUITABLE,” while \"with high probability\" captures the uncertainty in estimating model performance (due to sampling or variance). Once the high-probability condition is met—i.e., the model's performance on $D_{u}$ is within a margin $m$ of its performance on $D_\\text{test}$—the filter deterministically outputs “SUITABLE.”\n\n\n**C6: Complexity of notations and nomenclature.**\nWe tried to strike a balance between theoretical rigor and avoiding overly complex notations. Thus, in both equations 2 and 3, $x_1$ represents the first sample of the respective dataset. $D_\\text{sf}$ stands for the **s**uitability **f**ilter data that is used to learn the prediction correctness estimator.\n\n\n**C7: Meaningfulness of Theorem 4.2.**\nThis theorem, drawn from established results (Lehmann et al., 1986; Wellek, 2002), underpins our theoretical analysis. The assumption of independent, normally distributed samples is standard for many tests, allowing us to use these guarantees. Here, $\\alpha$ denotes the significance level, the probability of a false positive. \n\n\nLastly, we will consider changing the title to reflect that our work focuses on classification only as suggested by the reviewer. We hope that we have addressed the reviewer’s concerns and are looking forward to their response.", "claims_and_evidence": "In view of the experimental results, the method seems effective in practice.\n\nMy doubt is regarding the applicability/usefulness of the theoretical results. \nHow are these bounds used in practice? Are they actually verified from the experiments? I did not find answers to these questions in the manuscript.", "methods_and_evaluation_criteria": "The proposed evaluation is sufficient and rigorous.\nThey do not compare with existing literature but it is OK as the problem presented is new.\n\nHowever, I would have liked to see an ablation study to see the importance of the two contributions. \nFor instance, I would have liked to see how their signal fusion approach works to detect errors on individual samples.\nAs this is a widely studied problem, this would have allowed for comparison against the literature.", "theoretical_claims": "I did not check the details of the proofs.", "experimental_designs_or_analyses": "The experiments are sufficient and well designed. \nHowever, I believe that Section 5.1 is really hard to follow. Experimental description should be made more simple. Why introducing new variables (n, m, k) to present the experimental setup? It would be much clearer to simply replace these values with the true values that were actually used in the experiments.", "relation_to_broader_scientific_literature": "The literature in the broad topic of runtime monitoring of neural networks is becoming very broad, but the literature review proposed in the paper is sufficient to introduce the new problem and present the novelty of the work.", "essential_references_not_discussed": "Nothing specific", "other_strengths_and_weaknesses": "The ideas presented in this paper are very interesting.\nOverall the presented framework is generic and modular, which is good.\n\nHowever, I sometimes felt like the notations and concepts presentations were overly complex, even for simple ideas, making the paper hard to follow.\n\nRegarding the introduction, I think it would have been worth discussing further the relevance and applicability domain of the newly introduced problem. Unlike most existing methods, they propose to not consider data suitability at the sample level, but rather at the dataset level, by aggregating acceptance/rejection signals. This is an interesting take, which is interesting for some applications but not all. For example, the credit risk model presented in the introduction is a good use case where dataset level suitability is interesting. However, other cases such as autonomous systems might not respond very well to such kind of safety monitors. It would be good to give more examples and to delineate applicability scenarios in the introduction.", "other_comments_or_suggestions": "- This work focuses only on classification tasks. This should be reflected in the title: Suitability Filter: A Statistical Framework for Classification Model Evaluation in Real-World Deployment Settings.\n\n- Definition 3.2: The use of both \"if and only if\" and \"with high probability\" in the same sentence seem contradicting. \n\n- Why the name Dsf? What does it stand for?\n\n- In eq 2 and 3, same notations (x1) represent different things.\n\n- I am not sure to understand the meaningfulness of Theorem 4.2 and its practical relevance and the insights that it brings. Is the assumption of independant and normally distributed samples really relevant? Where is it used later on? If it is only used in the proofs, maybe it should be placed in the appendix.\n\n- Typo in caption of table 1: =rate \n \n- Two different definitions associated with parameter alpha: In the experiments it is the FP rate whereas in the theroem 4.2 it is different.", "questions_for_authors": "See questions and comments from other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The proposed suitability filter framework effectively addresses the crucial challenge of detecting model performance deterioration on unlabeled user data compared to labeled test data. All reviewers acknowledge the work's novelty, theoretical soundness, and practical utility for safety-critical applications. My recommendation is to accept the paper.
Decision: Accept (oral)