Counterfactually Comparing Abstaining Classifiers
Authors: Yo Joong Choe
Aditya Gangrade
Aaditya Ramdas
Venue: NeurIPS 2023 poster (2023)
Keywords: abstaining classifiers
black-box model evaluation
causal inference
missing data
TL;DR: We develop a novel framework for black-box evaluation and comparison of abstaining classifiers by casting the setting as a missing data problem.

Abstract: Abstaining classifiers have the option to abstain from making predictions on inputs that they are unsure about. These classifiers are becoming increasingly popular in high-stakes decision-making problems, as they can withhold uncertain predictions to improve their reliability and safety. When evaluating black-box abstaining classifier(s), however, we lack a principled approach that accounts for what the classifier would have predicted on its abstentions. These missing predictions matter when they can eventually be utilized, either directly or as a backup option in a failure mode. In this paper, we introduce a novel approach and perspective to the problem of evaluating and comparing abstaining classifiers by treating abstentions as missing data. Our evaluation approach is centered around defining the counterfactual score of an abstaining classifier, defined as the expected performance of the classifier had it not been allowed to abstain. We specify the conditions under which the counterfactual score is identifiable: if the abstentions are stochastic, and if the evaluation data is independent of the training data (ensuring that the predictions are missing at random), then the score is identifiable. Note that, if abstentions are deterministic, then the score is unidentifiable because the classifier can perform arbitrarily poorly on its abstentions. Leveraging tools from observational causal inference, we then develop nonparametric and doubly robust methods to efficiently estimate this quantity under identification. Our approach is examined in both simulated and real data experiments.

Counterfactually Comparing Abstaining Classifiers
Yo Joong Choe
Aditya Gangrade Department of EECS University of Michigan aditg@umich. ch. edu
Aaditya Ramdas
Data Science Institute
Dept. Statistics and Data Science Machine Learning Department
University of Chicago
yj zjchoe@cchicago.ee.
Carnegie Mellon University
aramdasccmu edu
Abstract
Abstaining classifiers have the option to abstain from making predictions on inputs that they are unsure about. These classifiers are becoming increasingly popular in high-stakes decision-making problems, as they can withhold uncertain predictions to improve their reliability and safety. When evaluating black-box abstaining classifier(s), however, we lack principled approach that accounts for what the classifier would have predicted on its abstentions. These missing predictions matter when they can eventually utilized, either directly or as a backup option a failure mode. In this paper, we introduce novel approach and perspective the problem evaluating and comparing abstaining classifiers by treating abstentions as missing data. Our evaluation approach is centered around defining the counterfactual score an abstaining classifier, defined as the expected performance of the classifier had it not been allowed to abstain. We specify the conditions under which the counterfactual score is identifiable: if the abstentions are stochastic, and if the evaluation data is independent of the training data (ensuring that the predictions are missing at random), then the score is identifiable. Note that, if abstentions are deterministic, then the score is unidentifiable because the classifier can perform arbitrarily poorly on its abstentions. Leveraging tools from observational causal inference, we then develop nonparametric and doubly robust methods efficiently estimate this quantity under identification. Our approach is examined in both simulated and real data experiments.
Introduction
Abstaining classifiers (Chow] 1957 EI- Yaniv and Wiener 2010), also known as selective classifiers or classifiers with a reject option, are classifiers that have the option abstain from making predictions on certain inputs. As their use continues to grow in safety-critical applications, such as medical imaging and autonomous driving, it natural ask how practitioner should evaluate and compare the predictive performance of abstaining classifiers under black- box access to their decisions.
In this paper, introduce the counterfactual score
evaluation metric for black-box abstaining classifiers. The counterfactual score is defined as the expected score an abstaining classifier's predictions, had it not been allowed to abstain. This score is of intrinsic importance when the potential predictions on abstaining inputs are relevant. We proceed with an illustrative example:
Example 1.1 (Free-trial ML APIs). Suppose compare different image classification APIs Each API has two versions: a free version that abstains, and a paid one that does not. Before paying for the full service, the user can query the free version for up to predictions on user-provided dataset, although it may choose to reject any input that it deems as requiring the paid service. Given two such APIs, how can the practitioner determine which of the paid mnon-abstaining) versions would be better on the population data source, given their abstaining predictions on sample?
This work submitted while this author was at Camegic Mellon University.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Example 1.1 exhibits why someone using black-box abstaining classifier would be interested in its counterfactual score: the user may want reward classifiers whose hidden predictions are also (reasonably) good, as those predictions may be utilized in the future. This is hypothetical example, but we can imagine other applications of abstaining classifiers where the counterfactual score is meaningful. In Appendix [. we give three additional examples, including safety-critical applications where the hidden predictions may be used as a backup option in a failure mode.
To formally define, identify, and estimate the counterfactual score, we cast the evaluation problem in Rubin (1976'ss missing data framework and treat abstentions as missing predictions. This novel viewpoint directly yields nonparametric methods for estimating the counterfactual score of an abstaining classifier, drawing upon methods for causal inference in observational studies (Rubin) 1974, Robins et al. 1994] van der Vaart 2000). and represents an interesting yet previously unutilized theoretical connection between selective classification, model evaluation, and causal inference.
The identification of the counterfactual score is guaranteed under two standard assumptions: the missing at random (MAR) condition, which satisfied as long as the evaluation data is independent of the classifier (or its training data), and the positivity condition. As with standard missing data settings, both the MAR and positivity conditions are necessary for identification. We later discuss each condition in detail, including when the positivity condition is met and how a policy-level approach may be necessary for safety-critical applications.
The counterfactual score can be viewed as an alter- native to the selective score (mean score nonah- stentions) and the coverage (1 minus the abstention rate) (EI- Yaniv and Wiener 2010), which are the
To CUNENE WARTS-BOO ARSTARNNG CLASSEERS TRAI COMBAAS BOOHBRARE
COMPARING BLACK ASSTANINGG CLASSITRRS
main existing metrics for evaluating black-box ab-
staining classifiers. As a two-dimensional metric,
ARE MIssING PREDICTIONS MEANINGFULL
No.
ARE SCOTES & COMBRACE
COMPEASBLE?
comparison on the basis of these is non-trivial. A common approach is assume a fixed cost for each abstention (Chow) 1970), but this is not always sat- isfactory since determining how to weigh absten- tions and errors against one another is a nontrivial question. Thus, in settings such as Example 1.I the notion of counterfactual score becomes necessary. Importantly, selective scores are not representative of the counterfactual performance, except in the (unrealistic) case wherein predictions are missing completely at random (MCAR) P Figure 1 ggives overview of scenarios where different metrics may be appropriate to compare abstaining classifiers.
COMpARE COUNTRRFNNTUA ScoReS (T:as WoRK)
COMPARS SELECTIVE SCORES
AND COVBRAGE
Figure 1: A schematic flowchart of comparing abstaining classifiers. In black-box setting where the evaluator does not have access to the training algorithms or the resources train them, the task can be viewed missing data problem. This work proposes the counterfactual score an evaluation metric.
nontrivial
The counterfactual score also offers practical benefits when comparing abstaining classifiers. Coun- terfactual scores are comparable even if abstaining classifiers are tuned to different abstention rates or selective scores. Moreover, compared to evaluation methods using the selective score-coverage curve (equivalent to re-training the classifier several times at different score/coverage levels), estimating the counterfactual score does not require re-training the classifier. Instead, we only need to estimate a pair of nuisance functions that can be learned using the observed predictions (nonabstentions) in the evaluation set. Let us further note that the setup is applicable generally to any form of prediction that can be scored, including regression and structured prediction. In this paper, we restrict our attention to classification for concreteness, as it is the most well-studied abstention framework.
Summary of contributions We first formalize the problem of comparing abstaining classifiers as a missing data problem and introduce the counterfactual score as a novel metric for abstaining classifiers. Next, we discuss how the counterfactual score can be identified under the MAR and positivity conditions. Then, we develop efficient nonparametric estimators for the counterfactual scores and their differences, namely doubly robust confidence intervals (DR CI). Finally, we validate our approach in simulated and real-data experriio experiments.
MMCA means the missing observations are simply a uniformly random subset of all observations, indepen- dently of the input/output. In contrast, MAR means there can be systematic differences between the missing and observed values, but these be explained by the input. Our method only requires MAR.

---

Related work on abstaining classifiers The training of abstaining classifiers has seen significant interest in the literature (e.g., Chow, 1970, 197 [2055] Bartlett and Wegkamp 2008 Cortes
et al. 2016 Geifman and HI- Yaniv 2017 2019 Gangrade et al. (021). We refer to Hendrickx et al.
(2021 ): Zhang et al. ) for recent surveys. For evaluation, aside from using some combination
of selective score and coverage, the most pertinent reference is the work of ofCCnnessa et al. (2017).
who propose the metric of "classifier quality" that is a somewhat inverse version of our counterfactual accuracy. This metric is the sum of the prediction accuracy when the classifier predicts, and prediction inaccuracy when it abstains, the idea being that a classifier is not abstaining needlessly, then points it abstains are very poor. While this view is
must hold that the underlying predictions relevant to the training of abstention rules, it is at odds with black-box settings where the underlying predictions may still be executed even when the method abstains, motivating the counterfactual score. In terms of estimating the score, Condessa et al. 20117) do not discuss the black-box setting that requires estimating counterfactual, they assume full knowledge of the classifier's predictions.
Related work on missing data, causal inference, and doubly robust estimation Our main approach to the estimability of counterfactual scores is driven by a reduction to an inference problem under missing data (or censoring) (Rubin] 1976; Little and Rubin (019). A missing data problem can be viewed equivalently as a causal inference problem in an observational study (Rubin 1976 Pearl (h0pitShpitser et al. 2015; Ding and Li 2018), and there exist well-established theories and methods for both identifying the counterfactual quantity of interest and efficiently estimating the identified target functional. For identification, unlike in some observational settings for estimating treatment effects, setup straightforwardly satisfies the standard assumption of consistency (non-interference). The MAR assumption is satisfied as long as independent evaluation data is used, and the positivity assumption translates to abstentions being stochastic. We discuss various implications of these conditions in Sections 2.2 and 5 For estimation, efficient methods for estimating targets such as the average treatment effect (ATE) have long been studied under semiparametric and nonparametric settings. Doubly robust (DR) estimators, in particular, are known to achieve the asymptotic minimax lower bound on the mean squared error. For details, we refer the reader to Bickel et al. (1993); ; Robins
ct al.] 1994); van der Vaart (2000] (2022); van der Laan and Robins (2003); Bang and Robins (2005); ;
Tsiatis (2006); CChrnnzhukov al. (2018); Kennedy (2022). Unlike the standard ATE estimation setup in observational studies, our setup contrasts two separate causal estimands, the counterfactual
scores of the two competing classifiers, that operate under their distinct missingness mechanisms.
2 Definition and identification of the counterfactual score
We formulate the problem of evaluating and comparing abstaining classifiers under the missing data framework (Rubin) 1976). We follow the standard approach of defining the target parameter (Section 2.T), identifying it with observable quantities (Section (.2), and estimating the identified parameter using data (Section 3 In each step, we first consider evaluating one abstaining classifier and then extend to comparing two abstaining classifiers. Below, X denotes the input space and )' = (1....,C7 C} is the set of possible classes, while C denotes the C-dimensional probability simplex on V
Abstaining classifiers
We define an abstaining classifier as a pair of functions (f. R), representing
its base classifier f : X + CC1 and abstention mechanism : X - [0, 1], respectively. Given query X, the classifier first forms a preliminary (probabilistic) prediction f(X). Then, potentially using the output f fXX), the classifier determines π (X). i.e., the abstention probability. Using x (X). the classifier then makes the binary abstention decision R π (X) ~ (x)()(X)), so that if R = 1 ("rejection")) the classifier abstains on the query, and if R 0, it reveals its prediction f(X). In some cases, we will explicitly define the source of randomness E (independent of the data) in deciding R. such that It = r(= (X),, E) for deterministic function r Neither f nor is assumed to be known to the evaluator, modeling the black-bor box access typically available to practitioners.
Scoring rules (Higher scores are better.) We measure the quality of a prediction f(x) for label y via a positively oriented scoring rule s: A C-1 x y + R. One simple scoring rule is classification accuracy, i.c., s(f()).)))) y) 1 (argmax.ey) f(x) y). but plethora of scores exist in the literature,
such the Brier 1950) score: s(f(x), y) = 1 Ecey (f)) - 1 (1 c))?
"Specifically, let & ~ Unif(0, 1] and At 1 (6 < =(X)). Then, R is function of only =(X) and E.
3

---

The evaluation setup For each labeled data point (X,Y) in an evaluation set, we observe the abstention decision R (() for some independent source of randomness ç used by the abstaining classifier. Then, its prediction f(X) is observed by the evaluator if and only if R = O. Let S == s(f(X), denote the score of the prediction I on the query X. irrespective of R. Because S is not observable when R 1, we refer to S as the po- tential score that would have been seen had the classifier not abstained. (See Appendix [A.2 for equivalent formulations that explicitly invoke Rubin (1974)'s potential outcomes model.) Since our evaluation based only on the score S, we can suppress the role of Y and assume that S is observed directly when R = 0. Similarly, of E, which is independent of the data. We let TP denote the law of 7 == (X; R, S)-
Table I: A summary of problem formulations and pro- posed approaches for evaluation and comparison of ab- staining classifiers. Our approaches avoid parametric assumptions and allow for black-box classifiers.
suppress the role
Evaluation
Comparison
Table I summarizes the problem formula- tions and the proposed approaches in our paper. The target definitions and the iden- tification conditions are discussed in the following subsections; the estimator and its optimality are discussed in Section 3
Classifier(s) Target
(f.T) = [S]
(f* T
A =
Identification Estimation
MAR & positivity
Doubly robust CI
Optimality
Nonparametrically efficient
2.1
The counterfactual score
We propose assess an abstaining classifier (f, with its (expected) counterfactual score:
/ = E (S),
(2.1)
where the expectation is taken w.r.t. PP. In words, t refers to the expected score of the abstaining classifier had it not been given the option to abstain. The counterfactual score captures the performance of an abstaining classifier via the score of its base classifier, making it suitable for cases where the evaluator is interested in the predictions without using an abstention mechanism.
Note that sb does not in general equal the selective score, i.e., [S| R = Oj. For example, when classifier abstains from making predictions its "weak points," i.e., inputs on which the clas- sifier performs poorly, the counterfactual score will be lower than the selective score. Also see Appendix AA.3 for direct comparison with Condessa et al. (2017)'s score, which rewards having bad hidden predictions, as opposed to (2.1). Our general framework allows estimating either score, although we focus 2.1 based our motivating examples (Example 1.] and Appendix A.
Comparison Counterfactual scores may also be used to compare two abstaining classifiers, (fa, ,7 and (f ,²), in the form of their counterfactual score difference: A AB = A v/ ElSAA S Here, the expectation is now taken over the joint law of 7AB = (X, RA SA RB SB).
2.2 Identifiability of the counterfactual score
Having defined the target parameters s/ and AB, we now discuss the assumptions under which these quantities become identifiable using only the observed random variables. In other words, these assumptions establish when the counterfactual quantity equals a statistical quantity. As in standard settings of counterfactual inference under missing data, the identifiability of counterfactual scores in this setting depends on two standard conditions: (i) the missing at random condition and (ii) positivity.
The missing at random (MAR) condition, also known as the ignorability or no unmeasured confound- ing condition, requires that the score S is conditionally independent of the abstention decision R given X, meaning that there are no unobserved confounders U that affect both the abstention decision R as well as the score S. Note that S is the potential score of what the classifier would get had not abstained - it is only observed when R = O. We formally state the MAR condition as follows: Assumption 2.1 (Scores are missing at random). S 1L R X.
In standard ML evaluation scenarios, where the evaluation set independent of the training set for the classifier, Assumption 2 is always met. We formalize this sufficient condition for MAR in the following proposition. Tet Dirain denote the collection of any training data used to learn the abstaining classifier (f, т) and, as before, (X, Y) denote an (i.id.) data point in the evaluation set.
4

---

Proposition 2.2 (Independent evaluation data ensures MAR). (X) 1L Derain then S TT R I.
This result is intuitive: given an independent test input X. the score S = s(f(X), Y) is deterministic function of the test label Y, and the abstention decision of classifier cannot depend on Y simply because the classifier has access to it. A short proof is given in Appendix [B.1 for completeness. In Appendix
C we also include causal graphs that visually illustrate how the MAR condition met.
If the evaluation data is not independent of P then the classifier already has information about the data on which generalization performance. Although the independence between the training and evaluation data is expected in standard MI. applications, may not be guaranteed when, e.g., using publicly available dataset that is used during the training of the classifier. These issues can be prevented by ensuring that the evaluation set is held out (e.g., hospital can use its own patient data to evaluate APIs).
tested, so generally speaking, no evaluation score will not accurately reflect its
The second condition, the positivity condition, is more substantial in our setting:
Assumption 2.3 (Positivity). There exists >> 0 Ouch that π (X) P (R 11
Assumption 2.3 says that, for each input X, there has to be at least a small probability that the classifier will not abstain (R = 0). Indeed, if the classifier deterministically abstains from making predictions on a specific input that has nonzero marginal density, then we have no hope of estimating an expectation over all possible values that X can take. When it comes to evaluating abstaining classifiers on safety-critical applications, we argue that this condition may need to be enforced at a policy level - we elaborate on this point in Section and Appendix [A.3 In practice, the exact value of is problem-dependent, and in Appendix E.4 we include additional experiments illustrating how our methods retain validity as long as the abstention rate is capped at 1 é for some >> 0.
Another justification for the positivity condition is that stochastically abstaining classifiers can achieve better performances than their deterministic counterparts, Kalai and Kanade (2021) illustrate how stochastic abstentions can improve the out-of-distribution (OOD) performance w.r.t. the Chow (1970) score (i.e., selective score + a coverage). SSchreuder and Chzhen (202I) also introduce randomness in their abstaining classifiers, which leverage abstentions as a means to improve their accuracy while satisfying a fairness constraint. The role of random abstentions in these examples mirrors the role of randomization the fairness literature (Barocas et al., 2019), where the optimal randomized fair predictors are known to outperform their deterministic counterparts (Agarwal and Deshpande] 2022) Grgié-Hlača et al. 2017). Given the effectiveness of randomized classifiers for fairness, it would not be surprising if fair abstaining classifier was randomized (in its decisions and abstentions).
With MAR and positivity in hand, we can show that the counterfactual score is indeed identifiable. Define no (I) := E [S] R = O, = x] as the regression function for the score under R = O.
Proposition 2.4 (Identification). Under Assumptions 2.7 and 2.3 v is identified as E (X)(
The proof, included in Appendix B.2 follows standard argument in causal inference. The identifica- tion of the target parameter v using Ho implies that we can estimate tb, the expectation of potential outcome, using only the observed inputs and scores. Specifically, the task of estimating & consistently reduces to the problem of estimating the regression function Mo, which only involves predictions that the classifier did not abstain from making. We note that, as standard causal inference, the task of identification, which concerns what to estimate, is largely orthogonal to the task of estimation, which concerns how to estimate the quantity. We discuss the latter problem Section 3
Comparison For the comparison task, given A ABB = a. 1B8 it immediately follows that if the MAR and positivity assumptions hold for each of (X, RA, SAA) and (X, R S5), then A is also
identified AB E (X) - a (X)]. where (z(x)) := EE [S* R 0, x] for E {A, B}. In
words, if (i) the evaluation data is independent of the training data for each classifier (Proposition 2.2
and (ii) each classifier has at least small chance of not abstaining on each input (Assumption 2.3)
then the counterfactual score difference, i.e., AB is identified as the expected difference in the expected scores over inputs conditional on non-abstentions, i.c., x - -1g (X)).
3 Nonparametric and doubly robust estimation of the counterfactual score
Having identified the counterfactual scores, we now focus on the problem of consistently estimating them. We estimate these quantities without resorting to parametric assumptions about the underlying
5

---

black-box abstention mechanisms. Instead, we reduce the problem to that of functional estimation and leverage techniques from nonparametric statistics. See Kennedy (2022) for a recent review.
3.1 Estimating the counterfactual score
Task Let {(X,, R., S))=== ~ P denote an i.d. evaluation set of size n. As before, we assume that we given access to the censored version of this sample, i.c., that observe S; if and only if R. 0. Using the observables, we seek form an estimate & of the counterfactual score v = E[S].
Doubly robust estimation Under identification (Proposition 2.4 we can estimate by estimating the regression function (x) on the data { Rg = 0j. However, the naive "plug-in" estimate suffers from an inflated bias due to the structure present in the abstention patterns. (See
Appendix (o.endix/A.44for details.) We instead develop a doubly robust (DR) estimator (Robins et al. 1994
Bang and Robins 2005), which is known to consistently estimate yé at the optimal nonparametric
efficincy rates, meaning that no other estimator based on the n observations can asymptotically
achieve a smaller mean squared error (van der Vaart 2002). The derivation below is relatively standard, explaining our brevity.
Formally, the DR estimator is defined using the (uncentered) efficient influence function (EIF) for the identified target functional ((P) = [x]]= IF(,, := po(z) + I-n(=s] (s (s-1o) (z))
I-c
(0/0- := 0). Here, and μ4 are the "nuisance" functions, representing the abstention mechanism and the score function under R = 0, respectively. The EIF can be computed as long as 8 is available when
0. An intuition for the EIF is that it is the first-order "distributional Taylor approximation" (Fisher and Kennedy (021) of the target functional, such that its bias is second-order.
Given that π and Ho are unknown, we define an estimate of the EIF, denoted as IF, by plugging in estimates â for and ño for Hoo Then, the DR estimator is simply the empirical mean of the EIF:
1 Σ dr 71
F(X R2 S.) = n Σ
po((.))
1- R (S: - Ro(X.))
(3.1)
1 1 =Xx))
This estimator is well-defined because S; is available precisely when R² = 0. Note that the first term is the (biased) plug-in estimator, and the second term represents the first-order correction term, which involves inverse probability weighting (IPW) (Horvitz and Thompson] 1952 [Rosenbaum 1995). In our experiments, we show how the DR estimator improves upon both the plug-in estimator, in terms of the bias, and the IPW-based estimator, which we recap in Section A.4 in terms of the variance.
The "double robustness" of thr translates to the following useful property: viar retains the parametric rate of convergence, Or(1/ / /n), even when the estimators jáo and â themselves converge at slower rates. This allows us to use nonparametric function estimators to estimate 140 and л, such as
stacking ensembles (Breiman 1996) like the super learner (van der Laan et al. (007) and regularized estimators like the Lasso (Tbshirani) 1996] [elloni et al. (014). Even for nonparametric models whose rates of convergence are not fully understood, such random forests (Breiman (200T) and deep neural networks (LeCun et al. 2015), we can empirically demonstrate valid coverage and efficiency (Section
In practice, the nuisance functions can be estimated via cross-fitting (Robins et al. 2008 Zheng and van der Laan| [011]] Chernozukoo et al. (2018), which is K ofoold generalization of sample splitting First, randomly split the data into K folds; then, fit á and An on K- 1 folds and use them to estimate the EIF on the remaining "evaluation" fold; repeat the process K times with each fold being the evaluation fold; finally, average the EIFs across all data points. The key benefit of using cross-fitting is to avoid any complexity restrictions individual nuisance functions without sacrificing sample efficiency. In the following, we let eùr be the estimator 3.1 obtained via cross-fitting
Now we are ready to present our first result, which states the asymptotic validity and efficiency of the DR estimator for v under identification and the DR condition.
Theorem 3.1 (DR estimation of the counterfactual score for abstaining classifier). Suppose that Assumptions 2.7 and 2.3 hold. Also, suppose that
|| - Tllza(P) -iã pollza(P) = (2(1/vn)
(3.2)

---

and that ||IF 10--111 -FFI = (1). Then,
vn (Dar -5) ~S N (0, Varp [IF)):
where Vare [IF] matches the nonparametric efficiency bound.
The proof adapts standard arguments in mathematical statistics, as found in, e.g., van der Vaart (2002); Kennedy (2022), to the abstaining classifier evaluation setup. We include proof sketch in Appendix B.3 Theorem tells
tells us that, under the identification and the DR condition (3.2) we can construct closed-form asymptotic confidence interval (CT) at level a E (0, 1) as follows:
Cn.a ( dr + 2a/2 Varg, [IF]
(3.3)
where fa/2 = (1 {) is the (1 -qquantile of a standard normal (e.g.. 1.96 for a 0.05). In Appendix D we describe asymptotic confidence sequence (AsympCS) Waudby-Smiii al. 2021). which is CI that is further valid under continuous monitoring (e.g., as more data is collected).
3.2 Estimating counterfactual score differences
Task Next, return to the problem of comparing two abstaining classifiers, (f^, x²)) and (f x xB), that each makes a decision to make a prediction on each input X₁ or abstain from doing so. That is, R T (X,) ~ Ber(x* (X,)), and observe St (X),,, Y.) if and only if R 0, for E (A, BJ. Recall that the target here is the score difference AB & B - SB
Doubly robust difference estimation If the parameters v.ah and e are each identified according to Proposition 2.4 then
estimate A AB ----jt LB for individual estimates eiA and JB The resulting EIF is simply the difference in the EIF for and B: IFAB (x, ,, 5A s5))) IF r S²) IF (2, , s.), where IF and IFB denote the EIF of the respective classifier. Thus, we arrive at analogous theorem that involves estimating the nuisance functions of each abstaining classifier and utilizing IFAB to obtain the limiting distribution of Á = d &B.
Theorem 3.2 (DR estimation of the counterfactual score difference). Suppose that Assumptions 2. and 2.3 hold for both (Xi, R.A. S.A)) and (X,; RR. SSP) Also, suppose that
||
(2))) Rig
1 (L)))))) + || π (I -onii---llleeeae (illz2()) (2(1/vm)
(3.4)
and that |||EAB IF
(?) = (1) Then,
Vn (A AB N (o, Varr [IFAB]
where [IF] matches the nonparametric efficiency bound.
A proof is given in Appendix B.4 As with evaluation, Theorem 3.2 yields closed-form asymptotic CI of the form (3.3) using the analogous estimate of EIF under MAR, positivity, and DR (3.4). Inverting this CI further yields a hypothesis test for Ho -.a dB Hi if * L
4 Experiments
The main goals of our experiments are to empirically check the validity and power of our estimation methods and to illustrate how methods be used in practice. First, in Section 4.1 present results on simulated data to examine the validity our proposed inference methods (CIs and hypothesis tests). Then, in Section 4.2 study three scenarios the CIFAR- 100 dataset that illustrate the practical use of our approach to real data settings. All code for the experiments is publicly available online at //ttts:://giiiiii
com/yj hwtp://hhooooaaaaggeege.e. Conppaargggbsssaiiiigiggaaaaaaa
4.1 Simulated experiments: Abstentions near the decision boundary
Setup (MAR but not MCAR) We first consider comparing two abstaining classifiers according to their accuracy scores, on simulated binary classification dataset with 2-dimensional inputs.
7

---

Given n = 000 i..dd imputs (X+ ~ (n. 1]²), each label Y, is decided using a linear boundary, f. (2), [2) 1), along with 15% i.i.d. label noise. Importantly. each classifier abstains near its decision boundary, such that its predictions and scores are MAR but not MCAR (because abstentions depend on the inputs). As a result, while the counterfactual score of A (vA = 0.86) is much higher than B (vB = 0.74), their selective scores are more similar (Sel* = 0.86, SelB 0.81) and coverage is lower for A (Cov = 0.55) than for B (Cov = 0.62). Another point to note here is that, even though both the outcome model and classifier A are linear, both the abstention mechanism T and the selective score functior { je are nonlinear functions of the inputs (similarly for T. and (z)). More generally, if classifier abstains near decision boundary, then both x and Ho could be at least as complex as the base classifier f itself. Further details of the setup, including a plot of the data, predictions, and abstentions, are provided in Appendix E.
Miscoverage rates and widths As our first experiment, we compare the miscov- erage rates and widths of the 95% DR CIs (Theorem 3.2 against two baseline esti- mators: the plug-in and the IPW (Rosen- baum 1 estimators (Appendix (A.4). For each method, the miscoverage rate of the CI C, is approximated via PBAAB & C.) R m " m 1 & cla)), where m is the number of simulations over re- peatedly sampled data. If the CI is valid, then this rate should approximately be 0.05. The miscoverage rate and the width of a CI, respectively, capture its bias and variance components. For the nuisance functions, we try linear predictors (L2-regularized lin- ear/logistic regression for ji//f), random forests, and super learners with k-NN, ker- nel SVM, and random forests.
Table 2: Miscoverage rates (and widths) of 95% CIs us- ing three estimation approaches and three nuisance func- tion (I and (4) estimators in a simulated experiment. Mcan and standard error computed over m = 1, 000 runs are shown; those within 2 standard errors of the intended level (0.05) are boldfaced. The sample size is n = 2, 000 in each run. The mean widths of CIs are shown in parentheses. DR estimation with either random forest or a super learner achieves control over the miscoverage rate, and the DR-based CI is twice as tight as the IPs-b CI in terms of their width.
No & Plug-in
IPW
DR
Linear
1.00 + 0.00
0.76 + 0.01
1.00 + 0.00 (0.04)
(0.00)
(0.09)
Random
0.64 + 0.02
0.14 + 0.01
0.05 + 0.01
forest
(0.02)
(0.13)
(0.07)
Super
0,91 = 0.01
0.03 + 0.01
0.05 + 0.01 (0.06)
We present our results in Table 2 First, using the random forest or the super learner,
learner
(0.01)
(0.12)
the DR CIs consistently achieve the intended coverage level of 0.95, over m = 1. 000 repeated simulations (standard error 0.01). This validates the asymptotic normality result of (3.2). Note that the CI with linear estimators does not achieve the intended coverage level: this is expected as neither f nor fúo can consistently estimate the nonlinear functions π or (oo violating the DR condition (3.4).
Second, when considering both the miscoverage rate and CI width, the DR estimator outperforms both the plug-in and IPW estimators. The plug-in estimator, despite having small CI width, has a very high miscoverage rate (0.91 with the super learner), meaning that it is biased even when ficxible nuisance learners are used. On the other hand, the IPW estimator has double the width of the DR estimator 12.12 to 0.06, with the super learner), meaning that it is not as efficient. Also, while the IPW estimator achieves the desired coverage level with the super learner (0.03), it fails with the random forest (0.14), which tends to make overconfident predictions of the abstention pattern and biases the resulting estimate. In contrast, the DR estimator retains its intended coverage level of 0.05 with the random forest, suggesting that it is amenable to overconfident nuisance learners.
Power analysis We conduct a power analysis of the statistical test for Ho AB vs. H A AB of 0 by inverting the DR CI. The results confirm that the power reaches I as either the sample size (n) or the absolute difference ((AAB) increases. This experiment is included in Appendix E.2
"The abstention mechanism (σ)) = P(R = X x) here separates the region below and above the decision boundary from the region near the boundary. Thus π nonlinear even when the boundary is linear. 'Given any input X for which the classifier did not abstain (At = 0) and its output Y. the score S s(f(x), Y) is nonlinear if either s(-, y) or f is nonlinear. Thus, even for linear f. nonlinear scores like the Brier score automatically make the selective score function pa (x) E[S R 0, X = x] nonlinear.
8

---

4.2 Comparing abstaining classifiers on CIFAR-100
To illustrate a real data use case, we compare abstaining classifiers on the CIFAR-100 image classification dataset (Krizhevsky) 2009). Observe that abstaining classifiers can behave differently not just when their base classifiers are different but also when their abstention mechanisms are different. In fact, two abstaining classifiers can have similarly effective base classifier but substantially different abstention mechanisms (c.g,, one more confident than the other). In such a case, the counterfactual score difference between the two classifiers is zero, but their selective scores and coverages are different. We examine such scenarios by comparing image classifiers that use the same pre-trained representation model but have different output layers and abstention mechanisms.
We start with the 512-dimensional final-layer representations of a VGG-16 convolutional neural network (Simonyan and Zisserman 2015), pre-trained on the CIFAR-100 training set, and compare different output layers and abstention mechanisms on the validation set. Generally, in real data setup, cannot verify whether statistical test a CI is correct; yet, in this experiment, we can still access to the base model of each abstaining classifier. This mcans that (a) if we compare abstaining classifiers that share the base classifier but differ in their abstention patterns, then we actually know that their counterfactual scores are exactly the same (AAB with different base classifiers, then i..dd. sampling error. This estimate is denoted by AB := n ("f (X,). Y,) (X)), Yi)].
AB = O); (b) if we compare abstaining classifiers compute their counterfactual scores accurately up to an
For all comparisons, we use the DR estimator, where the nuisance functions म and Ao for both classifiers are each an 1.2-regllarized linear layer learned on top of the pre-trained 1GG-16 features. The use of pre-trained representations for learning the nuisance functions is motivated by [Shi et al. (2019), who demonstrated the effectiveness of the approach in causal inference contexts. We also use the Brier score in this experiment. We defer other experiment details to Appendix E.3
In scenario I, we compare two abstaining classifiers that use the same softmax out- put layer but use a different threshold for abstentions. Specifically, both classifiers use the softmax response (SR) threshold- ing (Geifman and EI- Yaniv, 2017), i.e., ab- stain if max.ey [(X) < T for thresh- old >>> O, but A uses more conservative threshold (r = 0.8) than B (r = 0.5). As result, while their counterfactual scores are identical AB = 0), A has a higher selective score (+0.06) and lower cover- age (-0.20) than B. This is also a deter- ministic abstention mechanism, potentially challenging the premises of our setup. As shown in Table 3. we that the 95% DR CI is 0.005, 0.018) (n- 5, 000), confirming that there is no difference in counterfactual scores
Table 3: The 95% DR CIs and their corresponding hypothesis tests for Hg : AB at significance level a: = 0.05, for three different comparison scenarios on (half of) the CIFAR-100 test set (n 5, 000). The three scenarios compare different abstention mechanisms or predictors, as detailed in text; all comparisons use the Brier score. AB is the empirical counterfactual score difference without any abstentions. The result of each statistical test agrees with whether AB is O.
Scenarios
AB
95% DR CI
Reject Ho?
I 0.000
(-0.005, 0.018)
No
II 0.000
(-0.014, 0.008)
No
III
-0.099
(-0.051, -0.028)
Yes
Scenario II is similar to scenario I, except that the abstention mechanisms are now stochastic: A uses one minus the SR as the probability of making a prediction, i.c., 7 <<xx) = 11 1-maxxe f(X).. ). while
B uses one minus the Gini impurity as the probability of abstention, i.e., (x) = E f(X)² both clipped to (0.2, 0.8). The Gini impurity is an alternative measure of confidence to SR that is quadratic in the probabilities, instead of piecewise linear, and thus the Gini-based abstaining classifier (B) is more cautious than the SR-based one (A), particularly on uncertain predictions. In our setup, achieves higher coverage than B, while B achieves higher selective score than A. The 95% DR Cl (-0.014, 4, 0. 0.008), confirming that there is once again no difference in counterfactual scores. Scenarios and II both correspond to case (a).
In scenario III, we now examine a case where there is a difference in counterfactual scores between the two abstaining classifiers (case (b)). Specificallyy, compare the pre-trained VGG-16 model's
'Reproduced version, accessed from http::: www.w////w/.oo///oco//eoo/ wotpe/ffoptttoooo cifar- models "The reason why the DR CI correctly estimates the true A = 0, despite the fact that positivity is violated in this case, is because the two classifiers happen to abstain on similar examples (B abstains whenever A does) and their scores on their abstentions happen to be relatively similar (0.604 for A; 0.576 for B).

---

output layers (512-512-100) with the single softmax output layer that we considered in earlier scenarios. It turns out that the original model's multi-layer output model achieves a worse Brier score (0.758) than one with single output layer (0.787). likely because the probability predictions of the multi-layer model are too confident (miscalibrated). When using the same abstention mechanism (stochastic abstentions using SR, as in II), the overconfident original model correspondingyy achieves a higher coverage (and worse selective Brier score) than the Lingle-output-layer model. The Monte Carlo estimate of the true counterfactual score difference is given by A -0.029, and the 95% DR CI falls entirely negative with (-0.051, -0.028), rejecting the null of A AB 0 at a = 0.05.
Discussion
This paper lays the groundwork for addressing the challenging problem of counterfactually comparing black-box abstaining classifiers. Our solution casts the problem in the missing data framework, in which we treat abstentions as MAR predictions of the classifier()), and this allows to leverage nonparametrically efficient tools from causal inference. We close with discussions of two topics.
Addressing violations of the identification conditions At the conceptual level, the biggest chal- lenge arises from the positivity condition, which requires the classifiers to deploy a non-deterministic abstention mechanism. As mentioned in Section 2.2 the counterfactual score is unidentifiable without this assumption. We argue that this issue calls for a policy-level treatment, especially in auditing scenarios, where the evaluators may require vendors to supply a classifier that can abstain but has at least an e >> O chance of nonabstention. The level e can be mutually agreed upon by both parties. Such policy achieves a middle ground in which the vendors are not required fully reveal their proprietary classifiers. In Appendix A.5 we discuss this policy-level treatment in greater detail.
An alternative is explore existing techniques that aim to address positivity violations directly (Pe tersen et al. 2012: Ju et al., [2019] Léger et al.) 2022). Due to the unidentifiability result, these tech- niques have their own limitations. For example, we may consider applying sample trimming (Crump it al. 2009) to make valid inferences on a subpopulation, but the conclusions would not hold for the entire input domain (even when relevant). Nevertheless, it remains a meaningful future work adapt modern tools in causal inference for diagnosing and addressing positivity violations (e.g.. Lei et al.) 2021 Kennedy 2019). In the case of addressing MAR violations, we suspect the evaluation data is "compromised" and used in training, we can adapt sensitivity analysis methods that measure how much of the evaluation data is contaminated by the training data (Bonvini and Kennedy 2022).
Connections to learning-to-defer scenarios and cascading classifiers There are variants of abstaining classifiers in ML for which we can utilize the counterfactual score analogously. First, in the learning-to to-defer setting (Madras et al. 2018), where the algorithm is allowed to defer its decision to an expert that gives their own decision, the counterfactual score naturally corresponds to the expected score of the overall system had the classifier not deferred at all. The score is thus an evaluation metric primarily for the classifier, and it independent of the expert's predictions, even when the classifier is adaptive to the expert's s tendencies. In the case where the goal is assess the joint performance of the algorithm and the expert, then a variant of ooodessaa et al. (2017)'s ' score can be useful; in Appendix A.6 we discuss how the DR estimator can be utilized for estimating this variant, even when the expert's s score is random.
Another example would be cascading or multi-stage classifiers (Alpaydin and Kaynak 1998 Viola and Jones 2001). which are classifiers that coort-cireiit easy predictions by using simple models in their early stages to save computation. The counterfactual framework can assess the performance of cascading classifiers by treating each component as an abstaining classifier. For example, a two-stage cascading classifier, equipped with small model fo. a large model fi, and deferral mechanism π, can be viewed as pair of abstaining classifiers with tied abstention mechanisms, specifically (fo, x) and (fi, 7) where π = 1 1- Then, the corresponding counterfactual scores capture the performance of the cascading classifier had it been using only that (small or large) model. We can further extend this to general multi-stage classifiers if we are interested in estimating the score of a classifier in each stage individually. The two-stage setup is also relevant to the learning-to- model is imperfect expert on the task. In such setup, we can devise metric that combines (i) the selective score of the small model and (ii) the relative improvement by switching to the expert. Our approach can also estimate this new metric using essentially the same tools.
setup, where the large
10

---

Acknowledgments and Disclosure of Funding
The authors thank Edward Kennedy and anonymous reviewers for their helpful comments and suggestions. AR acknowledges support from NSF grants IIS-2229881 I and DMS-231011 This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548362. Specifically, it used the Bridges-2 system, which is supported by NSF award number DACI-1928101 147, at the Pittsburgh Supercomputing Center (PSC).
References
Agarwal, S. and Deshpande, A. (2022). On the power of randomization in fair classification and representation. 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1542-1551.
Alpaydin, and Kaynak, C. (1998). Cascading classifiers. Kybernetika, 34(4):369-31
Bang, and Robins, J. M. (2005), Doubly robust estimation missing data and causal inference models. Biometrics, 644:99723
Barocas, S., Hardt, M., and Narayanan, A. (2019). Fairness and machine learning: Limitations and opportumities. fairmlbook.org. attp://wwe www.wwwrwm...mmwwwwwwwwwwww.wwmw www.mmaimooooooooo org
Bartlett, P. L. and Wegkamp, M. H. (2008). Classification with reject option using hinge loss. Journal of Machine Learning Research, 9(8).
Belloni, A., Chernozhukov, V. and Hansen, C. (2014). Inference on treatment effects after selection among high-dimensional controls. The Review of Economic Studies, 81(2):608-633 -500.
Bickel, P. J., Klaassen, C. A., Bickel, P. J., Ritov, Y., Klaassen, J., Wellner, J. A., and Ritov, Y. (1993). Efficient and adaptive estimation for semiparametric models, volume 4. Johns Hopkins University Press Baltimore.
Bonvini, M. and Kennedy, E. H. (2022). Sensitivity analysis via the proportion of unmeasured confounding. Journal of the American Statistical Association, 1540530):1500-1550
Breiman, (1996). Stacked regressions. Machine Learning, 24:49-66
Breiman, L. (2001). Random forests. Machine Learning, 45:5-32.
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3.
Chemozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68.
Chow, C.-K. (1957). An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC-664))247-254.
Chow, C.-K. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions Information Theory, 16(1):41-46.
Condessa, F. Bioucas-Dias, J., and Kovačević, J. (2017). Performance measures for classification systems with rejection. Pattern Recognition, 63:437-450.
Cortes, C.. DeSalvo, G., and Mohri, M. (2016). Boosting with abstention. Advances in Neural Information Processing Systems, 29.
Crump, R. K., Hotz, J., Imbens, G. W. and Mitnik, O. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1):187-199
Ding, P. and Li, F. (2018). Causal inference: A missing data perspective. Statistical Science, 33(2).

---

El- El-Yaniv, R. and Wiener, Y. (2010). On the foundations of noise- free selective classification. Journal of Machine Learning Research, 11(5).
Fisher, A. and Kennedy, E. H. (2021). Visually communicating and teaching intuition for influence functions. The American Statistician, 1522):162-172.
Gangrade, A., Kag, A., and Saligrama, V. (2021). Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statistics, pages 2179-2187.
Geifman, Y. and El-Yaniv, R. (2017). Selective classification for deep neural networks. Advances in Neural Information Processing Systems, 30.
Geifman, Y. and El- Ea-ia,,v, R. (2019). SelectiveNet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference Machine Learning, volume 97, pages 2151-2159, PMLR.
Grgié-Hlačer N., Zafar, M. B., Gummadi, K. P., and Weller, A. (2017). On fairesss, diversity and randomness in algorithmic decision making. arXiv preprint arXiv:/706 100.1008...
Hendrickx, K., Perini, L.,, Van der Plas, D., Meert, W., and Davis, J. (2021). Machine learning with reject option: A survey. arXiv preprint 17X.v2210..11277.
Horvitz, D. G. and Thompson, D. J. (1952). generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47((60)::63366) 685.
Ju, C., Schwab, J., and van der Laan, M. J. (2019). On adaptive propensity score truncation in causal inference. Statistical Methods Medical Research, 17411777
Kalai, A. and Kanade, V. (2021). Towards optimally abstaining from prediction with ood test
examples. Advances in Neural Information Processing Systems, 34:12774-12785.
Kennedy, E. H. (2019). Nonparametric causal effects based on incremental propensity score interven- tions. Journal of the American Statistical Association, 114(526):645-655
Kennedy, E. H. (2022). Semiparametric doubly robust targeted double machine learning: A review. arXiv preprint arXiv:2203.06465
Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical Report, University of Toronto.
LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature, 521(7553)4436-444
Léger, M., Chatton, A., Le Borgne, F., Pirracchio, R., Lasocki, S., and Foucher, Y. (2022). Causal inference in case of near-violation of positivity: Comparison of methods. Biometrical Journal, 6498))138911433.
Lei, L., D' Amour, A., Ding, P., Feller, A., and Sekhon, (2021). Distribution-free assessment of population overlap in observational studies. Technical report, Stanford University.
Little, J. and Rubin, D. B. (2019). Statistical analysis with missing data, volume 793. John Wiley & Sons.
Madras, D., Pitassi, T., and Zemel, R. (2018). Predict responsibly: Improving fairness and accuracy by learning defer. Advances in Neural Information Processing Systems, 31.
Pearl, (2000). Models, reasoning and inference. Cambridge University Press, 19(2).
Petersen, M. L.,, Porter, K. E.,, Gruber, S., Wang, Y., and Van Der Laan, M. J. (2012). Diagnosing and responding to violations the positivity assumption. Statistical Methods in Medical Research, 21(1):31-54.
Pietraszek, T. (2005). Optimizing abstaining classifiers using ROC analysis. In Proceedings of the 22nd International Conference Machine Learning, pages 665-672.
12

---

Robins, J., Li, L.. Tchetgen, E., and van der Vaart, A. (2008). Higher order influence functions and minimax estimation of nonlinear functionals. Probability and Statistics: Essays in Honor of David A. Freedman, 2:335-422 421.
Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, (9447))8846 866.
Rosenbaum, P. R. (1995). Observational studies. Springer.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies, Journal of Educational Psychology, 66(5):688.
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3):581-592
Schreuder, N. and Chzhen, E. (2021). Classification with abstention but without disparities. In Uncertainty in Artificial Intelligence, pages 1227-1236. PMLR.
Shi, C., Blei, D., and Veitch, V. (2019). Adapting neural networks for the estimation of treatment effects. Advances in Neural Information Processing Systems, 32.
Shpitser, I,, Mohan, K., and Pearl, J. (2015). Missing data causal and probabilistic problem. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pages 802-811.
Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series (Methodological), 5811)22772888
Tsiatis, A. A. (2006). Semiparametric theory and missing data. Springer.
van der Laan, M. J., Polley, E. C., and Hubbard, A. E. (2007). Super learner. Statistical Applications in Genetics and Molecular Biology, 6(1).
van der Laan, M. and Robins, J. M. (2003). Unified methods for censored longitudinal data and causality. Springer.
van der Vaart, A. W. (2000), Asymptotic statistics, volume 3. Cambridge University Press.
van der Vaart, A. W. (2002). Semiparametric statisties. In Lectures Probability Theory and Statistics (Saint-Flone 1999), pages 331-457, Springer.
Viola, P. and Jones, M. (2001). Robust real-time object detection. International Journal of Computer Vision, 4(34-47):4.
Waudby-Smith, I., Arbour, D., Sinha, R., Kennedy, E. H., and Ramdas, (2021). Time-unifond central limit theory, asymptotic confidence sequences, and anytime-valid causal inference. arXiv preprint arXIv:2103.06476.
Zhang, X.-Y., Xie, G.-S., Li, X., Mei, T., and Liu, C.-L. (2023). survey on learning to reject. Proceedings of the IEEE, 11 111(2):15-2155
Zheng, W. and van der Laan, M. J. (2011). Cross-validatee targeted minimum- lostnnuum--os--ssseee estimation. In van der Laan, M. J. and Rose, S., editors, Targeted Learning: Causal Inference for Observational and Experimental Data, pages 459-474 Springer.
13


Reviews:
[{"review_id": "t82QdmfaLB", "reviewer": "Reviewer_paQd", "summary": "The paper proposes a new way to evaluate counterfactually, abstain classifiers. By considering the task of selective classification, the authors reformulate the problem into a Missing data problem which can thus be seen as a causal inference (Counterfactual) problem.\nThe authors also state the corresponding assumptions needed for the above to be identifiable and propose a doubly robust estimator for their proposed metric. ", "questions": "see in weakness\nI am more than happy to raise my score if the above are clarified.", "limitations": "Yes the authors have addressed limitations of their work", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper proposes a new view on evaluation of abstain classifiers through the lens of Rubins Counterfactuals\n- The paper clearly states the assumptions for the problem to be identifiable\n- The paper clearly formulates the approach of doubly robust estimations and how it can be used in their setting for abstain classifier comparisons.\n", "weaknesses": "- My biggest concern of the paper is the motivation. As stated in their example 1.1 the example they have given is quite contrived and not really applicable in most cases. I believe much stronger motivating examples would be useful. If the authors would clearly give me better examples (in addition to the appendix ones) that would greatly help me position this paper.\n- Secondly, I believe the experiments are not clearly written out to me. Could the authors tell me how exactly this metric is preferred over simply looking at the accuracy of the selected labels and the inaccuracy of the non-selected labels? I might have misunderstood this part, so if the authors could help me clarify this part it would be very helpful. The authors mention that this is an \"inverse\" problem, but not sure if I understood this part.\n- Lastly, I wonder if the authors have thought of implementing the second point mentioned in this review. I am confused about why there are no other baselines. I would have thought that the second point would constitute a simple baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our work and for acknowledging the novelty and clarity of our exposition. See below for our responses to each of your concerns.\n\n----\n\n> My biggest concern of the paper is the motivation. As stated in their example 1.1 the example they have given is quite contrived and not really applicable in most cases. I believe much stronger motivating examples would be useful. If the authors would clearly give me better examples (in addition to the appendix ones) that would greatly help me position this paper.\n\nPlease see our common response to all reviewers which includes our response to this point. \n\n----\n\n> Secondly, I believe the experiments are not clearly written out to me. Could the authors tell me how exactly this metric is preferred over simply looking at the accuracy of the selected labels and the inaccuracy of the non-selected labels? I might have misunderstood this part, so if the authors could help me clarify this part it would be very helpful. The authors mention that this is an \"inverse\" problem, but not sure if I understood this part.\n\nIn brief, if two classifiers make predictions on different subsets of the data, have different frequencies and patterns of abstaining, and have different accuracies on the ones they do predict, it is actually not so obvious how to combine that information to compare them in a sensible and fair manner.\n\nOur examples (1.1 and A.1-A.3) motivate cases in which it _hurts_ to be inaccurate on abstentions (non-selected labels):\n* If a free-trial classifier from Example 1.1 is highly inaccurate in its abstentions, and we (the evaluator) are impressed by its performance on non-abstentions and purchase the full non-abstaining service, then we will suffer a decrease in the total accuracy from the free trial phase to the paid use phase. \n* If the self-driving car from Ex. A.1 or the hospital from Ex. A.2 attempts to use the abstaining classifiers’ hidden predictions in a failure mode, then it will suffer more when the classifier is less accurate on its abstentions.\nThis is the “inverse” sense from a typical _learning_ scenario for abstaining classifiers: whereas learning objectives for abstaining classifiers often reward abstaining on inaccurate predictions, the counterfactual score is adequate for evaluation setups where such abstentions can be hurtful.\n\nThe metric that you discuss here roughly corresponds to Condessa et al. (2017)’s score, which we formally defined in Appendix A.2. We state there that:\n* The two metrics assess abstaining classifiers from different viewpoints, depending on whether it is good or bad for a classifier to perform poorly on its abstentions. \n* Our estimation approach can be applied to estimate Condessa et al.’s score in the black-box setting, in which their score is also a counterfactual (we do not know how well or poorly the classifier performs on its abstentions). Note that Condessa et al. primarily focused on defining their score in the “white-box” setting, in which the classifier’s accuracy on its abstentions is known, but we do not assume this.\n\nFinally, as for the experiments, given that our primary method is a statistical inference approach (a confidence interval), we focus on empirically examining its validity/coverage, that is, whether a 95% CI correctly covers the true parameter (fully known in simulated experiments) approximately 95% of the time across repeated simulations. The complementary metric to coverage is the power, as measured by how tight the CI is (a tighter CI would give more certainty to the user). Our real data experiment validates how the CI estimates a number that we expect to see in each scenario (zero if the base classifiers were the same; nonzero otherwise). These experiments are NOT meant to provide further justifications on why the counterfactual score is a viable alternative _as a metric_ to existing ones; they are simply meant to show that our method for estimating the counterfactual score is sensible and works well in practice.\n\nIn our revision, we clarified the main goals of our experiments up front (at the beginning of the section).\n\n----\n\n> Lastly, I wonder if the authors have thought of implementing the second point mentioned in this review. I am confused about why there are no other baselines. I would have thought that the second point would constitute a simple baseline.\n\nAs per our previous response, our primary baselines are other _estimation_ approaches, namely the plug-in and IPW estimators, and not other _metrics_. The metrics can be computed straightforwardly, but they do not reveal insights about whether our methods work with simulated and real data, and it is hard to compare different methods that are meant for different metrics. \n\nHaving said that, we already include the most commonly used metric for evaluating abstaining classifiers: a combination of the selective score and coverage. In our simulated experiments, these numbers are shown in the first paragraph of Sec. 4.1. Note that both of these quantities are just sample averages, and a standard CI can be computed for estimation. Another existing metric would be Condessa et al.’s score, which we recap in App. A.2 and in the above. For our simulated setup, this can also be computed straightforwardly: classifier A obtains 0.4145, while classifier B obtains 0.4345 (they both abstain quite a bit, and A is heavily penalized for abstaining on correct predictions).\n\n----\n\nWe sincerely hope that our responses address your concerns."}, {"review_id": "yMBObDdwGk", "reviewer": "Reviewer_fENC", "summary": "The authors focus on the problem of evaluating abstaining classifiers while also taking account their counterfactual predictions had they not abstained on certain data points. As an evaluation metric they consider the expected accuracy of the classifiers over both their actual and counterfactual predictions, which they define as the counterfactual score. They provide the conditions under which the counterfactual score is identifiable and also propose a doubly robust estimator to compute the counterfactual score of an abstaining classifier. Finally, they evaluate their estimator with simulations and experiments on  real data.", "questions": "N/A.", "limitations": "The authors adequately address the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea of considering the counterfactual predictions of the classifier had it not abstained for evaluation is conceptually quite interesting and also could have a significant impact in scenarios when the expert, for example,  who predicts when the classifier abstains is also uncertain and would still choose the classifier’s prediction.  The paper is very nicely written, well structured and organised, well motivated and convincing. The experimental evaluation includes both simulations and real data experiments, the setup and the results are clearly explained and also the code is provided for reproducibility.", "weaknesses": "There see to be in general no major weaknesses. It is a bit confusing the pointer to table 1 in line 86, where the notation has not yet been introduced. It would have been nicer perhaps not to include notation at that point or explain the notation in the caption. Moreover, the captions of the tables should have been above the tables according to the author instructions.\nTypos:\n\n-Line  87: ‘of significant interest’\n\n-Line 367: there seem to be a missing verb\n\n-Line 420: the sentence does not make sense, it seems that some part is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive and comprehensive feedback on our work. \n\nIn our revision, we incorporated all of your editorial notes, including moving Table 1 to the beginning of Section 2 and having all table captions appear before the tables themselves. \n"}, {"review_id": "sDv8eSwKtN", "reviewer": "Reviewer_9p41", "summary": "This paper proposes to compare black-box abstaining classifiers (neither the base classifier nor the abstention mechanism is known to the evaluator) by the counterfactual score, that is the expected score of the abstaining classifier had it not been given the option to abstain. They prove that this quantity is identifiable under two standard conditions, that are missing at random and positivity, and then develop nonparametric and doubly robust methods to estimate it. Experiments on simulated data and CIFAR-100 are conducted.", "questions": "1. In the learning-to-defer setting, where instead of rejecting an example, the example can be deferred to an expert and let that expert make the prediction, does the counterfactual score still make sense?\n\nminor:\n1. There is a redundant 'is' at the end of line 420.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well-organized and clearly written. Comparing abstaining classifiers is an important problem in machine learning and the counterfactual score proposed in this paper is a new evaluation metric that is relevant in practice.", "weaknesses": "1. As stated in the paper, the counterfactual score is unidentifiable if abstentions are deterministic which includes a fair amount of abstention methods in the literature.\n2. It would be better to briefly discuss some related works on abstention instead of just referring to surveys.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our work and for acknowledging the practical importance of our work and the problem. See below for our responses to each of your concerns and questions.  \n\n----\n\n> As stated in the paper, the counterfactual score is unidentifiable if abstentions are deterministic which includes a fair amount of abstention methods in the literature.\n\nWhile we acknowledge the concern from an applicability standpoint, we emphasize that this should not be considered a weakness from a methodology standpoint, given that the score is never identifiable under deterministic abstentions (lines 194–196). As we state in lines 424–428, if we are interested in the counterfactual score of an abstaining classifier, then no other method can estimate it under deterministic abstentions without resorting to restrictive modeling assumptions. This is why we first mention a policy-level approach that may require vendors to meet the positivity requirement in order to enable buyers to compare the products before choosing one (lines 419–423 and Appendix D). \n\nIn our opinion, the fact that our paper does not apply to all abstaining classifiers is perhaps ok, as long as we provide a solution for stochastically abstaining ones and we are upfront about this restriction. We’d also like to reiterate that there are recent papers that show the superiority of stochastically abstaining classifiers (lines 201–210), similar to the effectiveness of randomized classifiers in the fairness literature.  We believe that the counterfactual framework can be more relevant in the future when more abstaining classifiers adopt stochastic mechanisms.\n\n----\n\n> It would be better to briefly discuss some related works on abstention instead of just referring to surveys.\n\nWe have now added a few more references and details, but given that our approach is largely agnostic to the specific learning algorithms for abstaining classifiers, we put our emphasis on the evaluation side. We do explicitly mention directly relevant papers on abstention throughout the paper, including Chow (1970) and El-Yaniv and Wiener (2010) for their discussion of the selective score & coverage formulations, as well as the recent works that develop methods using stochastic abstentions (Kalai and Kanade, 2021; Schreuder and Chzhen, 2021). \n\n----\n\n> In the learning-to-defer setting, where instead of rejecting an example, the example can be deferred to an expert and let that expert make the prediction, does the counterfactual score still make sense?\n\nYes, the counterfactual score makes sense. In the learning-to-defer setting involving an expert, the counterfactual score would refer to the expected score of the overall system had the classifier not deferred at all. The counterfactual score is thus an evaluation metric primarily for the classifier, and it is independent of the expert’s predictions, even when the classifier is adaptive to the expert’s tendencies.\n\nIn the case where the goal is to assess the _joint_ performance of the algorithm and the expert, then it may be useful to estimate a variation of Condessa et al. (2017)’s score, which we summarize in both Section 2 and Appendix A.2. If we denote the expert’s score as $E$, then equation (1) in the Appendix can further be generalized to\n$$\n\\theta^E := \\mathbb{E}[S \\mid R=0] \\mathbb{P}(R=0) + \\mathbb{E}[E - S \\mid R=1] \\mathbb{P}(R=1).\n$$\n\nFor each rejection ($R=1$), we assess the system by the difference in the quality of expert prediction and the model prediction ($E-S$). If the expert is an oracle ($E=1$), then this recovers Condessa et al.’s score. Note that Condessa et al. primarily focused on defining and justifying $\\theta$ in the “white-box” setting, where the score on rejections is known, and it does not discuss the counterfactual estimation problem that arises in the black-box setting.\n\nWhen it comes to _estimation_, as we state in Appendix A.2, our method can analogously estimate Condessa et al.’s score $\\theta$. Thus, the approach can further estimate $\\theta^E$, whenever we can estimate $\\mathbb{E}[E \\mid R=1]$. If we are estimating the performance of a joint system, possibly learned using a learning-to-defer method, then we can estimate $\\theta^E$, as long as either the expert’s performance is known (e.g., $E=1$ for an oracle) or the deferral decision $R$ is recorded. \n"}, {"review_id": "0hhOwPsYsf", "reviewer": "Reviewer_9d5v", "summary": "This paper outlines a method for estimating the counterfactual performance of an abstain classifier on cases where it abstained (what would have happened if it had not abstained?). Using tools from causal inference and the potential outcomes framework in particular, they cast this is a missing data problem, and provide estimators for the relevant quantity, proving identifiability under standard conditions and showing convergence. Experimentally, they use semi-synthetic setups to show that this estimator produces good confidence intervals.", "questions": "- what are some other ways that missing data/causality tools can be adapted to this problem specifically? for instance, are there implications around confidence-based rejection that can be leveraged for estimation methods?\n- in some cases, the underlying prediction may not even be well defined when R = 1 - for instance in cascading models where computation is only proceeded with after the rejection decision is made (if necessary). how do these models fit into the conceptualization of this paper?", "limitations": "n/a", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- This paper is clearly presented and technically sound\n- the notion of connecting abstaining classifiers to missing data problems is interesting and I believe novel\n- the demonstration of the applicability of the missing at random (conditional on X) assumption to this problem is insightful and helpful\n- experiments are clear and I believe the practical effectiveness of this method", "weaknesses": "- Motivation: I'm not convinced that the problem this paper solves is a realistic one - the examples seem fairly contrived to me. I'm not sure why any API would be provided in the form described in 1.1 (with a free abstaining tier), and examples in the appendix all assume an  importance of the \"hidden\" predictions which doesn't seem super realistic. For instance, in A.3 it doesn't seem well motivated to want to evaluate a classifier's fairness based on its hidden predictions, as those do not actually result in impacts or harms on downstream populations. \n- Novelty: I think that the connection of this problem to causality + missing data is novel, but I'm not sure there's much novel insight here. I do think the argument around the missing at random assumption is good (Assumption 2.1) as stated previously, but there isn't much else in this paper which is specific to the abstaining classifier problem.\n\nSmaller notes:\nline 4: \"stake\" -> stakes\nline 50: unclear what \"provably unavoidable\" means\nline 213: is \\mu_0 defined anywhere?\nline 225: the word \"either\" here is confusing - probably want to use \"each\" instead\nline 324: the word \"nuisance\" is used many times in this section - is what it refers to defined?\nline 397: might be worth defining the Gini impurity more clearly and explaining why this is a good metric\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We sincerely hope that our point-by-point responses below address your concerns. \n\n---\n> Motivation\n\nPlease see our common response to all reviewers for this point.\n\n---\n> Novelty\n\nFirst, highlighting and formalizing the connection between abstaining classifier evaluation and causal inference is a central contribution of our work. Given that the connection is novel, it was imperative that we first clearly establish the correspondences across the topics, such as how the MAR condition translates to having independent evaluation data, and how the positivity condition reveals a need for a policy-level approach, before jumping onto new methods.\n\nWe also emphasize that, while the proof techniques are adapted from the causal inference literature, the framework and the results are new to the abstaining classifier literature, and the theorems are also not direct corollaries of any previously stated result in a different setup (e.g., for the ATE). \n\nAside from advocating for a causal point-of-view to the abstaining classifiers literature, our work presents a shift of focus from the _learning_ perspective (how to train), which represents the view of most papers in the literature, to the _(black-box) evaluation_ perspective (how to compare), which represents many users and regulators of the classifiers. The evaluation view reveals that there are cases when we want the hidden predictions to be good nevertheless, as opposed to the hidden predictions being bad in the learning view. In App. A.2, we make this contrast explicit by comparing our score with Condessa et al. (2017)’s score, which is an evaluation metric that rewards having low accuracy on abstentions. We state there that our method can also estimate this score, as it still involves a counterfactual (of the expected score under abstentions). Note that Condessa et al.’s paper still largely takes the learning view, as their approach requires white-box access to the underlying model and assumes that 'good' abstaining classifiers would have bad performance on their abstentions. \n\nYet another insight that we view as novel and important is in our discussion (Sec. 5 & App. D) relating the positivity condition to policy-level approaches in safety-critical contexts, in which abstaining classifiers are popularly used. The concrete suggestion we make about requiring an upper bound on the abstention rate is quite specific to abstaining classifiers.\n\nWe believe that these insights add substantially to the literature on evaluating black-box abstaining classifiers, and we hope that you reconsider your assessment of the novelty of our work.\n\n---\n> What are some other ways that missing data/causality tools can be adapted to this problem specifically?\n\nOne example in our paper is to use methods that handle positivity violations, such as sample trimming. These may yield valid inference under certain deterministic abstentions, at the cost of restricting inference to a subpopulation. Other adaptable tools include diagnostics for positivity violations (Petersen et al., 2012; Lei et al., 2021) and sensitivity analysis for MAR violations (e.g., measuring how much of the evaluation data is \"contaminated\" by training data; see Bonvini & Kennedy, 2020). Our framework opens up these avenues for future work (noted in the revision).\n\n> for instance, are there implications around confidence-based rejection that can be leveraged for estimation methods?\n\nIf a classifier makes confidence-based rejections, then the abstention mechanism $\\pi$ should reflect the confidence model across inputs. This means that our estimated abstention mechanism $\\hat\\pi$ can reveal the confidence regions of the classifier. Of course, this is just a learned binary classifier, and it is a byproduct of our overall estimation procedure. \n\n---\n> In some cases, the underlying prediction may not even be well defined when R = 1 - for instance in cascading models where computation is only proceeded with after the rejection decision is made (if necessary). how do these models fit into the conceptualization of this paper?\n\nThe broad answer here would be that the counterfactual score can always be defined whenever a supposed prediction on the classifier’s abstentions/rejections can be meaningful to the evaluator. If there are no supposed predictions on the rejections, then the counterfactual score would not make much sense as a metric.\n\nFor cascading models specifically, the counterfactual framework can assess the performance of cascading classifiers in a unique way. Imagine a two-stage cascading classifier that, in its first stage, uses a small model $f_0$ and then determines whether it is necessary to go through the expensive computation of the larger model $f_1$. Let $\\pi$ be the mechanism with which the model defers its predictions. The counterfactual score of the pair $(f_0, \\pi)$ can be defined straightforwardly as the expected score had only the small model been used, and it can be estimated as long as we know when the large model was invoked. For the large model, we can assess the counterfactual score of the pair $(f_1, \\bar\\pi)$, where $\\bar\\pi = 1 - \\pi$, and this would correspond to the expected score had only the large model been used. These ideas can be extended to 3+ stage cascading classifiers if we are interested in estimating the score of a classifier in each stage individually.\n\nWe note that the two-stage setup is relevant to the learning-to-defer setup, which Reviewer 9p41 pointed out, where the large model is an (imperfect) \"expert\" on the task. In such a setup, we can devise a target that combines (i) the selective score of the small model and (ii) the relative improvement by switching to the expert. As we point out in our response to R#9p41, our approach can also estimate this new target using essentially the same tools.\n\n---\n> Smaller notes\n\nWe incorporated many of these in our revised draft. We defined $\\mu_0$ in line 212 and nuisance functions in line 245."}]

Author rebuttal:
We appreciate all of our reviewers for their feedback on our work. We are particularly thankful to them for acknowledging the problem’s significance as well as the novelty of the connections we elucidate between abstaining classifiers, black-box evaluation, and causal inference. 

A main concern shared by Reviewers paQd and 9d5v is about the practicality of our motivating examples, so we give a common response here. Given the fairly recent advances in learning algorithms for abstaining classifiers, we find it imperative to lay out the tools with which we can evaluate and compare the resulting models in imaginable settings. We argue in the paper that existing metrics, such as Chow’s score and other combinations of selective score and coverage, are not adequate in many such scenarios. Thus, we take the first step in a different type of formal evaluation of black-box abstaining classifiers, by taking a counterfactual point of view. 

When it comes to practicality, we believe that not all works need to be directly motivated by an existing practical problem. One can foreshadow conceptual problems and work to address them in advance of them showing up in practice. As abstaining classifiers become more common, so will the desire to compare them, and especially ask counterfactual questions about their performance had they not abstained.

We specifically elaborate on two of our examples, in response to Reviewer 9d5v.
* Ex. 1.1: Closed-source ML APIs, like those offered by Azure, AWS, Google, and many startups, are becoming ever more popular, and the predictions they provide are clearly important assets of theirs. It is not difficult to imagine that these services will deem certain predictions as more valuable than others (think of classifying CT scans). This can incentivize them to allow users to try a free version of the software that abstains, such that potential buyers can get some sense of the quality of predictions but yet avoid giving away all valuable predictions.
* Ex. A.3: This is one example in which the hidden predictions are not directly used, but it rather illustrates a different use case in which the hidden predictions may explain the inner workings of an otherwise black-box prediction model. For example, the auditor may notice that certain demographic groups receive more abstentions by the model than others. In that case, the counterfactual score may reveal the extent to which the predictions on those groups are bad. If they are actually much worse, then better training data or fairer classifiers may be needed; if not, then only a fairer abstaining mechanism may be needed.

In response to Reviewer paQd: we think that the clarifications above may help address the motivation aspect more clearly than coming up with additional examples. Nevertheless, we are happy to discuss further during the discussion period about these and other examples. 

Aside from the motivation, the reviewers also raised interesting questions about our paper, including connections to the learning-to-defer settings and cascading models. Please see our point-by-point responses to each reviewer’s comments below. 


Meta-review: The authors introduce a method based on counterfactual inference to estimate the (counterfactual) performance of "abstaining" classifiers on sample where it has abstained. A majority of the reviewers are positive about the paper and think it should be accepted.
Decision: Accept (poster)