SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yo Joong Choe*
Data Science Institute
yjchoe@uchicago.edu
Aditya Gangrade
aditg@umich.edu
Dept. of Statistics and Data Science
aramdas@cmu.edu
Abstaining classifiers have the option to abstain from making predictions on inputs that they are unsure about. These classifiers are becoming increasingly popular in high-stakes decision-making problems, as they can withhold uncertain predictions to improve their reliability and safety. When evaluating black-box abstaining classifier(s), however, we lack a principled approach that accounts for what the classifier would have predicted on its abstentions. These missing predictions matter when they can eventually be utilized, either directly or as a backup option in a failure mode. In this paper, we introduce a novel approach and perspective to the problem of evaluating and comparing abstaining classifiers by treating abstentions as missing data. Our evaluation approach is centered around defining the counterfactual score of an abstaining classifier, defined as the expected performance of the classifier had it not been allowed to abstain. We specify the conditions under which the counterfactual score is identifiable: if the abstentions are stochastic, and if the evaluation data is independent of the training data (ensuring that the predictions are missing at random), then the score is identifiable. Note that, if abstentions are deterministic, then the score is unidentifiable because the classifier can perform arbitrarily poorly on its abstentions. Leveraging tools from observational causal inference, we then develop nonparametric and doubly robust methods to efficiently estimate this quantity under identification. Our approach is examined in both simulated and real data experiments.
Abstaining classifiers (Chow, 1957; El-Yaniv and Wiener, 2010), also known as selective classifiers or classifiers with a reject option, are classifiers that have the option to abstain from making predictions on certain inputs. As their use continues to grow in safety-critical applications, such as medical imaging and autonomous driving, it is natural to ask how a practitioner should evaluate and compare the predictive performance of abstaining classifiers under black-box access to their decisions.
In this paper, we introduce the counterfactual score as a new evaluation metric for black-box abstaining classifiers. The counterfactual score is defined as the expected score of an abstaining classifier's predictions, had it not been allowed to abstain. This score is of intrinsic importance when the potential predictions on abstaining inputs are relevant. We proceed with an illustrative example:
Example 1.1 (Free-trial ML APIs). Suppose we compare different image classification APIs. Each API has two versions: a free version that abstains, and a paid one that does not. Before paying for the full service, the user can query the free version for up to  $n$  predictions on a user-provided dataset, although it may choose to reject any input that it deems as requiring the paid service. Given two such APIs, how can the practitioner determine which of the two paid (non-abstaining) versions would be better on the population data source, given their abstaining predictions on a sample?
*This work was submitted while this author was at Carnegie Mellon University.
Example 1.1 exhibits why someone using a black-box abstaining classifier would be interested in its counterfactual score: the user may want to reward classifiers whose hidden predictions are also (reasonably) good, as those predictions may be utilized in the future. This is a hypothetical example, but we can imagine other applications of abstaining classifiers where the counterfactual score is meaningful. In Appendix A.1, we give three additional examples, including safety-critical applications where the hidden predictions may be used as a backup option in a failure mode.
To formally define, identify, and estimate the counterfactual score, we cast the evaluation problem in Rubin (1976)'s missing data framework and treat abstentions as missing predictions. This novel viewpoint directly yields nonparametric methods for estimating the counterfactual score of an abstaining classifier, drawing upon methods for causal inference in observational studies (Rubin, 1974; Robins et al., 1994; van der Vaart, 2000), and represents an interesting yet previously unutilized theoretical connection between selective classification, model evaluation, and causal inference.
The identification of the counterfactual score is guaranteed under two standard assumptions: the missing at random (MAR) condition, which is satisfied as long as the evaluation data is independent of the classifier (or its training data), and the positivity condition. As with standard missing data settings, both the MAR and positivity conditions are necessary for identification. We later discuss each condition in detail, including when the positivity condition is met and how a policy-level approach may be necessary for safety-critical applications.
The counterfactual score can be viewed as an alternative to the selective score (mean score on nonabstentions) and the coverage (1 minus the abstention rate) (El-Yaniv and Wiener, 2010), which are the main existing metrics for evaluating black-box abstaining classifiers. As a two-dimensional metric, comparison on the basis of these is non-trivial. A common approach is to assume a fixed cost for each abstention (Chow, 1970), but this is not always satisfactory since determining how to weigh abstentions and errors against one another is a nontrivial question. Thus, in settings such as Example 1.1, the notion of counterfactual score becomes necessary. Importantly, selective scores are not representative of the counterfactual performance, except in the (unrealistic) case wherein predictions are missing completely at random (MCAR).<sup>2</sup> Figure 1 gives an overview of scenarios where different metrics may be appropriate to compare abstaining classifiers.
Figure 1: A schematic flowchart of comparing abstaining classifiers. In a black-box setting where the evaluator does not have access to the training algorithms or the resources to train them, the task can be viewed as a nontrivial missing data problem. This work proposes the counterfactual score as an evaluation metric.
The counterfactual score also offers practical benefits when comparing abstaining classifiers. Counterfactual scores are comparable even if abstaining classifiers are tuned to different abstention rates or selective scores. Moreover, compared to evaluation methods using the selective score-coverage curve (equivalent to re-training the classifier several times at different score/coverage levels), estimating the counterfactual score does not require re-training the classifier. Instead, we only need to estimate a pair of nuisance functions that can be learned using the observed predictions (nonabstentions) in the evaluation set. Let us further note that the setup is applicable generally to any form of prediction that can be scored, including regression and structured prediction. In this paper, we restrict our attention to classification for concreteness, as it is the most well-studied abstention framework.
Summary of contributions We first formalize the problem of comparing abstaining classifiers as a missing data problem and introduce the counterfactual score as a novel metric for abstaining classifiers. Next, we discuss how the counterfactual score can be identified under the MAR and positivity conditions. Then, we develop efficient nonparametric estimators for the counterfactual scores and their differences, namely doubly robust confidence intervals (DR CI). Finally, we validate our approach in simulated and real-data experiments.
$^2$ MCAR means the missing observations are simply a uniformly random subset of all observations, independently of the input/output. In contrast, MAR means there can be systematic differences between the missing and observed values, but these can be explained by the input. Our method only requires MAR.
Related work on abstaining classifiers The training of abstaining classifiers has seen significant interest in the literature (e.g., Chow, 1970; Pietraszek, 2005; Bartlett and Wegkamp, 2008; Cortes et al., 2016; Geifman and El-Yaniv, 2017, 2019; Gangrade et al., 2021). We refer to Hendrickx et al. (2021); Zhang et al. (2023) for recent surveys. For evaluation, aside from using some combination of selective score and coverage, the most pertinent reference is the work of Condessa et al. (2017), who propose the metric of 'classifier quality' that is a somewhat inverse version of our counterfactual accuracy. This metric is the sum of the prediction accuracy when the classifier predicts, and prediction inaccuracy when it abstains, the idea being that if a classifier is not abstaining needlessly, then it must hold that the underlying predictions on points it abstains on are very poor. While this view is relevant to the training of abstention rules, it is at odds with black-box settings where the underlying predictions may still be executed even when the method abstains, motivating the counterfactual score. In terms of estimating the score, Condessa et al. (2017) do not discuss the black-box setting that requires estimating a counterfactual, as they assume full knowledge of the classifier's predictions.
Related work on missing data, causal inference, and doubly robust estimation Our main approach to the estimability of counterfactual scores is driven by a reduction to an inference problem under missing data (or censoring) (Rubin, 1976; Little and Rubin, 2019). A missing data problem can be viewed equivalently as a causal inference problem in an observational study (Rubin, 1976; Pearl, 2000; Shpitser et al., 2015; Ding and Li, 2018), and there exist well-established theories and methods for both identifying the counterfactual quantity of interest and efficiently estimating the identified target functional. For identification, unlike in some observational settings for estimating treatment effects, our setup straightforwardly satisfies the standard assumption of consistency (non-interference). The MAR assumption is satisfied as long as independent evaluation data is used, and the positivity assumption translates to abstentions being stochastic. We discuss various implications of these conditions in Sections 2.2 and 5. For estimation, efficient methods for estimating targets such as the average treatment effect (ATE) have long been studied under semiparametric and nonparametric settings. Doubly robust (DR) estimators, in particular, are known to achieve the asymptotic minimax lower bound on the mean squared error. For details, we refer the reader to Bickel et al. (1993); Robins et al. (1994); van der Vaart (2000, 2002); van der Laan and Robins (2003); Bang and Robins (2005); Tsiatis (2006); Chernozhukov et al. (2018); Kennedy (2022). Unlike the standard ATE estimation setup in observational studies, our setup contrasts two separate causal estimands, the counterfactual scores of the two competing classifiers, that operate under their distinct missingness mechanisms.
2 Definition and identification of the counterfactual score
We formulate the problem of evaluating and comparing abstaining classifiers under the missing data framework (Rubin, 1976). We follow the standard approach of defining the target parameter (Section 2.1), identifying it with observable quantities (Section 2.2), and estimating the identified parameter using data (Section 3). In each step, we first consider evaluating one abstaining classifier and then extend to comparing two abstaining classifiers. Below,  $\mathcal{X}$  denotes the input space and  $\mathcal{Y} = \{1,\dots ,C\}$  is the set of possible classes, while  $\Delta^{C - 1}$  denotes the  $C$  -dimensional probability simplex on  $\mathcal{Y}$ .
Abstaining classifiers We define an abstaining classifier as a pair of functions  $(f, \pi)$ , representing its base classifier  $f: \mathcal{X} \to \Delta^{C-1}$  and abstention mechanism  $\pi: \mathcal{X} \to [0, 1]$ , respectively. Given a query  $X$ , the classifier first forms a preliminary (probabilistic) prediction  $f(X)$ . Then, potentially using the output  $f(X)$ , the classifier determines  $\pi(X)$ , i.e., the abstention probability. Using  $\pi(X)$ , the classifier then makes the binary abstention decision  $R \mid \pi(X) \sim \operatorname{Ber}(\pi(X))$ , so that if  $R = 1$  ("rejection"), the classifier abstains on the query, and if  $R = 0$ , it reveals its prediction  $f(X)$ . In some cases, we will explicitly define the source of randomness  $\xi$  (independent of the data) in deciding  $R$ , such that  $R = r(\pi(X), \xi)$  for a deterministic function  $r$ . Neither  $f$  nor  $\pi$  is assumed to be known to the evaluator, modeling the black-box access typically available to practitioners.
Scoring rules (Higher scores are better.) We measure the quality of a prediction  $f(x)$  for a label  $y$  via a positively oriented scoring rule  $\mathsf{s} : \Delta^{C-1} \times \mathcal{Y} \to \mathbb{R}$ . One simple scoring rule is classification accuracy, i.e.,  $\mathsf{s}(f(x), y) = 1$  ( $\operatorname{argmax}_{c \in \mathcal{Y}} f(x)_c = y$ ), but a plethora of scores exist in the literature, such as the Brier (1950) score:  $\mathsf{s}(f(x), y) = 1 - \sum_{c \in \mathcal{Y}} (f(x)_c - 1(y = c))^2$ .
${}^{3}$  Specifically,let  $\xi  \sim$  Unif  $\left\lbrack  {0,1}\right\rbrack$  and  $R = \mathbf{1}\left( {\xi  \leq  \pi \left( X\right) }\right)$  . Then,  $R$  is a function of only  $\pi \left( X\right)$  and  $\xi$  .
The evaluation setup For each labeled data point  $(X,Y)$  in an evaluation set, we observe the abstention decision  $R = \mathsf{r}(\pi (X),\xi)$  for some independent source of randomness  $\xi$  used by the abstaining classifier. Then, its prediction  $f(X)$  is observed by the evaluator if and only if  $R = 0$ . Let  $S\coloneqq \mathsf{s}(f(X),Y)$  denote the score of the prediction  $f$  on the query  $X$ , irrespective of  $R$ . Because  $S$  is not observable when  $R = 1$ , we refer to  $S$  as the potential score that would have been seen had the classifier not abstained. (See Appendix A.2 for equivalent formulations that explicitly invoke Rubin (1974)'s potential outcomes model.)
Since our evaluation is based only on the score  $S$ , we can suppress the role of  $Y$  and assume that  $S$  is observed directly when  $R = 0$ . Similarly, we can suppress the role of  $\xi$ , which is independent of the data. We let  $\mathbb{P}$  denote the law of  $Z := (X, R, S)$ .
Table 1 summarizes the problem formulations and the proposed approaches in our paper. The target definitions and the identification conditions are discussed in the following subsections; the estimator and its optimality are discussed in Section 3.
Table 1: A summary of problem formulations and proposed approaches for evaluation and comparison of abstaining classifiers. Our approaches avoid parametric assumptions and allow for black-box classifiers.
Evaluation	Comparison
Classifier(s)	(f,π)	(fA,πA) & (fB,πB)
Target	ψ = E [S]	ΔAB = E [SA - SB]
Identification	MAR & positivity
Estimation	Doubly robust CI
Optimality	Nonparametrically efficient
2.1 The counterfactual score
We propose to assess an abstaining classifier  $(f, \pi)$  with its (expected) counterfactual score:
$$
\psi := \mathbb {E} [ S ], \tag {2.1}
$$
where the expectation is taken w.r.t.  $\mathbb{P}$ . In words,  $\psi$  refers to the expected score of the abstaining classifier had it not been given the option to abstain. The counterfactual score captures the performance of an abstaining classifier via the score of its base classifier, making it suitable for cases where the evaluator is interested in the predictions without using an abstention mechanism.
Note that  $\psi$  does not in general equal the selective score, i.e.,  $\mathbb{E}[S\mid R = 0]$ . For example, when a classifier abstains from making predictions on its "weak points," i.e., inputs on which the classifier performs poorly, the counterfactual score will be lower than the selective score. Also see Appendix A.3 for a direct comparison with Condessa et al. (2017)'s score, which rewards having bad hidden predictions, as opposed to (2.1). Our general framework allows estimating either score, although we focus on (2.1) based on our motivating examples (Example 1.1 and Appendix A.1).
Comparison Counterfactual scores may also be used to compare two abstaining classifiers,  $(f^{\mathsf{A}},\pi^{\mathsf{A}})$  and  $(f^{\mathsf{B}},\pi^{\mathsf{B}})$ , in the form of their counterfactual score difference:  $\Delta^{\mathsf{AB}}\coloneqq \psi^{\mathsf{A}} - \psi^{\mathsf{B}} = \mathbb{E}[S^{\mathsf{A}} - S^{\mathsf{B}}]$ . Here, the expectation is now taken over the joint law of  $Z^{\mathsf{AB}}\coloneqq (X,R^{\mathsf{A}},S^{\mathsf{A}},R^{\mathsf{B}},S^{\mathsf{B}})$ .
2.2 Identifiability of the counterfactual score
Having defined the target parameters  $\psi$  and  $\Delta^{\mathrm{AB}}$ , we now discuss the assumptions under which these quantities become identifiable using only the observed random variables. In other words, these assumptions establish when the counterfactual quantity equals a statistical quantity. As in standard settings of counterfactual inference under missing data, the identifiability of counterfactual scores in this setting depends on two standard conditions: (i) the missing at random condition and (ii) positivity.
The missing at random (MAR) condition, also known as the ignorability or no unmeasured confounding condition, requires that the score  $S$  is conditionally independent of the abstention decision  $R$  given  $X$ , meaning that there are no unobserved confounders  $U$  that affect both the abstention decision  $R$  as well as the score  $S$ . Note that  $S$  is the potential score of what the classifier would get had it not abstained — it is only observed when  $R = 0$ . We formally state the MAR condition as follows:
Assumption 2.1 (Scores are missing at random).  $S \perp R \mid X$ .
In standard ML evaluation scenarios, where the evaluation set is independent of the training set for the classifier, Assumption 2.1 is always met. We formalize this sufficient condition for MAR in the following proposition. Let  $\mathcal{D}_{\mathrm{train}}$  denote the collection of any training data used to learn the abstaining classifier  $(f,\pi)$  and, as before,  $(X,Y)$  denote an (i.i.d.) data point in the evaluation set.
Proposition 2.2 (Independent evaluation data ensures MAR). If  $(X,Y)\perp \mathcal{D}_{\mathrm{train}}$  , then  $S\perp R\mid X$
This result is intuitive: given an independent test input  $X$ , the score  $S = \mathsf{s}(f(X), Y)$  is a deterministic function of the test label  $Y$ , and the abstention decision  $R$  of a classifier cannot depend on  $Y$  simply because the classifier has no access to it. A short proof is given in Appendix B.1 for completeness. In Appendix C, we also include causal graphs that visually illustrate how the MAR condition is met.
If the evaluation data is not independent of  $\mathcal{D}_{\mathrm{train}}$ , then the classifier already has information about the data on which it is tested, so generally speaking, no evaluation score will not accurately reflect its generalization performance. Although the independence between the training and evaluation data is expected in standard ML applications, it may not be guaranteed when, e.g., using a publicly available dataset that is used during the training of the classifier. These issues can be prevented by ensuring that the evaluation set is held out (e.g., a hospital can use its own patient data to evaluate APIs).
The second condition, the positivity condition, is more substantial in our setting:
Assumption 2.3 (Positivity). There exists  $\epsilon >0$  such that  $\pi (X) = \mathbb{P}\left(R = 1\mid X\right)\leq 1 - \epsilon$
Assumption 2.3 says that, for each input  $X$ , there has to be at least a small probability that the classifier will not abstain ( $R = 0$ ). Indeed, if the classifier deterministically abstains from making predictions on a specific input that has nonzero marginal density, then we have no hope of estimating an expectation over all possible values that  $X$  can take. When it comes to evaluating abstaining classifiers on safety-critical applications, we argue that this condition may need to be enforced at a policy level — we elaborate on this point in Section 5 and Appendix A.5. In practice, the exact value of  $\epsilon$  is problem-dependent, and in Appendix E.4, we include additional experiments illustrating how our methods retain validity as long as the abstention rate is capped at  $1 - \epsilon$  for some  $\epsilon > 0$ .
Another justification for the positivity condition is that stochastically abstaining classifiers can achieve better performances than their deterministic counterparts. Kalai and Kanade (2021) illustrate how stochastic abstentions can improve the out-of-distribution (OOD) performance w.r.t. the Chow (1970) score (i.e., selective score  $+\alpha$  coverage). Schreuder and Chzhen (2021) also introduce randomness in their abstaining classifiers, which leverage abstentions as a means to improve their accuracy while satisfying a fairness constraint. The role of random abstentions in these examples mirrors the role of randomization in the fairness literature (Barocas et al., 2019), where the optimal randomized fair predictors are known to outperform their deterministic counterparts (Agarwal and Deshpande, 2022; Grgic-Hlača et al., 2017). Given the effectiveness of randomized classifiers for fairness, it would not be surprising if a fair abstaining classifier was randomized (in its decisions and abstentions).
With MAR and positivity in hand, we can show that the counterfactual score is indeed identifiable. Define  $\mu_0(x) \coloneqq \mathbb{E}\left[S \mid R = 0, X = x\right]$  as the regression function for the score under  $R = 0$ .
Proposition 2.4 (Identification). Under Assumptions 2.1 and 2.3,  $\psi$  is identified as  $\mathbb{E}\left[\mu_0(X)\right]$
The proof, included in Appendix B.2, follows a standard argument in causal inference. The identification of the target parameter  $\psi$  using  $\mu_0$  implies that we can estimate  $\psi$ , the expectation of a potential outcome, using only the observed inputs and scores. Specifically, the task of estimating  $\psi$  consistently reduces to the problem of estimating the regression function  $\mu_0$ , which only involves predictions that the classifier did not abstain from making. We note that, as in standard causal inference, the task of identification, which concerns what to estimate, is largely orthogonal to the task of estimation, which concerns how to estimate the quantity. We discuss the latter problem in Section 3.
Comparison For the comparison task, given  $\Delta^{\mathsf{AB}} = \psi^{\mathsf{A}} - \psi^{\mathsf{B}}$ , it immediately follows that if the MAR and positivity assumptions hold for each of  $(X,R^{\mathsf{A}},S^{\mathsf{A}})$  and  $(X,R^{\mathsf{B}},S^{\mathsf{B}})$ , then  $\Delta^{\mathsf{AB}}$  is also identified as  $\Delta^{\mathsf{AB}} = \mathbb{E}[\mu_0^{\mathsf{A}}(X) - \mu_0^{\mathsf{B}}(X)]$ , where  $\mu_0^\bullet (x)\coloneqq \mathbb{E}\left[S^\bullet |R^\bullet = 0,X = x\right]$  for  $\bullet \in \{\mathsf{A},\mathsf{B}\}$ . In words, if (i) the evaluation data is independent of the training data for each classifier (Proposition 2.2) and (ii) each classifier has at least a small chance of not abstaining on each input (Assumption 2.3), then the counterfactual score difference, i.e.,  $\Delta^{\mathsf{AB}}$ , is identified as the expected difference in the expected scores over inputs conditional on non-abstentions, i.e.,  $\mathbb{E}[\mu_0^{\mathsf{A}}(X) - \mu_0^{\mathsf{B}}(X)]$ .
3 Nonparametric and doubly robust estimation of the counterfactual score
Having identified the counterfactual scores, we now focus on the problem of consistently estimating them. We estimate these quantities without resorting to parametric assumptions about the underlying