SlowGuess/NeurIPS_2023_OCR huggingface dataset:
black-box abstention mechanisms. Instead, we reduce the problem to that of functional estimation and leverage techniques from nonparametric statistics. See Kennedy (2022) for a recent review.
3.1 Estimating the counterfactual score
Task Let  $\{(X_i,R_i,S_i)\}_{i = 1}^n\sim \mathbb{P}$  denote an i.i.d. evaluation set of size  $n$  . As before, we assume that we are given access to the censored version of this sample, i.e., that we observe  $S_{i}$  if and only if  $R_{i} = 0$  . Using the observables, we seek to form an estimate  $\hat{\psi}$  of the counterfactual score  $\psi = \mathbb{E}[S]$
Doubly robust estimation Under identification (Proposition 2.4), we can estimate  $\psi$  by estimating the regression function  $\mu_0(X)$  on the data  $\{(X_i, S_i): R_i = 0\}$ . However, the naïve "plug-in" estimate suffers from an inflated bias due to the structure present in the abstention patterns. (See Appendix A.4 for details.) We instead develop a doubly robust (DR) estimator (Robins et al., 1994; Bang and Robins, 2005), which is known to consistently estimate  $\psi$  at the optimal nonparametric efficiency rates, meaning that no other estimator based on the  $n$  observations can asymptotically achieve a smaller mean squared error (van der Vaart, 2002). The derivation below is relatively standard, explaining our brevity.
Formally, the DR estimator is defined using the (uncentered) efficient influence function (EIF) for the identified target functional  $\psi(\mathbb{P}) = \mathbb{E}_{\mathbb{P}}[\mu_0(X)]$ :  $\mathsf{IF}(x,r,s)\coloneqq \mu_0(x) + \frac{1 - r}{1 - \pi(x)} (s - \mu_0(x))$ $(0 / 0\coloneqq 0)$ . Here,  $\pi$  and  $\mu_0$  are the "nuisance" functions, representing the abstention mechanism and the score function under  $R = 0$ , respectively. The EIF can be computed as long as  $s$  is available when  $r = 0$ . An intuition for the EIF is that it is the first-order "distributional Taylor approximation" (Fisher and Kennedy, 2021) of the target functional, such that its bias is second-order.
Given that  $\pi$  and  $\mu_0$  are unknown, we define an estimate of the EIF, denoted as  $\hat{\mathsf{IF}}$ , by plugging in estimates  $\hat{\pi}$  for  $\pi$  and  $\hat{\mu}_0$  for  $\mu_0$ . Then, the DR estimator is simply the empirical mean of the EIF:
$$
\hat {\psi} _ {\mathrm {d r}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {I F} \left(X _ {i}, R _ {i}, S _ {i}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \hat {\mu} _ {0} \left(X _ {i}\right) + \frac {1 - R _ {i}}{1 - \hat {\pi} \left(X _ {i}\right)} \left(S _ {i} - \hat {\mu} _ {0} \left(X _ {i}\right)\right) \right]. \tag {3.1}
$$
This estimator is well-defined because  $S_{i}$  is available precisely when  $R_{i} = 0$ . Note that the first term is the (biased) plug-in estimator, and the second term represents the first-order correction term, which involves inverse probability weighting (IPW) (Horvitz and Thompson, 1952; Rosenbaum, 1995). In our experiments, we show how the DR estimator improves upon both the plug-in estimator, in terms of the bias, and the IPW-based estimator, which we recap in Section A.4, in terms of the variance.
The "double robustness" of  $\hat{\psi}_{\mathrm{dr}}$  translates to the following useful property:  $\hat{\psi}_{\mathrm{dr}}$  retains the parametric rate of convergence,  $O_{\mathbb{P}}(1 / \sqrt{n})$ , even when the estimators  $\hat{\mu}_0$  and  $\hat{\pi}$  themselves converge at slower rates. This allows us to use nonparametric function estimators to estimate  $\mu_0$  and  $\pi$ , such as stacking ensembles (Breiman, 1996) like the super learner (van der Laan et al., 2007) and regularized estimators like the Lasso (Tibshirani, 1996; Belloni et al., 2014). Even for nonparametric models whose rates of convergence are not fully understood, such as random forests (Breiman, 2001) and deep neural networks (LeCun et al., 2015), we can empirically demonstrate valid coverage and efficiency (Section 4).
In practice, the nuisance functions can be estimated via cross-fitting (Robins et al., 2008; Zheng and van der Laan, 2011; Chernozhukov et al., 2018), which is a  $K$ -fold generalization of sample splitting. First, randomly split the data into  $K$  folds; then, fit  $\hat{\pi}$  and  $\hat{\mu}_0$  on  $K - 1$  folds and use them to estimate the EIF on the remaining "evaluation" fold; repeat the process  $K$  times with each fold being the evaluation fold; finally, average the EIFs across all data points. The key benefit of using cross-fitting is to avoid any complexity restrictions on individual nuisance functions without sacrificing sample efficiency. In the following, we let  $\hat{\psi}_{\mathrm{dr}}$  be the estimator (3.1) obtained via cross-fitting.
Now we are ready to present our first result, which states the asymptotic validity and efficiency of the DR estimator for  $\psi$  under identification and the DR condition.
Theorem 3.1 (DR estimation of the counterfactual score for an abstaining classifier). Suppose that Assumptions 2.1 and 2.3 hold. Also, suppose that
$$
\left\| \hat {\pi} - \pi \right\| _ {L _ {2} (\mathbb {P})} \left\| \hat {\mu} _ {0} - \mu_ {0} \right\| _ {L _ {2} (\mathbb {P})} = o _ {\mathbb {P}} (1 / \sqrt {n}) \tag {3.2}
$$
and that  $\| \hat{\mathsf{IF}} -\mathsf{IF}\|_{L_2(\mathbb{P})} = o_{\mathbb{P}}(1)$ . Then,
$$
\sqrt {n} \left(\hat {\psi} _ {\mathrm {d r}} - \psi\right)\rightsquigarrow \mathcal {N} \left(0, \operatorname {V a r} _ {\mathbb {P}} [ \mathrm {I F} ]\right),
$$
where  $\mathsf{Var}_{\mathbb{P}}[\mathsf{IF}]$  matches the nonparametric efficiency bound.
The proof adapts standard arguments in mathematical statistics, as found in, e.g., van der Vaart (2002); Kennedy (2022), to the abstaining classifier evaluation setup. We include a proof sketch in Appendix B.3. Theorem 3.1 tells us that, under the identification and the DR condition (3.2), we can construct a closed-form asymptotic confidence interval (CI) at level  $\alpha \in (0,1)$  as follows:
$$
C _ {n, \alpha} = \left(\hat {\psi} _ {\mathrm {d r}} \pm z _ {\alpha / 2} \sqrt {n ^ {- 1} \mathsf {V a r} _ {\widehat {\mathbb {P}} _ {n}} [ \mathsf {I F} ]}\right), \tag {3.3}
$$
where  $z_{\alpha /2} = \Phi (1 - \frac{\alpha}{2})$  is the  $(1 - \frac{\alpha}{2})$  -quantile of a standard normal (e.g., 1.96 for  $\alpha = 0.05$ ). In Appendix D, we describe an asymptotic confidence sequence (AsympCS; Waudby-Smith et al., 2021), which is a CI that is further valid under continuous monitoring (e.g., as more data is collected).
3.2 Estimating counterfactual score differences
Task Next, we return to the problem of comparing two abstaining classifiers,  $(f^{\mathsf{A}},\pi^{\mathsf{A}})$  and  $(f^{\mathsf{B}},\pi^{\mathsf{B}})$ , that each makes a decision to make a prediction on each input  $X_{i}$  or abstain from doing so. That is,  $R_{i}^{\bullet}\mid \pi^{\bullet}(X_{i})\sim \mathrm{Ber}(\pi^{\bullet}(X_{i}))$ , and we observe  $S_{i}^{\bullet} = \mathfrak{s}(f^{\bullet}(X_{i}),Y_{i})$  if and only if  $R_{i}^{\bullet} = 0$ , for  $\bullet \in \{\mathsf{A},\mathsf{B}\}$ . Recall that the target here is the score difference  $\Delta^{\mathsf{AB}} = \psi^{\mathsf{A}} - \psi^{\mathsf{B}} = \mathbb{E}\left[S^{\mathsf{A}} - S^{\mathsf{B}}\right]$ .
Doubly robust difference estimation If the parameters  $\psi^{\mathsf{A}}$  and  $\psi^{\mathsf{B}}$  are each identified according to Proposition 2.4, then we can estimate  $\Delta^{\mathsf{AB}}$  as  $\hat{\Delta}^{\mathsf{AB}} = \hat{\psi}^{\mathsf{A}} - \hat{\psi}^{\mathsf{B}}$ , for individual estimates  $\hat{\psi}^{\mathsf{A}}$  and  $\hat{\psi}^{\mathsf{B}}$ . The resulting EIF is simply the difference in the EIF for A and B:  $|\mathsf{F}^{\mathsf{AB}}(x,r^{\mathsf{A}},r^{\mathsf{B}},s^{\mathsf{A}},s^{\mathsf{B}}) = |\mathsf{F}^{\mathsf{A}}(x,r^{\mathsf{A}},s^{\mathsf{A}}) - |\mathsf{F}^{\mathsf{B}}(x,r^{\mathsf{B}},s^{\mathsf{B}})$ , where  $|\mathsf{F}^{\mathsf{A}}$  and  $|\mathsf{F}^{\mathsf{B}}$  denote the EIF of the respective classifier. Thus, we arrive at an analogous theorem that involves estimating the nuisance functions of each abstaining classifier and utilizing  $|\mathsf{F}^{\mathsf{AB}}|$  to obtain the limiting distribution of  $\hat{\Delta}_{\mathrm{dr}}^{\mathsf{AB}} = \hat{\psi}_{\mathrm{dr}}^{\mathsf{A}} - \hat{\psi}_{\mathrm{dr}}^{\mathsf{B}}$ .
Theorem 3.2 (DR estimation of the counterfactual score difference). Suppose that Assumptions 2.1 and 2.3 hold for both  $(X_{i},R_{i}^{\mathrm{A}},S_{i}^{\mathrm{A}})$  and  $(X_{i},R_{i}^{\mathrm{B}},S_{i}^{\mathrm{B}})$ . Also, suppose that
$$
\| \hat {\pi} ^ {\mathsf {A}} - \pi^ {\mathsf {A}} \| _ {L _ {2} (\mathbb {P})} \| \hat {\mu} _ {0} ^ {\mathsf {A}} - \mu_ {0} ^ {\mathsf {A}} \| _ {L _ {2} (\mathbb {P})} + \| \hat {\pi} ^ {\mathsf {B}} - \pi^ {\mathsf {B}} \| _ {L _ {2} (\mathbb {P})} \| \hat {\mu} _ {0} ^ {\mathsf {B}} - \mu_ {0} ^ {\mathsf {B}} \| _ {L _ {2} (\mathbb {P})} = o _ {\mathbb {P}} (1 / \sqrt {n}) \qquad (3. 4)
$$
and that  $\| \tilde{\mathsf{I}}^{\mathsf{AB}} - \mathsf{I}\mathsf{F}^{\mathsf{AB}}\|_{L_2(\mathbb{P})} = o_{\mathbb{P}}(1)$ . Then,
$$
\sqrt {n} \left(\hat {\Delta} _ {\mathrm {d r}} ^ {\mathrm {A B}} - \Delta^ {\mathrm {A B}}\right)\rightsquigarrow \mathcal {N} \left(0, \operatorname {V a r} _ {\mathbb {P}} \left[ \mathrm {I F} ^ {\mathrm {A B}} \right]\right),
$$
where  $\mathsf{Var}_{\mathbb{P}}[\mathsf{IF}^{\mathsf{AB}}]$  matches the nonparametric efficiency bound.
A proof is given in Appendix B.4. As with evaluation, Theorem 3.2 yields a closed-form asymptotic CI of the form (3.3) using the analogous estimate of EIF under MAR, positivity, and DR (3.4). Inverting this CI further yields a hypothesis test for  $H_0: \psi^{\mathsf{A}} = \psi^{\mathsf{B}}$  vs.  $H_1: \psi^{\mathsf{A}} \neq \psi^{\mathsf{B}}$ .
The main goals of our experiments are to empirically check the validity and power of our estimation methods and to illustrate how our methods can be used in practice. First, in Section 4.1, we present results on simulated data to examine the validity of our proposed inference methods (CIs and hypothesis tests). Then, in Section 4.2, we study three scenarios on the CIFAR-100 dataset that illustrate the practical use of our approach to real data settings. All code for the experiments is publicly available online at https://github.com/yjchoe/ComparingAbstainingClassifiers.
4.1 Simulated experiments: Abstentions near the decision boundary
Setup (MAR but not MCAR) We first consider comparing two abstaining classifiers according to their accuracy scores, on a simulated binary classification dataset with 2-dimensional inputs.
Given  $n = 2,000$  i.i.d. inputs  $\{X_i\}_{i=1}^n \sim \mathrm{Unif}([0,1]^2)$ , each label  $Y_i$  is decided using a linear boundary,  $f_*(x_1, x_2) = \mathbf{1}(x_1 + x_2 \geq 1)$ , along with a  $15\%$  i.i.d. label noise. Importantly, each classifier abstains near its decision boundary, such that its predictions and scores are MAR but not MCAR (because abstentions depend on the inputs). As a result, while the counterfactual score of A  $(\psi^{\mathsf{A}} = 0.86)$  is much higher than B  $(\psi^{\mathsf{B}} = 0.74)$ , their selective scores are more similar  $(\mathsf{Sel}^{\mathsf{A}} = 0.86, \mathsf{Sel}^{\mathsf{B}} = 0.81)$  and coverage is lower for A  $(\mathsf{Cov}^{\mathsf{A}} = 0.55)$  than for B  $(\mathsf{Cov}^{\mathsf{B}} = 0.62)$ . Another point to note here is that, even though both the outcome model and classifier A are linear, both the abstention mechanism  $\pi^{\mathsf{A}}$  and the selective score function  $\mu_0^{\mathsf{A}}$  are nonlinear functions of the inputs (similarly for  $\pi^{\mathsf{B}}$  and  $\mu_0^{\mathsf{B}}$ ). More generally, if a classifier abstains near its decision boundary, then both  $\pi$  and  $\mu_0$  could be at least as complex as the base classifier  $f$  itself. Further details of the setup, including a plot of the data, predictions, and abstentions, are provided in Appendix E.1.
Miscoverage rates and widths As our first experiment, we compare the miscoverage rates and widths of the  $95\%$  DR CIs (Theorem 3.2) against two baseline estimators: the plug-in and the IPW (Rosenbaum, 1995) estimators (Appendix A.4). For each method, the miscoverage rate of the CI  $C_n$  is approximated via  $\mathbb{P}(\Delta^{\mathrm{AB}} \notin C_n) \approx m^{-1} \sum_{j=1}^{m} \mathbf{1}(\Delta^{\mathrm{AB}} \notin C_n^{(j)})$ , where  $m$  is the number of simulations over repeatedly sampled data. If the CI is valid, then this rate should approximately be 0.05. The miscoverage rate and the width of a CI, respectively, capture its bias and variance components. For the nuisance functions, we try linear predictors (L2-regularized linear/logistic regression for  $\hat{\mu}_0 / \hat{\pi}$ ), random forests, and super learners with  $k$ -NN, kernel SVM, and random forests.
We present our results in Table 2. First, using the random forest or the super learner,
Table 2: Miscoverage rates (and widths) of  $95\%$  CIs using three estimation approaches and three nuisance function  $(\pi$  and  $\mu_0)$  estimators in a simulated experiment. Mean and standard error computed over  $m = 1,000$  runs are shown; those within 2 standard errors of the intended level (0.05) are boldfaced. The sample size is  $n = 2,000$  in each run. The mean widths of CIs are shown in parentheses. DR estimation with either a random forest or a super learner achieves control over the miscoverage rate, and the DR-based CI is twice as tight as the IPW-based CI in terms of their width.
μ0 & π	Plug-in	IPW	DR
Linear	1.00 ± 0.00	0.76 ± 0.01	1.00 ± 0.00
(0.00)	(0.09)	(0.04)
Random forest	0.64 ± 0.02	0.14 ± 0.01	0.05 ± 0.01
(0.02)	(0.13)	(0.07)
Super learner	0.91 ± 0.01	0.03 ± 0.01	0.05 ± 0.01
(0.01)	(0.12)	(0.06)
the DR CIs consistently achieve the intended coverage level of 0.95, over  $m = 1,000$  repeated simulations (standard error 0.01). This validates the asymptotic normality result of (3.2). Note that the CI with linear estimators does not achieve the intended coverage level: this is expected as neither  $\hat{\pi}$  nor  $\hat{\mu}_0$  can consistently estimate the nonlinear functions  $\pi$  or  $\mu_0$ , violating the DR condition (3.4).
Second, when considering both the miscoverage rate and CI width, the DR estimator outperforms both the plug-in and IPW estimators. The plug-in estimator, despite having a small CI width, has a very high miscoverage rate (0.91 with the super learner), meaning that it is biased even when flexible nuisance learners are used. On the other hand, the IPW estimator has double the width of the DR estimator (0.12 to 0.06, with the super learner), meaning that it is not as efficient. Also, while the IPW estimator achieves the desired coverage level with the super learner (0.03), it fails with the random forest (0.14), which tends to make overconfident predictions of the abstention pattern and biases the resulting estimate. In contrast, the DR estimator retains its intended coverage level of 0.05 with the random forest, suggesting that it is amenable to overconfident nuisance learners.
Power analysis We conduct a power analysis of the statistical test for  $H_0: \Delta^{\mathrm{AB}} = 0$  vs.  $H_1: \Delta^{\mathrm{AB}} \neq 0$  by inverting the DR CI. The results confirm that the power reaches 1 as either the sample size  $(n)$  or the absolute difference  $(|\Delta^{\mathrm{AB}}|)$  increases. This experiment is included in Appendix E.2.
4The abstention mechanism  $\pi (x) = \mathbb{P}(R = 1\mid X = x)$  here separates the region below and above the decision boundary from the region near the boundary. Thus  $\pi$  is nonlinear even when the boundary is linear.
Given any input  $X$  for which the classifier did not abstain  $(R = 0)$  and its output  $Y$ , the score  $S = \mathsf{s}(f(X), Y)$  is nonlinear if either  $\mathsf{s}(\cdot, y)$  or  $f$  is nonlinear. Thus, even for linear  $f$ , nonlinear scores like the Brier score automatically make the selective score function  $\mu_0(x) = \mathbb{E}[S \mid R = 0, X = x]$  nonlinear.
4.2 Comparing abstaining classifiers on CIFAR-100
To illustrate a real data use case, we compare abstaining classifiers on the CIFAR-100 image classification dataset (Krizhevsky, 2009). Observe that abstaining classifiers can behave differently not just when their base classifiers are different but also when their abstention mechanisms are different. In fact, two abstaining classifiers can have a similarly effective base classifier but substantially different abstention mechanisms (e.g., one more confident than the other). In such a case, the counterfactual score difference between the two classifiers is zero, but their selective scores and coverages are different. We examine such scenarios by comparing image classifiers that use the same pre-trained representation model but have different output layers and abstention mechanisms.
We start with the 512-dimensional final-layer representations of a VGG-16 convolutional neural network (Simonyan and Zisserman, 2015), pre-trained $^6$  on the CIFAR-100 training set, and compare different output layers and abstention mechanisms on the validation set. Generally, in a real data setup, we cannot verify whether a statistical test or a CI is correct; yet, in this experiment, we can still access to the base model of each abstaining classifier. This means that (a) if we compare abstaining classifiers that share the base classifier but differ in their abstention patterns, then we actually know that their counterfactual scores are exactly the same  $(\Delta^{\mathsf{AB}} = 0)$ ; (b) if we compare abstaining classifiers with different base classifiers, then we can compute their counterfactual scores accurately up to an i.i.d. sampling error. This estimate is denoted by  $\bar{\Delta}^{\mathsf{AB}} \coloneqq n^{-1}\sum_{i=1}^{n}[\mathsf{s}(f^{\mathsf{A}}(X_i), Y_i) - \mathsf{s}(f^{\mathsf{B}}(X_i), Y_i)]$ .
For all comparisons, we use the DR estimator, where the nuisance functions  $\hat{\pi}$  and  $\hat{\mu}_0$  for both classifiers are each an L2-regularized linear layer learned on top of the pre-trained VGG-16 features. The use of pre-trained representations for learning the nuisance functions is motivated by Shi et al. (2019), who demonstrated the effectiveness of the approach in causal inference contexts. We also use the Brier score in this experiment. We defer other experiment details to Appendix E.3.
In scenario I, we compare two abstaining classifiers that use the same softmax output layer but use a different threshold for abstentions. Specifically, both classifiers use the softmax response (SR) thresholding (Geifman and El-Yaniv, 2017), i.e., abstain if  $\max_{c\in \mathcal{Y}}f(X)_c < \tau$  for a threshold  $\tau >0$ , but A uses a more conservative threshold  $(\tau = 0.8)$  than B  $(\tau = 0.5)$ . As a result, while their counterfactual scores are identical  $(\Delta^{\mathrm{AB}} = 0)$ , A has a higher selective score  $(+0.06)$  and a lower coverage  $(-0.20)$  than B. This is also a deterministic abstention mechanism, potentially challenging the premises of our setup. As shown in Table 3, we see that the  $95\%$  DR
Table 3: The  $95\%$  DR CIs and their corresponding hypothesis tests for  $H_0:\Delta^{\mathrm{AB}} = 0$  at significance level  $\alpha = 0.05$ , for three different comparison scenarios on (half of) the CIFAR-100 test set ( $n = 5,000$ ). The three scenarios compare different abstention mechanisms or predictors, as detailed in text; all comparisons use the Brier score.  $\bar{\Delta}^{\mathrm{AB}}$  is the empirical counterfactual score difference without any abstentions. The result of each statistical test agrees with whether  $\bar{\Delta}^{\mathrm{AB}}$  is 0.
Scenarios	ΔAB	95% DR CI	Reject H0?
I	0.000	(-0.005, 0.018)	No
II	0.000	(-0.014, 0.008)	No
III	-0.029	(-0.051, -0.028)	Yes
CI is  $(-0.005, 0.018)$  ( $n = 5, 000$ ), confirming that there is no difference in counterfactual scores. $^7$
Scenario II is similar to scenario I, except that the abstention mechanisms are now stochastic: A uses one minus the SR as the probability of making a prediction, i.e.,  $\pi^{\mathsf{A}}(x) = 1 - \max_{c\in [C]}f(X)_c$ , while B uses one minus the Gini impurity as the probability of abstention, i.e.,  $\pi^{\mathsf{B}}(x) = 1 - \sum_{c = 1}^{C}f(X)_c^2$  both clipped to (0.2, 0.8). The Gini impurity is an alternative measure of confidence to SR that is quadratic in the probabilities, instead of piecewise linear, and thus the Gini-based abstaining classifier (B) is more cautious than the SR-based one (A), particularly on uncertain predictions. In our setup, A achieves a higher coverage than B, while B achieves a higher selective score than A. The  $95\%$  DR CI is  $(-0.014, 0.008)$ , confirming that there is once again no difference in counterfactual scores. Scenarios I and II both correspond to case (a).
In scenario III, we now examine a case where there is a difference in counterfactual scores between the two abstaining classifiers (case (b)). Specifically, we compare the pre-trained VGG-16 model's
$^{6}$ Reproduced version, accessed from https://github.com/chenyafo/pytorch-cifar-models.
The reason why the DR CI correctly estimates the true  $\Delta^{\mathrm{AB}} = 0$ , despite the fact that positivity is violated in this case, is because the two classifiers happen to abstain on similar examples (B abstains whenever A does) and their scores on their abstentions happen to be relatively similar (0.604 for A; 0.576 for B).
output layers (512-512-100) with the single softmax output layer that we considered in earlier scenarios. It turns out that the original model's multi-layer output model achieves a worse Brier score (0.758) than one with a single output layer (0.787), likely because the probability predictions of the multi-layer model are too confident (miscalibrated). When using the same abstention mechanism (stochastic abstentions using SR, as in II), the overconfident original model correspondingly achieves a higher coverage (and worse selective Brier score) than the single-output-layer model. The Monte Carlo estimate of the true counterfactual score difference is given by  $\Delta^{\mathrm{AB}} = -0.029$ , and the  $95\%$  DR CI falls entirely negative with  $(-0.051, -0.028)$ , rejecting the null of  $\Delta^{\mathrm{AB}} = 0$  at  $\alpha = 0.05$ .
This paper lays the groundwork for addressing the challenging problem of counterfactually comparing black-box abstaining classifiers. Our solution casts the problem in the missing data framework, in which we treat abstentions as MAR predictions of the classifier(s), and this allows us to leverage nonparametrically efficient tools from causal inference. We close with discussions of two topics.
Addressing violations of the identification conditions At the conceptual level, the biggest challenge arises from the positivity condition, which requires the classifiers to deploy a non-deterministic abstention mechanism. As mentioned in Section 2.2, the counterfactual score is unidentifiable without this assumption. We argue that this issue calls for a policy-level treatment, especially in auditing scenarios, where the evaluators may require vendors to supply a classifier that can abstain but has at least an  $\epsilon >0$  chance of nonabstention. The level  $\epsilon$  can be mutually agreed upon by both parties. Such a policy achieves a middle ground in which the vendors are not required to fully reveal their proprietary classifiers. In Appendix A.5, we discuss this policy-level treatment in greater detail.