SlowGuess/NeurIPS_2023_OCR huggingface dataset:
An alternative is to explore existing techniques that aim to address positivity violations directly (Petersen et al., 2012; Ju et al., 2019; Leger et al., 2022). Due to the unidentifiability result, these techniques have their own limitations. For example, we may consider applying sample trimming (Crump et al., 2009) to make valid inferences on a subpopulation, but the conclusions would not hold for the entire input domain (even when relevant). Nevertheless, it remains a meaningful future work to adapt modern tools in causal inference for diagnosing and addressing positivity violations (e.g., Lei et al., 2021; Kennedy, 2019). In the case of addressing MAR violations, if we suspect the evaluation data is "compromised" and used in training, we can adapt sensitivity analysis methods that measure how much of the evaluation data is contaminated by the training data (Bonvini and Kennedy, 2022).
Connections to learning-to-defer scenarios and cascading classifiers There are variants of abstaining classifiers in ML for which we can utilize the counterfactual score analogously. First, in the learning-to-defer setting (Madras et al., 2018), where the algorithm is allowed to defer its decision to an expert that gives their own decision, the counterfactual score naturally corresponds to the expected score of the overall system had the classifier not deferred at all. The score is thus an evaluation metric primarily for the classifier, and it is independent of the expert's predictions, even when the classifier is adaptive to the expert's tendencies. In the case where the goal is to assess the joint performance of the algorithm and the expert, then a variant of Condessa et al. (2017)'s score can be useful; in Appendix A.6, we discuss how the DR estimator can be utilized for estimating this variant, even when the expert's score is random.
Another example would be cascading or multi-stage classifiers (Alpaydin and Kaynak, 1998; Viola and Jones, 2001), which are classifiers that short-circuit easy predictions by using simple models in their early stages to save computation. The counterfactual framework can assess the performance of cascading classifiers by treating each component as an abstaining classifier. For example, a two-stage cascading classifier, equipped with a small model  $f_{0}$ , a large model  $f_{1}$ , and a deferral mechanism  $\pi$ , can be viewed as a pair of abstaining classifiers with tied abstention mechanisms, specifically  $(f_{0}, \pi)$  and  $(f_{1}, \bar{\pi})$  where  $\bar{\pi} = 1 - \pi$ . Then, the corresponding counterfactual scores capture the performance of the cascading classifier had it been using only that (small or large) model. We can further extend this to general multi-stage classifiers if we are interested in estimating the score of a classifier in each stage individually. The two-stage setup is also relevant to the learning-to-defer setup, where the large model is an imperfect expert on the task. In such a setup, we can devise a metric that combines (i) the selective score of the small model and (ii) the relative improvement by switching to the expert. Our approach can also estimate this new metric using essentially the same tools.
The authors thank Edward H. Kennedy and anonymous reviewers for their helpful comments and suggestions. AR acknowledges support from NSF grants IIS-2229881 and DMS-2310718. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges-2 system, which is supported by NSF award number ACI-1928147, at the Pittsburgh Supercomputing Center (PSC).
Agarwal, S. and Deshpande, A. (2022). On the power of randomization in fair classification and representation. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1542-1551.
Alpaydin, E. and Kaynak, C. (1998). Cascading classifiers. Kybernetika, 34(4):369-374.
Bang, H. and Robins, J. M. (2005). Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4):962-973.
Barocas, S., Hardt, M., and Narayanan, A. (2019). *Fairness and machine learning: Limitations and opportunities*. fairmlbook.org. http://www.fairmlbook.org.
Bartlett, P. L. and Wegkamp, M. H. (2008). Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(8).
Belloni, A., Chernozhukov, V., and Hansen, C. (2014). Inference on treatment effects after selection among high-dimensional controls. The Review of Economic Studies, 81(2):608-650.
Bickel, P. J., Klaassen, C. A., Bickel, P. J., Ritov, Y., Klaassen, J., Wellner, J. A., and Ritov, Y. (1993). Efficient and adaptive estimation for semiparametric models, volume 4. Johns Hopkins University Press Baltimore.
Bonvini, M. and Kennedy, E. H. (2022). Sensitivity analysis via the proportion of unmeasured confounding. Journal of the American Statistical Association, 117(539):1540-1550.
Breiman, L. (1996). Stacked regressions. Machine Learning, 24:49-64.
Breiman, L. (2001). Random forests. Machine Learning, 45:5-32.
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3.
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68.
Chow, C.-K. (1957). An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC-6(4):247-254.
Chow, C.-K. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46.
Condessa, F., Bioucas-Dias, J., and Kovacevic, J. (2017). Performance measures for classification systems with rejection. Pattern Recognition, 63:437-450.
Cortes, C., DeSalvo, G., and Mohri, M. (2016). Boosting with abstention. Advances in Neural Information Processing Systems, 29.
Crump, R. K., Hotz, V. J., Imbens, G. W., and Mitnik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1):187-199.
Ding, P. and Li, F. (2018). Causal inference: A missing data perspective. Statistical Science, 33(2).
Robins, J., Li, L., Tchetgen, E., and van der Vaart, A. (2008). Higher order influence functions and minimax estimation of nonlinear functionals. *Probability and Statistics: Essays in Honor of David A. Freedman*, 2:335-421.
Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, 89(427):846-866.
Rosenbaum, P. R. (1995). Observational studies. Springer.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688.
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3):581-592.
Schreuder, N. and Chzhen, E. (2021). Classification with abstention but without disparities. In Uncertainty in Artificial Intelligence, pages 1227-1236. PMLR.
Shi, C., Blei, D., and Veitch, V. (2019). Adapting neural networks for the estimation of treatment effects. Advances in Neural Information Processing Systems, 32.
Shpitser, I., Mohan, K., and Pearl, J. (2015). Missing data as a causal and probabilistic problem. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pages 802-811.
Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288.
Tsiatis, A. A. (2006). Semiparametric theory and missing data. Springer.
van der Laan, M. J., Polley, E. C., and Hubbard, A. E. (2007). Super learner. Statistical Applications in Genetics and Molecular Biology, 6(1).
van der Laan, M. J. and Robins, J. M. (2003). Unified methods for censored longitudinal data and causality. Springer.
van der Vaart, A. W. (2000). Asymptotic statistics, volume 3. Cambridge University Press.
van der Vaart, A. W. (2002). Semiparametric statistics. In *Lectures on Probability Theory and Statistics* (Saint-Flour, 1999), pages 331–457. Springer.
Viola, P. and Jones, M. (2001). Robust real-time object detection. International Journal of Computer Vision, 4(34-47):4.
Waudby-Smith, I., Arbour, D., Sinha, R., Kennedy, E. H., and Ramdas, A. (2021). Time-uniform central limit theory, asymptotic confidence sequences, and anytime-valid causal inference. arXiv preprint arXiv:2103.06476.
Zhang, X.-Y., Xie, G.-S., Li, X., Mei, T., and Liu, C.-L. (2023). A survey on learning to reject. Proceedings of the IEEE, 111(2):185-215.
Zheng, W. and van der Laan, M. J. (2011). Cross-validated targeted minimum-loss-based estimation. In van der Laan, M. J. and Rose, S., editors, Targeted Learning: Causal Inference for Observational and Experimental Data, pages 459-474. Springer.