Towards optimally abstaining from prediction with OOD test examples

1 Introduction . For learning of a class of functions F of bounded complexity , statistical learning theory guarantees low error if test examples are distributed like training examples . Thus abstention is not necessary for standard realizable prediction . However , when the test distribution Q is not the same as the distribution of training examples P , abstaining from prediction may be beneficial if the cost of abstaining α is substantially less than the cost of an error . This is particularly important when there are “ blind spots ” where Q ( x ) > P ( x ) = 0 . Such blind spots may occur because of natural distribution shifts , or indeed may be due to adversarial attacks . Collectively , test examples that deviate from the training distribution are referred to as out of distribution ( OOD ) test examples . Such an extreme covariate shift scenario was analyzed for binary classification in recent work by Goldwasser , Kalais , and Montasser [ 10 ] ( henceforth GKKM ) . Abstaining from predicting may be useful because it is well-known to be impossible to guarantee accuracy on test examples from arbitrary Q 6= P ( abstention is unnecessary under the common assumption that Q ( x ) /P ( x ) is upper-bounded ) [ 7 ] . Whether one is classifying images or predicting the probability of success of a medical procedure , training data may miss important regions of test examples . As an example , Fang et al . [ 9 ] observed noticeable signs of COVID-19 in many lung scans ; any model trained on data pre-2019 would not have any such instances in its training dataset . In the case of image classification , a classifier may be trained on publicly available data but may be also used on people ’ s private images or even adversarial spam/phishing images [ 21 ] . In settings such as medical diagnoses or content moderation , abstention may be much less costly than a misclassification . In practice , if a model abstains from making a prediction , it could be followed up by a more elaborate ( and costly ) model or direct human intervention . The extra costs of human intervention , or as may be the case in a medical setting , a more expensive test , needs to be traded off with the potential costs of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . misclassification . For a regression example , a model may be used to predict the success probability of a medical treatment across a population , yet examples from people that are considered high-risk for the treatment may be absent from the training data . We begin by discussing binary classification , and then move to regression . In particular , for any distribution P over examples x ∈ X and any true classifier f : X → { 0 , 1 } in F of VC-dimension d , the so-called Fundamental Theorem of Statistical Learning ( FTSL ) guarantees w.h.p . Õ ( d/n ) error rate1 on future examples from P using any classifier h ∈ F that agrees with f on n noiseless labeled examples [ see , e.g. , 20 ] . In Chow ’ s original abstention model [ 6 ] , a selective classifier is allowed to either make a prediction ŷ at a loss of ` ( y , ŷ ) ≥ 0 or abstain from predicting at a fixed small loss α > 0 . Following GKKM , we consider a transductive abstention algorithm that takes as input n unlabeled test examples and n labeled training examples and predicts on a subset of the n test labels , abstaining on the rest . The goal is to minimize the average loss on the test set . ( Unfortunately , the natural idea to abstain on test points whose labels are not uniquely determined by the training data can lead to abstaining on all test points even when P = Q . ) GKKM give a algorithm with guarantees that naturally extend the FTSL to Q 6= P albeit at an additional cost . The word transductive refers to the prediction model where one wishes to classify a given test set rather a standard classifier that generalizes to future examples , though the two models are in fact equivalent in terms of expected error as we discuss . The term covariate shift is appropriate here as it describes settings in which Q 6= P but both train and test labels are consistent with the same f ∈ F . Without abstention , both transductive learning and covariate shift have been extensively studied [ see , e.g. , 1 ] . The principle behind our approach is illustrated through an example of Figure 1 . Suppose P is the distribution Q restricted to a set S ⊂ X which contains , say , 90 % of the test set , so that 10 % of the unlabeled test examples are in blind spots . Say we have learned a standard classifier h ∈ F from the n labeled training examples . Hypothetically , if we knew S , then we could predict h ( xi ) for test xi ∈ S and abstain from predicting on xi /∈ S. If abstaining costs α , then the FTSL would naturally guarantee test loss ≤ 0.1α + Õ ( d/n ) , because one abstains on 10 % of the test examples and the remaining test examples are distributed just like P . The difficulty is that S may be too complex to learn . To circumvent this , we suggest a conceptually simple but theoretically powerful approach : choose the set of points to abstain on so as to minimize the worst-case loss over any true function f ∈ F that is also consistent with the training data . In particular , given a predictor h and a set of test points not abstained on , ( ignoring efficiency ) one could compute the worst-case loss over all f ∈ F that are consistent with the labeled training examples . This approach achieves optimal worst-case guarantees and , in particular inherits the natural loss guarantee one attains from abstaining outside of S discussed above . Converting this theoretical insight into an efficient algorithm is the focus of this paper , and different algorithms are needed for the case of classification and regression . Interpretation . In the case of known P , Q , one may say there is a known unknown : the region of large Q ( x ) /P ( x ) . In our model , however , even though this region is an unknown unknown , we achieve essentially the same bounds as if we knew P and Q . Thus , perhaps surprisingly , there is little additional loss for not knowing P and Q. GKKM give related guarantees , but which suffer from not knowing P , Q . In particular , even when P = Q their guarantees are Õ ( √ d/n ) compared to the Õ ( d/n ) of FTSL , and they give a lower-bound showing Ω ( √ d/n ) is inherent in their “ PQ ” learning model . In the PQ model , the error rate on Q ( mistakes that are not abstained on ) and the abstention rate on future examples from P are separately bounded . In Appendix C , we show that the Chow model is stronger than PQ-learning in the sense that our guarantees imply PQ-bounds similar to theirs , but the reverse does not hold . Hence , the algorithms and guarantees in this paper extend the FTSL in both the Chow and PQ models . We also note that abstaining on predictions can either help reduce inaccuracies on marginal groups or be a source of unfairness towards a person or group of people , and it should not serve as an excuse not to collect representative data . 1.1 Classification results . For classification , suppose Y = { 0 , 1 } , ` ( y , ŷ ) : = |y− ŷ| is the 0-1 loss ( more general losses are considered in the body ) , and F is a family of functions of VC-dimension d. There is also a deterministic 1The Õ notation hides logarithmic factors . Empirical Risk Minimization ( ERM ) oracle that computes ERM ( x , y ) ∈ arg ming∈F ∑ i ` ( yi , g ( xi ) ) on any dataset x ∈ Xn , y ∈ Y n ( even noisy ) . While it is NP-hard to efficiently compute ERM for many simple classes like disjunctions , previous reductions to ERM have proven useful with off-the-shelf classifiers ( e.g. , neural networks ) albeit without theoretical guarantees . The inputs to the learner are labeled training examples x̄ ∈ Xn , ȳ = f ( x̄ ) ∈ Y n , and unlabeled test examples x ∈ Xn . For transductive learning , the goal is to predict labels for the n test examples x . A transductive abstention algorithm is also given the predictor h : = ERM ( x̄ , ȳ ) , which has 0 training error , and selects a vector a ∈ [ 0 , 1 ] n for the probability of abstaining on test examples x . Its loss is defined to be , ` x ( f , h , a ) : = 1 n n∑ i=1 ai α+ ( 1− ai ) ` ( f ( xi ) , h ( xi ) ) . ( 1 ) Our Min-Max Abstention ( MMA ) reduction , mentioned above , attempts to minimize the maximum test loss among classifiers consistent with the training data . In particular , it ( approximately ) solves the following convex optimization problem for a : min a∈ [ 0,1 ] n max g∈V ` x ( g , h , a ) . ( 2 ) Here , V : = { g ∈ F : ∀i g ( x̄i ) = f ( x̄i ) } is the version space of classifiers consistent with the training labels ; thus h , f ∈ V . In other words , MMA minimizes the worst-case test loss it could possibly incur based on the labeled training and unlabeled test data . A key insight is that ( 2 ) has no unknowns , so MMA will achieve a max in ( 2 ) as low as if it knew P , Q and even f . To solve ( 2 ) , one must be able to maximize loss over V . Fortunately , GKKM showed how to solve that using a simple subroutine , which we call FLIP , that calls ERM . Theorem 1 ( Classification ) . For Y = { 0 , 1 } , any n , d ∈ N , any F of VC dimension d , any f ∈ F , and any distributions P , Q over X , E x̄∼Pn , x∼Qn [ ` x ( f , h , â ) ] ≤ α|P −Q|TV + 2d lg 3n n , where h = ERM ( x̄ , f ( x̄ ) ) and â = MMA ( x̄ , f ( x̄ ) , x , h , FLIP ) ∈ [ 0 , 1 ] n can be computed in time poly ( n ) using the ERM oracle for F . The total variation distance |P − Q|TV , also called the statistical distance , is a natural measure of non-overlap that ranges from 0 when P = Q to 1 when P and Q have disjoint supports . The α|P − Q|TV term arises because the learner may need to abstain where P and Q do not overlap , and the other term derives directly from the same bound one gets in the case where P = Q ( so |P −Q|TV = 0 ) . These bounds are stated in terms of expected loss , because unfortunately as we show in Lemma 8 , high probability bounds require Ω ( α/ √ n ) loss because of the variance in how Q samples are distributed . All proofs , unless otherwise stated , are deferred to Appendix F. We generalize Theorem 1 in multiple ways . First , we point out a stronger bound in terms of a divergence Dk ( P‖Q ) , k ≥ 1 , that measures the excess of Q ( x ) over k · P ( x ) . It generalizes the total variation distance |P −Q|TV between distributions P , Q : Dk ( P‖Q ) : = ∑ x max ( Q ( x ) − k · P ( x ) , 0 ) ∈ [ 0 , 1 ] ( 3 ) |P −Q|TV : = D1 ( P‖Q ) = 1 2 ∑ x |P ( x ) −Q ( x ) | ( 4 ) Note that if Q ( x ) ≤ k · P ( x ) for all x , then Dk ( P‖Q ) = 0 . Second , we show this implies generalization by using a transductive abstaining algorithm to bound the expected loss with respect to future examples from Q . That is , to go alongside classifier h : X → { 0 , 1 } , we can output abstainer A : X → [ 0 , 1 ] that gives a probability of abstaining on each test example . Here , the generalization loss is , ` Q ( f , h , A ) : = E x∼Q [ αA ( x ) + ( 1−A ( x ) ) ` ( f ( x ) , h ( x ) ) ] . These two generalizations are summarized by the following theorem , which we state for classification but also has a regression analog . Theorem 2 ( Generalization for classification ) . Fix Y = { 0 , 1 } and F of VC dimension d. For any f ∈ F , and any distributions P , Q over X , E x̄∼Pn , x∼Qn [ ` Q ( f , h , A ) ] ≤ min k≥1 αDk ( P‖Q ) + 2dk lg 3n n ≤ α|P −Q|TV + 2d lg 3n n . where h = ERM ( x̄ , f ( x̄ ) ) and abstainer A : X → [ 0 , 1 ] can be computed in time poly ( n ) using ERM and is defined by A ( x′ ) : = MMA ( x̄ , f ( x̄ ) , ( x′ , x2 , . . . , xn ) , h , FLIP ) . These guarantees use the same algorithm—to predict on a new test example x′ , it simply runs MMA with a modified test set where we have replaced the first test example by x′ and returns the probability of abstaining on it . Third , as in GKKM , guarantees hold with respect to a “ white-box ” adversarial model in which there is only a training distribution P and an adversary who may corrupt any number of test examples ( but they are still labeled by f ) . More specifically , natural train and test sets x̄ , z ∼ Pn are drawn , and an adversary may form an arbitrary test set x ∈ Xn . We achieve guarantees as low as if one knew exactly which examples were corrupted and abstained on those : Theorem 3 ( Adversarial classification ) . For any f ∈ F with d = VC ( F ) , any n ∈ N , δ ≥ 0 and any distribution P over X , with probability ≥ 1− δ over x̄ , z ∼ Pn , the following holds simultaneously for all x ∈ Xn : ` x ( f , h , â ) ≤ α n | { i : xi 6= zi } |+ 2d lg 2n+ lg 1/δ n . where h = ERM ( x̄ , f ( x̄ ) ) and â = MMA ( x̄ , f ( x̄ ) , x , h , FLIP ) ∈ [ 0 , 1 ] n can be computed in time poly ( n ) using ERM . In their adversarial setting , GKKM again has Õ ( √ d/n ) bounds . Our presentation actually begins in this adversarial framework as it is in some sense most general–adversarial robustness implies robustness to covariate shift .

Summary: This paper considers the abstention model in which a classifier is able to abstain from predicting (and thus paying a fixed cost) as opposed to being forced to give a prediction.   They consider a formalism in which the algorithm is given a classifer $f$ with 0 training loss, and it must make predictions for an additional $n$ test points. The classifier incurs a loss of 1 for each error it makes on the test points, but incurs loss $\alpha$ if it chooses to abstain (where $\alpha \leq 1$). Furthermore, the points in the test set are drawn from some distribution that differs from the original training distribution.   In this formalism, they give a generalization bound of a polynomial time algorithm that uses empirical risk minimization as a subroutine. The basic gist is that the overall expected lost can be bounded by the total variation between the training and test distributions along with a $\frac{O(d\logn}{n}$ term where $d$ is the VC-dimension of the classifiers.   They then apply their ideas to the setting of adversarial classification, in which adversarial examples can be thought of as being drawn from a distribution $Q$ that is very close to the original data disribution $P$. They also apply their theorem to linear regression giving a bound on how well a linear classifier can be learned.