Towards optimally abstaining from prediction with OOD test examples

1 Introduction . For learning of a class of functions F of bounded complexity , statistical learning theory guarantees low error if test examples are distributed like training examples . Thus abstention is not necessary for standard realizable prediction . However , when the test distribution Q is not the same as the distribution of training examples P , abstaining from prediction may be beneficial if the cost of abstaining α is substantially less than the cost of an error . This is particularly important when there are “ blind spots ” where Q ( x ) > P ( x ) = 0 . Such blind spots may occur because of natural distribution shifts , or indeed may be due to adversarial attacks . Collectively , test examples that deviate from the training distribution are referred to as out of distribution ( OOD ) test examples . Such an extreme covariate shift scenario was analyzed for binary classification in recent work by Goldwasser , Kalais , and Montasser [ 10 ] ( henceforth GKKM ) . Abstaining from predicting may be useful because it is well-known to be impossible to guarantee accuracy on test examples from arbitrary Q 6= P ( abstention is unnecessary under the common assumption that Q ( x ) /P ( x ) is upper-bounded ) [ 7 ] . Whether one is classifying images or predicting the probability of success of a medical procedure , training data may miss important regions of test examples . As an example , Fang et al . [ 9 ] observed noticeable signs of COVID-19 in many lung scans ; any model trained on data pre-2019 would not have any such instances in its training dataset . In the case of image classification , a classifier may be trained on publicly available data but may be also used on people ’ s private images or even adversarial spam/phishing images [ 21 ] . In settings such as medical diagnoses or content moderation , abstention may be much less costly than a misclassification . In practice , if a model abstains from making a prediction , it could be followed up by a more elaborate ( and costly ) model or direct human intervention . The extra costs of human intervention , or as may be the case in a medical setting , a more expensive test , needs to be traded off with the potential costs of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . misclassification . For a regression example , a model may be used to predict the success probability of a medical treatment across a population , yet examples from people that are considered high-risk for the treatment may be absent from the training data . We begin by discussing binary classification , and then move to regression . In particular , for any distribution P over examples x ∈ X and any true classifier f : X → { 0 , 1 } in F of VC-dimension d , the so-called Fundamental Theorem of Statistical Learning ( FTSL ) guarantees w.h.p . Õ ( d/n ) error rate1 on future examples from P using any classifier h ∈ F that agrees with f on n noiseless labeled examples [ see , e.g. , 20 ] . In Chow ’ s original abstention model [ 6 ] , a selective classifier is allowed to either make a prediction ŷ at a loss of ` ( y , ŷ ) ≥ 0 or abstain from predicting at a fixed small loss α > 0 . Following GKKM , we consider a transductive abstention algorithm that takes as input n unlabeled test examples and n labeled training examples and predicts on a subset of the n test labels , abstaining on the rest . The goal is to minimize the average loss on the test set . ( Unfortunately , the natural idea to abstain on test points whose labels are not uniquely determined by the training data can lead to abstaining on all test points even when P = Q . ) GKKM give a algorithm with guarantees that naturally extend the FTSL to Q 6= P albeit at an additional cost . The word transductive refers to the prediction model where one wishes to classify a given test set rather a standard classifier that generalizes to future examples , though the two models are in fact equivalent in terms of expected error as we discuss . The term covariate shift is appropriate here as it describes settings in which Q 6= P but both train and test labels are consistent with the same f ∈ F . Without abstention , both transductive learning and covariate shift have been extensively studied [ see , e.g. , 1 ] . The principle behind our approach is illustrated through an example of Figure 1 . Suppose P is the distribution Q restricted to a set S ⊂ X which contains , say , 90 % of the test set , so that 10 % of the unlabeled test examples are in blind spots . Say we have learned a standard classifier h ∈ F from the n labeled training examples . Hypothetically , if we knew S , then we could predict h ( xi ) for test xi ∈ S and abstain from predicting on xi /∈ S. If abstaining costs α , then the FTSL would naturally guarantee test loss ≤ 0.1α + Õ ( d/n ) , because one abstains on 10 % of the test examples and the remaining test examples are distributed just like P . The difficulty is that S may be too complex to learn . To circumvent this , we suggest a conceptually simple but theoretically powerful approach : choose the set of points to abstain on so as to minimize the worst-case loss over any true function f ∈ F that is also consistent with the training data . In particular , given a predictor h and a set of test points not abstained on , ( ignoring efficiency ) one could compute the worst-case loss over all f ∈ F that are consistent with the labeled training examples . This approach achieves optimal worst-case guarantees and , in particular inherits the natural loss guarantee one attains from abstaining outside of S discussed above . Converting this theoretical insight into an efficient algorithm is the focus of this paper , and different algorithms are needed for the case of classification and regression . Interpretation . In the case of known P , Q , one may say there is a known unknown : the region of large Q ( x ) /P ( x ) . In our model , however , even though this region is an unknown unknown , we achieve essentially the same bounds as if we knew P and Q . Thus , perhaps surprisingly , there is little additional loss for not knowing P and Q. GKKM give related guarantees , but which suffer from not knowing P , Q . In particular , even when P = Q their guarantees are Õ ( √ d/n ) compared to the Õ ( d/n ) of FTSL , and they give a lower-bound showing Ω ( √ d/n ) is inherent in their “ PQ ” learning model . In the PQ model , the error rate on Q ( mistakes that are not abstained on ) and the abstention rate on future examples from P are separately bounded . In Appendix C , we show that the Chow model is stronger than PQ-learning in the sense that our guarantees imply PQ-bounds similar to theirs , but the reverse does not hold . Hence , the algorithms and guarantees in this paper extend the FTSL in both the Chow and PQ models . We also note that abstaining on predictions can either help reduce inaccuracies on marginal groups or be a source of unfairness towards a person or group of people , and it should not serve as an excuse not to collect representative data . 1.1 Classification results . For classification , suppose Y = { 0 , 1 } , ` ( y , ŷ ) : = |y− ŷ| is the 0-1 loss ( more general losses are considered in the body ) , and F is a family of functions of VC-dimension d. There is also a deterministic 1The Õ notation hides logarithmic factors . Empirical Risk Minimization ( ERM ) oracle that computes ERM ( x , y ) ∈ arg ming∈F ∑ i ` ( yi , g ( xi ) ) on any dataset x ∈ Xn , y ∈ Y n ( even noisy ) . While it is NP-hard to efficiently compute ERM for many simple classes like disjunctions , previous reductions to ERM have proven useful with off-the-shelf classifiers ( e.g. , neural networks ) albeit without theoretical guarantees . The inputs to the learner are labeled training examples x̄ ∈ Xn , ȳ = f ( x̄ ) ∈ Y n , and unlabeled test examples x ∈ Xn . For transductive learning , the goal is to predict labels for the n test examples x . A transductive abstention algorithm is also given the predictor h : = ERM ( x̄ , ȳ ) , which has 0 training error , and selects a vector a ∈ [ 0 , 1 ] n for the probability of abstaining on test examples x . Its loss is defined to be , ` x ( f , h , a ) : = 1 n n∑ i=1 ai α+ ( 1− ai ) ` ( f ( xi ) , h ( xi ) ) . ( 1 ) Our Min-Max Abstention ( MMA ) reduction , mentioned above , attempts to minimize the maximum test loss among classifiers consistent with the training data . In particular , it ( approximately ) solves the following convex optimization problem for a : min a∈ [ 0,1 ] n max g∈V ` x ( g , h , a ) . ( 2 ) Here , V : = { g ∈ F : ∀i g ( x̄i ) = f ( x̄i ) } is the version space of classifiers consistent with the training labels ; thus h , f ∈ V . In other words , MMA minimizes the worst-case test loss it could possibly incur based on the labeled training and unlabeled test data . A key insight is that ( 2 ) has no unknowns , so MMA will achieve a max in ( 2 ) as low as if it knew P , Q and even f . To solve ( 2 ) , one must be able to maximize loss over V . Fortunately , GKKM showed how to solve that using a simple subroutine , which we call FLIP , that calls ERM . Theorem 1 ( Classification ) . For Y = { 0 , 1 } , any n , d ∈ N , any F of VC dimension d , any f ∈ F , and any distributions P , Q over X , E x̄∼Pn , x∼Qn [ ` x ( f , h , â ) ] ≤ α|P −Q|TV + 2d lg 3n n , where h = ERM ( x̄ , f ( x̄ ) ) and â = MMA ( x̄ , f ( x̄ ) , x , h , FLIP ) ∈ [ 0 , 1 ] n can be computed in time poly ( n ) using the ERM oracle for F . The total variation distance |P − Q|TV , also called the statistical distance , is a natural measure of non-overlap that ranges from 0 when P = Q to 1 when P and Q have disjoint supports . The α|P − Q|TV term arises because the learner may need to abstain where P and Q do not overlap , and the other term derives directly from the same bound one gets in the case where P = Q ( so |P −Q|TV = 0 ) . These bounds are stated in terms of expected loss , because unfortunately as we show in Lemma 8 , high probability bounds require Ω ( α/ √ n ) loss because of the variance in how Q samples are distributed . All proofs , unless otherwise stated , are deferred to Appendix F. We generalize Theorem 1 in multiple ways . First , we point out a stronger bound in terms of a divergence Dk ( P‖Q ) , k ≥ 1 , that measures the excess of Q ( x ) over k · P ( x ) . It generalizes the total variation distance |P −Q|TV between distributions P , Q : Dk ( P‖Q ) : = ∑ x max ( Q ( x ) − k · P ( x ) , 0 ) ∈ [ 0 , 1 ] ( 3 ) |P −Q|TV : = D1 ( P‖Q ) = 1 2 ∑ x |P ( x ) −Q ( x ) | ( 4 ) Note that if Q ( x ) ≤ k · P ( x ) for all x , then Dk ( P‖Q ) = 0 . Second , we show this implies generalization by using a transductive abstaining algorithm to bound the expected loss with respect to future examples from Q . That is , to go alongside classifier h : X → { 0 , 1 } , we can output abstainer A : X → [ 0 , 1 ] that gives a probability of abstaining on each test example . Here , the generalization loss is , ` Q ( f , h , A ) : = E x∼Q [ αA ( x ) + ( 1−A ( x ) ) ` ( f ( x ) , h ( x ) ) ] . These two generalizations are summarized by the following theorem , which we state for classification but also has a regression analog . Theorem 2 ( Generalization for classification ) . Fix Y = { 0 , 1 } and F of VC dimension d. For any f ∈ F , and any distributions P , Q over X , E x̄∼Pn , x∼Qn [ ` Q ( f , h , A ) ] ≤ min k≥1 αDk ( P‖Q ) + 2dk lg 3n n ≤ α|P −Q|TV + 2d lg 3n n . where h = ERM ( x̄ , f ( x̄ ) ) and abstainer A : X → [ 0 , 1 ] can be computed in time poly ( n ) using ERM and is defined by A ( x′ ) : = MMA ( x̄ , f ( x̄ ) , ( x′ , x2 , . . . , xn ) , h , FLIP ) . These guarantees use the same algorithm—to predict on a new test example x′ , it simply runs MMA with a modified test set where we have replaced the first test example by x′ and returns the probability of abstaining on it . Third , as in GKKM , guarantees hold with respect to a “ white-box ” adversarial model in which there is only a training distribution P and an adversary who may corrupt any number of test examples ( but they are still labeled by f ) . More specifically , natural train and test sets x̄ , z ∼ Pn are drawn , and an adversary may form an arbitrary test set x ∈ Xn . We achieve guarantees as low as if one knew exactly which examples were corrupted and abstained on those : Theorem 3 ( Adversarial classification ) . For any f ∈ F with d = VC ( F ) , any n ∈ N , δ ≥ 0 and any distribution P over X , with probability ≥ 1− δ over x̄ , z ∼ Pn , the following holds simultaneously for all x ∈ Xn : ` x ( f , h , â ) ≤ α n | { i : xi 6= zi } |+ 2d lg 2n+ lg 1/δ n . where h = ERM ( x̄ , f ( x̄ ) ) and â = MMA ( x̄ , f ( x̄ ) , x , h , FLIP ) ∈ [ 0 , 1 ] n can be computed in time poly ( n ) using ERM . In their adversarial setting , GKKM again has Õ ( √ d/n ) bounds . Our presentation actually begins in this adversarial framework as it is in some sense most general–adversarial robustness implies robustness to covariate shift .

Summary: This paper studies agnostic transductive learning when train and test distributions have covariate shift. The objective follows Chow 57', where abstention suffers a fixed cost, and the goal is to minimize total costs of prediction error and abstentions. The paper proposes efficient algorithms for linear regression and binary classification, based on ideas from Goldwasser et al. The paper also proves lower bounds showing that the guarantee is near-optimal.