Online Selective Classification with Limited Feedback

1 Introduction . Consider a low-power or battery-limited edge device , such as a sensor or a smart-speaker that receives a stream of classification requests . Due to the resource limitations , such a device can not implement modern models that are needed for accurate decisions . Instead the device has access ( e.g . via an internet connection ) to an accurate but resource-intensive model implemented on a cloud server , and may send queries to the cloud server in order to retain accuracy . Of course , this incurs costs such as latency and battery drain due to communication . The ideal operation of such a device should thus be to learn a rule that classifies ‘ easy ’ instances locally , while sending harder ones to the cloud , thus maintaining accuracy whilst minimising the net resource consumption [ Xu+14 ; NS17 ] . Selective classification [ Cho57 ; Cho70 ] is a classical paradigm of relevance to such settings . The setup allows a predictor to abstain from classifying some instances ( without incurring a mistake ) . This abstention models adaptive decisions to invoke more resource-intensive methods on subtle cases , like in the above example . The solution concept is relevant widely - for instance , it is relevant to adaptively recommending further ( and costly ) tests rather than offering a diagnosis in a medical scenario , or to recommending a human review instead of an alarm-or-not decision in security contexts . Two aspects of such settings are of particular interest to us . Firstly , the cheaper methods are typically not sufficient to realise the true labels , due to which abstention may be a long-term necessity . Secondly , a-priori reliable labels can only be obtained by invoking the resource intensive option , and thus feedback on whether a non-abstaining decision was correct is unavailable . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We propose online selective classification , with an emphasis on ensuring very few mistakes , to account for the need for very accurate decisions . Concretely , an adversary sequentially produces contexts and labels ( Xt , Yt ) , and the learner uses the Xts to produce a decision Ŷt that may either be one of K classes , or an abstention , which we represent as ⊥ . Feedback in the form of Yt is provided if and only if Ŷt = ⊥ , and the learner incurs a mistake if Ŷt was non-abstaining and did not equal Yt . With the emphasis on controlling the total number of mistakes , we study regrets achievable when compared to the behaviour of the best-in-hindsight error-free selective classifier from a given class - that is , one that makes no mistakes , while abstaining the fewest number of times . Notice that our situation is non-realisable , and therefore this competitor may abstain in the long-run . The two metrics of importance here are the number of mistakes the learner makes , and its excess abstention over this competitor . An effective learner must control both abstention and mistakes , and it is not enough to make one small , e.g . a learner that makes a lot of mistakes but incurs a very negative excess abstention is no good . This simultaneous control of two regrets raises particular challenges . We construct a simple scheme that , when competing against finite classes , simultaneously guarantees O ( Tµ ) mistakes and O ( T 1−µ ) excess abstentions against adaptive adversaries ( for any µ ∈ [ 0 , 1 ] ) , and show that these rates are Pareto-tight [ OR94 ] . We further show that against stochastic adversaries , the same rates can be attained with improved dependence of the regret bounds on the size of the class , and we also describe schemes that enjoy similar improvements against adaptive adversaries , but at the cost of the T -dependence of the regret bounds . The main schemes randomly abstain at a given rate in order to gain information , and otherwise play Ŷt consistent with the ‘ version space ’ of classifiers that have not been observed to make mistakes . For the adversarial case , the analysis of the scheme relies on a new ‘ adversarial uniform law of large numbers ’ ( ALLN ) to argue that such methods can not incur too many mistakes . This ALLN uses a self-normalised martingale concentration bound , and further yields an adaptive continuous approximation guarantee for the Bernoulli-sampling sketch in the sense of Ben-Eliezer & Yogev [ BY20 ; Alo+21 ] . The theoretical exploration is complemented by illustrative experiments that implement our scheme on two benchmark datasets . 1.1 Related Work . Selective classification has been well studied in the batch setting , and many theoretical and methodological results have appeared [ e.g . HW06 ; BW08 ; EW10 ; WE11 ; KKM12 ; CDM16 ; Lei14 ; GE19 ; GKS21 ] . These batch results do not have strong implications for the online setting . Cortes et al . have studied selective classification in the online setting [ Cor+18 ] , but with two differences from our setting . Firstly , rather than individually controlling mistakes and abstentions , the regret is defined according to the Chow loss , which adds up the number of mistakes and c times the number of abstentions , where c is a fixed cost parameter . Secondly ( and more importantly ) it is assumed that feedback is provided only when the learner does not abstain , rather than only when it does . This difference arises from the underlying situations being modelled - Cortes et al . view the abstention as a decision given to a user in which case no feedback is forthcoming , while we view it as a decision to invoke further processing . Both of the scenarios are reasonable , and so both of these explorations are valid , however it is unclear what implications one set of results have for the other . A similar decision and feedback model as ours was proposed by Li et al . in the ‘ knows what it knows ’ ( KWIK ) framework [ LLWS11 ] . The KWIK model , however , fundamentally views abstentions as a short term action , typically arguing that only a finite number of these are made . This is viable since Li et al . study this model in an essentially realisable setting , wherein the optimal labels are known to be essentially realised by a given class - notice that in such a case , a single abstention at an instance x determines what value should be played there in the long run . Our interest however lies in the situation where this data can not be represented in such a way , and such strategies are not viable since the labels may be noisy . Our work thus generalises the KWIK setting to non-realisable data , and to situations wherein abstention is a valid long-term action , as motivated in the introduction , by studying behaviour against competitors that may abstain.1 While Szita and Szepesvári have extended the KWIK formulation to the agnostic case in a regression setting [ SS11 ] , this work also focuses of limiting the number of abstentions to be finite rather than 1The KWIK model also bears other significant differences . It posits an input parameter ε , and requires that the learner either abstains , or produces an ε-accurate response . A notion of competitor is not invoked , and rather than studying regret , the number of abstentions needed to achieve this ε-accuracy is studied . long-run abstentions . Concretely it is assumed that Yt = g ( Xt ) + noise , for some function g , and the learner knows a class H , and a bound ∆ such some h ∈ H is ∆-close to g ( in an appropriate norm ) . Using the knowledge of ∆ , they describe schemes that have limited abstention , but at the cost of mistakes , by producing responses Ŷt that are up to ( 2 + o ( 1 ) ) ∆ separated from Yt . In contrast , in our formulation , contexts Xt for which no function inH can represent the ground truth g well would always be abstained upon . In addition to this work , trade-offs between mistakes and abstentions in a relaxed version of the KWIK framework have been considered [ ZC16 ; SZB10 ; DZ13 ] , and in particular the agnostic case has been explored by Zhang and Chaudhuri [ ZC16 ] , but unlike our situation this relaxed KWIK model requires full-feedback to be available whether or not the learner abstains . Neu and Zhivotovskiy [ NZ20 ] also work in this relaxed model , and show that when comparing the standard loss of a non-abstaining classifier against the Chow loss of an abstaining learner , regrets independent of time can be obtained . Due to the limited feedback , our setting is related to partial-monitoring [ LS20 , Ch . 37 ] . Viewing actions as choices over functions , our setting has feedback graphs [ MS11 ] that connect abstaining actions to every other action and themselves . The novelty with respect to partial-monitoring arises from the fact that we individually control two notions of losses , rather than a single one . It ’ s unclear how to apply the generic partial-monitoring setup to this situation - indeed , naïvely , our game is only weakly observable in the sense of Alon et al . [ ACDK15 ] , and one would expect Ω ( T 2/3 ) regrets , while we can control both mistakes and excess abstention to Õ ( √ T ) . A limited feedback setting where two ‘ losses ’ are individually controlled is label-efficient prediction [ CLS05 ] , where a learner must query in order to get feedback . However , in our setting , abstentions are both a way to gather feedback , and also necessary to prevent mistakes . That is , our competitor may abstain regularly , but makes few mistakes , while in this prior work the competitor does not abstain , but may make many mistakes . The resulting scenario is both qualitatively and quantitatively distinct , e.g . in label-efficient prediction , the smallest symmetric rate of number of queries and excess mistakes is again Θ ( T 2/3 ) . 2 Setting , and Problem Formulation . Setup Let X be a feature space , Y a finite set of labels , and F a finite class of selective classifiers , which are Y ∪ { ⊥ } valued . For simplicity , we assume that F contains the all abstaining classifier ( i.e . the function f⊥ such that ∀x , f⊥ ( x ) = ⊥ ) . We will denote |F| = N . The setting may be described as a game between a learner and an adversary ( or more prosaically , a data generating mechanism ) proceeding in T rounds . Also for simplicity , we will assume that T is known to both the learner and the adversary in advance . The objects in this game are the context process , Xt ∈ X , the label process Yt ∈ Y , the action process Ŷt ∈ Y ∪ { ⊥ } and the feedback process Zt ∈ Y ∪ { ∗ } , where ∗ 6∈ Y is a trivial symbol . The information sets of the adversary and learner up to the tth round are respectively H At−1 : = { ( Xs , Ys , Ŷs ) : s < t } , and H Lt−1 : = { ( Xs , Ŷs , Zs ) : s < t } . The Game For each round t ∈ [ 1 : T ] , the adversary produces a context and a label ( Xt , Yt ) on the basis its history H At−1 . The learner observes only the context , Xt , and on the basis of this and its history H Lt−1 , produces an action Ŷt . We will say that this action is an abstention if Ŷt = ⊥ , and that it is a prediction otherwise . If the action was an abstention , set Zt = Yt , and otherwise to ∗ . The learner then observes Zt , and the round concludes . Notice that since Zt is a deterministic function of Yt and Ŷt , and since the adversary observes both , H Lt−1 can be determinstically generated from H A t−1 . Due to the same reason , Ŷt and Yt are conditionally independent given ( Xt , H At−1 ) . Adversaries are characterised by a sequence of conditional laws on ( Xt , Yt ) given H At−1 ( and T , F ) . In the following we will explicitly consider two classes of such laws : • Stochastic Adversary : ( Xt , Yt ) are drawn according to a fixed law , P , unknown to the learner , independently of H At−1 . • Adaptive Adversary : ( Xt , Yt ) are arbitrary random variables with H At−1-measurable laws . We will denote a generic class of adversaries as C . Performance Metrics The two principal quantities of interest are the number of mistakes made by the learner , and the number of times it has abstained . We will denote these as MT : = ∑ t≤T 1 { Ŷt 6∈ { ⊥ , Yt } } , and AT : = ∑ t≤T 1 { Ŷt = ⊥ } . As previously discussed , the performance of a learner is measured in terms of regret with respect to the best-in-hindsight abstaining classifier from F that makes no mistakes , that is f∗ ∈ arg min f∈F ∑ t≤T 1 { f ( Xt ) = ⊥ } s.t . ∑ t≤T 1 { f ( Xt ) 6∈ { ⊥ , Yt } } = 0 . Note that such an f∗ is always realised , since the class is finite , and since it contains the all abstaining classifier . Let A∗T : = ∑ t≤T 1 { f∗ ( Xt ) = ⊥ } denote the value of the minimum above . The principal metrics of interest to us are the abstention regret AT −A∗T , and the total mistakes MT . Solution Concept The two performance metrics naturally involve a tradeoff - for instance , making some mistakes may allow a learner to drastically reduce its abstention regret to the point that it is negative . We pursue the trade-off between the worst possible behaviour of either regret . Definition ( Regret Achievability ) For functions ϕ , ψ : N2 → R , we say that expected regret bounds of ( ϕ , ψ ) are achievable against a class of adversaries C if there exists a learner such that for every adversary in C , E [ AT −A∗T ] ≤ ϕ ( T , N ) and E [ MT ] ≤ ψ ( T , N ) . As is common , we are interested in the growth rates of achievable bounds with T . We thus define Definition ( Achievable rates ) we say that asymptotic expected-regret rates of ( α , µ ) ∈ [ 0 , 1 ] 2 are achievable against a class of adversaries C if an expected regret bound of ( ϕ , ψ ) can be achieved against it for functions ϕ , ψ , said to be witnesses for the rate , such that lim sup T→∞ logϕ ( T , N ) log T ≤ α and lim sup T→∞ logψ ( T , N ) log T ≤ µ . Notice that if ( α , µ ) is an achievable rate , so is ( α′ , δ′ ) for α′ ≥ α , δ′ ≥ δ . As a result , the lower boundary of the set of achievable rates is well defined , and we will refer to this as the Pareto frontier of achievable rates . This is equivalently characterised by the function α ( µ ) : = inf { α : ( α , µ ) is an achievable rate } . This is well defined since ∀µ , ( 1 , µ ) is achievable by always abstaining .

Summary: This manuscript proposes a framework for selective classification in an online fashion. In particular, it proposes strategies that make few mistakes while not abstaining too many times. A tight regret analysis is well established and experiments on real datasets demonstrate the power of the approach.