Online Selective Classification with Limited Feedback

1 Introduction . Consider a low-power or battery-limited edge device , such as a sensor or a smart-speaker that receives a stream of classification requests . Due to the resource limitations , such a device can not implement modern models that are needed for accurate decisions . Instead the device has access ( e.g . via an internet connection ) to an accurate but resource-intensive model implemented on a cloud server , and may send queries to the cloud server in order to retain accuracy . Of course , this incurs costs such as latency and battery drain due to communication . The ideal operation of such a device should thus be to learn a rule that classifies ‘ easy ’ instances locally , while sending harder ones to the cloud , thus maintaining accuracy whilst minimising the net resource consumption [ Xu+14 ; NS17 ] . Selective classification [ Cho57 ; Cho70 ] is a classical paradigm of relevance to such settings . The setup allows a predictor to abstain from classifying some instances ( without incurring a mistake ) . This abstention models adaptive decisions to invoke more resource-intensive methods on subtle cases , like in the above example . The solution concept is relevant widely - for instance , it is relevant to adaptively recommending further ( and costly ) tests rather than offering a diagnosis in a medical scenario , or to recommending a human review instead of an alarm-or-not decision in security contexts . Two aspects of such settings are of particular interest to us . Firstly , the cheaper methods are typically not sufficient to realise the true labels , due to which abstention may be a long-term necessity . Secondly , a-priori reliable labels can only be obtained by invoking the resource intensive option , and thus feedback on whether a non-abstaining decision was correct is unavailable . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We propose online selective classification , with an emphasis on ensuring very few mistakes , to account for the need for very accurate decisions . Concretely , an adversary sequentially produces contexts and labels ( Xt , Yt ) , and the learner uses the Xts to produce a decision Ŷt that may either be one of K classes , or an abstention , which we represent as ⊥ . Feedback in the form of Yt is provided if and only if Ŷt = ⊥ , and the learner incurs a mistake if Ŷt was non-abstaining and did not equal Yt . With the emphasis on controlling the total number of mistakes , we study regrets achievable when compared to the behaviour of the best-in-hindsight error-free selective classifier from a given class - that is , one that makes no mistakes , while abstaining the fewest number of times . Notice that our situation is non-realisable , and therefore this competitor may abstain in the long-run . The two metrics of importance here are the number of mistakes the learner makes , and its excess abstention over this competitor . An effective learner must control both abstention and mistakes , and it is not enough to make one small , e.g . a learner that makes a lot of mistakes but incurs a very negative excess abstention is no good . This simultaneous control of two regrets raises particular challenges . We construct a simple scheme that , when competing against finite classes , simultaneously guarantees O ( Tµ ) mistakes and O ( T 1−µ ) excess abstentions against adaptive adversaries ( for any µ ∈ [ 0 , 1 ] ) , and show that these rates are Pareto-tight [ OR94 ] . We further show that against stochastic adversaries , the same rates can be attained with improved dependence of the regret bounds on the size of the class , and we also describe schemes that enjoy similar improvements against adaptive adversaries , but at the cost of the T -dependence of the regret bounds . The main schemes randomly abstain at a given rate in order to gain information , and otherwise play Ŷt consistent with the ‘ version space ’ of classifiers that have not been observed to make mistakes . For the adversarial case , the analysis of the scheme relies on a new ‘ adversarial uniform law of large numbers ’ ( ALLN ) to argue that such methods can not incur too many mistakes . This ALLN uses a self-normalised martingale concentration bound , and further yields an adaptive continuous approximation guarantee for the Bernoulli-sampling sketch in the sense of Ben-Eliezer & Yogev [ BY20 ; Alo+21 ] . The theoretical exploration is complemented by illustrative experiments that implement our scheme on two benchmark datasets . 1.1 Related Work . Selective classification has been well studied in the batch setting , and many theoretical and methodological results have appeared [ e.g . HW06 ; BW08 ; EW10 ; WE11 ; KKM12 ; CDM16 ; Lei14 ; GE19 ; GKS21 ] . These batch results do not have strong implications for the online setting . Cortes et al . have studied selective classification in the online setting [ Cor+18 ] , but with two differences from our setting . Firstly , rather than individually controlling mistakes and abstentions , the regret is defined according to the Chow loss , which adds up the number of mistakes and c times the number of abstentions , where c is a fixed cost parameter . Secondly ( and more importantly ) it is assumed that feedback is provided only when the learner does not abstain , rather than only when it does . This difference arises from the underlying situations being modelled - Cortes et al . view the abstention as a decision given to a user in which case no feedback is forthcoming , while we view it as a decision to invoke further processing . Both of the scenarios are reasonable , and so both of these explorations are valid , however it is unclear what implications one set of results have for the other . A similar decision and feedback model as ours was proposed by Li et al . in the ‘ knows what it knows ’ ( KWIK ) framework [ LLWS11 ] . The KWIK model , however , fundamentally views abstentions as a short term action , typically arguing that only a finite number of these are made . This is viable since Li et al . study this model in an essentially realisable setting , wherein the optimal labels are known to be essentially realised by a given class - notice that in such a case , a single abstention at an instance x determines what value should be played there in the long run . Our interest however lies in the situation where this data can not be represented in such a way , and such strategies are not viable since the labels may be noisy . Our work thus generalises the KWIK setting to non-realisable data , and to situations wherein abstention is a valid long-term action , as motivated in the introduction , by studying behaviour against competitors that may abstain.1 While Szita and Szepesvári have extended the KWIK formulation to the agnostic case in a regression setting [ SS11 ] , this work also focuses of limiting the number of abstentions to be finite rather than 1The KWIK model also bears other significant differences . It posits an input parameter ε , and requires that the learner either abstains , or produces an ε-accurate response . A notion of competitor is not invoked , and rather than studying regret , the number of abstentions needed to achieve this ε-accuracy is studied . long-run abstentions . Concretely it is assumed that Yt = g ( Xt ) + noise , for some function g , and the learner knows a class H , and a bound ∆ such some h ∈ H is ∆-close to g ( in an appropriate norm ) . Using the knowledge of ∆ , they describe schemes that have limited abstention , but at the cost of mistakes , by producing responses Ŷt that are up to ( 2 + o ( 1 ) ) ∆ separated from Yt . In contrast , in our formulation , contexts Xt for which no function inH can represent the ground truth g well would always be abstained upon . In addition to this work , trade-offs between mistakes and abstentions in a relaxed version of the KWIK framework have been considered [ ZC16 ; SZB10 ; DZ13 ] , and in particular the agnostic case has been explored by Zhang and Chaudhuri [ ZC16 ] , but unlike our situation this relaxed KWIK model requires full-feedback to be available whether or not the learner abstains . Neu and Zhivotovskiy [ NZ20 ] also work in this relaxed model , and show that when comparing the standard loss of a non-abstaining classifier against the Chow loss of an abstaining learner , regrets independent of time can be obtained . Due to the limited feedback , our setting is related to partial-monitoring [ LS20 , Ch . 37 ] . Viewing actions as choices over functions , our setting has feedback graphs [ MS11 ] that connect abstaining actions to every other action and themselves . The novelty with respect to partial-monitoring arises from the fact that we individually control two notions of losses , rather than a single one . It ’ s unclear how to apply the generic partial-monitoring setup to this situation - indeed , naïvely , our game is only weakly observable in the sense of Alon et al . [ ACDK15 ] , and one would expect Ω ( T 2/3 ) regrets , while we can control both mistakes and excess abstention to Õ ( √ T ) . A limited feedback setting where two ‘ losses ’ are individually controlled is label-efficient prediction [ CLS05 ] , where a learner must query in order to get feedback . However , in our setting , abstentions are both a way to gather feedback , and also necessary to prevent mistakes . That is , our competitor may abstain regularly , but makes few mistakes , while in this prior work the competitor does not abstain , but may make many mistakes . The resulting scenario is both qualitatively and quantitatively distinct , e.g . in label-efficient prediction , the smallest symmetric rate of number of queries and excess mistakes is again Θ ( T 2/3 ) . 2 Setting , and Problem Formulation . Setup Let X be a feature space , Y a finite set of labels , and F a finite class of selective classifiers , which are Y ∪ { ⊥ } valued . For simplicity , we assume that F contains the all abstaining classifier ( i.e . the function f⊥ such that ∀x , f⊥ ( x ) = ⊥ ) . We will denote |F| = N . The setting may be described as a game between a learner and an adversary ( or more prosaically , a data generating mechanism ) proceeding in T rounds . Also for simplicity , we will assume that T is known to both the learner and the adversary in advance . The objects in this game are the context process , Xt ∈ X , the label process Yt ∈ Y , the action process Ŷt ∈ Y ∪ { ⊥ } and the feedback process Zt ∈ Y ∪ { ∗ } , where ∗ 6∈ Y is a trivial symbol . The information sets of the adversary and learner up to the tth round are respectively H At−1 : = { ( Xs , Ys , Ŷs ) : s < t } , and H Lt−1 : = { ( Xs , Ŷs , Zs ) : s < t } . The Game For each round t ∈ [ 1 : T ] , the adversary produces a context and a label ( Xt , Yt ) on the basis its history H At−1 . The learner observes only the context , Xt , and on the basis of this and its history H Lt−1 , produces an action Ŷt . We will say that this action is an abstention if Ŷt = ⊥ , and that it is a prediction otherwise . If the action was an abstention , set Zt = Yt , and otherwise to ∗ . The learner then observes Zt , and the round concludes . Notice that since Zt is a deterministic function of Yt and Ŷt , and since the adversary observes both , H Lt−1 can be determinstically generated from H A t−1 . Due to the same reason , Ŷt and Yt are conditionally independent given ( Xt , H At−1 ) . Adversaries are characterised by a sequence of conditional laws on ( Xt , Yt ) given H At−1 ( and T , F ) . In the following we will explicitly consider two classes of such laws : • Stochastic Adversary : ( Xt , Yt ) are drawn according to a fixed law , P , unknown to the learner , independently of H At−1 . • Adaptive Adversary : ( Xt , Yt ) are arbitrary random variables with H At−1-measurable laws . We will denote a generic class of adversaries as C . Performance Metrics The two principal quantities of interest are the number of mistakes made by the learner , and the number of times it has abstained . We will denote these as MT : = ∑ t≤T 1 { Ŷt 6∈ { ⊥ , Yt } } , and AT : = ∑ t≤T 1 { Ŷt = ⊥ } . As previously discussed , the performance of a learner is measured in terms of regret with respect to the best-in-hindsight abstaining classifier from F that makes no mistakes , that is f∗ ∈ arg min f∈F ∑ t≤T 1 { f ( Xt ) = ⊥ } s.t . ∑ t≤T 1 { f ( Xt ) 6∈ { ⊥ , Yt } } = 0 . Note that such an f∗ is always realised , since the class is finite , and since it contains the all abstaining classifier . Let A∗T : = ∑ t≤T 1 { f∗ ( Xt ) = ⊥ } denote the value of the minimum above . The principal metrics of interest to us are the abstention regret AT −A∗T , and the total mistakes MT . Solution Concept The two performance metrics naturally involve a tradeoff - for instance , making some mistakes may allow a learner to drastically reduce its abstention regret to the point that it is negative . We pursue the trade-off between the worst possible behaviour of either regret . Definition ( Regret Achievability ) For functions ϕ , ψ : N2 → R , we say that expected regret bounds of ( ϕ , ψ ) are achievable against a class of adversaries C if there exists a learner such that for every adversary in C , E [ AT −A∗T ] ≤ ϕ ( T , N ) and E [ MT ] ≤ ψ ( T , N ) . As is common , we are interested in the growth rates of achievable bounds with T . We thus define Definition ( Achievable rates ) we say that asymptotic expected-regret rates of ( α , µ ) ∈ [ 0 , 1 ] 2 are achievable against a class of adversaries C if an expected regret bound of ( ϕ , ψ ) can be achieved against it for functions ϕ , ψ , said to be witnesses for the rate , such that lim sup T→∞ logϕ ( T , N ) log T ≤ α and lim sup T→∞ logψ ( T , N ) log T ≤ µ . Notice that if ( α , µ ) is an achievable rate , so is ( α′ , δ′ ) for α′ ≥ α , δ′ ≥ δ . As a result , the lower boundary of the set of achievable rates is well defined , and we will refer to this as the Pareto frontier of achievable rates . This is equivalently characterised by the function α ( µ ) : = inf { α : ( α , µ ) is an achievable rate } . This is well defined since ∀µ , ( 1 , µ ) is achievable by always abstaining .

Summary: The authors propose online selective classification and explore strategies with a goal to make the least number of mistakes and abstain the fewest number of times. Concretely, they explore the adversarial case and the stochastic case. They argue that the regret bounds in the ‘versioned uniform explorer’ (VUE) scheme in the adversarial case can be improved logarithmically in N in the stochastic setting by using the VUE-PROD scheme.