Learning with Labeling Induced Abstentions

1 Introduction . In this paper , we consider a system that relies on automated predictions made by a machine learning model . We assume this system has a limited budget for requesting ground-truth labels ( e.g . from a domain expert ) . In practice , such request can be used , among other purposes , to gather additional training data for the machine learning model . If the system asks for a label for an example , then it no longer needs the model ’ s prediction for that particular example . Thus , the pattern of label queries effectively defines the distribution that the model will learn with and predict on . Take for example a large-scale video-hosting website . The website wants to automatically detect videos that violate its community guidelines . In order to acquire labels for this task , some of these videos are evaluated by a finite pool of human reviewers . There are two consequences of this evaluation . Firstly , the reviewer provides a training label for the model . Secondly , the human intervention makes it so that the machine learning algorithm is no longer tasked with making predictions on these examples . As a result , the goal is then optimize the the model ’ s performance in the domain where it will be executed . We take the perspective of a system designer who wants to understand ( and optimize for ) the performance of the automated system on the examples that it will be asked to evaluate . Let r : X ! { 0 , 1 } be a rule governing whether the system requests a label for x 2 X . Let h : X ! R be a hypothesis describing the automated system . We seek to minimize E [ L ( h ( x ) , y ) | r ( x ) = 0 ] , where L is a loss function . We can think of this setting as combining the objective studied in abstention learning ( minimizing E [ L ( h ( x ) , y ) | r ( x ) = 0 ] ) with the feedback model studied in active learning ( labels are only available when r ( x ) = 1 ) . We introduce a new framework , which we call dual purpose learning framework , that combines these elements . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We first analyze the proper dual purpose labeling framework in an online setting , which proceeds as follows . At the start of each round t , the learner selects both a requester function rt from some class R⇢ , where ⇢ is an upper bound on the request-rate of functions in this class , and a hypothesis ht from some class H using all the feedback available from the past . The learner ’ s expected loss on this round is given by E [ L ( ht ( x ) , y ) | rt ( x ) = 0 ] . The function rt also determines the feedback available to the algorithm . If rt ( x ) = 1 , the learner observes ( x , y ) , which was drawn i.i.d . from an unknown distribution , and otherwise , only x is revealed and y is censored . The goal of the learner is to compete with the optimal choice of hypothesis and requester by minimizing the excess loss LH , R⇢ ( ht , rt ) = E [ L ( ht ( x ) , y ) | rt ( x ) = 0 ] inf ( h⇤ , r⇤ ) 2H⇥R⇢ E [ L ( h⇤ ( x ) , y ) | r⇤ ( x ) = 0 ] . Our first main result is the surprising fact that , under mild assumptions , bounds on the excess loss that match the O ( 1/ p t ) generalization rates of full-feedback passive learning are not possible in the proper dual purpose labeling framework . This lower bound also suggests a relaxation of the proper dual purpose labeling framework , which we call improper dual purpose labeling . In the improper setting , the learner is still interested in learning ht , rt that minimize LH , R⇢ . However , the learner is allowed to request labels using a more powerful requester class than R⇢ during training . As in classical PAC-learning results , improper learning allows the circumvention of the impossibility result . As a practical matter , the improper setting is useful when the system designer is willing to spend more resources during training . In our motivating example , the designer of the abuse-detection system might be willing to implement a more complicated system during training , which might include a larger budget ( in dollars or man-power ) for human intervention . However , after a time horizon T , training stops , and the designer commits to some hT , rT for rT 2 R⇢ . From then on , examples satisfyng rT ( x ) = 0 are routed to hT , and thus , we wish to characterize LH , R⇢ ( hT , rT ) . In the improper setting , we demonstrate that IWAL , an algorithm from the active learning literature [ Beygelzimer et al. , 2009 ] , can be adapted to our setting into an algorithm which we call DPL-IWAL . Since our objective is no longer an expectation of some loss function , but rather , the conditional expectation evaluated on the event that r ( x ) = 0 , IWAL ’ s standard analysis does not apply . A key technical hurdle is proving that estimates of this conditional loss concentrate at the right rate in order to attain generalization guarantees for the pair ( h , r ) returned by DPL-IWAL . We show that over a time horizon T , DPL-IWAL algorithm requests O ( ( ⇢ + ⌘ ) T ) examples where ⌘ = min ( h , r ) 2H⇥R⇢ E [ L ( h ( x ) , y ) | r ( x ) = 0 ] is the optimum value of our objective . At the same time our main lower bound demonstrates that ⌦ ( ( ⇢+ ⌘ ) T ) requests are in fact necessary to compete with the best policy in H⇥R⇢ . Finally , we conduct a thorough exploration of these techniques on a number of datasets . We first undertake an ablation study , using a finite hypothesis class , showing that DPL-IWAL outperforms baselines that either ignore the active learning or abstention learning aspects of the problem . DPLIWAL is not computationally efficient to implement since just like IWAL , it maintains a version space , that is a set of candidate hypotheses , which is non-trivial to optimize over in general . We therefore also conduct a number of experiments with an efficient heuristic inspired by our results using continuous hypothesis classes , outperforming natural baselines including margin sampling [ Lewis and Gale , 1994 , Balcan et al. , 2007 ] , which admits state-of-the-art performance for active learning problems in practice [ Yang and Loog , 2016 , Mussmann and Liang , 2018 , Chuang et al. , 2019 ] . 1.1 Related Work . Our setting encompasses an objective considered in abstention learning ( sometimes called selective classification ) , where the learner controls its evaluation region , that is the region of the domain where the learner is evaluated , by abstaining on the complement . In our setting , the evaluation region is where the learner does not request a label and the complementing abstention region is where the learner makes a request . At the same time , our setting considers the feedback mechanism from active learning . These are tied in a specific way in our framework : feedback is only available on the abstention ( i.e . requesting ) region and no information is revealed about the evaluation region . Neither an abstention algorithm nor a standard active learning algorithm would work in our setting and the algorithms in these settings can lead to incorrect solutions ( see Appendix C ) . Nevertheless , we survey some of the relevant literature . Learning with abstention was first studied by Chow [ 1957 , 1970 ] for specific practical applications . Subsequently , several authors analyzed algorithms [ Bartlett and Wegkamp , 2008 , Grandvalet et al. , 2008 , Yuan and Wegkamp , 2010 , Yuang and Wegkamp , 2011 ] for this setting with an emphasis on developing margin-based rules for abstention . Along a different line of work , El-Yaniv and Wiener [ 2010 , 2011 ] analyze the theoretical trade-off between the coverage of an abstention function and a classifier ’ s performance when not abstaining . All these works share in common the assumption that the learner has offline access to fully labeled samples , unlike the setting considered in this work . A more recent line of work Cortes et al . [ 2016a , b , 2018 ] considers a setting where the algorithms learn over two classes of functions , a hypothesis class and an abstention class . These work either assume full feedback is always available ( e.g . Cortes et al . [ 2016a , b ] ) , or like in Cortes et al . [ 2018 ] , the feedback is only available in the evaluation region , which is the exact opposite feedback mechanism than that of our setting since we receive feedback in the abstention region . Similarly , Shekhar et al . [ 2021 ] consider a setting where feedback is always available . In the active learning literature , several authors focus on analyzing margin-based active learning which requests the labels for points close to a learned model classification surface [ Dasgupta et al. , 2005 , Balcan et al. , 2007 , Balcan and Long , 2013 , Awasthi et al. , 2014 , 2015 , Zhang , 2018 , Huang et al. , 2019 , Zhang et al. , 2020 ] . Other algorithms admit generalization guarantees on the same order as passive learning while proving that their algorithm ’ s label complexity , i.e . the number of points requested during learning , is bounded by a favorable rate . Beygelzimer et al . [ 2009 ] derived an algorithm , called IWAL , for general loss functions with strong theoretical guarantees . We wish to use this algorithm for our setting with the abstention learning objective . However , the loss function we consider is in fact a conditional expectation evaluated on an event , which is not amenable to standard IWAL . Our analysis and corresponding algorithm , DPL-IWAL , describes how to apply IWAL to such a setting . Finally , ideas from the abstention framework have been applied to the active learning previously . Zhang and Chaudhuri [ 2014 ] used confidence-based predictors as a subroutine of an active learning algorithm . El-Yaniv and Wiener [ 2012 ] applied an abstention strategy from El-Yaniv and Wiener [ 2010 ] to the CAL algorithm of Cohn et al . [ 1994 ] proving theoretical guarantees , but only under specific model and distributional assumptions , which we do not make in our setting . 2 Setting and Preliminaries . In the dual purpose labeling framework , a learner is given a hypothesis class H with finite VCdimension d , and a class of deterministic requester functions R ⇢ { X ! { 0 , 1 } } . Nature fixes a distribution D over X ⇥ Y , unknown to the learner . Given the marginal over X , and ⇢ > 0 , we denote R⇢ ⇢ R as the subset of R with bounded request-rate Ex [ r ( x ) = 1 ] ⇢ for all r 2 R⇢ . Ex [ r ( x ) = 1 ] can be well estimated using unlabeled data , which is generally readily available , allowing a bound on request-rate to be enforced in practice . The interaction between the learner and nature proceeds through a sequence of rounds t. The learner first selects ( ht , rt ) 2 H⇥R⇢ as a function of the past . Nature then draws an independent sample ( xt , yt ) from D , where xt is revealed to the learner . If rt ( xt ) = 1 , the learner additionally observes yt , but the performance of ht is not evaluated . If rt ( xt ) = 0 , the learner does not observe yt , but the performance of ht is evaluated . Thus , the choice of rt serves dual purposes : it determines whether the algorithm receives feedback , and whether its performance will be evaluated . Formalizing this further , we suppose that the learner is given a loss function L : Y ⇥ Y ! R. The learner seeks to output ht that generalizes well on the region where rt dictates that ht should be evaluated . We therefore seek to bound the ( conditional ) excess loss LH , R⇢ ( ht , rt ) = E [ L ( ht ( x ) , y ) | rt ( x ) = 0 ] inf ( h⇤ , r⇤ ) 2H⇥R⇢ E [ L ( h⇤ ( x ) , y ) | r⇤ ( x ) = 0 ] . In particular we seek O ( 1/ p t ) bounds on LH , R⇢ ( ht , rt ) , matching the generalization rate of full-feedback passive learning . The conditional loss of the best pair in H⇥R⇢ plays an important role in our lower bounds , and so it is useful to define ⌘ = inf ( h⇤ , r⇤ ) 2H⇥R⇢ Ex , y [ L ( h ⇤ ( x ) , y ) | r⇤ ( x ) = 0 ] . Finally , we call this the proper dual purpose framework since the algorithm is attempting to generalize well with respect to the class H⇥R⇢ and labels are generated according to functions in R⇢ . In the subsequent section , we will see that O ( 1/ p t ) bounds on LH , R⇢ ( ht , rt ) are impossible in general . This motivates the improper dual purpose framework as introduced in the following section .

Summary: The authors propose a new learning problem that combines aspects of active learning and learning with abstention. They formalize the setting, provide an algorithm. They also prove an upper bound on the algorithm’s label complexity and a matching lower bound for any algorithm in this setting. Experiments are presented, too.