Learning with Labeling Induced Abstentions

1 Introduction . In this paper , we consider a system that relies on automated predictions made by a machine learning model . We assume this system has a limited budget for requesting ground-truth labels ( e.g . from a domain expert ) . In practice , such request can be used , among other purposes , to gather additional training data for the machine learning model . If the system asks for a label for an example , then it no longer needs the model ’ s prediction for that particular example . Thus , the pattern of label queries effectively defines the distribution that the model will learn with and predict on . Take for example a large-scale video-hosting website . The website wants to automatically detect videos that violate its community guidelines . In order to acquire labels for this task , some of these videos are evaluated by a finite pool of human reviewers . There are two consequences of this evaluation . Firstly , the reviewer provides a training label for the model . Secondly , the human intervention makes it so that the machine learning algorithm is no longer tasked with making predictions on these examples . As a result , the goal is then optimize the the model ’ s performance in the domain where it will be executed . We take the perspective of a system designer who wants to understand ( and optimize for ) the performance of the automated system on the examples that it will be asked to evaluate . Let r : X ! { 0 , 1 } be a rule governing whether the system requests a label for x 2 X . Let h : X ! R be a hypothesis describing the automated system . We seek to minimize E [ L ( h ( x ) , y ) | r ( x ) = 0 ] , where L is a loss function . We can think of this setting as combining the objective studied in abstention learning ( minimizing E [ L ( h ( x ) , y ) | r ( x ) = 0 ] ) with the feedback model studied in active learning ( labels are only available when r ( x ) = 1 ) . We introduce a new framework , which we call dual purpose learning framework , that combines these elements . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We first analyze the proper dual purpose labeling framework in an online setting , which proceeds as follows . At the start of each round t , the learner selects both a requester function rt from some class R⇢ , where ⇢ is an upper bound on the request-rate of functions in this class , and a hypothesis ht from some class H using all the feedback available from the past . The learner ’ s expected loss on this round is given by E [ L ( ht ( x ) , y ) | rt ( x ) = 0 ] . The function rt also determines the feedback available to the algorithm . If rt ( x ) = 1 , the learner observes ( x , y ) , which was drawn i.i.d . from an unknown distribution , and otherwise , only x is revealed and y is censored . The goal of the learner is to compete with the optimal choice of hypothesis and requester by minimizing the excess loss LH , R⇢ ( ht , rt ) = E [ L ( ht ( x ) , y ) | rt ( x ) = 0 ] inf ( h⇤ , r⇤ ) 2H⇥R⇢ E [ L ( h⇤ ( x ) , y ) | r⇤ ( x ) = 0 ] . Our first main result is the surprising fact that , under mild assumptions , bounds on the excess loss that match the O ( 1/ p t ) generalization rates of full-feedback passive learning are not possible in the proper dual purpose labeling framework . This lower bound also suggests a relaxation of the proper dual purpose labeling framework , which we call improper dual purpose labeling . In the improper setting , the learner is still interested in learning ht , rt that minimize LH , R⇢ . However , the learner is allowed to request labels using a more powerful requester class than R⇢ during training . As in classical PAC-learning results , improper learning allows the circumvention of the impossibility result . As a practical matter , the improper setting is useful when the system designer is willing to spend more resources during training . In our motivating example , the designer of the abuse-detection system might be willing to implement a more complicated system during training , which might include a larger budget ( in dollars or man-power ) for human intervention . However , after a time horizon T , training stops , and the designer commits to some hT , rT for rT 2 R⇢ . From then on , examples satisfyng rT ( x ) = 0 are routed to hT , and thus , we wish to characterize LH , R⇢ ( hT , rT ) . In the improper setting , we demonstrate that IWAL , an algorithm from the active learning literature [ Beygelzimer et al. , 2009 ] , can be adapted to our setting into an algorithm which we call DPL-IWAL . Since our objective is no longer an expectation of some loss function , but rather , the conditional expectation evaluated on the event that r ( x ) = 0 , IWAL ’ s standard analysis does not apply . A key technical hurdle is proving that estimates of this conditional loss concentrate at the right rate in order to attain generalization guarantees for the pair ( h , r ) returned by DPL-IWAL . We show that over a time horizon T , DPL-IWAL algorithm requests O ( ( ⇢ + ⌘ ) T ) examples where ⌘ = min ( h , r ) 2H⇥R⇢ E [ L ( h ( x ) , y ) | r ( x ) = 0 ] is the optimum value of our objective . At the same time our main lower bound demonstrates that ⌦ ( ( ⇢+ ⌘ ) T ) requests are in fact necessary to compete with the best policy in H⇥R⇢ . Finally , we conduct a thorough exploration of these techniques on a number of datasets . We first undertake an ablation study , using a finite hypothesis class , showing that DPL-IWAL outperforms baselines that either ignore the active learning or abstention learning aspects of the problem . DPLIWAL is not computationally efficient to implement since just like IWAL , it maintains a version space , that is a set of candidate hypotheses , which is non-trivial to optimize over in general . We therefore also conduct a number of experiments with an efficient heuristic inspired by our results using continuous hypothesis classes , outperforming natural baselines including margin sampling [ Lewis and Gale , 1994 , Balcan et al. , 2007 ] , which admits state-of-the-art performance for active learning problems in practice [ Yang and Loog , 2016 , Mussmann and Liang , 2018 , Chuang et al. , 2019 ] . 1.1 Related Work . Our setting encompasses an objective considered in abstention learning ( sometimes called selective classification ) , where the learner controls its evaluation region , that is the region of the domain where the learner is evaluated , by abstaining on the complement . In our setting , the evaluation region is where the learner does not request a label and the complementing abstention region is where the learner makes a request . At the same time , our setting considers the feedback mechanism from active learning . These are tied in a specific way in our framework : feedback is only available on the abstention ( i.e . requesting ) region and no information is revealed about the evaluation region . Neither an abstention algorithm nor a standard active learning algorithm would work in our setting and the algorithms in these settings can lead to incorrect solutions ( see Appendix C ) . Nevertheless , we survey some of the relevant literature . Learning with abstention was first studied by Chow [ 1957 , 1970 ] for specific practical applications . Subsequently , several authors analyzed algorithms [ Bartlett and Wegkamp , 2008 , Grandvalet et al. , 2008 , Yuan and Wegkamp , 2010 , Yuang and Wegkamp , 2011 ] for this setting with an emphasis on developing margin-based rules for abstention . Along a different line of work , El-Yaniv and Wiener [ 2010 , 2011 ] analyze the theoretical trade-off between the coverage of an abstention function and a classifier ’ s performance when not abstaining . All these works share in common the assumption that the learner has offline access to fully labeled samples , unlike the setting considered in this work . A more recent line of work Cortes et al . [ 2016a , b , 2018 ] considers a setting where the algorithms learn over two classes of functions , a hypothesis class and an abstention class . These work either assume full feedback is always available ( e.g . Cortes et al . [ 2016a , b ] ) , or like in Cortes et al . [ 2018 ] , the feedback is only available in the evaluation region , which is the exact opposite feedback mechanism than that of our setting since we receive feedback in the abstention region . Similarly , Shekhar et al . [ 2021 ] consider a setting where feedback is always available . In the active learning literature , several authors focus on analyzing margin-based active learning which requests the labels for points close to a learned model classification surface [ Dasgupta et al. , 2005 , Balcan et al. , 2007 , Balcan and Long , 2013 , Awasthi et al. , 2014 , 2015 , Zhang , 2018 , Huang et al. , 2019 , Zhang et al. , 2020 ] . Other algorithms admit generalization guarantees on the same order as passive learning while proving that their algorithm ’ s label complexity , i.e . the number of points requested during learning , is bounded by a favorable rate . Beygelzimer et al . [ 2009 ] derived an algorithm , called IWAL , for general loss functions with strong theoretical guarantees . We wish to use this algorithm for our setting with the abstention learning objective . However , the loss function we consider is in fact a conditional expectation evaluated on an event , which is not amenable to standard IWAL . Our analysis and corresponding algorithm , DPL-IWAL , describes how to apply IWAL to such a setting . Finally , ideas from the abstention framework have been applied to the active learning previously . Zhang and Chaudhuri [ 2014 ] used confidence-based predictors as a subroutine of an active learning algorithm . El-Yaniv and Wiener [ 2012 ] applied an abstention strategy from El-Yaniv and Wiener [ 2010 ] to the CAL algorithm of Cohn et al . [ 1994 ] proving theoretical guarantees , but only under specific model and distributional assumptions , which we do not make in our setting . 2 Setting and Preliminaries . In the dual purpose labeling framework , a learner is given a hypothesis class H with finite VCdimension d , and a class of deterministic requester functions R ⇢ { X ! { 0 , 1 } } . Nature fixes a distribution D over X ⇥ Y , unknown to the learner . Given the marginal over X , and ⇢ > 0 , we denote R⇢ ⇢ R as the subset of R with bounded request-rate Ex [ r ( x ) = 1 ] ⇢ for all r 2 R⇢ . Ex [ r ( x ) = 1 ] can be well estimated using unlabeled data , which is generally readily available , allowing a bound on request-rate to be enforced in practice . The interaction between the learner and nature proceeds through a sequence of rounds t. The learner first selects ( ht , rt ) 2 H⇥R⇢ as a function of the past . Nature then draws an independent sample ( xt , yt ) from D , where xt is revealed to the learner . If rt ( xt ) = 1 , the learner additionally observes yt , but the performance of ht is not evaluated . If rt ( xt ) = 0 , the learner does not observe yt , but the performance of ht is evaluated . Thus , the choice of rt serves dual purposes : it determines whether the algorithm receives feedback , and whether its performance will be evaluated . Formalizing this further , we suppose that the learner is given a loss function L : Y ⇥ Y ! R. The learner seeks to output ht that generalizes well on the region where rt dictates that ht should be evaluated . We therefore seek to bound the ( conditional ) excess loss LH , R⇢ ( ht , rt ) = E [ L ( ht ( x ) , y ) | rt ( x ) = 0 ] inf ( h⇤ , r⇤ ) 2H⇥R⇢ E [ L ( h⇤ ( x ) , y ) | r⇤ ( x ) = 0 ] . In particular we seek O ( 1/ p t ) bounds on LH , R⇢ ( ht , rt ) , matching the generalization rate of full-feedback passive learning . The conditional loss of the best pair in H⇥R⇢ plays an important role in our lower bounds , and so it is useful to define ⌘ = inf ( h⇤ , r⇤ ) 2H⇥R⇢ Ex , y [ L ( h ⇤ ( x ) , y ) | r⇤ ( x ) = 0 ] . Finally , we call this the proper dual purpose framework since the algorithm is attempting to generalize well with respect to the class H⇥R⇢ and labels are generated according to functions in R⇢ . In the subsequent section , we will see that O ( 1/ p t ) bounds on LH , R⇢ ( ht , rt ) are impossible in general . This motivates the improper dual purpose framework as introduced in the following section .

Summary: Summary: This paper sets and explores relations between learning and labeling. The article develops a new method requesting labeling combining abstention and active learning. The authors develop a new method and detail their corresponding algorithm DPL-IWAL. They also provide asymptotic statistical guarantees concerning their approach to show the efficacy of their approach (lower and upper bounds).