Online Active Learning with Surrogate Loss Functions

1 Introduction . While unlabeled data is generated at increasing rates , supervised learning often requires large portions of this data to be annotated by human raters , which carry significant costs . Active learning seeks to significantly alleviate this problem through algorithms that attain comparable performance to passive learning ( i.e. , to full supervision ) while using considerably fewer labels . Because of the ever increasing use of labeled data in modern machine learning approaches , this calls for the development of more and more effective active learning techniques . In this paper , we are specifically interested in binary classification problems in the so-called streaming ( or online ) setting of active learning . In this setting , learning proceeds in a sequence of rounds , where in each round an unlabeled sample is received , and the learner decides on-the-fly whether or not to observe the associated binary label . Now , it is common practice , in active learning or machine learning in general , that even for binary classification problems , the loss used both for training and for evaluating statistical performance is not the zero-one loss , but some surrogate function thereof . This results in more tractable ( e.g . convex ) optimization problems in the training phase , but also in more calibrated metrics in the evaluation phase . For instance , we may be compelled to output estimates of class probabilities , say , the success rate of a given set of decisions , and thus deploy a maximum posterior probability estimator for such probabilities . Then , given a training set with labels y ∈ { ±1 } and if p̂ ( x ) is an estimate of the conditional class probability P [ y = 1 |x ] , a standard goal is to minimize the log-loss 1+y2 log 1 p̂ ( x ) + 1−y 2 log 1 1−p̂ ( x ) . In these cases , the logistic link function p̂ ( x ) = 1 1+e−h ( x ) is often adopted to model conditional class probabilities where h is some ( unknown ) function in a given hypothesis class . In this and many other situations that arise in practice , it is thus advisable to resort to surrogate loss functions for both training and evaluation . The online active learning literature contains a number of proposed algorithms that admit theoretical guarantees for both generalization error and label complexity , that is , the expected number of label requests . The theoretical analyses found in these works are either tailored to the zero-one loss or to general surrogate losses , with much more emphasis historically placed on the former . The papers that 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . analyze zero-one loss introduce novel ideas tackling important theoretical questions in active learning , but the theory does not easily extend to general loss functions – see discussion in Henneke and Yang [ 2019 ] . Nevertheless , the algorithmic solutions we propose borrow several ideas from one such work , namely [ Dasgupta et al. , 2008 ] . In the agnostic setting , Dasgupta et al . [ 2008 ] developed an algorithm for the zero-one loss based on the disagreement between functions , that carefully constructs pseudo-labels , that is , weak labels generated by the algorithm itself . We call this algorithm DHM , after the name of the three authors . For general surrogate losses , Beygelzimer et al . [ 2009 ] designed an algorithm based on constructing importance weighted predictors with theoretical guarantees in the agnostic setting . Cortes et al . [ 2019a , b , 2020 ] further developed these importance-weighted algorithms enhancing their guarantees and used them as building blocks for meta-algorithms . We refer to these algorithms as IWAL ( Importance Weighted Active Learning ) . On a different line of work , Henneke and Yang [ 2019 ] presents a theoretical analysis of an active learning algorithm that uses surrogate loss functions for binary classification . Among the theoreticallyoriented contributions in active learning , this work appears to be the closest reference to our paper when it comes to motivations . Yet , it is fair to say that the authors make the very strong assumption that the Bayes optimal function for the surrogate loss at hand lies in the hypothesis class used for learning , and they explicitly state that this realizability condition can not be relaxed without resorting to a significantly different approach . Moreover , unlike our work and that of IWAL-based algorithms , Henneke and Yang [ 2019 ] are only focusing on deriving bounds on the zero-one risk . And finally , their algorithm operates in a pool-based setting of active learning , where the whole set of unlabeled points is available to the learner , which is more flexible than the streaming setting analyzed here . In practice , the Margin Algorithm , or Uncertainty Sampling , of Lewis and Gale [ 1994 ] attains a substantial performance improvement compared to passive learning as well as other active learning algorithms , and is thus often favored by practitioners ( e.g . Schohn and Cohn [ 2000 ] , Tong and Koller [ 2001 ] , Brinker [ 2003 ] , Culotta and McCallum [ 2005 ] , Joshi et al . [ 2009 ] , Mussmann and Liang [ 2018 ] ) . Moreover , the margin algorithm is flexible in that it can be used in both the pool and streaming setting . In the pool setting , several papers have recently tailored active learning algorithms to neural networks , always comparing their algorithm ’ s empirical performance to that of the margin algorithm [ Geifman and El-Yaniv , 2017 , Gal et al. , 2017 , Savarese et al. , 2018 , Ducoffe and Precioso , 2018 , Ash et al. , 2020 , Moon et al. , 2020 , Schroder and Niekler , 2020 ] . Overall , the margin algorithm is often found to be a competitive contender to this new collection of practical active learning algorithms . Additionally , two extensive empirical studies found that the margin algorithm outperforms many recent active learning algorithms [ Yang and Loog , 2016 , Chuang et al. , 2019 ] . On the theoretical side , several authors developed active learning algorithms with theoretical guarantees for specific classes of functions ( e.g. , linear separators ) based on sampling along the margin [ Dasgupta et al. , 2005 , Balcan et al. , 2007 , Balcan and Long , 2013 , Awasthi et al. , 2014 , 2015 , Zhang , 2018 , Zhang et al. , 2020 ] , but the guarantees of these margin-based algorithms only hold under strong assumptions on the realizability of the problem . Our goal is thus to derive an algorithm that admits theoretical guarantees without relying on realizability assumptions and that , at the same time , attains in practice a performance that is better than that of the margin algorithm in the streaming setting . 1.1 Contributions . In this paper , we introduce a novel active learning algorithm , called ALPS ( Actively Learning over Pseudo-labels for Surrogate losses ) , in the streaming setting for binary classification tasks that trains a model by optimizing a surrogate loss on the joint set of labeled and self-constructed pseudo-labeled points ( Section 3 ) . Crucially , we prove that ALPS admits theoretical guarantees with respect to the surrogate loss in the general agnostic setting while at the same time surpassing the training performance of the margin algorithm as well as other baselines . More concretely , we prove theoretical guarantees for ALPS in terms of both generalization and label complexity , under assumptions on the interplay between the function class and the data distribution which are not captured by traditional low noise assumptions often formulated in active learning ( Section 4 ) . We show that ALPS is able to leverage pseudo-labels in a principled manner even though the loss function of interest is not the zero-one loss , but a surrogate loss thereof . We then complement our theoretical findings with a thorough experimental investigation on 18 real-world datasets using a class of neural networks , Multi-layer Perceptrons . We test our algorithm against the IWAL algorithm of Beygelzimer et al . [ 2009 ] since it admits theoretical guarantees for general loss functions , and against passive learning since we want to quantify the improvement over supervised methods . Additionally , we generate as competitors a pool of margin-based uncertainty samplers by varying their label request threshold , and show that on almost all datasets our algorithm outperforms the ex-post best among such algorithms ( Section 5 ) . This paper follows the general research agenda of reducing the gap between theory and practice , which has been widening at a faster rate in the recent active learning literature . As outlined above , we try to bridge this disconnect by deriving an algorithm that not only works well in practice , but also admits solid theoretical guarantees . In doing so , we also introduce novel algorithmic ideas , which could independently lead to new directions of research . 1.2 Main Ideas . The main mechanism behind ALPS is to query the label of the current instance based on the disagreement between hypotheses and to construct pseudo-labels for the non-queried instances . The algorithm then trains a model on the joint set of labeled and pseudo-labeled data . The sample used for training will be unbiased with respect to marginal distribution on the instance space , but label noise will be introduced whenever the pseudo-label differs from the true ( unrevealed ) label . This noise affects both the generalization ability of the algorithm and its label complexity . Controlling this noise is not only crucial , but also quite challenging , for it relies on assessing the expected reliability of the pseudo-labels . In order to control this noise , we introduce the novel idea of using requester functions : ALPS learns to query labels on instances with unreliable ( that is , potentially noisy ) pseudo-labels by using a requester function chosen to minimize an importance-weighted estimate of the noise . For the sake of our results , the requester functions are assumed to lie in some generic function class . One natural example of this class is that of margin-based functions , meaning functions that request along the margin of a given prediction model . In this case , our algorithm is related to the margin algorithm but , unlike the theoretical guarantees for the margin algorithm ( e.g . Balcan et al . [ 2007 ] ) , we do not rely on specific distributional assumptions on the instance space . The requester functions are reminiscent of the abstention functions introduced in Cortes et al . [ 2016 ] , but serve an entirely different purpose in active learning . ALPS can be seen as an extension of the DHM algorithm [ Dasgupta et al. , 2008 ] to surrogate loss functions , which seeks to address the open question posed by Dasgupta et al . [ 2008 ] of whether there are active learning algorithms that only require solving tractable optimization problems in agnostic scenarios . At the same time , these two algorithms differ on how they treat the label noise introduced by pseudo-labeling . In DHM , the label noise can be easily bypassed by a simple property of the zero-one loss . In contrast , for general surrogate loss functions , this noise is unavoidable , and if not controlled as is done in the ALPS algorithm , the learned hypothesis can be arbitrarily far from the best-in-class . 2 Preliminaries and Notation . We consider an active learning framework in the streaming setting for binary classification . Learning proceeds in a sequence of rounds . In each round n , the learner receives from the environment an instance ( or feature vector ) xn from an instance ( or feature ) space X . Based on what has been observed so far , the learner can then decide whether or not to request the true label yn associated with xn . Label yn is assumed to lie in the output space Y = { ±1 } . The pairs ( x1 , y1 ) , ( x2 , y2 ) , . . . are drawn i.i.d . from a joint distribution D over X× Y . In order to evaluate the statistical performance of the learner , we consider any bounded surrogate loss function ` : R × Y → [ 0 , B ] that upper bounds the zero-one loss , and let ` B ( · , · ) = ` ( · , · ) /B be a [ 0 , 1 ] -normalized version of this loss . Let H denote a hypothesis class of functions h : X→ R. For simplicity of exposition , we assume that H is a finite class , but our analysis can be extended to classes with finite VC dimension by standard covering arguments . The true ( expected ) risk of hypothesis h ∈ H is defined as err ( h ) = E ( x , y ) ∼D [ ` ( h ( x ) , y ) ] and h∗ = argminh∈H err ( h ) denotes the best-in-class hypothesis . Moreover , we denote by err ( h , Z ) = 1n ∑ ( x , y ) ∈Z ` ( h ( x ) , y ) the empirical risk of h over a training set Z = { ( x1 , y1 ) , ( x2 , y2 ) , . . . , ( xn , yn ) } . Given round n ≥ 1 , we denote by Tn the set of labeled data points { ( xs , ys ) } from rounds s ∈ [ n ] : = { 1 , . . . , n } where the label was requested by the learning algorithm , and denote by Sn its complement , that is , the set of labeled data points { ( xs , ys ) } from rounds s ∈ [ n ] where the label was not requested . Unlike Tn , the set Sn is not fully known to the learner . On those rounds , the learner replaces the true unobserved label ys by pseudo-label ŷs , which is a suitable proxy to ys generated by the learner itself . We denote by Ŝn the set containing data points { ( xs , ŷs ) } from rounds s ∈ [ n ] where the label is not requested , and the true label gets replaced by the corresponding pseudo-label . For notational convenience ( and unless otherwise specified ) , we set ŷs = ys in rounds s where the true label is known to the learner . Then the following two short-hands are used for empirical risk up to round n using ground-truth labels and the empirical risk up to round n using pseudo-labels : err n ( h ) = err ( h , Sn ∪ Tn ) = 1 n n∑ s=1 ` ( h ( xs ) , ys ) ; êrr n ( h ) = err ( h , Ŝn ∪ Tn ) = 1 n n∑ s=1 ` ( h ( xs ) , ŷs ) . In the sequel , we will also need the following definition of consistency of hypothesis h ∈ H . Definition 1 . We say that h ∈ H is consistent with a labeled data point ( x , y ) ∈ X×Y if sgn ( h ( x ) ) = y and , by extension , that h is consistent with a set of labeled data points Z if it is consistent with all data points in Z . Two hypotheses h , h′ ∈ H are consistent with one another on a set of ( unlabeled ) points if sgn ( h ( x ) ) = sgn ( h′ ( x ) ) for all x in that set of points . At each round n , the learner is compelled to output a hypothesis hn ∈ H , and we seek to bound its true risk , err ( hn ) , in terms of the true risk , err ( h∗ ) , of the best-in-class hypothesis h∗ , as a function of the total number of streamed points n thus far . In addition , we want a bound on the label complexity of the learner , that is , the number of labels requested by the learner during the course of training . This bound should be provably smaller than n , which is the label complexity of a passive learner observing all labels . Further ancillary definitions will be introduced in later sections .

Summary: This paper presents Actively Learning over Pseudo-Labels for Surrogate Losses (ALPS), an active learning algorithm for binary classification with surrogate losses in the streaming setting. The main idea is to generate pseudo-labels for points whose labels are not requested and train on the union of data points with ground-truth labels and those with pseudo-labels. Since the pseudo-labels may not be accurate, the authors propose requestor functions that have a final say in whether a pseudo-label is used, and in effect, enable control over the noise generated by erroneous pseudo-labeling. Under an assumption involving the requestor function class, the authors prove bounds on the generalization error and label complexity of ALPS that are roughly on the same order of prior work (e.g., IWAL).