On Statistical Bias In Active Learning: How and When to Fix It

1 INTRODUCTION . In modern machine learning , unlabelled data can be plentiful while labelling requires scarce resources and expert attention , for example in medical imaging or scientific experimentation . A promising solution to this is active learning—picking the most informative datapoints to label that will hopefully let the model be trained in the most sample-efficient way possible ( Atlas et al. , 1990 ; Settles , 2010 ) . However , active learning has a complication . By picking the most informative labels , the acquired dataset is not drawn from the population distribution . This sampling bias , noted by e.g. , MacKay ( 1992 ) ; Dasgupta & Hsu ( 2008 ) , is worrying : key results in machine learning depend on the training data being identically and independently distributed ( i.i.d . ) samples from the population distribution . For example , we train neural networks by minimizing a Monte Carlo estimator of the population risk . If training data are actively sampled , that estimator is biased and we optimize the wrong objective . The possibility of bias in active learning has been considered by e.g. , Beygelzimer et al . ( 2009 ) ; Chu et al . ( 2011 ) ; Ganti & Gray ( 2012 ) , but the full problem is not well understood . In particular , methods that remove active learning bias have been restricted to special cases , so it has been impossible to even establish whether removing active learning bias is helpful or harmful in typical situations . To this end , we show how to remove the bias introduced by active learning with minimal changes to existing active learning methods . As a stepping stone , we build a Plain Unbiased Risk Estimator , R̃PURE , which applies a corrective weighting to actively sampled datapoints in pool-based active learning . Our Levelled Unbiased Risk Estimator , R̃LURE , builds on this and has lower variance and additional desirable finite-sample properties . We prove that both estimators are unbiased and consistent for arbitrary functions , and characterize their variance . Interestingly , we find—both theoretically and empirically—that our bias corrections can simultaneously also reduce the variance of the estimator , with these gains becoming larger for more effective acquisition strategies . We show that , in turn , these combined benefits can sometimes lead to significant improvements for both model evaluation and training . The benefits are most pronounced in underparameterized models where each datapoint affects the learned function globally . For example , in linear regression adopting our weighting allows better estimates of the parameters with less data . On the other hand , in cases where the model is overparameterized and datapoints mostly affect the learned function locally—like deep neural networks—we find that correcting active learning bias can be ineffective or even harmful during model training . Namely , even though our corrections typically produce strictly superior statistical estimators , we find that the bias from standard active learning can actually be helpful by providing a regularising effect that aids generalization . Through this , our work explains the known empirical successes of existing active learning approaches for training deep models ( Gal et al. , 2017b ; Shen et al. , 2018 ) , despite these ignoring the bias this induces . ∗Equal contribution . Corresponding author sebastian.farquhar @ cs.ox.ac.uk . To summarize , our main contributions are : 1 . We offer a formalization of the problem of statistical bias in active learning . 2 . We introduce active learning risk estimators , R̃PURE and R̃LURE , and prove both are unbiased , consistent , and with variance that can be less than the naive ( biased ) estimator . 3 . Using these , we show that active learning bias can hurt in underparameterized cases like linear regression but help in overparameterized cases like neural networks and explain why . 2 BIAS IN ACTIVE LEARNING . We begin by characterizing the bias introduced by active learning . In supervised learning , generally , we aim to find a decision rule fθ corresponding to inputs , x , and outputs , y , drawn from a population data distribution pdata ( x , y ) which , given a loss function L ( y , fθ ( x ) ) , minimizes the population risk : r = Ex , y∼pdata [ L ( y , fθ ( x ) ) ] . The population risk can not be found exactly , so instead we consider the empirical distribution for some dataset of N points drawn from the population . This gives the empirical risk : an unbiased and consistent estimator of r when the data are drawn i.i.d from pdata and are independent of θ , R̂ = 1 N ∑N n=1 L ( yn , fθ ( xn ) ) . In pool-based active learning ( Lewis & Gale , 1994 ; Settles , 2010 ) , we begin with a large unlabelled dataset , known as the pool dataset Dpool ≡ { xn|1 ≤ n ≤ N } , and sequentially pick the most useful points for which to acquire labels . The lack of most labels means we can not evaluate R̂ directly , so we use the sub-sample empirical risk evaluated using the M actively sampled labelled points : R̃ = 1 M ∑M m=1 L ( ym , fθ ( xm ) ) . ( 1 ) Though almost all active learning research uses this estimator ( see Appendix D ) , it is not an unbiased estimator of either R̂ or r when the M points are actively sampled . Under active—i.e . non–uniform— sampling the M datapoints are not drawn from the population distribution , resulting in a bias which we formally characterize in §4 . See Appendix A for a more general overview of active learning . Note an important distinction between what we will call “ statistical bias ” and “ overfitting bias. ” The bias from active learning above is a statistical bias in the sense that using R̃ biases our estimation of r , regardless of θ . As such , optimizing θ with respect to R̃ induces bias into our optimization of θ . In turn , this breaks any consistency guarantees for our learning process : if we keep M/N fixed , take M →∞ , and optimize for θ , we no longer get the optimal θ that minimizes r. Almost all work on active learning for neural networks currently ignores the issue of statistical bias . However , even without this statistical bias , indeed even if we use R̂ directly , the training process itself also creates an overfitting bias : evaluating the risk using training data induces a dependency between the data and θ . This is why we usually evaluate the risk on held-out test data when doing model selection . Dealing with overfitting bias is beyond the scope of our work as this would equate to solving the problem of generalization . The small amount of prior work which does consider statistical bias in active learning entirely ignores this overfitting bias without commenting on it . In §3-6 , we focus on statistical bias in active learning , so that we can produce estimators that are valid and consistent , and let us optimize the intended objective , not so they can miraculously close the train–test gap . From a more formal perspective , our results all assume that θ is chosen independently of the training data ; an assumption that is almost always ( implicitly ) made in the literature . This ensures our estimators form valid objectives , but also has important implications that are typically overlooked . We return to this in §7 , examining the interaction between statistical and overfitting bias . 3 UNBIASED ACTIVE LEARNING : R̃PURE AND R̃LURE . We now show how to unbiasedly estimate the risk in the form of a weighted expectation over actively sampled data points . We denote the set of actively sampled points Dtrain ≡ { ( xm , ym ) |1 ≤ m ≤M } , where ∀m : xm ∈ Dpool . We begin by building a “ plain ” unbiased risk estimator , R̃PURE , as a stepping stone—its construction is quite natural in that each term is individually an unbiased estimator of the risk . We then use it to construct a “ levelled ” unbiased risk estimator , R̃LURE , which is an unbiased and consistent estimator of the population risk just like R̃PURE , but which reweights individual terms to produce lower variance and resolve some pathologies of the first approach . Both estimators are easy to implement and have trivial compute/memory requirements . 3.1 R̃PURE : PLAIN UNBIASED RISK ESTIMATOR . For our estimators , we introduce an active sampling proposal distribution over indices rather than the more typical distribution over datapoints . This simplifies our proofs , but the two are algorithmically equivalent for pool-based active learning because of the one–to–one relationship between datapoints and indices . We define the probability mass for each index being the next to be sampled , once Dtrain contains m− 1 points , as q ( im ; i1 : m−1 , Dpool ) . Because we are learning actively , the proposal distribution depends on the indices sampled so far ( i1 : m−1 ) and the available data ( Dpool , note though that it does not depend on the labels of unsampled points ) . The only requirement on this proposal distribution for our theoretical results is that it must place non-zero probability on all of the training data : anything else necessarily introduces bias . Considerations for the acquisition proposal distribution are discussed further in §3.3 . We first present the estimator before proving the main results : R̃PURE ≡ 1 M ∑M m=1 am ; where am ≡ wmLim + 1 N ∑m−1 t=1 Lit , ( 2 ) where the loss at a point Lim ≡ L ( yim , fθ ( xim ) ) , the weights wm ≡ 1/Nq ( im ; i1 : m−1 , Dpool ) and im ∼ q ( im ; i1 : m−1 , Dpool ) . For practical implementation , R̃PURE can further be written in the following more computationally friendly form that avoids a double summation : R̃PURE = 1 M M∑ m=1 ( 1 Nq ( im ; i1 : m−1 , Dpool ) + M −m N ) Lim . ( 3 ) However , we focus on the first form for our analysis because am in ( 2 ) has some beneficial properties not shared by the weighting factors in ( 3 ) . In particular , in Appendix B.1 we prove that : Lemma 1 . The individual terms am of R̃PURE are unbiased estimators of the risk : E [ am ] = r. The motivation for the construction of am directly originates from constructing an estimator where Lemma 1 holds while only making use of the observed losses Li1 , . . . , Lim , taking care with the fact that each new proposal distribution does not have support over points that have already been acquired . Except for trivial problems , am is essentially unique in this regard ; naive importance sampling ( i.e . 1M ∑M m=1 wmLim ) does not lead to an unbiased , or even consistent , estimator . However , the overall estimator R̃PURE is not the only unbiased estimator of the risk , as we discuss in §3.2 . We can now characterize the behaviour of R̃PURE as follows ( see Appendix B.2 for proof ) Theorem 1 . R̃PURE as defined above has the properties : E [ R̃PURE ] = r , Var [ R̃PURE ] = Var [ L ( y , fθ ( x ) ) ] N + 1 M2 M∑ m=1 EDpool , i1 : m−1 [ Var [ wmLim |i1 : m−1 , Dpool ] ] . ( 4 ) Remark 1 . The first term of ( 4 ) is the variance of the loss on the whole pool , while the second term accounts for the variance originating from the active sampling itself given the pool . This second term is O ( N/M ) times larger and so will generally dominate in practice as typically M N . Armed with Theorem 1 , we can prove the consistency of R̃PURE under standard assumptions : R̃PURE converges in expectation ( i.e . its mean squared error tends to zero ) as M →∞ under the assumptions that N > M , L ( y , fθ ( x ) ) is integrable , and q ( im ; i1 : m−1 , Dpool ) is a valid proposal in the sense that it puts non-zero mass on each unlabelled datapoint . Formally , as proved in Appendix B.3 , Theorem 2 . Let α = N/M and assume that α > 1 . If E [ L ( y , fθ ( x ) ) 2 ] < ∞ and ∃β > 0 : min n∈ { 1 : N\i1 : m−1 } q ( im = n ; i1 : m−1 , Dpool ) ≥ β/N ∀N ∈ Z+ , m ≤ N , then R̃PURE converges in its L2 norm to r as M →∞ , i.e. , limM→∞ E [ ( R̃PURE − r ) 2 ] = 0 .

Summary: The authors consider the bias (in the risk) introduced by active sampling strategies with respect to the true underlying data generating distribution. They then propose two estimators of the risk -- SURE and PURE -- that are unbiased and asymptotically consistent under certain assumptions. The authors consider two toy examples where they show the existence of active learning bias, and the effect of removing this. In particular, the authors notice that while training a linear regression (based on actively acquired datapoints) with their unbiased risks improves test loss, the opposite happens for a Bayesian Neural Network (BNN). They then provide a potential explanation for this phenomenon.