Neural Active Learning with Performance Guarantees

1 Introduction . Supervised learning is a fundamental paradigm in machine learning and is at the core of modern breakthroughs in deep learning [ 29 ] . A machine learning system trained via supervised learning requires access to labeled data collected via recruiting human experts , crowdsourcing , or running expensive experiments . Furthermore , as the complexity of current deep learning architectures grows , their requirement for labeled data increases significantly . The area of active learning aims to reduce this data requirement by studying the design of algorithms that can learn and generalize from a small carefully chosen subset of the training data [ 13 , 40 ] . The two common formulations of active learning are pool based active learning , and sequential ( or streaming ) active learning . In the pool based setting [ 30 ] , the learning algorithm has access to a large unlabeled set of data points , and the algorithm can ask for a subset of the data to be labeled . In contrast , in the sequential setting , data points arrive in a streaming manner , either adversarially or drawn i.i.d . from a distribution , and the algorithm must decide whether to query the label of a given point or not [ 14 ] . From a theoretical perspective , active learning has typically been studied under models inspired by the probably approximately correct ( PAC ) model of learning [ 41 ] . Here one assumes that there is a pre-specified classH of functions such that the target function mapping examples to their labels 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . either lies in H or has a good approximation inside the class . Given access to unlabeled samples generated i.i.d . from the distribution , the goal is to query for a small number of labels and produce a hypothesis of low error . In the parametric setting , namely , when the class of functionsH has finite VC-dimension ( or finite disagreement coefficient ) [ 21 ] , the rate of convergence of active learning , i.e. , the rate of decay of the regret as a function of the number of label queries ( N ) , is of the form ν N−1/2 + e− √ N , where ν is the population loss of the best function in class H. This simple finding shows that active learning behaves like passive learning when ν > 0 , while very fast rates can only be achieved under low noise ( ν ≈ 0 ) conditions . This has been worked out in , e.g. , [ 19 , 15 , 5 , 4 , 6 , 38 ] . While the parametric setting comes with methodological advantages , the above shows that in order to unleash the true power of active learning , two properties are desirable : ( 1 ) A better interplay between the input distribution and the label noise and , ( 2 ) a departure from the parametric setting leading us to consider wider classes of functions ( so as to reduce the population loss ν to close to 0 ) . To address the above , there has also been considerable theoretical work in recent years on non-parametric active learning [ 10 , 33 , 31 ] . However , these approaches suffer from the curse of dimensionality and do not lead to computationally efficient algorithms . A popular approach that has been explored empirically in recent works is to use Deep Neural Networks ( DNNs ) to perform active learning ( e.g. , [ 37 , 26 , 39 , 3 , 44 ] ) . While these works empirically demonstrate the power of the DNN-based approach to active learning , they do not come with provable guarantees . The above discussion raises the following question : Is provable and computationally efficient active learning possible in non-parametric settings ? We answer the above question in the affirmative by providing the first , to the best of our knowledge , computationally efficient algorithm for active learning based on Deep Neural Networks . Similar to non-parametric active learning , we avoid fixing a function class a-priori . However , in order to achieve computational efficiency , we instead propose to use over-parameterized DNNs , where the amount of over-parameterization depends on the input data at hand . We work in the sequential setting , and propose a simple active learning algorithm that forms an uncertainty estimate for the current data point based on the output of a DNN , followed by a gradient descent step to update the network parameters if the data point is queried . We show that under standard low-noise assumptions [ 32 ] our proposed algorithm achieves fast rates of convergence . In order to analyze our algorithm , we use tools from the theory of Neural Tangent Kernel ( NTK ) approximation [ 24 , 2 , 18 ] that allows us to analyze the dynamics of gradient descent by considering a linearization of the network around random initialization . Since we study the non-parametric regime , the convergence rates of our algorithm depend on a data-dependent complexity term that is expected to be small in practical settings , but could be very large in worst-case scenarios . Furthermore , the algorithm itself needs an estimate of complexity term in order to form accurate uncertainty estimates . We show that one can automatically adapt to the magnitude of the unknown complexity term by designing a novel model selection algorithm inspired by recent works in model selection in multiarmed bandit settings [ 36 , 35 ] . Yet , several new insights are needed to ensure that the model selection algorithm can simultaneously achieve low generalization error without spending a significant amount of budget on label queries . 2 Preliminaries and Notation . Let X denote the input space , Y the output space , and D an unknown distribution over X × Y . We denote the corresponding random variables by x and y . We also denote by DX the marginal distribution of D over X , and by DY|x0 the conditional distribution of random variable y given x = x0 . Moreover , given a function f ( sometimes called a hypothesis or a model ) mapping X to Y , the conditional population loss ( often referred to as conditional risk ) of f is denoted by L ( f |x ) , and defined as L ( f |x ) = Ey∼DY|x [ ` ( f ( x ) , y ) |x ] , where ` : Y × Y → [ 0 , 1 ] is a loss function . For ease of presentation , we restrict to a binary classification setting with 0-1 loss , whence Y = { −1 , +1 } , and ` ( a , y ) = 1 { a 6= y } ∈ { 0 , 1 } , 1 { · } being the indicator function of the predicate at argument . When clear from the surrounding context , we will omit subscripts like “ y ∼ DY|x '' from probabilities and expectations . We investigate a non-parametric setting of active learning where the conditional distribution of y given x is defined through an unknown function h : X 2 → [ 0 , 1 ] such that P ( y = 1 |x ) = h ( ( x , 0 ) ) P ( y = −1 |x ) = h ( ( 0 , x ) ) , ( 1 ) where 0 ∈ X , ( x1 , x2 ) denotes the concatenation ( or pairing ) of the two instances x1 and x2 ( so that ( x , 0 ) and ( 0 , x ) are in X 2 ) and , for all x ∈ X we have h ( ( x , 0 ) ) + h ( ( 0 , x ) ) = 1 . We make no explicit assumptions on h , other than its well-behavedness w.r.t . the data { xt } Tt=1 at hand through the formalism of Neural Tangent Kernels ( NTK ) – see below . As a simple example , in the linear case , X is the d-dimensional unit ball , h ( · , · ) is parametrized by an unknown unit vector θ ∈ Rd , and h ( ( x1 , x2 ) ) = 1+〈 ( θ , −θ ) , ( x1 , x2 ) 〉 2 , so that h ( ( x , 0 ) ) = 1+〈θ , x〉 2 and h ( ( 0 , x ) ) = 1−〈θ , x〉 2 , where 〈· , ·〉 is the usual dot product in Rd . We consider a streaming setting of active learning where , at each round t ∈ [ T ] = { 1 , . . . , T } , a pair ( xt , yt ) ∈ X × Y is drawn i.i.d . from D. The learning algorithm receives as input only xt , and is compelled to both issue a prediction at for yt and , at the same time , decide on-the-fly whether or not to observe yt . These decisions can only be based on past observations . Let Et denote the conditional expectation E [ · | ( x1 , y1 ) . . . , ( xt−1 , yt−1 ) , xt ] , and we introduce the shorthand xt , a = { ( xt , 0 ) if a = 1 ( 0 , xt ) if a = −1 . Notice that with this notation E [ ` ( a , yt ) |xt ] = 1− h ( xt , a ) , for all a ∈ Y . We quantify the accuracy of the learner ’ s predictions through its ( pseudo ) regret , defined as RT = T∑ t=1 ( Et [ ` ( at , yt ) |xt ] − E [ ` ( a∗t , yt ) |xt ] ) = T∑ t=1 ( h ( xt , a∗t ) − h ( xt , at ) ) , where a∗t is the Bayesian-optimal classifier on instance xt , that is , a ∗ t = arg maxa∈Y h ( xt , a ) . Additionally , we are interested in bounding the number of labels NT the algorithm decides to request . Our goal is to simultaneously bound RT and NT with high probability over the generation of the sample { ( xt , yt ) } t=1 , ... , T . Throughout this work , we consider the following common low-noise condition on the marginal distributionDX ( Mammen-Tsybakov low noise condition [ 32 ] ) : There exist absolute constants c > 0 , and α ≥ 0 such that for all ∈ ( 0 , 1/2 ) we have P ( |h ( ( x , 0 ) ) − 1 2 | < ) ≤ c α . In particular , α =∞ gives the so-called hard margin condition P ( |h ( ( x , 0 ) ) − 12 | < ) = 0. while , at the opposite extreme , exponent α = 0 ( and c = 1 ) results in no assumptions whatsoever on DX . For simplicity , we shall assume throughout that the above low-noise condition holds for1 c = 1 . Our techniques are inspired by the recent work [ 45 ] from which we also borrow some notation . We are learning the class of functions { h } by means of fully connected neural networks f ( x , θ ) = √ mWnσ ( ... σ ( W1x ) ) , where σ is a ReLU activation function σ ( x ) = max { 0 , x } , m is the width of the network and n ≥ 2 is its depth . In the above , θ ∈ Rp collectively denotes the set of weights { W1 , W2 , . . . , Wn } of the network , where p = m+ 2md+m2 ( n− 2 ) is their number , and the input x at training time should be thought of as some xt , a ∈ X 2 . With any depth-n network and data points { xt , a } t=1 , ... , T , a=±1 we associate a depth-n NTK matrix as follows [ 24 ] . First , rename { xt , a } t=1 , ... , T , a=±1 as { x ( i ) } i=1 , ... ,2T . Then define matrices H̃ ( 1 ) = [ H ( 1 ) i , j ] 2T×2T i , j=1 Σ ( 1 ) = [ Σ ( 1 ) i , j ] 2T×2T i , j=1 with H ( 1 ) i , j = Σ ( 1 ) i , j = 〈x ( i ) , x ( j ) 〉 , and then , for any k ≤ n and i , j = 1 , . . . , 2T , introduce the bivariate covariance matrix A ( k ) i , j = [ Σ ( k ) i , i Σ ( k ) i , j Σ ( k ) i , j Σ ( k ) j , j ] 1A more general formulation requires the above to hold only for ≤ 0 , where 0 ∈ ( 0 , 1/2 ) is a third parameter . We shall omit this extra parameter from our presentation . by which we recursively define Σ ( k+1 ) i , j = 2E ( u , v ) ∼N ( 0 , A ( k ) i , j ) [ σ ( u ) σ ( v ) ] and H̃ ( k+1 ) i , j = 2H̃ ( k ) i , j E ( u , v ) ∼N ( 0 , A ( k ) i , j ) [ 1 { u ≥ 0 } 1 { v ≥ 0 } ] + Σ ( k+1 ) i , j . The 2T × 2T -dimensional matrix H = 12 ( H̃ ( n ) + Σ ( n ) ) is called the Neural Tangent Kernel ( NTK ) matrix of depth n ( and infinite width ) over the set of points { xt , a } t=1 , ... , T , a=±1 . The reader is referred to [ 24 ] for more details on NTK . In order to avoid heavy notation , we assume ||xt|| = 1 for all t. Matrix H is positive semi-definite by construction but , as is customary in the NTK literature ( e.g. , [ 2 , 9 , 17 ] ) , we assume it is actually positive definite ( hence invertible ) with smallest eigenvalue λ0 > 0 . This is a mild assumption that can be shown to hold if no two vectors xt are aligned to each other . We measure the complexity of the function h at hand in a way similar to [ 45 ] . Using the same rearrangement of { xt , a } t=1 , ... , T , a=±1 into { x ( i ) } i=1 , ... ,2T as above , let h be the 2T -dimensional ( column ) vector whose i-th component is h ( x ( i ) ) . Then , we define the complexity ST , n ( h ) of h over { xt , a } t=1 , ... , T , a=±1 w.r.t . an NTK of depth n as ST , n ( h ) = √ h > H−1h . Notice that this notion of ( data-dependent ) complexity is consistent with the theoretical findings of [ 2 ] , who showed that for a two-layer network the bound on the generalization performance is dominated by y > H−1y , where y is the vector of labels . Hence if y is aligned with the top eigenvectors of H the learning problem becomes easier . In our case , vector h plays the role of vector y . Also observe that S2T , n ( h ) can in general be as big as linear in T ( in which case learning becomes hopeless with our machinery ) . In the special case where h belongs to the RKHS induced by the NTK , one can upper bound ST , n ( h ) by the norm of h in the RKHS . The complexity term ST , n ( h ) is typically unknown to the learning algorithm , and it plays a central role in both regret and label complexity guarantees . Hence the algorithm needs to learn this value as well during its online functioning . Apparently , this aspect of the problem has been completely overlooked by [ 45 ] ( as well as by earlier references on contextual bandits in RKHS , like [ 12 ] ) , where a ( tight ) upper bound on ST , n ( h ) is assumed to be available in advance . We will cast the above as a model selection problem in active learning , where we adapt and largely generalize to active learning the regret balancing technique from [ 36 , 35 ] . In what follows , we use the short-hand g ( x ; θ ) = ∇θf ( x , θ ) and , for a vector g ∈ Rp and matrix Z ∈ Rp×p , we often write √ g > Zg as ||g||Z , so that ST , n ( h ) = ||h||H−1 .

Summary: NB: The original submission contains an error. The supplementary material contains the fixed version of the submission.   This paper produces the first NTK based active learning method with provable guarantees.