Neural Active Learning with Performance Guarantees

1 Introduction . Supervised learning is a fundamental paradigm in machine learning and is at the core of modern breakthroughs in deep learning [ 29 ] . A machine learning system trained via supervised learning requires access to labeled data collected via recruiting human experts , crowdsourcing , or running expensive experiments . Furthermore , as the complexity of current deep learning architectures grows , their requirement for labeled data increases significantly . The area of active learning aims to reduce this data requirement by studying the design of algorithms that can learn and generalize from a small carefully chosen subset of the training data [ 13 , 40 ] . The two common formulations of active learning are pool based active learning , and sequential ( or streaming ) active learning . In the pool based setting [ 30 ] , the learning algorithm has access to a large unlabeled set of data points , and the algorithm can ask for a subset of the data to be labeled . In contrast , in the sequential setting , data points arrive in a streaming manner , either adversarially or drawn i.i.d . from a distribution , and the algorithm must decide whether to query the label of a given point or not [ 14 ] . From a theoretical perspective , active learning has typically been studied under models inspired by the probably approximately correct ( PAC ) model of learning [ 41 ] . Here one assumes that there is a pre-specified classH of functions such that the target function mapping examples to their labels 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . either lies in H or has a good approximation inside the class . Given access to unlabeled samples generated i.i.d . from the distribution , the goal is to query for a small number of labels and produce a hypothesis of low error . In the parametric setting , namely , when the class of functionsH has finite VC-dimension ( or finite disagreement coefficient ) [ 21 ] , the rate of convergence of active learning , i.e. , the rate of decay of the regret as a function of the number of label queries ( N ) , is of the form ν N−1/2 + e− √ N , where ν is the population loss of the best function in class H. This simple finding shows that active learning behaves like passive learning when ν > 0 , while very fast rates can only be achieved under low noise ( ν ≈ 0 ) conditions . This has been worked out in , e.g. , [ 19 , 15 , 5 , 4 , 6 , 38 ] . While the parametric setting comes with methodological advantages , the above shows that in order to unleash the true power of active learning , two properties are desirable : ( 1 ) A better interplay between the input distribution and the label noise and , ( 2 ) a departure from the parametric setting leading us to consider wider classes of functions ( so as to reduce the population loss ν to close to 0 ) . To address the above , there has also been considerable theoretical work in recent years on non-parametric active learning [ 10 , 33 , 31 ] . However , these approaches suffer from the curse of dimensionality and do not lead to computationally efficient algorithms . A popular approach that has been explored empirically in recent works is to use Deep Neural Networks ( DNNs ) to perform active learning ( e.g. , [ 37 , 26 , 39 , 3 , 44 ] ) . While these works empirically demonstrate the power of the DNN-based approach to active learning , they do not come with provable guarantees . The above discussion raises the following question : Is provable and computationally efficient active learning possible in non-parametric settings ? We answer the above question in the affirmative by providing the first , to the best of our knowledge , computationally efficient algorithm for active learning based on Deep Neural Networks . Similar to non-parametric active learning , we avoid fixing a function class a-priori . However , in order to achieve computational efficiency , we instead propose to use over-parameterized DNNs , where the amount of over-parameterization depends on the input data at hand . We work in the sequential setting , and propose a simple active learning algorithm that forms an uncertainty estimate for the current data point based on the output of a DNN , followed by a gradient descent step to update the network parameters if the data point is queried . We show that under standard low-noise assumptions [ 32 ] our proposed algorithm achieves fast rates of convergence . In order to analyze our algorithm , we use tools from the theory of Neural Tangent Kernel ( NTK ) approximation [ 24 , 2 , 18 ] that allows us to analyze the dynamics of gradient descent by considering a linearization of the network around random initialization . Since we study the non-parametric regime , the convergence rates of our algorithm depend on a data-dependent complexity term that is expected to be small in practical settings , but could be very large in worst-case scenarios . Furthermore , the algorithm itself needs an estimate of complexity term in order to form accurate uncertainty estimates . We show that one can automatically adapt to the magnitude of the unknown complexity term by designing a novel model selection algorithm inspired by recent works in model selection in multiarmed bandit settings [ 36 , 35 ] . Yet , several new insights are needed to ensure that the model selection algorithm can simultaneously achieve low generalization error without spending a significant amount of budget on label queries . 2 Preliminaries and Notation . Let X denote the input space , Y the output space , and D an unknown distribution over X × Y . We denote the corresponding random variables by x and y . We also denote by DX the marginal distribution of D over X , and by DY|x0 the conditional distribution of random variable y given x = x0 . Moreover , given a function f ( sometimes called a hypothesis or a model ) mapping X to Y , the conditional population loss ( often referred to as conditional risk ) of f is denoted by L ( f |x ) , and defined as L ( f |x ) = Ey∼DY|x [ ` ( f ( x ) , y ) |x ] , where ` : Y × Y → [ 0 , 1 ] is a loss function . For ease of presentation , we restrict to a binary classification setting with 0-1 loss , whence Y = { −1 , +1 } , and ` ( a , y ) = 1 { a 6= y } ∈ { 0 , 1 } , 1 { · } being the indicator function of the predicate at argument . When clear from the surrounding context , we will omit subscripts like “ y ∼ DY|x '' from probabilities and expectations . We investigate a non-parametric setting of active learning where the conditional distribution of y given x is defined through an unknown function h : X 2 → [ 0 , 1 ] such that P ( y = 1 |x ) = h ( ( x , 0 ) ) P ( y = −1 |x ) = h ( ( 0 , x ) ) , ( 1 ) where 0 ∈ X , ( x1 , x2 ) denotes the concatenation ( or pairing ) of the two instances x1 and x2 ( so that ( x , 0 ) and ( 0 , x ) are in X 2 ) and , for all x ∈ X we have h ( ( x , 0 ) ) + h ( ( 0 , x ) ) = 1 . We make no explicit assumptions on h , other than its well-behavedness w.r.t . the data { xt } Tt=1 at hand through the formalism of Neural Tangent Kernels ( NTK ) – see below . As a simple example , in the linear case , X is the d-dimensional unit ball , h ( · , · ) is parametrized by an unknown unit vector θ ∈ Rd , and h ( ( x1 , x2 ) ) = 1+〈 ( θ , −θ ) , ( x1 , x2 ) 〉 2 , so that h ( ( x , 0 ) ) = 1+〈θ , x〉 2 and h ( ( 0 , x ) ) = 1−〈θ , x〉 2 , where 〈· , ·〉 is the usual dot product in Rd . We consider a streaming setting of active learning where , at each round t ∈ [ T ] = { 1 , . . . , T } , a pair ( xt , yt ) ∈ X × Y is drawn i.i.d . from D. The learning algorithm receives as input only xt , and is compelled to both issue a prediction at for yt and , at the same time , decide on-the-fly whether or not to observe yt . These decisions can only be based on past observations . Let Et denote the conditional expectation E [ · | ( x1 , y1 ) . . . , ( xt−1 , yt−1 ) , xt ] , and we introduce the shorthand xt , a = { ( xt , 0 ) if a = 1 ( 0 , xt ) if a = −1 . Notice that with this notation E [ ` ( a , yt ) |xt ] = 1− h ( xt , a ) , for all a ∈ Y . We quantify the accuracy of the learner ’ s predictions through its ( pseudo ) regret , defined as RT = T∑ t=1 ( Et [ ` ( at , yt ) |xt ] − E [ ` ( a∗t , yt ) |xt ] ) = T∑ t=1 ( h ( xt , a∗t ) − h ( xt , at ) ) , where a∗t is the Bayesian-optimal classifier on instance xt , that is , a ∗ t = arg maxa∈Y h ( xt , a ) . Additionally , we are interested in bounding the number of labels NT the algorithm decides to request . Our goal is to simultaneously bound RT and NT with high probability over the generation of the sample { ( xt , yt ) } t=1 , ... , T . Throughout this work , we consider the following common low-noise condition on the marginal distributionDX ( Mammen-Tsybakov low noise condition [ 32 ] ) : There exist absolute constants c > 0 , and α ≥ 0 such that for all ∈ ( 0 , 1/2 ) we have P ( |h ( ( x , 0 ) ) − 1 2 | < ) ≤ c α . In particular , α =∞ gives the so-called hard margin condition P ( |h ( ( x , 0 ) ) − 12 | < ) = 0. while , at the opposite extreme , exponent α = 0 ( and c = 1 ) results in no assumptions whatsoever on DX . For simplicity , we shall assume throughout that the above low-noise condition holds for1 c = 1 . Our techniques are inspired by the recent work [ 45 ] from which we also borrow some notation . We are learning the class of functions { h } by means of fully connected neural networks f ( x , θ ) = √ mWnσ ( ... σ ( W1x ) ) , where σ is a ReLU activation function σ ( x ) = max { 0 , x } , m is the width of the network and n ≥ 2 is its depth . In the above , θ ∈ Rp collectively denotes the set of weights { W1 , W2 , . . . , Wn } of the network , where p = m+ 2md+m2 ( n− 2 ) is their number , and the input x at training time should be thought of as some xt , a ∈ X 2 . With any depth-n network and data points { xt , a } t=1 , ... , T , a=±1 we associate a depth-n NTK matrix as follows [ 24 ] . First , rename { xt , a } t=1 , ... , T , a=±1 as { x ( i ) } i=1 , ... ,2T . Then define matrices H̃ ( 1 ) = [ H ( 1 ) i , j ] 2T×2T i , j=1 Σ ( 1 ) = [ Σ ( 1 ) i , j ] 2T×2T i , j=1 with H ( 1 ) i , j = Σ ( 1 ) i , j = 〈x ( i ) , x ( j ) 〉 , and then , for any k ≤ n and i , j = 1 , . . . , 2T , introduce the bivariate covariance matrix A ( k ) i , j = [ Σ ( k ) i , i Σ ( k ) i , j Σ ( k ) i , j Σ ( k ) j , j ] 1A more general formulation requires the above to hold only for ≤ 0 , where 0 ∈ ( 0 , 1/2 ) is a third parameter . We shall omit this extra parameter from our presentation . by which we recursively define Σ ( k+1 ) i , j = 2E ( u , v ) ∼N ( 0 , A ( k ) i , j ) [ σ ( u ) σ ( v ) ] and H̃ ( k+1 ) i , j = 2H̃ ( k ) i , j E ( u , v ) ∼N ( 0 , A ( k ) i , j ) [ 1 { u ≥ 0 } 1 { v ≥ 0 } ] + Σ ( k+1 ) i , j . The 2T × 2T -dimensional matrix H = 12 ( H̃ ( n ) + Σ ( n ) ) is called the Neural Tangent Kernel ( NTK ) matrix of depth n ( and infinite width ) over the set of points { xt , a } t=1 , ... , T , a=±1 . The reader is referred to [ 24 ] for more details on NTK . In order to avoid heavy notation , we assume ||xt|| = 1 for all t. Matrix H is positive semi-definite by construction but , as is customary in the NTK literature ( e.g. , [ 2 , 9 , 17 ] ) , we assume it is actually positive definite ( hence invertible ) with smallest eigenvalue λ0 > 0 . This is a mild assumption that can be shown to hold if no two vectors xt are aligned to each other . We measure the complexity of the function h at hand in a way similar to [ 45 ] . Using the same rearrangement of { xt , a } t=1 , ... , T , a=±1 into { x ( i ) } i=1 , ... ,2T as above , let h be the 2T -dimensional ( column ) vector whose i-th component is h ( x ( i ) ) . Then , we define the complexity ST , n ( h ) of h over { xt , a } t=1 , ... , T , a=±1 w.r.t . an NTK of depth n as ST , n ( h ) = √ h > H−1h . Notice that this notion of ( data-dependent ) complexity is consistent with the theoretical findings of [ 2 ] , who showed that for a two-layer network the bound on the generalization performance is dominated by y > H−1y , where y is the vector of labels . Hence if y is aligned with the top eigenvectors of H the learning problem becomes easier . In our case , vector h plays the role of vector y . Also observe that S2T , n ( h ) can in general be as big as linear in T ( in which case learning becomes hopeless with our machinery ) . In the special case where h belongs to the RKHS induced by the NTK , one can upper bound ST , n ( h ) by the norm of h in the RKHS . The complexity term ST , n ( h ) is typically unknown to the learning algorithm , and it plays a central role in both regret and label complexity guarantees . Hence the algorithm needs to learn this value as well during its online functioning . Apparently , this aspect of the problem has been completely overlooked by [ 45 ] ( as well as by earlier references on contextual bandits in RKHS , like [ 12 ] ) , where a ( tight ) upper bound on ST , n ( h ) is assumed to be available in advance . We will cast the above as a model selection problem in active learning , where we adapt and largely generalize to active learning the regret balancing technique from [ 36 , 35 ] . In what follows , we use the short-hand g ( x ; θ ) = ∇θf ( x , θ ) and , for a vector g ∈ Rp and matrix Z ∈ Rp×p , we often write √ g > Zg as ||g||Z , so that ST , n ( h ) = ||h||H−1 .

Summary: Using a combination of the Neural Tangent Kernel (NTK) approximation and contextual bandit approaches, the paper first derives a neural active learning algorithm for the streaming setting where the data points come sequentially and the algorithm has to decide whether to query for the label of each data point. A second similar algorithm is developed for model selection so that the model complexity does not have to be known beforehand. Both algorithms have counterparts which train the neural networks with stochastic gradient descent. The goal of the algorithms is to balance the regret and the number of queried data points. Using the mentioned theoretical frameworks, joint bounds on the regret and the number of queries are derived.