On The Difficulty of Warm-Starting Neural Network Training

We design a new algorithm for batch active learning with deep neural network models . Our algorithm , Batch Active learning by Diverse Gradient Embeddings ( BADGE ) , samples groups of points that are disparate and high-magnitude when represented in a hallucinated gradient space , a strategy designed to incorporate both predictive uncertainty and sample diversity into every selected batch . Crucially , BADGE trades off between diversity and uncertainty without requiring any hand-tuned hyperparameters . While other approaches sometimes succeed for particular batch sizes or architectures , BADGE consistently performs as well or better , making it a useful option for real world active learning problems . 1 INTRODUCTION . In recent years , deep neural networks have produced state-of-the-art results on a variety of important supervised learning tasks . However , many of these successes have been in domains where large amounts of data are available . A promising approach for minimizing labeling effort is active learning , a supervised learning protocol where labels can be requested by the algorithm in a sequential feedback-driven fashion . Active learning algorithms aim to identify and label only maximally informative samples , so that a highperforming classifier can be trained with minimal labeling effort . As such , a robust active learning algorithm for deep neural networks may considerably expand the domains where these models are applicable . How should a practical , general-purpose , label-efficient active learning algorithm for deep neural networks be designed ? Theory for active learning suggests a version-space-based approach ( Balcan et al. , 2006 ) , which explicitly or implicitly maintains a set of plausible models , and queries examples for which two models that make different predictions . But when using highly-expressive models , these algorithms degenerate to querying every example . Further , the computational overhead of training deep neural networks preclude algorithms that update the model to best fit the data after each label query , as is often done ( exactly or approximately ) for linear methods ( Beygelzimer et al. , 2010 ; Cesa-Bianchi et al. , 2009 ) . Unfortunately , the theory provides little guidance for these models . One option is to use the network ’ s uncertainty to inform a query strategy , for example by labeling samples for which the model is least confident . However , in a batch setting this creates a pathological scenario where data in the batch are nearly identical , a clear inefficiency . Remedying this issue , we could select samples to maximize batch diversity , but this might choose points that provide little useful new information to the model . For these reasons , methods that exploit just uncertainty or diversity do not consistently work well across model architectures , batch sizes , or datasets . An algorithm that performs well when using a ResNet , for example , might perform poorly when using a multilayer-perceptron . A diversity-based approach might work well when the batch size is very large , but poorly when the batch size is small . Even what in practice constitutes a “ big ” or “ small ” batch size is largely a function of the statistical properties of the data in question . These weaknesses pose a major problem for real , practical batch active learning situations , where data are unfamiliar and potentially unstructured . There is no way to know which active learning algorithm is best to use . Further , in a real active learning scenario , every change of hyperparameters typically causes the algorithm to label examples not chosen under other hyperparameters , provoking substantial labeling inefficiency . That is , hyperparameter sweeps in active learning can be label expensive . Because of this , active learning algorithms need to “ just work ” , given fixed hyperparameters , to a greater extent than is typical for supervised learning . Based on these observations , we design an approach which creates diverse batches of examples about which the current model is uncertain . We measure uncertainty as the gradient magnitude with respect to parameters in the final ( output ) layer , which is computed using the most likely label according to the model . To capture diversity , we collect a batch of examples where these gradients span a diverse set of directions . More specifically , we build up the batch of query points based on these hallucinated gradients using the k-MEANS++ initialization ( Arthur and Vassilvitskii , 2007 ) , which simultaneously captures both the magnitude of a candidate gradient and its distance from previously included points in the batch . We name the resulting approach Batch Active learning by Diverse Gradient Embeddings ( BADGE ) . We show that BADGE is robust to architecture choice , batch size , and dataset , generally performing as well as or better than the best baseline across our experiments , which vary all of the aforementioned environmental conditions . We begin by introducing our notation and setting , followed by a description of the BADGE algorithm in Section 3 , and experiments in Section 4 . We defer discussion of related work to Section 5 . 2 NOTATION AND SETTING . Define [ K ] : = { 1 , 2 , . . . , K } . Denote byX the instance space , andY the label space . In this work we consider multiclass classification , so that Y = [ K ] . Denote by D the distribution from which examples are drawn , DX the unlabeled data distribution , andDY|X the conditional distribution over labels given examples . We consider the pool-based active learning setup , where the learner receives an unlabeled dataset U sampled according to DX , and can request labels sampled according to DY|X for any x ∈ U . We use ED to denote expectation under the data distribution D. Given a classifier h : X → Y , which maps examples to labels , and a labeled example ( x , y ) , we denote the 0/1 error of h on ( x , y ) as ` 01 ( h ( x ) , y ) = I ( h ( x ) 6= y ) . The performance of a classifier h is measured by its expected 0/1 error , i.e . ED [ ` 01 ( h ( x ) , y ) ] = Pr ( x , y ) ∼D ( h ( x ) 6= y ) . The goal of pool-based active learning is to find a classifier with a small expected 0/1 error , using as few label queries as possible . Given a set S of labeled examples ( x , y ) , where each x ∈ S is picked from U , followed by a label query , we use ES as the sample averages over S. In this paper , we consider classifiers h parameterized by underlying neural networks f of fixed architecture , with the weights in the network denoted by θ . We abbreviate the classifier with parameters θ as hθ since the architectures are fixed in any given context , and our classifiers take the form hθ ( x ) = argmaxy∈ [ K ] f ( x ; θ ) y , where f ( x ; θ ) ∈ RK is a vector of scores assigned to candidate labels , given the example x and parameters θ . We optimize the parameters by minimizing the cross-entropy loss ES [ ` CE ( f ( x ; θ ) , y ) ] over the labeled examples , where ` CE ( p , y ) = ∑K i=1 I ( y = i ) ln 1/pi . 3 ALGORITHM . BADGE , described in Algorithm 1 , starts by drawing an initial set of M examples uniformly at random from U and asking for their labels . It then proceeds iteratively , performing two main computations at each step t : a gradient embedding computation and a sampling step . Specifically , at each step t , for every x in the pool U , we compute the label ŷ ( x ) preferred by the current model , and the gradient gx of the loss on ( x , ŷ ( x ) ) with respect to the parameters of the last layer of the network . Given these gradient embedding vectors { gx : x ∈ U } , BADGE selects a set of points by sampling via the k-MEANS++ initialization scheme ( Arthur and Vassilvitskii , 2007 ) . The algorithm queries the labels of these examples , retrains the model , and repeats . We now describe the main computations — the embedding and sampling steps — in more detail . Algorithm 1 BADGE : Batch Active learning by Diverse Gradient Embeddings Require : Neural network f ( x ; θ ) , unlabeled pool of examples U , initial number of examples M , number of iterations T , number of examples in a batch B . 1 : Labeled dataset S ←M examples drawn uniformly at random from U together with queried labels . 2 : Train an initial model θ1 on S by minimizing ES [ ` ( f ( x ; θ ) , y ) ] . 3 : for t = 1 , 2 , . . . , T : do 4 : For all examples x in U \ S : 1 . Compute its hypothetical label ŷ ( x ) = hθt ( x ) . 2 . Compute gradient embedding gx = ∂∂θout ` ( f ( x ; θ ) , ŷ ( x ) ) |θ=θt , where θout refers to parameters of the final ( output ) layer . 5 : Compute St , a random subset of U \S , using the k-MEANS++ seeding algorithm , on { gx : x ∈ U \ S } , and query for their labels . 6 : S ← S ∪ St. 7 : Train a model θt+1 on S by minimizing ES [ ` CE ( f ( x ; θ ) , y ) ] . 8 : end for 9 : return Final model θT+1 . The gradient embedding . Since deep neural networks are optimized using gradient-based methods , we capture uncertainty about an example through the lens of gradients . In particular , we consider the model uncertain about an example if knowing the label induces a large gradient of the loss with respect to the model parameter , and hence a large update to the model . A difficulty with this reasoning is that we need to know the label to compute the gradient . As a proxy , we compute the gradient as if the model ’ s current prediction on the example is the true label . We show in Proposition 1 that , assuming a common structure satisfied by most natural loss functions , the gradient norm with respect to the last layer using this label provides a lower bound on the gradient norm induced by any other label . In addition , under that assumption , the length of this hypothetical gradient vector captures the uncertainty of the model on the example : if the model is highly certain about the example ’ s label , then the example ’ s gradient embedding will have a small norm ( see example below ) . Thus , the gradient embedding conveys information both about the model ’ s uncertainty and potential update direction upon receiving a label at an example . Consider a neural network f where the last nonlinearity is a softmax , i.e . σ ( o ) i = e oi/ ∑K j=1 e oj . Specifically , f is parametrized by θ = ( W , V ) , where θout = W = ( W > 1 , . . . , W > K ) > ∈ RK×d are the weights of the last layer , and V consists of weights of all earlier layers . This means that f ( x ; θ ) = σ ( W · z ( x ; V ) ) , where z is the nonlinear function that maps an input x to the output of the network ’ s penultimate layer . We define gyx = ∂ ∂W ` CE ( f ( x ; θ ) , y ) for a label y and gx = g ŷ x as the gradient embedding in our algorithm , where ŷ = argmaxi∈ [ K ] pi . Fixing an unlabeled sample x , we define pi = f ( x ; θ ) i . Proposition 1 . For all y ∈ { 1 , . . . , K } , denote by gyx = ∂∂W ` CE ( f ( x ; θ ) , y ) . Then ‖gyx‖2 = ( K∑ i=1 p2i + 1− 2py ) ‖z ( x ; V ) ‖2 . Consequently , ŷ = argminy∈ [ K ] ‖gyx‖ . Proof . Observe that by Equation ( 1 ) , ‖gyx‖2 = K∑ i=1 ( pi − I ( y = i ) ) 2‖z ( x ; V ) ‖2 = ( K∑ i=1 p2i + 1− 2py ) ‖z ( x ; V ) ‖2 . The second part follows from the fact that ŷ = argmaxy∈ [ K ] py . The sampling step . We want the newly-acquired labeled dataset to induce large and diverse changes to the model . To this end , we want the selection procedure to favor both sample magnitude and batch diversity . Specifically , we want to avoid the pathology of , for example , selecting a batch of k similar samples where even just a single single label could alleviate our uncertainty on all remaining samples . A natural way of making this selection without introducing additional hyperparameters is to sample from a k-DPP ( Kulesza and Taskar , 2011 ) . That is , to select a batch of k points with a probability proportional to the determinant of their Gram matrix . Recently , Dereziński and Warmuth ( 2018 ) showed that in experimental design for least square linear regression settings , learning from samples drawn from a k-DPP can have much smaller mean square prediction error than learning from any iid samples . In this process , when the batch size is very low , the selection will naturally favor points with a large length , which corresponds to uncertainty in our space . When the batch size is large , the sampler chooses points that are diverse because linear dependence makes the Gram determinant equal to zero . Unfortunately , sampling from a k-DPP is not trivial . Many sampling algorithms ( Kang , 2013 ; Anari et al. , 2016 ) rely on MCMC , where mixing time poses a significant computational hurdle . The state-of-the-art algorithm of ( Dereziński , 2018 ) has a high-order polynomial running time in the batch size and the embedding dimension . To overcome this computational hurdle , we suggest instead sampling using the k-MEANS++ seeding algorithm ( Arthur and Vassilvitskii , 2007 ) , originally made to produce a good initialization for k-means clustering . k-MEANS++ seeding selects centroids by sampling points in proportion to their squared distances from the nearest centroid that has already been chosen . For completeness , we give a formal description of the k-MEANS++ seeding algorithm in Appendix A . This simple sampler tends to produce diverse batches similar to a k-DPP . As shown in Figure 1 , switching between the two samplers does not affect the active learner ’ s statistical performance while greatly improving the computational performance . A thorough comparison on the running times and test accuracies of k-MEANS++ and k-DPP based gradient embedding sampling can be found in Appendix G. Figure 2 describes the batch diversity and average gradient magnitude per selected batch for a variety of sampling strategies . As expected , both k-DPPs and k-MEANS++ tend to select samples that are diverse ( as measured by the magnitude of their Gram determinant with batches ) and high magnitude . Other samplers , like furthest-first traversal for k-Center clustering ( FF-k-CENTER ) , do not seem to have this property . The FF-k-CENTER algorithm is the sampling choice of the CORESET approach to active learning , which we describe in the proceeding section ( Sener and Savarese , 2018 ) . Appendix F discusses diversity with respect to uncertainty-based approaches . Example : multiclass classification with softmax activation . Recall that the neural network f has the form f ( x ; θ ) = σ ( W · z ( x ; V ) ) , where σ ( o ) i = eoi/∑Kj=1 eoj , and the loss function is the cross entropy , ` CE ( p , y ) = ∑K i=1 I ( y = i ) ln 1/pi . By substitution , ` CE ( f ( x ; θ ) , y ) = ln ( K∑ j=1 eWj ·z ( x ; V ) ) −Wy · z ( x ; V ) . Given this , the i-th block of gx is equal to ( gx ) i = ∂ ∂Wi ` CE ( f ( x ; θ ) , ŷ ) = ( pi − I ( ŷ = i ) ) z ( x ; V ) . ( 1 ) Based on this expression , we can make the following observations : 1 . Each block of gx is a scaling of z ( x ; V ) , the output of the network ’ s penultimate layer . In this respect , gx captures the representation information of z ( x ; V ) , the space in which diversity-based algorithms operate ( Sener and Savarese , 2018 ) . 2 . Proposition 1 shows that the norm of gx is a lower bound on the norm of the loss gradient induced by the example with true label y with respect to the last-layer weights , i.e . ‖gx‖ ≤ ‖gyx‖ . This suggests that the norm of gx conservatively estimates the example ’ s influence on the current model . 3 . If the current model θ is highly confident about x , i.e . vector p is skewed towards a standard basis vector ej , then ŷ = j , and vector ( pi − I ( ŷ = i ) ) Ki=1 has a small length . Therefore , gx has a small length as well . Consequently , such high-confidence examples tend to have gradient embeddings of small magnitude , which are unlikely to be repeatedly selected by k-MEANS++ at iteration t. For the special case of binary logistic regression ( K = 2 and z ( x ; V ) = x ) , we provide further justifications on why BADGE yields better updates than vanilla uncertainty sampling in Appendix B .

Summary: This paper examines the problem of warm-starting the training of neural networks. In particular, a generalization gap arises when the network is trained on the full training set from the start versus being warm-started, where the network is initially (partially) trained on a subset of the training set, then switched to the full training set. This problem is practical, as it is often preferable to train online while data is collected to make up-to-date predictions for tasks (such as in online advertising or recommendation systems), but it has been found that retraining is necessary in order to obtain optimal performance. The paper also mentions active learning, domain shift, and transfer learning as two other relevant and important problems.