LLBoost: Last Layer Perturbation to Boost Pre-trained Neural Networks

In this work , we present LLBoost , a theoretically-grounded , computationallyefficient method to boost the validation accuracy of pre-trained overparameterized models without impacting the original training accuracy . LLBoost adjusts the last layer of a neural network by adding a term that is orthogonal to the training feature matrix , which is constructed by applying all layers but the last to the training data . We provide an efficient implementation of LLBoost on the GPU and demonstrate that LLBoost , run using only 1 GPU , improves the test/validation accuracy of pre-trained models on CIFAR10 , ImageNet32 , and ImageNet . In the over-parameterized linear regression setting , we prove that LLBoost reduces the generalization error of any interpolating solution with high probability without affecting training error . 1 INTRODUCTION . Over the past decade , deep networks have produced a number of state-of-the-art results including surpassing human performance on the ImageNet classification task ( 26 ; 14 ) . However , tuning hyperparameters to produce the best possible neural network in a given application remains a computationally expensive task . State-of-the-art results often involve selecting the best model using multiple random seeds ( 14 ; 27 ; 12 ; 28 ) or ensembling ( 15 ) , and training even a single model can take several days even when using multiple GPUs . Hence , it is critical to identify computationally efficient approaches to improve the performance of pre-trained models without the need of re-training them . In this work , we present LLBoost , a theoretically-grounded , computationally-efficient method to boost the validation accuracy of pre-trained , over-parameterized models without impacting the training accuracy . Figure 1 provides an overview of our method as well as the main results . As shown in Figure 1A , our method adjusts the last fully-connected layer of a neural network by selecting the best performing perturbation out of several orthogonal to the training feature matrix , which is constructed by applying all but the last layer to the training data . In Figure 1B , we provide an example showing how our method applied to a model trained under a poorly chosen random seed can boost validation accuracy comparable to that of a model trained under a better seed . Lastly , Figure 1C shows that our method can significantly improve the validation accuracy of pre-trained neural networks on large datasets using a fraction of the training time . The intuition for our method is based on characterizing the benefit of random initialization in overparameterized linear regression . In particular , consider a dataset ( X , y ) ⊂ Rd×n×R1×n with n < d for which there exists w∗ ∈ R1×d such that y = w∗X . In order to estimate w∗ from the data , we use gradient descent with learning rate η and initialization w ( 0 ) to minimize the squared loss , i.e . to solve : arg min w∈Rd 1 2 ‖y − wX‖2 . It is well-known ( 8 ) that gradient descent converges to the following closed-form solution : wr = w ( 0 ) ( I −X ( XTX ) †XT ) + yX† , where A† is the pseudo-inverse of A ( see Appendix A for a proof ) . In this work , we prove that when n < d , sampling w ( 0 ) on a hyper-sphere leads to wr with lower generalization error than the minimum norm solution yX† with constant probability . Since the last layer of modern deep networks is usually linear , we apply this result from regression to the last layer of networks to arrive at our method , LLBoost . We end the introduction with a summary of our main contributions : 1 . We present LLBoost , a computationally efficient method for boosting the validation accuracy of pre-trained neural networks without affecting training accuracy . Additionally , we provide an efficient implementation of LLBoost on the GPU . 2 . We provide a wealth of empirical evidence that LLBoost improves the validation accuracy of pre-trained neural networks and prove that it does not affect training accuracy for overparameterized models . 3 . We provide evidence that LLBoost yields a computational advantage over random seed search for training neural networks . 4 . In the over-parameterized linear regression setting , we prove that LLBoost reduces the test error of any interpolating solution with high probability without affecting training error . 2 RELATED WORK . Understanding the benefits of over-parameterization is a recent topic of major interest in machine learning . ( 2 ) introduced the double descent curve showing that when increasing model capacity past the interpolation threshold , test error can drop a second time . ( 20 ) noted that this phenomenon had been noticed empirically several decades prior to the work of ( 2 ) . A line of recent work have provided a theoretical analysis of double descent in linear regression ( 13 ; 3 ; 1 ; 22 ; 5 ; 21 ) . In particular , ( 13 ; 3 ) analyzed the generalization error for the minimum norm solution in over-parameterized linear models . These works proved that the minimum norm solution can yield lower generalization error in the over-parameterized regime than in the under-parameterized regime . Our theoretical analysis primarily builds on that of ( 3 ) , but we analyze the generalization error of random-initialization interpolating solutions instead of the minimum-norm solution . Various methods , including random seed search and ensembling , are standardly used to improve the performance of machine learning models through training . ( 4 ) recommends using 5-10 random seeds if computational power is available , and this is typically done in state-of-the-art models ; for example , ( 14 ) considered 5 random seeds for ResNet-110 and ( 12 ) considered at least 2 random seeds on CIFAR10 ( 17 ) . ( 9 ) rigorously analyzed the impact of random seeds on model performance by considering between 50-400 random seeds for models of varying depth on MNIST ( 18 ) . Figure 1 from their work demonstrates that random seeds can affect the validation accuracy of models by up to a few percentage points on MNIST . Since this comes at a significant computational price , it is critical to identify methods that can obtain such boosts without having to perform random seed search , which is the topic of this paper . Another popular approach to improving neural network performance is ensembling , for example via bagging ( 6 ) or boosting ( 10 ) . Recently , ( 15 ) presented an approach to ensembling , which involved training a single network and saving the model parameters each time training stopped at a local minimum . ( 15 ) built upon ( 30 ) , which introduced horizontal and vertical voting to ensemble a model across training epochs . LLBoost can be used in combination with all the above approaches to further boost their performance , since it only adjusts the last layer of a pre-trained model without requiring any training . 3 PRELIMINARIES AND METHOD . We now present the preliminaries relevant for formally introducing LLBoost . Since our method is derived from over-parameterized linear regression , we first describe this problem setting . Noiseless Regression . Let ( X , y ) ⊂ Rd×n×R1×n denote the training dataset where d is the number of dimensions and n is the number of samples . In noiseless linear regression there exists w∗ ∈ R1×d such that y = w∗X . In order to estimate w∗ from the data , we use gradient descent to minimize the squared loss . Formally , we solve : arg min w∈Rd 1 2 ‖y − wX‖2 ( 1 ) using gradient descent with learning rate η , which proceeds according to the updates : w ( t+1 ) = w ( t ) + η ( y − w ( t ) X ) XT . The following well-known theorem ( 8 ) gives the solution , w ( ∞ ) , for the objective ( 1 ) given by gradient descent . Theorem 1 . Let ( X , y ) ⊂ Rd×n × R1×n with X of rank r < d , let λmax denote the largest eigenvalue ofXXT , and letA† denote the pseudo-inverse of a matrixA . Given initializationw ( 0 ) ∈ Rd , gradient descent used to solve : arg min w∈Rd 1 2 ‖y − wX‖2 with learning rate η < λ−1max converges to w ( ∞ ) = w ( 0 ) ( I −X ( XTX ) †XT ) + yX† . When w ( 0 ) = 0 ( zero initialization ) , w ( ∞ ) is the minimum ` 2-norm solution for solving the linear system wX = y . The following property of w ( ∞ ) is used to demonstrate that LLBoost does not impact training error . Lemma 1 . Let w ( ∞ ) = w ( 0 ) ( I−X ( XTX ) †XT ) + yX† . Then , w ( 0 ) ( I−X ( XTX ) †XT ) ⊥ yX† . The proof follows directly by substituting inX = UΣV T given by the singular value decomposition ( SVD ) . Lemma 1 implies that LLBoost does not affect training predictions since it only ever adds a component orthogonal to the span of the training feature matrix . We primarily focus on the noiseless regression setting , but we note that our theory directly extends to the Gaussian model from ( 3 ) and ( 7 ) . We now present a result from random matrix theory , which is at the basis of our results . The proof of the following lemma relies on the rotation invariance of the multivariate Gaussian distribution and is presented in Appendix B. Lemma 2 . If X ∈ Rd×n with columns x ( i ) ∼ N ( 0 , Id×d ) , then the singular vectors of X are uniformly distributed on the unit sphere Sd−1 . In particular , if x ∼ N ( 0 , Id×d ) , then x‖x‖ is uniformly distributed on the unit sphere Sd−1 . Method Overview . Given the above preliminaries , we present LLBoost in Algorithm 1 . LLBoost takes in the following inputs : ( 1 ) the training feature matrix X ; ( 2 ) the validation feature matrix Xt ; ( 3 ) the validation labels yt ; ( 4 ) the last layer weights w ; ( 5 ) the number of samples to consider T ; and ( 6 ) the radius of the hyper-sphere to sample from γ. LLBoost begins by constructing the projection matrix P = I −X ( XTX ) †XT . Then , LLBoost samples z ∼ N ( 0 , I ) , normalizes z and adds the perturbation γzP to w. LLBoost repeats this process and returns the perturbation that most improves the validation accuracy . Remarks . As shown in Figure 1A , X and Xt are constructed by applying all but the last layer of the neural network to the training/test data . If X is not full rank ( as is the case for neural networks that are not over-parameterized ) , we can create a low rank X by truncating the lower singular values of X . This correction only mildly affects training error as is shown in Section 4 . For classification settings , yt should be converted to 1-hot vectors . When the last layer has a bias , one can simply append a 1 to w and append an extra row of 1 ’ s to X and Xt . In most of our experiments , we consider up to T = 100 , 000 samples . When selecting γ , one can utilize a binary search approach starting with γ = 1√ d and either doubling or halving γ depending on whether the validation accuracy increased or decreased respectively . However , as a simple heuristic one can start with γ = 1√ d ‖w‖ ( as is recommended by Theorem 3 ) . Lastly , when the network has multiple outputs ( as in multiclass problems ) , we let the entries of the weight matrix be i.i.d . standard normal variables and then normalize by the Frobenius norm . LLBoost does not impact training error/accuracy . Since the perturbations constructed by LLBoost are always projected into the space orthogonal to the training features X , Lemma 1 implies that LLBoost does not alter the training predictions . This is an important aspect of our algorithm since otherwise , LLBoost could just overfit the test data . Algorithm 1 LLBoost ( X , Xt , yt , w , T , γ ) Input : X : = rank r training feature matrix ; Xt : = val . feature matrix ; yt : = val . labels ; w : = last layer weights ; T : = number of samples ; γ : = radius of hyper-sphere Output : wbest : = adjusted last layer weights 1 : wbest = None 2 : accbest = 0 ; ( 1 , d ) = w.shape ; U , Σ , V T ← SVD ( X ) 3 : P ← Id×d − UIr×rUT 4 : for t← 1 to T do 5 : z ← N ( 0 , Id×d ) 6 : z ← z‖z‖ 7 : wr ← γzP + w 8 : acc = GetV alAcc ( Xt , yt , wr ) 9 : if acc > accbest then 10 : wbest = wr ; accbest = acc 11 : end if 12 : end for 13 : return wbest GPU Implementation . We can easily remove the loop in Algorithm 1 by using the GPU since the dominating computations are vector-matrix products . A full implementation is provided using PyTorch ( 25 ) and NumPy ( 23 ; 29 ) in the footnote below1 . In practice , computing the validation accuracy for each sample of wr is the limiting factor in LLBoost . For large datasets , the functions used to compute the SVD and the projection matrix P need to be selected carefully . Note that with respect to SVD , the reduced SVD is sufficient for our algorithm . While for the ImageNet training features from FixResNext-101 the full SVD would require constructing a matrix with roughly 1012 entries , the reduced SVD only requires 108 entries .

Summary: This paper proposes LLBoost that enables adjusting the last linear layer without impacting the training accuracy under the assumption that the last linear layer is in an over-parametrized situation. When the last layer is not over-parametrized, LLBoost first applies the low rank approximation to the training feature matrix through the SVD decomposition, which may affect the original training accuracy. The reason why LLBoost does not change the training accuracy is explained as follows: In an over-parametrized noiseless linear regression, a solution of a linear system $y = wX$ obtained by the gradient descent with an initial value of $w^{(0)}$ is given in a closed form of $\hat{w} = w^{(0)} (I - X(X^\top X)^\dagger X^\top) +  yX^\dagger$. Therefore, we can compute a solution of $y = wX$ by simply generating $w^{(0)}$ randomly and applying this formula. It is also experimentally verified that LLBoost can adjust the last linear layer without impacting the training accuracy (after appriximated with SVD when necessary). The authors also present theoretical results that sampling $w^{(0)}$ uniformly on the hyper-shpere of appropriate radius leads to a solution that is better than the minimum norm solution ($yX^\dagger$) with constant probability.