Can We Use Gradient Norm as a Measure of Generalization Error for Model Selection in Practice?

1 INTRODUCTION . Generalization performance of deep learning through stochastic gradient descent-based optimization has been widely studied in recent work ( Li et al. , 2020 ; Chatterjee , 2020 ; Negrea et al. , 2019 ; Thomas et al. , 2019 ; Zhu et al. , 2019 ; Hu et al. , 2019 ; Zhang et al. , 2016 ; Mou et al. , 2017 ) . These studies initialize the analysis from the learning dynamics ’ perspectives and then extend to the upper bound estimation of generalization errors ( Mou et al. , 2017 ) . More recently , researchers have shifted their focuses onto providing some theoretical or empirical measures on generalization performance ( Li et al. , 2020 ; Negrea et al. , 2019 ; Thomas et al. , 2019 ; Cao & Gu , 2019 ; He et al. , 2019 ; Frei et al. , 2019 ) with respect to deep architectures , hyper-parameters , data distributions , learning dynamics and so on . This work studies the use of generalization performance measures ( Li et al. , 2020 ; Negrea et al. , 2019 ) for model selection purposes ( Cawley & Talbot , 2010 ) . Prior to the endeavor of deep learning , the generalization gap has been used as a straightforward measure . Given a set of N training samples { x1 , x2 , x3 , . . . xN } , a deep learning model θ and a loss function L ( θ ; x ) based on the sample x , the generalization gap G is defined as G ( θ ) = E x∼X L ( θ ; x ) − 1 N N∑ i=1 L ( θ ; xi ) , ( 1 ) where X is defined as the distribution of data , and Ex∼X L ( θ ; x ) refers to the expected loss . To quantify the generalization gap , validation or testing samples have been frequently used to measure the expected loss ( Kohavi et al. , 1995 ) . Such that given a validation/testing dataset with M samples { y1 , y2 , y3 , . . . , yM } , the empirical generalization gap ĜM is estimated as ĜM ( θ ) = 1 M M∑ i=1 L ( θ ; yi ) − 1 N N∑ i=1 L ( θ ; xi ) , ( 2 ) and limM→∞ ĜM = G. However , due to the limited amount of samples , the accurate estimation of generalization gap is not always available ( Cawley & Talbot , 2010 ) . It has been evidenced that performance tuning based on validation set frequently causes overfitting to the validation set ( Recht et al. , 2018 ) in deep learning settings . Rather than the use of empirical generalization gap , some advanced measure has been proposed ( Li et al. , 2020 ; Negrea et al. , 2019 ; Thomas et al. , 2019 ) to provide data-dependent characterization on the generalization performance for deep learning . For example , Thomas et al . ( 2019 ) derived the Takeuchi information criterion ( TIC ) using the Hessian and covariance matrices of loss gradients with low-complexity approximation . They then propose to use TIC as an empirical metric that correlates to the generalization gap . As the calculation of TIC relies on the use of validation set , TIC for deep learning is a posterior measure . Further , Negrea et al . ( 2019 ) improved mutual information bounds for Stochastic Gradient Langevin Dynamics via some data-dependent measure . More specifically , the squared norm of gradients have been used as the data-dependent priors to tightly bound the generalization gap through measuring the flatness of empirical risk surface . More recently , the squared norm of gradients over the learning dynamics has been studied as the measure to upper-bound the generalization gap ( Li et al. , 2020 ; Negrea et al. , 2019 ) . All above methods connect the generalization gap of deep learning to the gradients and Hessians of loss functions . While Thomas et al . ( 2019 ) is a posterior measure relying on the validation datasets , the two studies ( Li et al. , 2020 ; Negrea et al. , 2019 ) provide “ prior ” measure that uses training datasets . More specifically , Li et al . ( 2020 ) proves that the generalization error is upper bounded by the empirical squared gradient norm along the optimization path . Formally , they show that , given a model trained with n samples in dataset X = { x1 , x2 . . . xn } for a total of T iterations using a C-bounded loss function is used , the theoretical generalization gap of the DNN θT is bounded as follows , G ( θT ) ≤ 2 √ 2C n √√√√E X [ T∑ t=1 γ2t σ2t ge ( t ) ] , ( 3 ) where ge ( t ) = Eθt−1 [ 1n ∑n i=1 ‖∇L ( θt−1 , xi ) ‖ 2 2 ] is the empirical squared gradient norm in the t th iteration , γt is the learning rate and σt indicates the standard deviation of Gaussian noise in the stochastic process . Their result directly shows that empirical squared gradient norm along the optimization path is a good indicator of a model ’ s generalization ability . Therefore , we are interested in using this signal as a criterion in model selection , so as to find an optimal set of hyper-parameters . Our Contributions . In this work , we follow the theoretical investigation in ( Li et al. , 2020 ) and try to use the squared gradient norms ( GN ) over the optimization path as a data-dependent generalization performance measure for DNN model selection . In an empirical manner , the proposed metric GN here should be able to measure the generalization gap of the DNN model θT , which has been trained with T iterations and N training samples , as follows GN ( θT ) = T∑ t=1 [ 1 N N∑ i=1 ‖∇L ( θt ; xi ) ‖22 ] , ( 4 ) where ∇L ( θt ; xi ) refers to the loss gradient using the model of the tth iteration and the ith training sample in { x1 , x2 , . . . , xN } . Based on the data-dependent measure GN , our work make three pieces of significant technical contributions as follows ( 1 ) Approximated Gradient Norm ( AGN ) as an accelerated and low-complexity approximation to GN . Despite the fact that GN can measure the generalization error of a DNN model , the time consumption for GN is high , due to the complexity of gradient estimation using every training sample in every iteration . To lower the computational complexity , we propose Approximated Gradient Norm ( AGN ) that only uses the summation of loss gradients of the Fully-Connected ( FC ) Layer in DNN by the end of every epoch ( rather than every iteration ) as an approximation of GN . Furthermore , the calculation of FC Layer gradient per sample could be further accelerated by Goodfellow ( 2015 ) with extremely low costs . Our empirical evaluation finds that , over various DNN models trained with different hyper-parameters , the metrics GN ( θT ) and AGN ( θT ) behave identically with respect to empirical generalization gap ĜM ( θT ) . This approximation makes it feasible to carry out experiments that evaluate the effectiveness of GN for model selection . ( 2 ) To validate the correlations between generalization performance and AGN ( θT ) , we carry out extensive experiments using various deep neural networks , such as Multi-Layer Perception ( MLP ) , LeNet ( LeCun et al. , 1998 ) , and ResNet ( He et al. , 2016 ) , on top of benchmark datasets including MNIST ( LeCun et al. , 2010 ) , Fashion-MNIST ( Xiao et al. , 2017 ) , SVHN ( Netzer et al. , 2011 ) , and CIFAR ( Krizhevsky et al. , 2009 ) . Observation 1 : It has been observed that , when the models are well-fitted ( i.e. , with low training loss or high training accuracy ) , AGN ( θT ) well corresponds with the empirical generalization gap ĜM ( θT ) while models with lower/higher AGN ( θT ) are consistently with lower/higher ĜM ( θT ) and better/poorer generalization performance . On the other hand , when models are not well-fitted or namely under-fitted , either due to the use of inappropriate hyperparameters or a training process not converged ( e.g. , T is small ) , the correlations between ĜM ( θT ) and AGN ( θT ) are not always consistent while the direction of correlation somtimes is even opposite to the theoretical investigation in Li et al . ( 2020 ) and Eq . ( 3 ) . ( 3 ) To understand the effectiveness and efficiency of using AGN ( θT ) for model selection , we extend our experiments to use AGN ( θT ) as an objective for hyper-parameter selection , under both blackbox optimization ( Escalante et al. , 2009 ; Loshchilov & Hutter , 2016 ) and bandit-based search ( Li et al. , 2017 ; Falkner et al. , 2018 ) settings . Observation 2 : We find that , through the combination with training or validation loss , AGN ( θT ) can help black-box optimization algorithms , such as particle swarm ( PSO ) ( Escalante et al. , 2009 ) or covariance matrix adaption based evolving strategies ( CMA-ES ) ( Loshchilov & Hutter , 2016 ) , search the models ( hyper-parameters ) with equivalent or marginally better performance than using validation loss/accuracy as the search objective . The training procedure can somehow avoid the potential overfitting to the validation set ( Recht et al. , 2018 ) . However the use of AGN ( θT ) requests more prior knowledge ( additional parameters ) to balance the weights of training/validation loss and AGN ( θT ) in the combined objective during the search , which might be sensitive for model selection . Observation 3 : Further , our research finds that bandit-based search algorithms , such as Bayesian optimization over HyperBand ( BOHB ) ( Falkner et al. , 2018 ) , can not work well with gradient norms . Because BOHB selects and drops models during the learning process with respect to the intermediate measures on performance , while within an end-to-end learning process AGN ( θt ) ( for θ1 , θ2 , θ3 . . . , θT ) can not always provide consistent measures on the generalization performance . Observation 4 : In addition to hyper-parameter selection for deep learning , our experiment finally finds that AGN ( θT ) fails to predict the the generalization performance of the models based on different deep architectures , in comparison to some state of the art algorithms and metrics ( Jiang et al. , 2019 ; Nagarajan & Kolter , 2019 ) . We believe the most relevant studies to this work are those done by Li et al . ( 2020 ) ; Negrea et al . ( 2019 ) ; Thomas et al . ( 2019 ) ( that measure the generalization performance of DNN using derivatives , such as gradients and Hessian matrices , of the loss ) , and the contribution made by our work is unique . Compared to Thomas et al . ( 2019 ) , the measure AGN studied here is also a data-dependent metric to characterize the generalization performance . AGN uses gradients and avoids the use of validation sets , while Thomas et al . ( 2019 ) is based on Hessian matrices and relies on validation data . Compared to Li et al . ( 2020 ) ; Negrea et al . ( 2019 ) , our work studies the feasibility of using gradient norms over learning process for model selection in empirical settings , where we first provide AGN as a low-complexity implementation to accelerate the computation of gradient norms , then we conduct extensive results to demonstrate the pros and cons of using gradient norms or AGN for model selection in two major hyper-parameter search settings . Our result demonstrates the potentials but also limitations in the adoption of current theoretical results ( Li et al. , 2020 ; Negrea et al. , 2019 ) as an objective for model selection .

Summary: This paper studies the gradient norm as a measure of generalization in deep learning. The authors first an approximation to the gradient norm (GN) that is the norm of the gradients for only fully connected layers (AGN). Then they empirically evaluate the correlation between AGN and GN as well as GN and the generalization error. In Section 2.1, the authors conclude that AGN is highly correlated with GN and both are correlated with generalization error. In Section 3, the authors conclude that the correlation between AGN and generalization error is not consistent in a wider family of models. In Section 4, authors propose to use AGN for model selection and conclude that AGN is not good for model selection unless the hyperparameter for mixing AGN with another metric is optimal.