Can We Use Gradient Norm as a Measure of Generalization Error for Model Selection in Practice?

1 INTRODUCTION . Generalization performance of deep learning through stochastic gradient descent-based optimization has been widely studied in recent work ( Li et al. , 2020 ; Chatterjee , 2020 ; Negrea et al. , 2019 ; Thomas et al. , 2019 ; Zhu et al. , 2019 ; Hu et al. , 2019 ; Zhang et al. , 2016 ; Mou et al. , 2017 ) . These studies initialize the analysis from the learning dynamics ’ perspectives and then extend to the upper bound estimation of generalization errors ( Mou et al. , 2017 ) . More recently , researchers have shifted their focuses onto providing some theoretical or empirical measures on generalization performance ( Li et al. , 2020 ; Negrea et al. , 2019 ; Thomas et al. , 2019 ; Cao & Gu , 2019 ; He et al. , 2019 ; Frei et al. , 2019 ) with respect to deep architectures , hyper-parameters , data distributions , learning dynamics and so on . This work studies the use of generalization performance measures ( Li et al. , 2020 ; Negrea et al. , 2019 ) for model selection purposes ( Cawley & Talbot , 2010 ) . Prior to the endeavor of deep learning , the generalization gap has been used as a straightforward measure . Given a set of N training samples { x1 , x2 , x3 , . . . xN } , a deep learning model θ and a loss function L ( θ ; x ) based on the sample x , the generalization gap G is defined as G ( θ ) = E x∼X L ( θ ; x ) − 1 N N∑ i=1 L ( θ ; xi ) , ( 1 ) where X is defined as the distribution of data , and Ex∼X L ( θ ; x ) refers to the expected loss . To quantify the generalization gap , validation or testing samples have been frequently used to measure the expected loss ( Kohavi et al. , 1995 ) . Such that given a validation/testing dataset with M samples { y1 , y2 , y3 , . . . , yM } , the empirical generalization gap ĜM is estimated as ĜM ( θ ) = 1 M M∑ i=1 L ( θ ; yi ) − 1 N N∑ i=1 L ( θ ; xi ) , ( 2 ) and limM→∞ ĜM = G. However , due to the limited amount of samples , the accurate estimation of generalization gap is not always available ( Cawley & Talbot , 2010 ) . It has been evidenced that performance tuning based on validation set frequently causes overfitting to the validation set ( Recht et al. , 2018 ) in deep learning settings . Rather than the use of empirical generalization gap , some advanced measure has been proposed ( Li et al. , 2020 ; Negrea et al. , 2019 ; Thomas et al. , 2019 ) to provide data-dependent characterization on the generalization performance for deep learning . For example , Thomas et al . ( 2019 ) derived the Takeuchi information criterion ( TIC ) using the Hessian and covariance matrices of loss gradients with low-complexity approximation . They then propose to use TIC as an empirical metric that correlates to the generalization gap . As the calculation of TIC relies on the use of validation set , TIC for deep learning is a posterior measure . Further , Negrea et al . ( 2019 ) improved mutual information bounds for Stochastic Gradient Langevin Dynamics via some data-dependent measure . More specifically , the squared norm of gradients have been used as the data-dependent priors to tightly bound the generalization gap through measuring the flatness of empirical risk surface . More recently , the squared norm of gradients over the learning dynamics has been studied as the measure to upper-bound the generalization gap ( Li et al. , 2020 ; Negrea et al. , 2019 ) . All above methods connect the generalization gap of deep learning to the gradients and Hessians of loss functions . While Thomas et al . ( 2019 ) is a posterior measure relying on the validation datasets , the two studies ( Li et al. , 2020 ; Negrea et al. , 2019 ) provide “ prior ” measure that uses training datasets . More specifically , Li et al . ( 2020 ) proves that the generalization error is upper bounded by the empirical squared gradient norm along the optimization path . Formally , they show that , given a model trained with n samples in dataset X = { x1 , x2 . . . xn } for a total of T iterations using a C-bounded loss function is used , the theoretical generalization gap of the DNN θT is bounded as follows , G ( θT ) ≤ 2 √ 2C n √√√√E X [ T∑ t=1 γ2t σ2t ge ( t ) ] , ( 3 ) where ge ( t ) = Eθt−1 [ 1n ∑n i=1 ‖∇L ( θt−1 , xi ) ‖ 2 2 ] is the empirical squared gradient norm in the t th iteration , γt is the learning rate and σt indicates the standard deviation of Gaussian noise in the stochastic process . Their result directly shows that empirical squared gradient norm along the optimization path is a good indicator of a model ’ s generalization ability . Therefore , we are interested in using this signal as a criterion in model selection , so as to find an optimal set of hyper-parameters . Our Contributions . In this work , we follow the theoretical investigation in ( Li et al. , 2020 ) and try to use the squared gradient norms ( GN ) over the optimization path as a data-dependent generalization performance measure for DNN model selection . In an empirical manner , the proposed metric GN here should be able to measure the generalization gap of the DNN model θT , which has been trained with T iterations and N training samples , as follows GN ( θT ) = T∑ t=1 [ 1 N N∑ i=1 ‖∇L ( θt ; xi ) ‖22 ] , ( 4 ) where ∇L ( θt ; xi ) refers to the loss gradient using the model of the tth iteration and the ith training sample in { x1 , x2 , . . . , xN } . Based on the data-dependent measure GN , our work make three pieces of significant technical contributions as follows ( 1 ) Approximated Gradient Norm ( AGN ) as an accelerated and low-complexity approximation to GN . Despite the fact that GN can measure the generalization error of a DNN model , the time consumption for GN is high , due to the complexity of gradient estimation using every training sample in every iteration . To lower the computational complexity , we propose Approximated Gradient Norm ( AGN ) that only uses the summation of loss gradients of the Fully-Connected ( FC ) Layer in DNN by the end of every epoch ( rather than every iteration ) as an approximation of GN . Furthermore , the calculation of FC Layer gradient per sample could be further accelerated by Goodfellow ( 2015 ) with extremely low costs . Our empirical evaluation finds that , over various DNN models trained with different hyper-parameters , the metrics GN ( θT ) and AGN ( θT ) behave identically with respect to empirical generalization gap ĜM ( θT ) . This approximation makes it feasible to carry out experiments that evaluate the effectiveness of GN for model selection . ( 2 ) To validate the correlations between generalization performance and AGN ( θT ) , we carry out extensive experiments using various deep neural networks , such as Multi-Layer Perception ( MLP ) , LeNet ( LeCun et al. , 1998 ) , and ResNet ( He et al. , 2016 ) , on top of benchmark datasets including MNIST ( LeCun et al. , 2010 ) , Fashion-MNIST ( Xiao et al. , 2017 ) , SVHN ( Netzer et al. , 2011 ) , and CIFAR ( Krizhevsky et al. , 2009 ) . Observation 1 : It has been observed that , when the models are well-fitted ( i.e. , with low training loss or high training accuracy ) , AGN ( θT ) well corresponds with the empirical generalization gap ĜM ( θT ) while models with lower/higher AGN ( θT ) are consistently with lower/higher ĜM ( θT ) and better/poorer generalization performance . On the other hand , when models are not well-fitted or namely under-fitted , either due to the use of inappropriate hyperparameters or a training process not converged ( e.g. , T is small ) , the correlations between ĜM ( θT ) and AGN ( θT ) are not always consistent while the direction of correlation somtimes is even opposite to the theoretical investigation in Li et al . ( 2020 ) and Eq . ( 3 ) . ( 3 ) To understand the effectiveness and efficiency of using AGN ( θT ) for model selection , we extend our experiments to use AGN ( θT ) as an objective for hyper-parameter selection , under both blackbox optimization ( Escalante et al. , 2009 ; Loshchilov & Hutter , 2016 ) and bandit-based search ( Li et al. , 2017 ; Falkner et al. , 2018 ) settings . Observation 2 : We find that , through the combination with training or validation loss , AGN ( θT ) can help black-box optimization algorithms , such as particle swarm ( PSO ) ( Escalante et al. , 2009 ) or covariance matrix adaption based evolving strategies ( CMA-ES ) ( Loshchilov & Hutter , 2016 ) , search the models ( hyper-parameters ) with equivalent or marginally better performance than using validation loss/accuracy as the search objective . The training procedure can somehow avoid the potential overfitting to the validation set ( Recht et al. , 2018 ) . However the use of AGN ( θT ) requests more prior knowledge ( additional parameters ) to balance the weights of training/validation loss and AGN ( θT ) in the combined objective during the search , which might be sensitive for model selection . Observation 3 : Further , our research finds that bandit-based search algorithms , such as Bayesian optimization over HyperBand ( BOHB ) ( Falkner et al. , 2018 ) , can not work well with gradient norms . Because BOHB selects and drops models during the learning process with respect to the intermediate measures on performance , while within an end-to-end learning process AGN ( θt ) ( for θ1 , θ2 , θ3 . . . , θT ) can not always provide consistent measures on the generalization performance . Observation 4 : In addition to hyper-parameter selection for deep learning , our experiment finally finds that AGN ( θT ) fails to predict the the generalization performance of the models based on different deep architectures , in comparison to some state of the art algorithms and metrics ( Jiang et al. , 2019 ; Nagarajan & Kolter , 2019 ) . We believe the most relevant studies to this work are those done by Li et al . ( 2020 ) ; Negrea et al . ( 2019 ) ; Thomas et al . ( 2019 ) ( that measure the generalization performance of DNN using derivatives , such as gradients and Hessian matrices , of the loss ) , and the contribution made by our work is unique . Compared to Thomas et al . ( 2019 ) , the measure AGN studied here is also a data-dependent metric to characterize the generalization performance . AGN uses gradients and avoids the use of validation sets , while Thomas et al . ( 2019 ) is based on Hessian matrices and relies on validation data . Compared to Li et al . ( 2020 ) ; Negrea et al . ( 2019 ) , our work studies the feasibility of using gradient norms over learning process for model selection in empirical settings , where we first provide AGN as a low-complexity implementation to accelerate the computation of gradient norms , then we conduct extensive results to demonstrate the pros and cons of using gradient norms or AGN for model selection in two major hyper-parameter search settings . Our result demonstrates the potentials but also limitations in the adoption of current theoretical results ( Li et al. , 2020 ; Negrea et al. , 2019 ) as an objective for model selection .

Summary: In this paper, they provide the empirical studies  to understand the effectiveness and efficiency of the use of the gradient norm (induced by [the Li et al., 2020]) as the model selection criterion. To speed up the calculation process the of the gradient norm, they first propose an approximate gradient norm (AGN) based on the depth-wise, sample-wise and epoch-wise accelerations.  Their empirical studies find that the use of AGN can select the models with lower generalization error, but fails for bandit-based or population-based algorithms, and fails to predict the generalization performance of models based on different architectures.  In conclusion, they do not recommend using (approxiamte) gradient norm for model selection in practice.