Identifying Generalization Properties in Neural Networks

1 INTRODUCTION . Deep models have proven to work well in applications such as computer vision ( Krizhevsky et al. , 2012 ) ( He et al. , 2014 ) ( Karpathy et al. , 2014 ) , speech recognition ( Mohamed et al. , 2012 ) ( Hinton et al. , 2012 ) , and natural language processing ( Socher et al. , 2013 ) ( Graves , 2013 ) ( McCann et al. , 2018 ) . Many deep models have millions of parameters , which is more than the number of training samples , but the models still generalize well ( Huang et al. , 2017 ) . On the other hand , classical learning theory suggests the model generalization capability is closely related to the “ complexity ” of the hypothesis space , usually measured in terms of number of parameters , Rademacher complexity or VC-dimension . This seems to be a contradiction to the empirical observations that over-parameterized models generalize well on the test data1 . Indeed , even if the hypothesis space is complex , the final solution learned from a given training set may still be simple . This suggests the generalization capability of the model is also related to the property of the solution . Keskar et al . ( 2017 ) and Chaudhari et al . ( 2017 ) empirically observe that the generalization ability of a model is related to the spectrum of the Hessian matrix∇2L ( w∗ ) evaluated at the solution , and large eigenvalues of the ∇2L ( w∗ ) often leads to poor model generalization . Also , ( Keskar et al. , 2017 ) , ( Chaudhari et al. , 2017 ) and ( Novak et al. , 2018b ) introduce several different metrics to measure the “ sharpness ” of the solution , and demonstrate the connection between the sharpness metric and the generalization empirically . Dinh et al . ( 2017 ) later points out that most of the Hessian-based sharpness measures are problematic and can not be applied directly to explain generalization . In particular , they show that the geometry of the parameters in RELU-MLP can be modified drastically by re-parameterization . Another line of work originates from Bayesian analysis . Mackay ( 1995 ) first introduced Taylor expansion to approximate the ( log ) posterior , and considered the second-order term , characterized by the Hessian of the loss function , as a way of evaluating the model simplicity , or “ Occam factor ” . Recently Smith & Le ( 2018 ) use this factor to penalize sharp minima , and determine the optimal batch size . Germain et al . ( 2016 ) connect the PAC-Bayes bound and the Bayesian marginal likelihood when the loss is ( bounded ) negative log-likelihood , which leads to an alternative perspective on Occam ’ s razor . ( Langford & Caruana , 2001 ) , and more recently , ( Harvey et al. , 2017 ) ( Neyshabur et al. , 2017 ) ( Neyshabur et al. , 2018 ) use PAC-Bayes bound to analyze the generalization behavior of the deep models . Since the PAC-Bayes bound holds uniformly for all “ posteriors ” , it also holds for some particular “ posterior ” , for example , the solution parameter perturbed with noise . This provides a natural 1For example over-parameterized neural network can fit any function of sample size n , making the Rademacher complexity large , but empirically those neural networks generalizes well . ( Zhang et al. , 2016 ) way to incorporate the local property of the solution into the generalization analysis . In particular , Neyshabur et al . ( 2017 ) suggests to use the difference between the perturbed loss and the empirical loss as the sharpness metric . Dziugaite & Roy ( 2017 ) tries to optimize the PAC-Bayes bound instead for a better model generalization . Still some fundamental questions remain unanswered . In particular we are interested in the following question : How is model generalization related to local “ smoothness ” of a solution ? In this paper we try to answer the question from the PAC-Bayes perspective . Under mild assumptions on the Hessian of the loss function , we prove the generalization error of the model is related to this Hessian , the Lipschitz constant of the Hessian , the scales of the parameters , as well as the number of training samples . The analysis also gives rise to a new metric for generalization . Based on this , we can approximately select an optimal perturbation level to aid generalization which interestingly turns out to be related to Hessian as well . Inspired by this observation , we propose a perturbation based algorithm that makes use of the estimation of the Hessian to improve model generalization . 2 PAC-BAYES AND MODEL GENERALIZATION . We consider the supervised learning in PAC-Bayes scenario ( McAllester , 2003 ) ( McAllester , 1998 ) ( McAllester , 1999 ) ( Langford & Shawe-Taylor , 2002 ) . Suppose we have a labeled data set S = { si = ( xi , yi ) | i ∈ { 1 , . . . , n } , xi ∈ Rd , yi ∈ { 0 , 1 } k } , where ( xi , yi ) are sampled i.i.d . from a distribution xi , yi ∼ Ds . 2The variables from different layers are shared so that the model only has two free parameters w1 and w2 . The bound in ( a ) is approximated with η = 39 using inequality ( 8 ) The PAC-Bayes paradigm assumes probability measures over the function class F : X → Y . In particular , it assumes a “ posterior ” distribution Df as well as a “ prior ” distribution πf over the function class F. We are interested in minimizing the expected loss , in terms of both the random draw of samples as well as the random draw of functions : L ( Df , Ds ) = Ef∼DfEx , y∼Ds l ( f , x , y ) . Correspondingly , the empirical loss in the PAC-Bayes paradigm is the expected loss over the draw of functions from the posterior : L̂ ( Df , S ) = Ef∼Df 1 n n∑ i=1 l ( f , xi , yi ) ( 1 ) PAC-Bayes theory suggests the gap between the expected loss and the empirical loss is bounded by a term that is related to the KL divergence between Df and πf ( McAllester , 1999 ) ( Langford & Shawe-Taylor , 2002 ) . In particular , if the function f is parameterized as f ( w ) with w ∈ W , when Dw is perturbed around any w , we have the following PAC-Bayes bound ( Seldin et al. , 2012a ) ( Seldin et al. , 2012b ) ( Neyshabur et al. , 2017 ) ( Neyshabur et al. , 2018 ) : Theorem 1 ( PAC-Bayes-Hoeffding Perturbation ) . Let l ( f , x , y ) ∈ [ 0 , 1 ] , and π be any fixed distribution over the parametersW . For any δ > 0 and η > 0 , with probability at least 1 − δ over the draw of n samples , for any w and any random perturbation u , Eu [ L ( w + u ) ] ≤ Eu [ L̂ ( w + u ) ] + KL ( w + u||π ) + log 1δ η + η 2n ( 2 ) One may further optimize η to get a bound that scales approximately as Eu [ L ( w+u ) ] . Eu [ L̂ ( w+ u ) ] + 2 √ KL ( w+u||π ) +log 1δ 2n ( Seldin et al. , 2012b ) . 3 A nice property of the perturbation bound ( 2 ) is it connects the generalization with the local properties around the solution w through some perturbation u around w. In particular , suppose L̂ ( w∗ ) is a local optimum , when the perturbation level of u is small , Eu [ L̂ ( w∗ + u ) ] tends to be small , but KL ( w∗ + u||π ) may be large since the posterior is too “ focused ” on a small neighboring area around w∗ , and vice versa . As a consequence , we may need to search for an “ optimal ” perturbation level for u so that the bound is minimized . 3 MAIN RESULT . While some researchers have already discovered empirically the generalization ability of the models is related to the second order information around the local optima , to the best of our knowledge there is no work on how to connect the Hessian matrix∇2L̂ ( w ) with the model generalization rigorously . In this section we introduce the local smoothness assumption , as well as our main theorem . It may be unrealistic to assume global smoothness properties for the deep models . Usually the assumptions only hold in a small local neighborhood Neigh ( w∗ ) around a reference point w∗ . In this paper we define the neighborhood set as Neighκ ( w ∗ ) = { w | |wi − w∗i | ≤ κi ∀i } where κi ∈ R+ is the “ radius ” of the i-th coordinate . In our draft we focus on a particular type of radius κi ( w∗ ) = γ|w∗i |+ , but our argument holds for other types of radius , too . In order to get a control of the deviation of the optimal solution we need to assume inNeighγ , ( w∗ ) , the empirical loss function L̂ in ( 1 ) is Hessian Lipschitz , which is defined as : Definition 1 ( Hessian Lipschitz ) . A twice differentiable function f ( · ) is ρ-Hessian Lipschitz if : ∀w1 , w2 , ‖∇2f ( w1 ) −∇2f ( w2 ) ‖ ≤ ρ‖w1 − w2‖ , ( 3 ) where ‖ · ‖ is the operator norm . The Hessian Lipschitz condition has been used in the numeric optimization community to model the smoothness of the second-order gradients ( Nesterov & Polyak , 2006 ) ( Carmon et al. , 2018 ) ( Jin et al. , 2018 ) . In the rest of the draft we always assume the following : 3Since η can not depend on the data , one has to build a grid and use the union bound . Assumption 1 . In Neighκ ( w∗ ) the empirical loss L̂ ( w ) defined in ( 1 ) is convex , and ρ-Hessian Lipschitz . For the uniform perturbation , the following theorem holds : Theorem 2 . Suppose the loss function l ( f , x , y ) ∈ [ 0 , 1 ] , and model weights are bounded |wi| + κi ( w ) ≤ τi ∀i . With probability at least 1 − δ over the draw of n samples , for any w̌ ∈ Rm such that assumption 1 holds Eu [ L ( w̌ + u ) ] ≤ L̂ ( w̌ ) +O √m+∑i log τiσ̌i + log 1δ n where ui ∼ U ( −σ̌i , σ̌i ) are i.i.d . uniformly distributed random variables , and σ̌i ( w̌ , η , γ ) = min ( √ 1 √ mn ( ∇2i , iL̂ ( w̌ ) /3 + ρm1/2κi ( w̌ ) /9 ) , κi ( w̌ ) ) ( 4 ) Theorem 2 says if we choose the perturbation levels carefully , the expected loss of a uniformly perturbed model is controlled . The bound is related to the diagonal element of Hessian ( logarithmic ) , the Lipschitz constant ρ of the Hessian ( logarithmic ) , the neighborhood scales characterized by κ ( logarithmic ) , the number of parameters m , and the number of samples n. Also roughly the perturbation level is inversely related to √ ∇2i , iL̂ , suggesting the model be perturbed more along the coordinates that are “ flat ” .4 Similar argument can be made on the truncated Gaussian perturbation , which is presented in Appendix B . In the next section we walk through some intuitions of our arguments .

Summary: The authors study generalization capabilities of neural networks local minimums thanks to a PAC-Bayesian analysis that grasps the local smoothness properties. Even if some assumptions are made along the way, their analysis provides a metric that gives insight on the accuracy of a solution, as well as an optimization algorithm. Both of these result show good empirical behavior.