Information-Theoretic Local Minima Characterization and Regularization

1 INTRODUCTION . Recently , there has been a surge in the interest of acquiring a theoretical understanding over deep neural network ’ s behavior . Breakthroughs have been made in characterizing the optimization process , showing that learning algorithms such as stochastic gradient descent ( SGD ) tend to end up in one of the many local minima which have close-to-zero training loss ( Choromanska et al. , 2015 ; Dauphin et al. , 2014 ; Kawaguchi , 2016 ; Nguyen & Hein , 2018 ; Du et al. , 2018 ) . However , these numerically similar local minima typically exhibit very different behaviors in terms of generalizability . It is , therefore , natural to ask two closely related questions : ( a ) What kind of local minima can generalize better ? ( b ) How to find those better local minima ? To our knowledge , existing work focused only on one of the two questions . For the “ what ” question , various definitions of “ flatness/sharpness ” have been introduced and analyzed ( Keskar et al. , 2017 ; Neyshabur et al. , 2018 ; 2017 ; Wu et al. , 2017 ; Liang et al. , 2017 ) . However , they suffer from one or more of the problems : ( 1 ) being mostly theoretical with no or poor empirical evaluations on modern neural networks , ( 2 ) lack of theoretical analysis and understanding , ( 3 ) in practice not applicable to finding better local minima . Regarding the “ how ” question , existing approaches ( Hochreiter & Schmidhuber , 1997 ; Sokolić et al. , 2017 ; Chaudhari et al. , 2017 ; Hoffer et al. , 2017 ; Neyshabur et al. , 2015a ; Izmailov et al. , 2018 ) share some of the common drawbacks : ( 1 ) derived only from intuitions but no specific metrics provided to characterize local minima , ( 2 ) no or weak analysis of such metrics , ( 3 ) not applicable or no consistent generalization improvement for modern DNNs . In this paper , we tackle both the “ what ” and the “ how ” questions in a unified manner . Our answer provides both the theory and applications for the generalization problems across different local minima . Based on the determinant of Fisher information estimated from the training set , we propose a metric that solves all the aforementioned issues . The metric can well capture properties that characterize local minima of different generalization ability . We provide its theoretical analysis , primarily a generalization bound based on PAC-Bayes ( McAllester , 1999b ; a ) . For modern DNNs in practice , it is necessary to provide a tractable approximation of our metric . We propose an intuitive and efficient approximation to compare it across different local minima . Our empirical evaluations fully illustrate the effectiveness of the metric as a strong indicator of local minima ’ s generalizability . Moreover , from the metric we further derive and design a practical regularization technique that guides the optimization process in finding better generalizable local minima . The experiments on image classification datasets demonstrate that our approach gives consistent generalization boost for a range of DNN architectures . 2 RELATED WORK . It has been empirically shown that larger batch sizes lead to worse generalization ( Keskar et al. , 2017 ) . Hoffer et al . ( 2017 ) analyzed how the training dynamics is affected by different batch sizes and presented a perturbed batch normalization technique for better generalization . While it effectively improves generalization for large-batch training , a specific metric that indicates the generalizability is missing . Similarly , Elsayed et al . ( 2018 ) employed a structured margin loss to improve performance of DNNs w.r.t . noise and adversarial attack yet no metric was proposed . Furthermore , this approach essentially provided no generalization gain in the normal training setup . The local entropy of the loss landscape was proposed to measure “ flatness ” in Chaudhari et al . ( 2017 ) , which also designed an entropy-guided SGD that achieves faster convergence in training DNNs . However , the method does not consistently improve generalization , e.g. , a decrease of performance on CIFAR-10 ( Krizhevsky & Hinton , 2009 ) . Another method that focused on modifying the optimization process is the Path-SGD proposed by Neyshabur et al . ( 2015a ) . Specifically , the authors derived an approximate steepest descent algorithm that utilizes the path-wise norm regularization to achieve better generalization . The authors only evaluated it on a two-layer neural network , very likely since the path norm is computationally expensive to optimize during training . A flat minimum search algorithm was proposed by Hochreiter & Schmidhuber ( 1997 ) based on the “ flatness ” of local minima defined as the volume of local boxes . Yet since the boxes have their axes aligned to the axes of the model parameters , their volumes could be significant underestimations of “ flatness ” for over-parametrized networks , due to the specific spectral density of Hessian of DNNs studied in Pennington & Worah ( 2018 ) ; Sagun et al . ( 2018 ) . The authors of Wu et al . ( 2017 ) also characterized the “ flatness ” by volumes . They considered the inverse volume of the basin of attraction and proposed to use the Frobenius norm of Hessian at the local minimum as a metric . In our experiments , we show that their metric does not accurately capture the generalization ability of local minima under different scenarios . Moreover , they have not derived a regularizer from their metric . Based on a “ robustness ” metric , Sokolić et al . ( 2017 ) derived a regularization technique that successfully improves generalization on multiple image classification datasets . Nevertheless , we show that their metric fails to capture the generalizability across different local minima . By using the Bayes factor , MacKay ( 1992 ) studied the generalization ability of different local minima obtained by varying the coefficient of L2 regularization . It derived a formula involving the determinant of Hessian , similar to the one in ours . Whereas , this approach has restricted settings and , without proposing an efficient approximation , its metric is not applicable to modern DNNs , let alone serving as a regularizer . A generalization bound is missing in MacKay ( 1992 ) as well . In a broader context of the “ what ” question , properties that capture the generalization of neural networks have been extensively studied . Various complexity measures for DNNs have been proposed based on norm , margin , Lipschitz constant , compression and robustness ( Bartlett & Mendelson , 2002 ; Neyshabur et al. , 2015b ; Sokolić et al. , 2017 ; Xu & Mannor , 2012 ; Bartlett et al. , 2017 ; Zhou et al. , 2019 ; Dziugaite & Roy , 2017 ; Arora et al. , 2018 ; Jiang et al. , 2019 ) . While some of them aimed to provide tight generalization bounds and some of them to provide better empirical results , none of the above approaches explored the “ how ” question at the same time . Very recently , Karakida et al . ( 2019 ) and Sun & Nielsen ( 2019 ) studied the Fisher information of the neural network through the lens of its spectral density . In specific , Karakida et al . ( 2019 ) applied mean field theory to study the statistics of the spectrum and the appropriate size of the learning rate . Also an information-theoretic approach , Sun & Nielsen ( 2019 ) derived a novel formulation of the minimum description length in the context of deep learning by utilizing tools from singular semi-Riemannian geometry . 3 OUTLINE AND NOTATIONS . In a typical K-way classification setting , each sample x ∈ X belongs to a single class denoted cx ∈ { 0 , 1 , ... , K } according to the probability vector y ∈ Y , where Y is the k-dimensional probability simplex so that p ( cx = i ) = yi and ∑ i yi = 1 . Denote a feed-forward DNN parametrized by w ∈ RW as fw : X → Y , which uses nonlinear activation functions and a softmax layer at the end . Denote the cross entropy loss as ` ( fw ( x ) , y ) = − ∑ i yi ln fw ( x ) i. De- note the training set as S , defined over X × Y with |S|= N . The training objective is given as L ( S , w ) = 1N ∑ ( x , y ) ∼S ` ( fw ( x ) , y ) . Assume S is sampled from some true data distribution denoted D , we can define expected loss L ( D , w ) = E ( x , y ) ∼D [ ` ( fw ( x ) , y ) ] . Throughout this paper , we refer a local minimum of L ( S , w ) corresponding to a local minimizer w0 as just the local minimum w0 . Given such w0 , our paper ’ s outline as well as our main achievements are : • In Section 4 we relates Fisher information to neural network training as a prerequisite . • In Section 5.1 we propose a metric γ ( w0 ) that well captures local minima ’ s generalizability . • In Section 5.2 we provide a generalization bound related to γ ( w0 ) . • In Section 5.3 we propose an approximation γ̂ ( w0 ) for γ ( w0 ) , which is shown to be very effective in Section 7.1 via extensive empirical evaluations . • In Section 6 we devise a practical regularizer from γ ( w0 ) that consistently improves gen- eralizability across different DNNs , as evaluated in Section 7.2 . 3.1 OTHER NOTATIONS . Denote∇w as gradient , Jw [ · ] as Jacobian matrix , ∇2w as Hessian , DKL ( ·‖· ) as KL divergence , ‖·‖2 as spectrum or Euclidean norm , ‖·‖F as Frobenius norm , |·| as determinant , tr ( · ) as trace norm , ρ ( · ) as spectral radius , `` S ( w ) as log-likelihood on S , and [ · ] i for selecting the ith entry . We define ` x ( w ) ∈ RK whose ith entry is− ln fw ( x ) i so that ` ( fw ( x ) , y ) = ` x ( w ) T y . We define ȳ as argmax ( y ) and ỹ ∈ RK the one-hot vector whose ȳ-th entry is 1 and otherwise 0 . Then we define L̃ ( S , w ) ∈ RN as the “ simplified ” loss vector of S whose entries are ` ( fw ( x ) , ỹ ) for ( x , y ) ∈ S , i.e. , we approximate the cross entropy loss ` ( fw ( x ) , y ) by ` ( fw ( x ) , ỹ ) . 4 LOCAL MINIMUM AND FISHER INFORMATION . First of all , if y is strictly one-hot , no local minimum will even exist with 100 % training accuracy , since the cross entropy loss will always be positive . To admit good local minima in the first place , we assume the widely used label smoothing ( Szegedy et al. , 2016 ) is applied to train all models in our analysis . Label smoothing enables us to assume a local minimum w0 ( in this case , also a global minimum ) of the training loss with ∑ ( x , y ) ∈S DKL ( fw0 ( x ) ‖y ) = 0 . Each sample ( x , y ) ∈ S has its label cx sampled by p ( cx = i|x ) = yi , denoted as ( x , cx ) ∼ S. The joint probability p ( x , cx ) modeled by the DNN is p ( x , cx = i ; w ) = p ( cx = i|x ; w ) p ( x ) = [ fw ( x ) ] i p ( x ) with p ( x ) = 1N . We can relate the training loss L ( S , w ) to the negative log-likelihood − `` S ( w ) = − ∑ ( x , y ) ∈S Ecx∼y ln p ( x , cx ; w ) by : L ( S , w ) = 1 N ∑ ( x , y ) ∈S ` x ( w ) T y = − 1 N ∑ ( x , y ) ∈S E cx∼y ln p ( cx|x ; w ) = − 1 N `` S ( w ) + ln 1 N And so w0 also corresponds to a local maximum of the likelihood function . The observed Fisher information evaluated at w0 is defined as I ( w0 ) = − 1N∇ 2 w `` S ( w0 ) . We can further derive : I ( w0 ) = ∇2wL ( S , w0 ) = E ( x , cx ) ∼S [ ∇w ln p ( cx|x ; w0 ) ∇w ln p ( cx|x ; w0 ) T ] ( 1 ) The first equality is straightforward ; the second has its proof in Appendix A . Since p ( cx = i|x ) = yi and ln p ( cx = i|x ; w0 ) = [ ` x ( w0 ) ] i , we can further simplify the Equation 1 to : I ( w0 ) = 1 N ∑ ( x , y ) ∈S K∑ i=1 ( ∇w [ ` x ( w0 ) ] i ) ( ∇w [ ` x ( w0 ) ] i ) T ( 2 ) Remark : When we assume global optimality , we have ∇w ` ( fw0 ( x ) , y ) = 0 as DKL ( fw0 ( x ) ‖y ) = 0 ; yet it does not indicate I ( w0 ) ∈ RW×W 6= 0 in Equation 2 .

Summary: This paper contributes to the deep learning generalization theory, mainly from the theoretical perspective with experimental verifications. The key proposition is given by the unnumbered simple equation in the middle of page 4 (please number it), where \mathcal{I} is the Fisher information matrix. According to the authors, this simple metric, which is the log-determinant of the Fisher information matrix, can characterize the generalization of a DNN.