How noise affects the Hessian spectrum in overparameterized neural networks

1 INTRODUCTION . Deep neural networks have achieved remarkable success in the past decade on tasks that were out of reach prior to the era of deep learning . Yet fundamental questions remain regarding the strong performance of over-parameterized models and optimization schemes that typically involve only first-order information , such as stochastic gradient descent ( SGD ) and its variants . Regarding generalization , it has been noted that flat minima with small curvature tend to generalize better than sharp minima ( Hochreiter & Schmidhuber , 1997 ; Keskar et al. , 2017 ) . This has been argued by nonvacuous PAC-Bayes bounds ( Dziugaite & Roy , 2017 ) and Bayesian evidence ( Smith & Le , 2018 ) . But why does SGD bias learning towards flat minima ? One possible explanation was proposed by Zhang et al . ( 2018 ) in terms of energy-entropy competition . Jastrzbski et al . ( 2018 ) also suggests that isotropic noise in SGD helps networks escape sharp minima with large Hessian determinant . However , previous theoretical analyses assume that minima are isolated and non-singular . In contrast , Sagun et al . ( 2017b ) finds that most of the eigenvalues of the Hessian of the loss function at a minimum are close to zero , indicating highly degenerate minima . The degeneracy is further supported by results on mode connectedness ( Garipov et al. , 2018 ; Draxler et al. , 2018 ) . Furthermore , it is shown that minima found from different initializations and the same optimization scheme are connected with essentially no barriers between them . Nguyen ( 2019 ) also shows theoretically that for a class of deep overparameterized neural nets with piecewise linear activation functions , all of the global minima are connected within a unique and potentially very large global valley . In this paper , we prove that for models whose loss has a ” minimal valley ” structure , defined below , optimization via SGD decreases the trace of Hessian of the loss , by utilizing recent fluctuationdissipation relations ( Yaida , 2019 ) . Furthermore , we derive the noise covariance matrix that would result in the reduction of other potentially desired quantities such as the Hessian determinant , leading towards the design of new optimization algorithms . We present experiments on toy models and deep neural networks to confirm our predictions . 2 MAIN THEOREM . In this section , we present our main theorem - how noise during optimization affects the Hessian of the loss function when the loss landscape locally takes the shape of a degenerate valley . More specifically , let N and n be the dimension of the total parameter space and non-degenerate space , respectively , and consider a general loss function L ( w ) wherew ∈ RN are the model parameters.1 Since the curvature around the minima can vary , we approximate the loss function in such a valley around a degenerate minimum with a modified quadratic form L ( w ) = L∗ + 12 ( w − P ( w ) ) TH ( P ( w ) ) ( w − P ( w ) ) where P is a function that projects a point w in parameter space to the nearest minimum , and H is the Hessian , a function of the location of the projected minimum . We have the following lemma : Lemma 1 . For an arbitrary point w and its neighborhood in the valley , there exists an orthogonal transformation Q and a translation vector v such that the loss function in the new coordinate system θ = Qw − v has the following form , L ( θ ) = L∗ + 1 2 n∑ i=1 θ2i λi ( θn+1 , ... , θN ) , ( 1 ) where λis are the positive eigenvalues of the loss function Hessian for the non-degenerate space and depend on the position in the degenerate space . Also , note that the gradient descent equation is invariant under this transformation . A detailed , constructive proof of this lemma can be found in Appendix A.1 . Notice that this is a generalization of Morse ’ s Lemma ( Callahan , 2010 ) . In the rest of this section , we denote the nondegenerate and degenerate subspaces by θ̄ = ( θ1 , ... , θn ) T and θ̂ = ( θn+1 , ... , θN ) T , respectively . Similarly , the gradients of θ̄ and θ̂ are denoted by ∇̄ and ∇̂ , respectively . Notice that at a minimum where θ̄ = 0 , the λis are the only nonzero Hessian eigenvalues . Next we provide a quick review of the relevant fluctuation-dissipation relation formalism for stochastic gradient descent ( Yaida , 2019 ) . We denote the loss for a random batchB of training sample as LB ( θ ) . Clearly we have [ [ ∇LB ( θ ) ] ] m.b . = ∇L ( θ ) , ( 2 ) where [ [ • ] ] m.b . represents the average over all mini-batch realizations . If there exists a stationary state for θ̄ , pss ( θ̄ ) and the stationary-average is defined as 〈O ( θ̄ ) 〉 ≡ ∫ dθ̄pss ( θ̄ ) O ( θ̄ ) where O ( θ̄ ) is any observable of θ̄ , it is straightforward to derive the following master equation : 〈O ( θ̄ ) 〉 = 〈 [ [ O [ θ̄ − η∇̄LB ( θ ) ] ] ] m.b.〉 . ( 3 ) We denote the two-point noise matrix as C̃i , j ( θ ) ≡ [ [ ∂θiLB ( θ ) ∂θjLB ( θ ) ] ] m.b . and the noise covariance matrix Ci , j ( θ ) ≡ C̃i , j ( θ ) − ∂θiL ( θ ) ∂θjL ( θ ) . Now we present our main theorem : Theorem 1 . When the loss can be locally approximated as in Equation 1 , assuming that the nondegenerate space θ̄ is in a stationary state at time t and that the noise covariance matrix is aligned with the Hessian , we have 〈 [ [ Tt+1 − Tt ] ] m.b.〉 ≤ 0 , where T = ∑n i=1 λi is the trace of the Hessian and Tt represents the trace at training step t. Equality holds when∇LB ( θ ) = ∇L ( θ ) or ∇̂T ( θ̂ ) = 0 . Theorem 1 indicates that the change of the Hessian trace during SGD optimization is non-positive on average . A trivial condition for equality is optimization with gradient descent ( GD ) instead of its stochastic variant because the stationary state for noiseless optimization is a constant state with vanishing gradient . An alternate condition for equality is that the trace function of θ̄ reaches a minimum or saddle point . Before proving Theorem 1 , we will first analyze the assumptions made . We emphasize at the outset that we expect our assumptions to be valid only after an early phase of training corresponding to ∼ 10 epochs in our experiments . 2.1 EVIDENCE FOR THE ASSUMPTIONS MADE . In this subsection , we will analyze the assumptions in Theorem 1 and present some experiments that support their validity . 1Here the non-degenerate space corresponds to the subspace of the Hessian with non-zero eigenvalues . Note that we do not require any specific relationship between n and N , e.g . n N . 2.1.1 MINIMAL VALLEY . Our theory relies on the existence of a degenerate valley in the loss function defined previously via Equation 1 . In addition to degeneracy , we furthermore assume that there exists a region of connected degenerate minima in the landscape of the loss function , in particular for overparameterized models . SGD or its variants then selects one of the degenerate solutions . Such degeneracy has been observed empirically where most of the eigenvalues of the loss Hessian from various models and tasks are zero or close to zero ( Sagun et al. , 2017a ; b ) . Furthermore , the connectedness is further supported by results of mode connectedness ( Garipov et al. , 2018 ; Draxler et al. , 2018 ) . Nguyen ( 2019 ) also shows theoretically that for a class of deep overparameterized neural nets with piecewise linear activation functions , all of the global minima are connected within a unique and potentially very large global valley . Here we present additional experiments to support the existence of such minimal valleys , i.e . basins of attraction of connected , degenerate minima that can be locally described by Equation 1 . These experiments will be also used to analyze and confirm our theory later . We consider classification of CIFAR10 with label-smoothing cross entropy ( Szegedy et al. , 2016 ) 2 , constant learning rate of 0.1 , momentum 0.9 , batch size 500 and total training epochs 150 . The network architectures are Preact-ResNet18 ( He et al. , 2016 ) and VGG16 ( Simonyan & Zisserman , 2015 ) . The training and validation loss are shown in Figure 1 ( inset ) . After 150 training epochs , the networks fluctuates with small training loss , and the minimal training loss for both case is ∼ 0.005 . In order to confirm the existence of minimal valleys , we would like to project each point along the training path to the corresponding closest minimum 3 and check whether the projected path crosses any barriers . To determine whether two minima can be connected by a line segment in a minimal valley implying no barriers in between , we use line interpolation to connect these two minima and measure the training loss along the interpolating line . In practice , we call it line connectedness if we sample evenly 10 points in the line segment between consecutive minima and the training loss is smaller than a threshold for all 10 points . To project a state to a closest minimum , the model is trained from that state with the same hyperparameters except using GD instead of SGD to remove noise . The stopping criterion for these auxiliary training tasks is that the training loss is less than the minimal training loss , which is ∼ 0.005 . Ideally , we would project states after each iteration and check for line connectedness , but this involves large unnecessary calculations . In practice , if state B is obtained from n training iterations after state A , and A and B are not line connected , we insert C which is obtained from n/2 training iterations after A and check for line connectedness between A and C and between C and B . This process stops when there is no extra iteration between A and B or they are line connected . Starting from each pair of consecutive epochs as A and B , we obtain the projected path recursively . The training and validation losses on these two projected paths are shown in Figure 1 ( main plot ) . The left path has 2700 projected points and the right has 4020 . We see that after a few epochs , the training loss along the path remains close to zero ( and hence minimal ) , which means that there exists a least one minimal valley connected to the model state found by regular SGD . Also SGD optimization happens within such a valley after the first few epochs . Furthermore , the validation loss varies along the valley . More experiments can be found in Appendix A.2 . Further discussion of violation of this minimal valley assumption can be found in Appendix A.6 .

Summary: This paper studies the behavior of SGD after it has reached a steady state and settled in a minimal valley. The authors show that even if SGD has reached a minimal valley, the noisy updates provided by a minibatch of samples result in the trace of the Hessians reducing with further SGD iterations. The authors also show that by changing the type of noise that is added to GD, one can get different functions of the Hessian to reduce with SGD iterations. The theoretical conclusions are then verified empirically with synthetic examples and real deep learning examples