A Walk with SGD: How SGD Explores Regions of Deep Network Loss?

1 INTRODUCTION . The non-convexity of the deep neural network ( DNN ) loss surface makes the behavior of optimization algorithms less intuitive compared to the convex setting . Moreover , optimization in DNNs is no longer about finding any minimum , but rather about finding ones that generalizes well ( Keskar et al. , 2016 ) . Since deep networks are initialized randomly , finding such minima will require exploration of different regions of the loss surface . This intuition has been formalized in recent papers that study stochastic gradient descent ( SGD ) as a diffusion process ( Hoffer et al. , 2017 ; Smith & Le , 2017 ; Jastrzebski et al. , 2017 ; Chaudhari & Soatto , 2017 ) . Briefly , these papers show that SGD simulates a discrete approximation of stochastic differential equation ( SDE ) , and hence performs a random walk on the potential induced by the DNN loss surface . In this work , we complement the diffusion perspective of SGD with a qualitative view of how SGD explores different regions of the non-convex loss landscape of deep neural networks through empirical evidence . Intuitively , when performing random walk on a potential , one would expect barriers being crossed quite often during the process . We show in this work that SGD rarely crosses any barriers along its path during the course of training . By this observation , we do not claim that SGD does not simulate diffusion . Through experimental deductions , we show an alternate mechanism that SGD seems to dominantly use to explore different regions of the non-convex loss landscape . Further , it is known that larger batch-sizes slow down the diffusion process ( Hoffer et al. , 2017 ) . We show the qualitative reason behind this slow down to be an oscillation behavior of SGD which prevents it from moving far away from initialization . This behavior is a result of the mini-batch gradients becoming increasingly aligned with the top eigenvectors of the Hessian for larger batch-sizes . This behavior is known to slow down convergence in optimization theory ( for instance consider the motivation behind momentum ( Polyak , 1964 ; Sutskever et al. , 2013 ) ) . We discuss how it also slows down explorations in the non-convex setting of deep network loss surface . Experiments are conducted on multiple data sets , architectures and hyper-parameter settings . The findings mentioned above hold true on all of them . 2 SETUP . We now describe the details of how we study the existence of barriers along the optimization path of SGD . The main tool we use for studying the DNN loss surface along SGD ’ s path is to interpolate the loss surface between parameters before and after each training update . We note that this strategy of interpolating the loss surface between parameters was introduced by Goodfellow et al . ( 2014 ) . In their paper , the interpolation is conducted between initial and final ( after training ) parameter values for analysis purposes . In contrast , we compute interpolations before and after each training update because this interpolation precisely tells us whether or not SGD crosses a barrier during an update step . We say a barrier is crossed when we see a point in the parameter space interpolated between the parameters just before and after an update step , such that the loss at the barrier point is higher than the loss at both the other points . Consider that the parameters θ of a neural network are initialized to a value θ0 . When using an optimization method to update these parameters , the tth update step takes the parameter from θt to θt+1 using estimated gradient gt as , θt+1 = θt − ηgt ( 1 ) where η is the learning rate . Notice the tth update step implies the tth epoch only in the case when using the full batch gradient descent ( GD ) . In the case of stochastic gradient descent , one iteration is an update from gradient computed from a mini-batch . We then interpolate the DNN loss between the convex combination of θt and θt+1 by considering parameter vectors θαt = ( 1− α ) θt + αθt+1 , where α ∈ [ 0 , 1 ] is chosen such that we obtain 10 samples uniformly placed between these two parameter points . We note that even though the updates are performed using mini-batches for SGD , the training loss values we compute for the interpolation use the full dataset to visualize the actual loss landscape . 3 BARRIERS AND EXPLORATION DURING SGD TRAINING . For this section , we perform experiments on MNIST ( Lecun & Cortes ) and CIFAR-10 ( Krizhevsky , 2009 ) datasets , and use multi-layer perceptrons ( MLP ) , VGG-11 ( Simonyan & Zisserman , 2014 ) and Resnet-56 ( He et al. , 2016 ) architectures with various batch sizes and learning rates . We discuss our observations for VGG-11 architecture on CIFAR-10 dataset ( figure 1 ) as a reference but the same conclusions hold for experiments on MLP trained on MNIST ( figure 3 ) and Resnet-56 trained on CIFAR-10 ( figure 2 ) . We train VGG-11 on CIFAR-10 with a batch size of 100 and fixed learning rate of 0.1 . We report the visualization of loss interpolation between consecutive iterations for 40 iterations from epochs 1 , 2 , 25 and 100 for visual clarity . The interpolation is shown in figure 1 . To be clear , the x-axis is calibrated by the number of iterations , and there are 10 interpolated loss values between each consecutive iteration ( vertical gray lines ) in the training loss plots . In these plots , we find two interesting behavior of SGD . First , we find that the interpolated loss between every consecutive update from SGD optimization update appears to be a quadratic-like structure with a minimum in between . Note that while this is not visible for epochs 25 and 100 , we later show quantitative measurements that ensures this claim . This plot thus shows that in the iterations plotted , SGD rarely crosses barriers . Second , we observe how the minimum of each interpolation evolves as training progresses . This is highlighted in figure 1 ( a ) with a dashed orange line . We find that this minimum has ups and downs along the path of SGD for all our interpolation plots . To draw deductions from this observation , consider a simple example that helps us understand this scenario concretely . Let parameter points θA , θB and θC be a result of three consecutive SGD updates with loss values ` A , ` B and ` C ( using full training set ) . Note that since these are only three points , they exist in a two dimensional subspace and the loss value can be imagined along the third dimension . Then corresponding to the behavior in the plot , there is a parameter point θAB between θA and θB on the line connecting these two points , which has a loss value ` AB < ` A , ` B . Similarly there is a point θBC between θB and θC on the line connecting these two points , which has a loss value ` BC < ` B , ` C . Given this construction , for any configuration of θA , θB and θC on the two dimensional plane , it is easy to see that if ` AB < ` BC , any path from θAB to θBC will have loss values that must increase at some point . Hence , what this construction essentially represents ( as we refer to it ) , is a situation where SGD has moved over a barrier . Therefore , the ups and downs of the minimum between loss interpolations in figure 1 ( a , b , c , d ) represents SGD moving over barriers . In this way we find that when running SGD on the loss surface of deep networks , instead of crossing barriers , a more dominant way SGD performs exploration is by moving over them . The same qualitative analysis for SGD with different hyper-parameters are also shown in section 1 in appendix . The observations we described here remain consistent for all these experiments . So far we showed qualitative visualizations to make the claim that SGD rarely crosses barriers . In order to show that the claim extends to the rest of the training instead of only a few iterations we showed above , we now quantitatively measure how many barriers are crossed for the entire epoch in different phase of training . This result is shown in table 1 for VGG-11 and Resnet-56 trained on CIFAR-10 ( trained for 100 epochs ) and an MLP trained on MNIST ( trained for 40 epochs ) . We note that each case , an epoch consists of more than 450 iterations . As we see , a negligible number of barriers are crossed for most parts of the training compared to the number of iterations performed during each epoch . For concreteness , we further compute the number of barriers crossed for the first 40 epochs for VGG-11 on CIFAR-10 as shown in Figure 4 and reach the same conclusion . at either points . Note that even for these barriers , their heights ( defined by L ( θt ) +L ( θt+1 ) −2L ( θ min t ) 2 ) are substantially smaller compared with the value of loss at the corresponding iterations ( not mentioned here ) , meaning they are not significant barriers . 4 THE EFFECT OF BATCH-SIZE ON EXPLORATION . 4.1 ANALYSIS . Hoffer et al . ( 2017 ) discuss that SGD training with different batch-sizes leads to different diffusion rates ( very large batch sizes being slower ) . Further , when training for the same number of epochs , a larger batch training performs less number of iterations . Combining these two observations , they reach the conclusion that large batch training makes the diffusion process slow . As empirical evidence , they show that the distance of parameters from initialization evolves logarithmically in the number of iterations . We now present a complementary optimization perspective to their observation . To continue , we introduce the following notations . Let pi ( θ ) denote the predicted probability output ( of the correct class in the classification setting for instance ) of a DNN parameterized by θ for the ith data sample ( in total N samples ) . Then the negative log likelihood loss for the ith sample is given by Li ( θ ) = − log ( pi ( θ ) ) . The gradient gB ( θ ) from mini-batch SGD at a parameter value θ is expressed as , gB ( θ ) = 1B ∑ i∈B ∂Li ( θ ) ∂θ , ḡ ( θ ) denotes the expected gradient using all training samples , B is the mini-batch size ( and we have also overloaded it to mean the mini-batch set ) and C ( θ ) is the gradient covariance matrix at θ . Then the relation between the Hessian H ( θ ) and the dataset gradient covariance C ( θ ) for negative log likelihood loss is described by the Gauss-Newton decomposition as follows , H ( θ ) = C ( θ ) + ḡ ( θ ) ḡ ( θ ) T + 1 N N∑ i=1 ∂Li ( θ ) ∂pi ( θ ) · ∂ 2pi ( θ ) ∂w2 ( 2 ) where H ( θ ) is the Hessian of the loss . The derivation can be found in section B of the appendix . To continue with our argument , we note that it has been discussed by Shwartz-Ziv & Tishby ( 2017 ) that early on during training , the mean gradient over the training set is larger in magnitude compared to the variance in gradients . The above argument essentially says that the scale of mean gradient ḡ ( θ ) is larger compared with the scale of C ( θ ) . Ignoring the second order term in the Gauss-Newton decomposition above , we see that the mean gradient must be aligned with the top eigenvectors of the Hessian since the scale of gradient covariance is much smaller early on during training . Finally , we note that using large batch-sizes makes the mini-batch gradient closer to the mean gradient by reducing the scale of mini-batch gradient covariance as shown by Hoffer et al . ( 2017 ) , cov ( gB ( θ ) , gB ( θ ) ) = ( 1 B − 1 N ) C ( θ ) ( 3 ) The two arguments together imply that gradients from larger batch-sizes are likely to be more aligned with the high curvature directions of the loss surface especially early on during training . In convex optimization theory , when gradients point along the top eigenvectors of the Hessian ( also referred to as the sharp directions of the loss surface ) , optimization exhibits under-damped convergence , meaning it oscillates along the sharp directions in the case when the learning rate is smaller than a certain threshold . Applying this idea to non-convex loss landscapes , a large alignment between the mini-batch gradient and the sharp directions should also lead to oscillations . At this point , we depart from the conclusions of the convex setting and recall our observation in the previous section that the interpolation between consecutive iterations has a quadratic like shape and SGD moves over barriers for the deep network loss surface . We thus hypothesize that a lower alignment between mini-batch gradients and the sharp directions of the loss surface makes SGD exploration faster by exhibiting less oscillation , and vice-versa .

Summary: The subject of how a given algorithm explores the landscape is still a poorly understood area in training neural networks. There is a large body of recent work that attempts to shed light on this puzzle, and each one tries to claim their share in the furthering of the understanding of the relationship between the geometry of the landscape and the dynamics that one chooses in optimization. The present paper is a fine addition to the literature with interesting observations and novel questions, however, it falls short in many core areas: An apparent work in progress that has a great potential.