Directional Analysis of Stochastic Gradient Descent via von Mises-Fisher Distributions in Deep Learning

1 INTRODUCTION . Stochastic gradient descent ( SGD ) has been a driving force behind the recent success of deep learning . Despite a series of work on improving SGD by incorporating the second-order information of the objective function ( Roux et al. , 2008 ; Martens , 2010 ; Dauphin et al. , 2014 ; Martens & Grosse , 2015 ; Desjardins et al. , 2015 ) , SGD is still the most widely used optimization algorithm for training a deep neural network . The learning dynamics of SGD , however , has not been well characterized beyond that it converges to an extremal point ( Bottou , 1998 ) due to the non-convexity and highdimensionality of a usual objective function used in deep learning . Gradient stochasticity , or the signal-to-noise ratio ( SNR ) of the stochastic gradient , has been proposed as a tool for analyzing the learning dynamics of SGD . Shwartz-Ziv & Tishby ( 2017 ) identified two phases in SGD based on this . In the first phase , “ drift phase ” , the gradient mean is much higher than its standard deviation , during which optimization progresses rapidly . This drift phase is followed by the “ diffusion phase ” , where SGD behaves similarly to Gaussian noise with very small means . Similar observations were made by Li & Yuan ( 2017 ) and Chee & Toulis ( 2018 ) who have also divided the learning dynamics of SGD into two phases . Shwartz-Ziv & Tishby ( 2017 ) have proposed that such phase transition is related to information compression . Unlike them , we notice that there are two aspects to the gradient stochasticity . One is the L2 norm of the minibatch gradient ( the norm stochasticity ) , and the other is the directional balance of minibatch gradients ( the directional stochasticity ) . SGD converges or terminates when either the norm of the minibatch gradient vanishes to zeros , or when the angles of the minibatch gradients are uniformly distributed and their non-zero norms are close to each other . That is , the gradient stochasticity , or the SNR of the stochastic gradient , is driven by both of these aspects , and it is necessary for us to investigate not only the holistic SNR but also the SNR of the minibatch gradient norm and that of the minibatch gradient angles . In this paper , we use a von Mises-Fisher ( vMF hereafter ) distribution , which is often used in directional statistics ( Mardia & Jupp , 2009 ) , and its concentration parameter κ to characterize the directional balance of minibatch gradients and understand the learning dynamics of SGD from the perspective of directional statistics of minibatch gradients . We prove that SGD increases the direc- tional balance of minibatch gradients . We empirically verify this with deep convolutional networks with various techniques , including batch normalization ( Ioffe & Szegedy , 2015 ) and residual connections ( He et al. , 2015 ) , on MNIST and CIFAR-10 ( Krizhevsky & Hinton , 2009 ) . Our empirical investigation further reveals that the proposed directional stochasticity is a major drive behind the gradient stochasticity compared to the norm stochasticity , suggesting the importance of understanding the directional statistics of the stochastic gradient . Contribution We analyze directional stochasticity of the minibatch gradients via angles as well as the concentration parameter of the vMF distribution . Especially , we theoretically show that the directional uniformity of the minibatch gradients modeled by the vMF distribution increases as training progresses , and verify this by experiments . In doing so , we introduce gradient norm stochasticity as the ratio of the standard deviation of the minibatch gradients to their expectation and theoretically and empirically show that this gradient norm stochasticity decreases as the batch size increases . Related work Most studies about SGD dynamics have been based on two-phase behavior ( Shwartz-Ziv & Tishby , 2017 ; Li & Yuan , 2017 ; Chee & Toulis , 2018 ) . Li & Yuan ( 2017 ) investigated this behavior by considering a shallow neural network with residual connections and assuming the standard normal input distribution . They showed that SGD-based learning under these setups has two phases ; search and convergence phases . Shwartz-Ziv & Tishby ( 2017 ) on the other hand investigated a deep neural network with tanh activation functions , and showed that SGD-based learning has drift and diffusion phases . They have also proposed that such SNR transition ( drift + diffusion ) is related to the information transition divided into empirical error minimization and representation compression phases . However , Saxe et al . ( 2018 ) have reported that the information transition is not generally associated with the SNR transition with ReLU ( Nair & Hinton , 2010 ) activation functions . Chee & Toulis ( 2018 ) instead looked at the inner product between successive minibatch gradients and presented transient and stationary phases . Unlike our work here , the experimental verification of the previous work conducted under limited settings – the shallow network ( Li & Yuan , 2017 ) , the specific activation function ( Shwartz-Ziv & Tishby , 2017 ) , and only MNIST dataset ( Shwartz-Ziv & Tishby , 2017 ; Chee & Toulis , 2018 ) – that conform well with their theoretical assumptions . Moreover , their work does not offer empirical result about the effect of the latest techniques including both batch normalization ( Ioffe & Szegedy , 2015 ) layers and residual connections ( He et al. , 2015 ) . 2 PRELIMINARIES . Norms and Angles Unless explicitly stated , a norm refers to L2 norm . ‖ · ‖ and 〈· , ·〉 thus correspond to L2 norm and the Euclidean inner product on Rd , respectively . We use xn ⇒ x to indicate that “ a random variable xn converges to x in distribution. ” Similarly , xn P→ x means convergence in probability . An angle θ between d-dimensional vectors u and v is defined by θ = 180π cos −1 ( 〈u , v〉 ‖u‖‖v‖ ) . Loss functions A loss function of a neural network is written as f ( w ) = 1n ∑n i=1 fi ( w ) , where w ∈ Rd is a trainable parameter . fi is “ a per-example loss function ” computed on the i-th data point . We use I and m to denote a minibatch index set and its batch size , respectively . Further , we call fI ( w ) = 1m ∑ i∈I fi ( w ) “ a minibatch loss function given I ” . In Section 3.1 , we use gi ( w ) and ĝ ( w ) to denote −∇wfi ( w ) and −∇wfI ( w ) , respectively . In Section 3.3 , the index i is used for the corresponding minibatch index set Ii . For example , the negative gradient of fIi ( w ) is written as ĝi ( w ) . During optimization , we denote a parameter w at the i-th iteration in the t-th epoch as wit , and w00 is an initial parameter . We use nb to refer to the number of minibatches in a single epoch . von Mises-Fisher Distribution We use the von Mises-Fisher ( vMF ) distribution to model the directions of vectors . The definition of the vMF distribution is as follows : Definition 1 . ( von Mises-Fisher Distribution , Banerjee et al . ( 2005 ) ) The pdf of the vMF ( µ , κ ) is given by fd ( x ; µ , κ ) = Cd ( κ ) exp ( κµ > x ) on the hypersphere Sd−1 ⊂ Rd . Here , the concentration parameter κ determines how the samples from this distribution are concentrated on the mean direction µ and Cd ( κ ) is constant determined by d and κ . If κ is zero , then it is a uniform distribution on the unit hypersphere , and as κ → ∞ , it becomes a point mass on the unit hypersphere ( Figure 1 ) . The maximum likelihood estimates for µ and κ are µ̂ = ∑n i=1 xi ‖ ∑n i=1 xi‖ and κ̂ ≈ r̄ ( d−r̄ 2 ) 1−r̄2 where xi ’ s are random samples from the vMF distribution and r̄ = ‖ ∑n i=1 xi‖ n . The formula for κ̂ is approximate since the exact computation is intractable ( Banerjee et al. , 2005 ) . 3 THEORETICAL MOTIVATION . 3.1 ANALYSIS OF THE GRADIENT NORM STOCHASTICITY . It is a usual practice for SGD to use a minibatch gradient ĝ ( w ) = −∇wfI ( w ) instead of a full batch gradient g ( w ) = −∇wf ( w ) . The minibatch index set I is drawn from { 1 , . . . , n } randomly . ĝ ( w ) satisfies E [ ĝ ( w ) ] = g ( w ) and Cov ( ĝ ( w ) , ĝ ( w ) ) ≈ 1mn ∑n i=1 gi ( w ) gi ( w ) > for n m where n is the number of full data points and gi ( w ) = −∇wfi ( w ) ( Hoffer et al. , 2017 ) . As the batch size m increases , the randomness of ĝ ( w ) decreases . Hence E‖ĝ ( w ) ‖ tends to ‖g ( w ) ‖ , and Var ( ‖ĝ ( w ) ‖ ) , which is the variance of the norm of the minibatch gradient , vanishes . The convergence rate analysis is as the following : Theorem 1 . Let ĝ ( w ) be a minibatch gradient induced from the minibatch index set I of batch size m from { 1 , . . . , n } and suppose γ = maxi , j∈ { 1 , ... , n } | 〈gi ( w ) , gj ( w ) 〉 | . Then 0 ≤ E‖ĝ ( w ) ‖ − ‖g ( w ) ‖ ≤ 2 ( n−m ) m ( n− 1 ) × γ E‖ĝ ( w ) ‖+ ‖g ( w ) ‖ ≤ ( n−m ) γ m ( n− 1 ) ‖g ( w ) ‖ and Var ( ‖ĝ ( w ) ‖ ) ≤ 2 ( n−m ) m ( n− 1 ) γ . Hence , √ . Var ( ‖ĝ ( w ) ‖ ) E‖ĝ ( w ) ‖ ≤ √ 2 ( n−m ) m ( n− 1 ) × γ ‖g ( w ) ‖2 . ( 1 ) Proof . See Supplemental A . According to Theorem 1 , a large batch sizem reduces the variance of ‖ĝ ( w ) ‖ centered at E‖ĝ ( w ) ‖ with convergence rate O ( 1/m ) . We empirically verify this by estimating the gradient norm stochasticity at random points while varying the minibatch size , using a fully-connected neural network ( FNN ) with MNIST , as shown in Figure 2 ( a ) ( see Supplemental E for more details . ) This theorem however only demonstrate that the gradient norm stochasticity is ( l.h.s . of ( 1 ) ) is low at random initial points . It may blow up after SGD updates , since the upper bound ( r.h.s . of ( 1 ) ) is inversely proportional to ‖g ( w ) ‖ . This implies that the learning dynamics and convergence of SGD , measured in terms of the vanishing gradient , i.e. , ∑nb i=1 ĝi ( w ) ≈ 0 , is not necessarily explained by the vanishing norms of minibatch gradients , but rather by the balance of the directions of ĝi ( w ) ’ s , which motivates our investigation of the directional statistics of minibatch gradients . See Figure 2 ( b ) as an illustration .

Summary: This work provides an analysis of the directional distribution of of stochastic gradients in SGD. The basic claim is that the distribution, when modeled as a von Mises-Fisher distribution, becomes more uniform as training progresses. There is experimental verification of this claim, and some results suggesting that the SNR is more correlated with their measure of uniformity than with the norm of the gradients.