SGD Converges to Global Minimum in Deep Learning via Star-convex Path

Stochastic gradient descent ( SGD ) has been found to be surprisingly effective in training a variety of deep neural networks . However , there is still a lack of understanding on how and why SGD can train these complex networks towards a global minimum . In this study , we establish the convergence of SGD to a global minimum for nonconvex optimization problems that are commonly encountered in neural network training . Our argument exploits the following two important properties : 1 ) the training loss can achieve zero value ( approximately ) , which has been widely observed in deep learning ; 2 ) SGD follows a star-convex path , which is verified by various experiments in this paper . In such a context , our analysis shows that SGD , although has long been considered as a randomized algorithm , converges in an intrinsically deterministic manner to a global minimum . 1 INTRODUCTION . Training neural networks has been proven to be NP-hard decades ago Blum & Rivest ( 1988 ) . At that time , the limited computation power makes neural network training a “ mission impossible ” . However , as the development of computing device achieves several revolutionary milestones ( e.g . GPUs ) , deep neural networks are found to be practically trainable and can generalize well on real datasets Krizhevsky et al . ( 2017 ) . At the same time , deep learning technique starts to beat the performance of other conventional approaches in a variety of challenging tasks , e.g. , computer vision , classification , natural language processing , etc . Modern neural network training is typically performed by applying first-order algorithms such as stochastic gradient descent ( SGD ) ( a.k.a backpropagation ) ( Linnainmaa , 1976 ) or its variants , e.g. , Adam ( Kingma & Ba , 2014 ) , Adagrad ( Duchi et al. , 2011 ) , etc . Traditional analysis of SGD in nonconvex optimization guarantees the convergence to a stationary point Bottou et al . ( 2016 ) ; Ghadimi et al . ( 2016 ) . Recently , it has been shown that SGD has the capability to escape strict saddle points Ge et al . ( 2015 ) ; Jin et al . ( 2017 ) ; Reddi et al . ( 2018a ) ; Daneshmand et al . ( 2018 ) , and can escape even sharp local minima Kleinberg et al . ( 2018 ) . While these works provide different insights towards understanding the performance of SGD , they can not explain the success of SGD that has been widely observed in deep learning applications . Specifically , it is known that SGD is able to train a variety of deep neural networks to achieve zero training loss ( either exactly or approximately ) for non-negative loss functions . This implies that SGD can find a global minimum of deep neural networks at ease . The major challenges towards understanding this phenomenon are in two-fold : 1 ) deep neural networks have complex landscape that can not be fully understood analytically ( Zhou & Liang , 2018 ) ; 2 ) the randomness nature of SGD makes it hard to characterize its convergence on such a complex landscape . These factors prohibit a good understanding of the practical success of SGD in deep learning from a traditional optimization aspect . In this study , we analyze the convergence of SGD in the above phenomenon by exploiting the following two critical properties . First , the fact that SGD can train neural networks to zero loss value implies that the non-negative loss functions on all data samples share a common global minimum . Second , our experiments establish strong empirical evidences that SGD ( when training the loss to zero value ) follows a star-convex path . Based on these properties , we formally establish the convergence of SGD to a global minimizer . Our work conveys a useful insight that although the landscape of neural networks can be complicated , the actual optimization path that SGD takes turns out to be remarkably simple and sufficient to guarantee the converge to a global minimum . 1.1 OUR CONTRIBUTIONS . We focus on the empirical observation that SGD can train various neural networks to achieve zero loss , and validate that SGD follows an epochwise star-convex path in empirical optimization processes . Based on such a property , we prove that the Euclidean distance between the variable sequence generated by SGD and a global minimizer decreases at an epoch level . We also show that the subsequences of iterations that correspond to the same data sample is a minimizing sequence of the loss corresponding to that sample . By further empirical exploration , we validate that SGD follows an iterationwise star-convex path during the major part of the training process . Based on such a property , we prove that the entire variable sequence generated by SGD converges to a global minimizer of the objective function . Then , we show that the convergence of SGD induces a self-regularization on its variance , i.e. , the variance of stochastic gradients vanishes as SGD converges in such a context . From a technical perspective , we characterize the intrinsic deterministic convergence property of SGD when the optimization path is well regularized , rather than the performance on average or in probability established in the existing studies . Our results provide a novel and promising aspect to understand SGD-based optimization in deep learning . Furthermore , our analysis of SGD explores the limiting convergence property of the subsequences that correspond to individual data samples , which is in sharp contrast to the traditional treatment of SGD that depends on its random nature and bounds on variance . Hence , our proof technique can be of independent interest to the community . 1.2 RELATED WORK . As there are extensive literature on SGD , we only mention the highly relevant studies here . Theoretical foundations of SGD have been developed in the optimization community ( Robbins & Monro , 1951 ; Nemirovski et al. , 2009 ; Lan , 2012 ; Ghadimi et al. , 2016 ; Ghadimi & Lan , 2016 ) , and have attracted much attention from the machine learning community in the past decade ( Schmidt et al. , 2017 ; Defazio et al. , 2014 ; Johnson & Zhang , 2013 ; Li et al. , 2017 ; Wang et al. , 2018a ) . In general nonconvex optimization , it is known that SGD converges to a stationary point under a bounded variance assumption and a diminishing learning rate Bottou et al . ( 2016 ) ; Ghadimi et al . ( 2016 ) . Other variants of SGD that are designed for deep learning have been proposed , e.g. , Adam ( Kingma & Ba , 2014 ) , AMSgrad Reddi et al . ( 2018b ) , Adagrad ( Duchi et al. , 2011 ) , etc , and their convergence properties have been studied in the context of online convex regret minimization . Needell et al . ( 2014 ) ; Moulines & Bach ( 2011 ) show that SGD converges at a linear rate when the objective function is strongly convex and has a unique common global minimum . Recently , several studies show that SGD has the capability to escape strict saddle points Ge et al . ( 2015 ) ; Jin et al . ( 2017 ) ; Reddi et al . ( 2018a ) ; Daneshmand et al . ( 2018 ) . Other cubic-regularization-based methods have also been shown to be able to escape strict saddle points Nesterov & Polyak ( 2006 ) ; Zhou et al . ( 2018 ) ; Wang et al . ( 2018b ) . Kleinberg et al . ( 2018 ) considers functions with one-point strong convexity , and shows that the randomness of SGD has an intrinsic smoothing effect that can avoid convergence to sharp minimum . Our paper exploits a very different notion of star-convexity path of SGD , which is a much weaker condition than those in the previous studies . 2 PROBLEM SETUP AND PRELIMINARIES . Neural network training can be formulated as the following finite-sum optimization problem . min x∈Rd f ( x ) : = 1 n n∑ i=1 ` i ( x ) , ( P ) where there are in total n training data samples . The loss function that corresponds to the i-th data sample is denoted by ` i : Rd → R for i = 1 , . . . , n , and the vector to be minimized in the problem ( e.g. , network weights in deep learning ) are denoted by x ∈ Rd . In general , problem ( P ) is a nonconvex optimization problem , and we make the following standard assumptions regarding the objective function . Assumption 1 . The loss functions ` i , i = 1 , . . . , n in problem ( P ) satisfy : 1 . They are continuously differentiable , and their gradients are L-Lipschitz continuous ; 2 . For every i = 1 , . . . , n , infx∈Rd ` i ( x ) > 0 . The conditions imposed by Assumption 1 are standard in analysis of nonconvex optimization . In specific , item 1 is a standard smoothness assumption on nonconvex loss functions . Item 2 assumes that the loss functions are bounded below , which is satisfied by many loss functions in deep learning , e.g. , MSE loss , crossentropy loss , NLL loss , etc , which are all non-negative . Next , we introduce the following fact that is widely observed in training overparameterized deep neural networks , which we assume to hold throughout the paper . Observation 1 ( Global minimum in deep learning ) . The objective function f in problem ( P ) with non-negative loss can achieve zero value at certain x∗ . Thus , x∗ is also a common global minimizer for all individual loss { ` i } ni=1 . More formally , denote X ∗i as the set of global minimizers of ` i for i = 1 , . . . , n. Then , the set of common global minimizers , i.e. , X ∗ : = ∩ni=1X ∗i , is non-empty and bounded . Observation 1 is a common observation in deep learning applications , because deep neural networks ( especially in the overparameterized regime ) typically have enough capacity to fit all training data samples , and therefore the model has a global minimum shared by the loss functions on all data samples . To elaborate more , if the total loss f attains zero value at x∗ , then ` i for all i must achieve zero value at x∗ as they are non-negative . Thus , x∗ is a common global minimum of all individual loss ` i ( x ) for all i . Such a fact plays a critical role in understanding the convergence of SGD in training neural networks . Next , we introduce our algorithm of interest – stochastic gradient descent ( SGD ) . Specifically , to solve problem ( P ) , SGD starts at an initial vector x0 ∈ Rd and generates a variable sequence { xk } k according to the following update rule . ( SGD ) xk+1 = xk − η∇ ` ξk ( xk ) , k = 0 , 1 , ... , where η > 0 denotes the learning rate , and ξk ∈ { 1 , . . . , n } corresponds to the sampled data index at iteration k. In this study , we consider the practical cyclic sampling scheme with reshuffle ( also referred to as the random sampling without replacement ) to generate the random variable ξk . To elaborate the notation , we rewrite every iteration number k as nB + t , where B = 0 , 1 , 2 , ... denotes the index of epoch that iteration k belongs to and t ∈ { 0 , 1 , ... , n − 1 } denotes the corresponding iteration index in that epoch . We further denote πB as the random permutation of 1 , ... , n in the B-th epoch , and denote πB ( j ) as its j-th element . Then , the sampled data index at iteration k can be expressed as ( Cyclic sampling with reshuffle ) : ξk = πB ( t+ 1 ) , t = 0 , ... , n− 1 .

Summary: The paper proposes a new approach to explain the effective behavior of SGD in training deep neural networks by introducing the notion of star-convexity. A function h is star-convex if its global minimum lies on or above any plane tangent to the function, namely h* >= h(x) + < h'(x), x*-x> for any x. Under such condition, the paper shows that the empirical loss goes to zero and the iterates generated by SGD converges to a global minimum. Extensive experiments has been conducted to empirically validate the assumption.