On the expected running time of nonconvex optimization with early stopping

1 INTRODUCTION . This work considers the minimization of a differentiable and possible nonconvex objective function : min x∈Rd f ( x ) . ( 1 ) A generally accepted success criteria for algorithms that use only first-order information is that an approximate stationary point is generated . These are points x ∈ Rd at which the function f has a small gradient . In a typical machine learning scenario , f is the average loss over a dataset of training examples , and the method of choice involves using some form of stochastic gradient method , for instance , stochastic gradient descent , or SGD ( see Algorithm 1 ) . The success of SGD in machine learning problems has led to many extensions of the algorithm , including variance-reduced and distributed variants ( reviewed in Section 1.1 ) . A common approach to stopping optimization in practice is to use early stopping based on a validation function . In this scenario , a stopping criterion is periodically evaluated on the validation function , and the algorithm stops once this criterion is met . The validation and training sets often are disjoint ( although this is not required in the present work ) . Although this approach is used frequently , there is little theoretical work on the runtime of nonconvex optimization using early stopping based on a validation function . In general , the runtime and performance will depend on several factors , including the relation between the validation and training functions , and the desired level of solution accuracy . In this work , the stopping criterion is that the algorithm has generated a point , which is an approximate stationary point for the validation function . Our analysis focuses on bounding the number of iterations and gradient evaluations used until the algorithm meets the stopping criteria . Formally , we consider the stopping time defined as the first time an iterate has the property of being an approximate stationary point for the validation function , and we derive upper bounds on the expected value of this stopping time . Using bounds on the Wasserstein distance between the training and validation sets , we also may derive a bound on the stationary gap of the training function at the resulting iterate . As an extension , we also describe how Wasserstein concentration bounds can be used to bound the stationarity gap with respect to the testing distribution to which both the training and validation datasets are drawn . We apply our analysis to several settings , including stochastic gradient descent ( SGD ) , stochastic variance reduced gradient ( SVRG ) , decentralized SGD ( DSGD ) and stacked SGD ( SSGD ) . The result is new bounds on the expected number of Incremental First-order Oracle ( IFO ) calls needed to generate approximate stationary points for some known algorithms ( SGD ; DSGD ; SVRG ) , as well as a new algorithm ( SSGD ) . Main contributions Our main contributions include : – We present a non-asymptotic analysis of SGD with early stopping that leads to a bound on the expected number of gradient evaluations needed to find approximate stationary points of the training function ( Corollary 3.5 ) . The analysis allows for biases in the update direction , subject to a Lyapunov-type inequality on the error terms . – We rigorously analyze the expected running time of two distributed SGD algorithms : Decentralized SGD , and Stacked SGD ( Algorithm 2 ) , a new decentralized form of SGD designed to exploit connectivity patterns consisting of a network of parameter-server type clusters ( Corollary 4.4 . ) – We apply the analysis to nonconvex SVRG to obtain a bound on the expected number of IFO calls for this algorithm ( Corollary 6.2 ) . – We demonstrate how Wasserstein concentration bounds can be leveraged to bound the generalization performance of parameters returned by SGD with early stopping ( Corollary 7.2 ) . 1.1 RELATED WORK . The study of stochastic optimization goes back ( at least ) to the pioneering efforts of Robbins and Monro Robbins & Monro ( 1951 ) . Subsequent developments include the ordinary differential equation ( ODE ) method Ljung ( 1977 ) and stochastic approximation Kushner & Clark ( 1978 ) , which emphasizes the asymptotic behavior of the algorithms . The subject of non-asymptotic performance guarantees has attracted interest as well , including lower bounds on algorithm performance Nemirovski & Yudin ( 1983 ) . For a review of non-asymptotic guarantees for convex SGD , the reader may consult Nemirovski et al . ( 2009 ) ; Rakhlin et al . ( 2012 ) ; Bach & Moulines ( 2013 ) . Extensions such as distributed Zinkevich et al . ( 2010 ) and asynchronous Zhang et al . ( 2015 ) ; Agarwal & Duchi ( 2011 ) variants of the algorithms also have been investigated . Many machine learning optimization problems involve an objective represented as a finite sum of functions , and , for this case , variance reduction techniques lead to improved rates of convergence over SGD Johnson & Zhang ( 2013 ) ; L. Roux et al . ( 2012 ) ; Defazio et al . ( 2014 ) . The randomized stochastic gradient ( RSG ) method Ghadimi & Lan ( 2013 ) uses randomization to obtain a non-asymptotic performance guarantee for SGD . The randomization technique has became a standard tool for analyzing optimization algorithms in the nonconvex setting Ghadimi & Lan ( 2016 ) ; Lian et al . ( 2017 ) ; Allen-Zhu ( 2018a ; b ) ; Reddi et al . ( 2016b ; a ) ; Zhang et al . ( 2016 ) ; Reddi et al . ( 2016b ) ; Lei et al . ( 2017 ) ; Lian et al . ( 2015 ) . Follow-up works have included analysis of nonconvex optimization in more sophisticated algorithmic settings , such as asynchronous Lian et al . ( 2015 ) and decentralized Lian et al . ( 2017 ) optimization . Analysis of variance reduced optimization has extended beyond convex functions beginning with an application to principal components analysis Shamir ( 2015 ) and later to general nonconvex functions Allen-Zhu & Hazan ( 2016 ) ; Reddi et al . ( 2016b ; a ) ; Lei et al . ( 2017 ) . A highlight in this area is that the IFO complexity of SVRG for nonconvex functions is superior to that of RSG Allen-Zhu & Hazan ( 2016 ) ; Reddi et al . ( 2016a ) . Algorithms with better convergence rates than SVRG also have been developed Allen-Zhu ( 2018a ; b ) . We are particularly interested in results for biased SGD , e.g. , Bertsekas & Tsitsiklis ( 2000 ) which considers the asymptotic convergence of biased SGD to stationary points . In this work , we are interested in a similar scenario but instead focus on the non-asymptotic behavior of the algorithm . Several recent works also have explored the average amount of resources needed to reach a desired performance level in optimization . The expected running time of a stochastic trust region algorithm ( STORM ) is given in Blanchet et al . ( 2016 ) . A similar methodology also has been used to analyze stochastic line search methods Paquette & Scheinberg ( 2018 ) . Our convergence analysis is similar in spirit to these works as we also are interested in the expected amount of time or other resources required to meet the performance guarantee . However , the algorithms and assumptions in this work differ . Other recent work has analyzed the theoretical aspects of early stopping . For instance , Duvenaud et al . ( 2016 ) developed an interpretation of early stopping in terms of variational Bayesian inference . Early stopping for a least squares problem in a reproducing kernel Hilbert space has been treated in Lin & Rosasco ( 2016 ) . The implications of early stopping generalization were studied in Hardt et al . ( 2016 ) . However , to our knowledge , this work is the first to analyze the algorithms ’ runtimes when using a validation function for early stopping in nonconvex optimization . 2 PRELIMINARIES . Let f : Rq×Rd → R be a loss function whose value we denote by f ( y , x ) . Intuitively , the variable y represents an input/output pair , and x represents the model parameters . Throughout , we shall assume the gradient of the objective function with respect to x is Lipschitz continuous ( defined as follows ) . Assumption 2.1 . The function f : Rq × Rd → R is bounded from below by f∗ ∈ R , and the derivative of f with respect to x is L-Lipschitz continuous : ∀y ∈ Rq , x1 , x2 ∈ Rd , ‖∇xf ( y , x1 ) −∇xf ( y , x2 ) ‖ ≤ L‖x1 − x2‖ . At times , we will make a distinction between the training function fT , which is used to calculate gradients , and a validation function fV used to decide when to stop training . Assumption 2.2 . The training function fT is defined using a set YT ⊆ Rq of nT elements as fT ( x ) = 1 nT ∑ y∈YT f ( y , x ) , and the validation function fV is defined using a set YV ⊆ Rq of nV elements as fV ( x ) = 1nV ∑ y∈YV f ( y , x ) . To guarantee that the early stopping rule leads to a well-defined algorithm , we will assume a relation between the training and validation functions . Intuitively , the functions fT and fV will be similar when the datasets YT and YV are similar . Formally , the datasets YT and YV determine probability measures µT and µY , defined as µT = 1nT ∑ y∈YT δy and µV = 1 nV ∑ y∈YV δy , respectively , where δy is the delta measure δy ( A ) = 1y∈A for all sets A . We can compare these measures using the Wasserstein distance as in : For q ≥ 1 , p ≥ 1 , we denote by Pp ( Rq ) the probability measures on Rq with finite moments of order p. Recall that a coupling of probability measures µ1 and µ2 is a probability measure γ on Rq × Rq such that for all measurable sets A , γ ( A × Rq ) = µ1 ( A ) and γ ( Rq × A ) = µ2 ( A ) . Intuitively , a coupling transforms data distributed like µ1 into a dataset that is distributed according to µ2 . The p-Wasserstein distance on Pp ( Rq ) , denoted by dp , is defined as : dp ( µ1 , µ2 ) = inf γ∈Γ ( µ1 , µ2 ) ( E ( x1 , x2 ) ∼γ [ ‖x1 − x2‖p ] ) 1/p , ( 2 ) where Γ ( µ1 , µ2 ) is the set of all couplings of µ1 and µ2 . For more details the reader is referred to Villani ( 2008 ) . Assumption 2.3 . There is a constant G ≥ 0 such that ∀x ∈ Rd , ‖∇fV ( x ) −∇fT ( x ) ‖ ≤ Gd1 ( µV , µT ) . There are several cases in which this assumption will be satisfied . It is trivially satisfied if the training and validation sets are the same as µT = µV . As a consequence of the Kantorovich duality formula ( Villani ( 2008 ) , Remark 6.5 ) , it is also satisfied if the function y 7→ ∇xf ( y , x ) is a G-Lipschitz function , uniformly for all x . Consider the following example : Example 2.4 . Suppose that g : Rq × Rd → R is a smooth function . Let h : Rd → R be the function that applies the hyperbolic tangent function to each of its components : h ( x ) = ( tanh ( x1 ) , . . . , tanh ( xd ) ) , and define f ( y , x ) = g ( y , h ( x ) ) . Further suppose that the training data are bounded : ‖y‖ ≤ J for all y ∈ YT . Then Assumption 2.3 is satisfied because y 7→ ∇xf ( x , y ) is a Lipschitz function with G = sup‖y‖≤J , ‖x‖≤√d ‖ ∂ 2g ∂x∂y ( y , x ) ‖ . In our analyses the notion of success is that an algorithm generates an approximate stationary point : Definition 2.5 . A point x ∈ Rd is an -approximate stationary point of f if ‖∇f ( x ) ‖2 ≤ . We measure the complexity of algorithms according to how many function value and gradient queries they make . Formally , an IFO ) is defined as follows ( Agarwal & Bottou , 2015 ) : Definition 2.6 . An IFO takes a parameter x and an input y and returns the pair ( f ( y , x ) , ∇xf ( y , x ) ) . In Appendix A.1 , we briefly recall the notion of filtration , stopping times , and other concepts from stochastic processes that will be used in our analyses . We also refer readers to Williams ( 1991 ) for more details . 3 BIASED SGD Algorithm 1 SGD with early stopping 1 : input : Initial point x1 ∈ Rd 2 : t = 1 3 : while ‖∇fV ( xt ) ‖2 > do 4 : for n = t to t+m− 1 do 5 : xn+1 = xn − ηhn 6 : end 7 : t = t+m 8 : end 9 : return xt This section details our analysis of SGD with early stopping shown on the right in Algorithm 1 . Starting from an initial point x1 , at each iteration , the parameter is updated with an approximate gradient hn using a step-size η . The gradient norm of the validation function is evaluated every m iterations , and the algorithm ends when the norm decreases below a threshold . We assume that the update direction ht is a sum of two components , vt and ∆t , that represent an unbiased gradient estimate and an error term , respectively : ht = vt + ∆t . ( 3 ) Let { Ft } t≥0 be a filtration such that x1 is F0-measurable , and for all t > 1 , the variables vt , ∆t are Ft-measurable . Our assumptions on the vt are as follows . Assumption 3.1 . For any t ≥ 1 , it holds that E [ vt −∇fT ( xt ) | Ft−1 ] = 0 , ( 4 ) E [ ‖vt −∇fT ( xt ) ‖2 | Ft−1 ] ≤ σ2v . ( 5 ) Assumption 3.1 states that the update directions vt are valid approximations to the gradient∇fT ( xt ) , and it also bounds the error in the approximation . For the variables ∆t we assume the following : Assumption 3.2 . There is a sequence of random variables V1 , V2 , . . . , and U1 , U2 , . . . such that for all t ≥ 1 the variables Vt and Ut are Ft-measurable , ‖∆t‖2 ≤ Vt , and the Vt satisfy the following Lyapunov-type inequality : For constants α ∈ [ 0 , 1 ) and β ≥ 0 , V1 ≤ β , ( 6 ) ∀t ≥ 2 , Vt ≤ αVt−1 + Ut−1 , ( 7 ) ∀t ≥ 1 , E [ Ut | Ft−1 ] ≤ β . ( 8 ) Assumption 3.2 models a scenario where the gradient error dynamics is a combination of contracting and expanding behaviors . Contraction shrinks the error and is represented by a factor α . External noise , represented by the Ut terms , prevents the error from vanishing completely . Note that the assumption would be satisfied in the unbiased case by simply setting Vt = 0 . We can now state our result on the expected number of iterations for SGD with early stopping : Proposition 3.3 . Let { xt } t≥1 be as in Algorithm 1 . Let Assumptions 2.1 , 2.2 , 2.3 , 3.1 , and 3.2 hold . For > 0 , let τ ( ) be the stopping time τ ( ) = inf { n ≥ 1 | n ≡ 1 ( modm ) and ‖∇fV ( xn ) ‖2 ≤ } . Suppose that η ≤ 1L and − 2Lmησ2v − 2mβ/ ( 1− α ) −G2d1 ( µV , µT ) 2 > 0 . Then E [ τ ( ) ] ≤ ηG 2d1 ( µV , µT ) 2 + 2 ( fT ( x1 ) − f∗ ) + η /m+ 2ηβ/ ( 1− α ) η /m− 2Lη2σ2v − 2ηβ/ ( 1− α ) − ηG2d1 ( µV , µT ) 2/m . ( 9 ) Furthermore , it holds with probability 1 that ‖∇fT ( xτ ( ) ) ‖2 ≤ 2 + 2G2d1 ( µV , µT ) 2 . This result can be strengthened by assuming a coupling between the step-size and the expansion bound β , then the result can be strengthened as demonstrated in the next corollary . Corollary 3.4 . Let Assumptions 2.1 , 2.2 , 2.3 , 3.1 , and 3.2 hold . In the context of Proposition 3.3 , let the constant β be of the form β = ηR for some R ≥ 0 , and suppose that > G2d1 ( µV , µT ) 2 . Let c ∈ ( 0 , 1 ) and let the step-size be η = c ·min { 1 L , −G2d1 ( µV , µT ) 2 m ( 2Lσ2v + 2R/ ( 1− α ) ) } . ( 10 ) Then E [ τ ( ) ] = O ( m2 ( 1 +R/ ( 1− α ) ) ( 1− c ) c ( −G2d1 ( µV , µT ) 2 ) 2 ) ( 11 ) See the appendix for the complete formula , including lower order terms . From this corollary we see that when β is proportional to the step-size η , the complexity bound is of the order O ( 1/ 2 ) . As we shall see below , this coupling assumption is satisfied for Stacked SGD . For the case of using SGD to minimize a finite sum using unbiased , we obtain the following : Corollary 3.5 . Let Assumptions 2.1 , 2.2 , and 2.3 hold . Suppose each gradient estimate is obtained by selecting a data point yt ∈ YT uniformly at random and setting vt = ∇xf ( yt , xt ) and that there is a σ2v ≥ 0 such that ∀ y′ ∈ YT , x ∈ Rd , ∥∥∥∇xf ( y′ , x ) − 1nT ∑y∈YT ∇xf ( y , x ) ∥∥∥2 ≤ σ2v . If the step-sizes are defined according to Equation equation 20 using c = 1/2 then the expected number of IFO calls used by SGD before reaching an -approximate stationary point is E [ IFO ( ) ] = O ( mnV +m 2 ( −G2d1 ( µV , µT ) 2 ) 2 + nV ) See the appendix for the complete formula , including lower-order terms . Note that when d ( µV , µT ) is on the order of √ , this result states that the expected IFO complexity is O ( 1/ ( 2 ) ) . This can be compared with the RSG algorithm , where O ( 1/ ( 2 ) ) iterations are sufficient for the expected ( squared ) norm of the gradient at a random iterate to be at most ( Cor . 2.2 in Ghadimi & Lan ( 2013 ) ) . 4 STACKED SGD Cluster 1 Cluster 2 Cluster 3 Cluster 4 In this section , we introduce SSGD for decentralized optimization , which involves a distributed system with two types of nodes : workers and communicators . The communication pattern is shown on the right for a hypothetical system consisting of 16 workers and 4 communicators . Workers ( circles ) are grouped into clusters , each containing a communicator ( triangles ) . Algorithm 2 depicts the steps of SSGD . For every m epochs , the gradient of the validation function is computed at the mean of the communication node parameters . The algorithm stops when the norm of this gradient drops below a threshold . This step is naturally carried out by one of the communication nodes because they can store the average computed during Line 9 of the algorithm . For workers , each iteration begins on Line 6 with one step of SGD . Then , on Line 7 , an averaging step is performed to partially synchronize the model parameters within the local cluster . The steps for a communication node begin in Line 9 with an averaging step , among the other communication nodes from each cluster . On Line 10 , there is a partial synchronization with the worker nodes in the cluster . The key to the algorithm ’ s efficiency is that the averaging among communication nodes can happen in parallel with the gradient descent step as it occurs within the cluster . In the naive approach to parallelizing SGD , all nodes must block after each iteration for synchronization of their parameters . We assume there are M ≥ 1 clusters , each containing K ≥ 1 computation nodes and 1 communication node . Thus , there are M ( K + 1 ) total nodes . Given a worker node 1 ≤ i ≤ KM , we let Algorithm 2 SSGD with early stopping 1 : input : Node id i , initial parameters xi1 ( for workers ) , or x̂ i 1 ( for communicators ) 2 : t = 1 3 : while ∥∥∥∇fV ( 1M ∑Mj=1 x̂jt ) ∥∥∥2 > do 4 : for n = t to t+m− 1 do 5 : if node i is a computation node then 6 : xi n+ 12 = xin − ηvin 7 : xin+1 = 1 K+1 ( x̂ c ( i ) n+ 12 + ∑K j=1 x j n+ 12 ) 8 : else ( node i is a communication node ) 9 : x̂i n+ 12 = 1M ∑M j=1 x̂ j n 10 : x̂in+1 = 1 K+1 ( x̂i n+ 12 + ∑ j∈c−1 ( i ) x j n+ 12 ) 11 : end if 12 : end 13 : t = t+m 14 : end 15 : return x̂t+ 12 4 8 16 32 64 128 256 Number of nodes 104 105 D at a p er se co n d SGD SSGD 4 8 16 32 64 128 256 Number of nodes 0 1 2 3 f T ( x t ) SGD SSGD SSGD results for a climate modeling task . Top : throughput vs number of nodes .. Bottom : comparison of the loss of the algorithms . c ( i ) ∈ { 1 , . . . , M } denote the index of the communication node of the group containing worker i . For analysis , we define the filtration { Ft } t≥0 as follows : F0 = σ ( { xi1 ∣∣ 1 ≤ i ≤ KM } ∪ { x̂i1 ∣∣ 1 ≤ i ≤M } ) , ∀t ≥ 1 , Ft = σ ( { xi1 , v i n ∣∣ 1 ≤ n ≤ t , 1 ≤ i ≤ KM } ∪ { x̂i1 ∣∣ 1 ≤ i ≤M } ) . We assume that the gradient estimates used in SSGD are unbiased and have bounded variance : Assumption 4.1 . For any t ≥ 1 and 1 ≤ i ≤ KM , E [ vit −∇fT ( xit ) | Ft−1 ] = 0 , ( 12 ) E [ ∥∥vit −∇fT ( xit ) ∥∥2 | Ft−1 ] ≤ σ2v . ( 13 ) The first step in our analysis is a bound on the dispersion of the parameters across the system . Proposition 4.2 . Let Assumption 2.1 , 2.2 , and 4.1 hold , and let the variables x̂it be as defined in Line 6 of Algorithm 2 . Suppose the step-size satisfies η < 1/ ( 2LK ) . Define the variables V1 , U1 , V2 , U2 , . . . and the constants α , β as follows : Vt = L2 M2 M∑ i=1 M∑ j=1 ∥∥∥x̂it − x̂jt∥∥∥2 , ( 14a ) Ut = ηKL ( K + 5/8 ) 1 M2 M∑ i=1 M∑ j=1 ∥∥∥∥∥ ( 1 K ∑ k∈c−1 ( i ) vkt −∇fT ( x̂it ) ) − ( 1 K ∑ k∈c−1 ( j ) vkt −∇fT ( x̂jt ) ) ∥∥∥∥∥ 2 ( 14b ) α = ( K + 3/4 ) 2 ( K + 1 ) 2 , ( 14c ) β = η · 4Lσ 2 v ( K + 5/8 ) . ( 14d ) Then , for all t ≥ 1 , it holds that Vt+1 ≤ αVt + Ut and E [ Ut | Ft−1 ] ≤ β . In the preceding definitions , Vt represents the dispersion of the parameter values across different nodes . An averaging step tends to reduce the dispersion by a factor of α , while the independent gradient updates at each node may increase parameter dispersion by an amount β . This result allows us to model SSGD as a form of biased SGD , leading to the following : Proposition 4.3 . Let Assumptions 2.1 , 2.2 , 2.3 , and 4.1 hold . Assume that the initial parameters at every node are equal : xi1 = x j 1 , for all 1 ≤ i , j ≤ KM and x̂i1 = x̂j1 for 1 ≤ i , j ≤ M . For some c < 12K , suppose that the step-size η is η = c ·min { 1 L , −G2d1 ( µV , µT ) 2 m ( 2Lσ2v/K + 32Lσ 2 v ( K + 1 ) / ( K + 5/8 ) ) } . ( 15 ) Let x̂t be the average of the communicator states at time t : x̂t = 1M ∑M i=1 x̂ i t. For > 0 , define τ ( ) to be the stopping time τ ( ) = inf { n ≥ 1 | n ≡ 1 ( mod m ) and ‖∇fV ( x̂n ) ‖2 ≤ } . Then , E [ τ ( ) ] = O ( m2 ( 1− c ) c ( −G2d1 ( µV , µT ) 2 ) 2 ) . Refer to the Appendix for the complete formula , including lower-order terms . This leads to a bound on the expected IFO complexity to minimize a finite sum using SSGD : Corollary 4.4 . Let Assumptions 2.1 , 2.2 , and 2.3 hold . Suppose each gradient estimate is obtained by selecting a data point yjt ∈ YT uniformly at random and setting vjt = ∇xf ( yjt , xjt ) and that there is a σ2 ≥ 0 , such that ∀ y′ ∈ YT , x ∈ Rd , ∥∥∥∇xf ( y′ , x ) − 1nT ∑y∈YT ∇xf ( y , x ) ∥∥∥2 ≤ σ2v . If the step-sizes are defined as in Equation equation 15 with c = 1/ ( 4K ) , then the expected number of IFO calls used by SSGD before reaching an -approximate stationary point is E [ IFO ( ) ] = O ( mK ( nV +mK ) ( −G2d1 ( µV , µT ) 2 ) 2 + nV ) . Experimental Result SSGD has been implemented to train a neural network model as part of research into spatio-temporal data analysis for climate research . The model is LSTNet , a neural network architecture that includes a convolutional neural network to extract short-term local dependency patterns from spatial variables and a recurrent neural network component to discover long-term patterns from time series trends Lai et al . ( 2018 ) . LSTNet is trained for the prediction of solar radiation from past sensor measurements . Fig . 4 shows results from the experiment . The upper plot compares the throughput of SGD and SSGD as the number of nodes increases . We see that the performance of SGD degrades after 64 nodes , while SSGD maintains near-linear scalability . The lower plot shows that although SSGD involves biased gradients , it does not sacrifice accuracy , and yields similar prediction errors compared to SGD . See Appendix G.1 for more details about the experimental methodology .

Summary: In this paper, the authors consider stochastic optimization in the setting where a validation function is used to guide the termination of the algorithm. In more details, the algorithm terminates if the gradient of the validation function at an iterate is smaller than a threshold. In this framework, the authors consider several variants of SGD, including distributed variant and SVRG, for each of which the authors study the expected number of iterations for a prescribed accuracy under an assumption between the training and validation set.