On the expected running time of nonconvex optimization with early stopping

1 INTRODUCTION . This work considers the minimization of a differentiable and possible nonconvex objective function : min x∈Rd f ( x ) . ( 1 ) A generally accepted success criteria for algorithms that use only first-order information is that an approximate stationary point is generated . These are points x ∈ Rd at which the function f has a small gradient . In a typical machine learning scenario , f is the average loss over a dataset of training examples , and the method of choice involves using some form of stochastic gradient method , for instance , stochastic gradient descent , or SGD ( see Algorithm 1 ) . The success of SGD in machine learning problems has led to many extensions of the algorithm , including variance-reduced and distributed variants ( reviewed in Section 1.1 ) . A common approach to stopping optimization in practice is to use early stopping based on a validation function . In this scenario , a stopping criterion is periodically evaluated on the validation function , and the algorithm stops once this criterion is met . The validation and training sets often are disjoint ( although this is not required in the present work ) . Although this approach is used frequently , there is little theoretical work on the runtime of nonconvex optimization using early stopping based on a validation function . In general , the runtime and performance will depend on several factors , including the relation between the validation and training functions , and the desired level of solution accuracy . In this work , the stopping criterion is that the algorithm has generated a point , which is an approximate stationary point for the validation function . Our analysis focuses on bounding the number of iterations and gradient evaluations used until the algorithm meets the stopping criteria . Formally , we consider the stopping time defined as the first time an iterate has the property of being an approximate stationary point for the validation function , and we derive upper bounds on the expected value of this stopping time . Using bounds on the Wasserstein distance between the training and validation sets , we also may derive a bound on the stationary gap of the training function at the resulting iterate . As an extension , we also describe how Wasserstein concentration bounds can be used to bound the stationarity gap with respect to the testing distribution to which both the training and validation datasets are drawn . We apply our analysis to several settings , including stochastic gradient descent ( SGD ) , stochastic variance reduced gradient ( SVRG ) , decentralized SGD ( DSGD ) and stacked SGD ( SSGD ) . The result is new bounds on the expected number of Incremental First-order Oracle ( IFO ) calls needed to generate approximate stationary points for some known algorithms ( SGD ; DSGD ; SVRG ) , as well as a new algorithm ( SSGD ) . Main contributions Our main contributions include : – We present a non-asymptotic analysis of SGD with early stopping that leads to a bound on the expected number of gradient evaluations needed to find approximate stationary points of the training function ( Corollary 3.5 ) . The analysis allows for biases in the update direction , subject to a Lyapunov-type inequality on the error terms . – We rigorously analyze the expected running time of two distributed SGD algorithms : Decentralized SGD , and Stacked SGD ( Algorithm 2 ) , a new decentralized form of SGD designed to exploit connectivity patterns consisting of a network of parameter-server type clusters ( Corollary 4.4 . ) – We apply the analysis to nonconvex SVRG to obtain a bound on the expected number of IFO calls for this algorithm ( Corollary 6.2 ) . – We demonstrate how Wasserstein concentration bounds can be leveraged to bound the generalization performance of parameters returned by SGD with early stopping ( Corollary 7.2 ) . 1.1 RELATED WORK . The study of stochastic optimization goes back ( at least ) to the pioneering efforts of Robbins and Monro Robbins & Monro ( 1951 ) . Subsequent developments include the ordinary differential equation ( ODE ) method Ljung ( 1977 ) and stochastic approximation Kushner & Clark ( 1978 ) , which emphasizes the asymptotic behavior of the algorithms . The subject of non-asymptotic performance guarantees has attracted interest as well , including lower bounds on algorithm performance Nemirovski & Yudin ( 1983 ) . For a review of non-asymptotic guarantees for convex SGD , the reader may consult Nemirovski et al . ( 2009 ) ; Rakhlin et al . ( 2012 ) ; Bach & Moulines ( 2013 ) . Extensions such as distributed Zinkevich et al . ( 2010 ) and asynchronous Zhang et al . ( 2015 ) ; Agarwal & Duchi ( 2011 ) variants of the algorithms also have been investigated . Many machine learning optimization problems involve an objective represented as a finite sum of functions , and , for this case , variance reduction techniques lead to improved rates of convergence over SGD Johnson & Zhang ( 2013 ) ; L. Roux et al . ( 2012 ) ; Defazio et al . ( 2014 ) . The randomized stochastic gradient ( RSG ) method Ghadimi & Lan ( 2013 ) uses randomization to obtain a non-asymptotic performance guarantee for SGD . The randomization technique has became a standard tool for analyzing optimization algorithms in the nonconvex setting Ghadimi & Lan ( 2016 ) ; Lian et al . ( 2017 ) ; Allen-Zhu ( 2018a ; b ) ; Reddi et al . ( 2016b ; a ) ; Zhang et al . ( 2016 ) ; Reddi et al . ( 2016b ) ; Lei et al . ( 2017 ) ; Lian et al . ( 2015 ) . Follow-up works have included analysis of nonconvex optimization in more sophisticated algorithmic settings , such as asynchronous Lian et al . ( 2015 ) and decentralized Lian et al . ( 2017 ) optimization . Analysis of variance reduced optimization has extended beyond convex functions beginning with an application to principal components analysis Shamir ( 2015 ) and later to general nonconvex functions Allen-Zhu & Hazan ( 2016 ) ; Reddi et al . ( 2016b ; a ) ; Lei et al . ( 2017 ) . A highlight in this area is that the IFO complexity of SVRG for nonconvex functions is superior to that of RSG Allen-Zhu & Hazan ( 2016 ) ; Reddi et al . ( 2016a ) . Algorithms with better convergence rates than SVRG also have been developed Allen-Zhu ( 2018a ; b ) . We are particularly interested in results for biased SGD , e.g. , Bertsekas & Tsitsiklis ( 2000 ) which considers the asymptotic convergence of biased SGD to stationary points . In this work , we are interested in a similar scenario but instead focus on the non-asymptotic behavior of the algorithm . Several recent works also have explored the average amount of resources needed to reach a desired performance level in optimization . The expected running time of a stochastic trust region algorithm ( STORM ) is given in Blanchet et al . ( 2016 ) . A similar methodology also has been used to analyze stochastic line search methods Paquette & Scheinberg ( 2018 ) . Our convergence analysis is similar in spirit to these works as we also are interested in the expected amount of time or other resources required to meet the performance guarantee . However , the algorithms and assumptions in this work differ . Other recent work has analyzed the theoretical aspects of early stopping . For instance , Duvenaud et al . ( 2016 ) developed an interpretation of early stopping in terms of variational Bayesian inference . Early stopping for a least squares problem in a reproducing kernel Hilbert space has been treated in Lin & Rosasco ( 2016 ) . The implications of early stopping generalization were studied in Hardt et al . ( 2016 ) . However , to our knowledge , this work is the first to analyze the algorithms ’ runtimes when using a validation function for early stopping in nonconvex optimization . 2 PRELIMINARIES . Let f : Rq×Rd → R be a loss function whose value we denote by f ( y , x ) . Intuitively , the variable y represents an input/output pair , and x represents the model parameters . Throughout , we shall assume the gradient of the objective function with respect to x is Lipschitz continuous ( defined as follows ) . Assumption 2.1 . The function f : Rq × Rd → R is bounded from below by f∗ ∈ R , and the derivative of f with respect to x is L-Lipschitz continuous : ∀y ∈ Rq , x1 , x2 ∈ Rd , ‖∇xf ( y , x1 ) −∇xf ( y , x2 ) ‖ ≤ L‖x1 − x2‖ . At times , we will make a distinction between the training function fT , which is used to calculate gradients , and a validation function fV used to decide when to stop training . Assumption 2.2 . The training function fT is defined using a set YT ⊆ Rq of nT elements as fT ( x ) = 1 nT ∑ y∈YT f ( y , x ) , and the validation function fV is defined using a set YV ⊆ Rq of nV elements as fV ( x ) = 1nV ∑ y∈YV f ( y , x ) . To guarantee that the early stopping rule leads to a well-defined algorithm , we will assume a relation between the training and validation functions . Intuitively , the functions fT and fV will be similar when the datasets YT and YV are similar . Formally , the datasets YT and YV determine probability measures µT and µY , defined as µT = 1nT ∑ y∈YT δy and µV = 1 nV ∑ y∈YV δy , respectively , where δy is the delta measure δy ( A ) = 1y∈A for all sets A . We can compare these measures using the Wasserstein distance as in : For q ≥ 1 , p ≥ 1 , we denote by Pp ( Rq ) the probability measures on Rq with finite moments of order p. Recall that a coupling of probability measures µ1 and µ2 is a probability measure γ on Rq × Rq such that for all measurable sets A , γ ( A × Rq ) = µ1 ( A ) and γ ( Rq × A ) = µ2 ( A ) . Intuitively , a coupling transforms data distributed like µ1 into a dataset that is distributed according to µ2 . The p-Wasserstein distance on Pp ( Rq ) , denoted by dp , is defined as : dp ( µ1 , µ2 ) = inf γ∈Γ ( µ1 , µ2 ) ( E ( x1 , x2 ) ∼γ [ ‖x1 − x2‖p ] ) 1/p , ( 2 ) where Γ ( µ1 , µ2 ) is the set of all couplings of µ1 and µ2 . For more details the reader is referred to Villani ( 2008 ) . Assumption 2.3 . There is a constant G ≥ 0 such that ∀x ∈ Rd , ‖∇fV ( x ) −∇fT ( x ) ‖ ≤ Gd1 ( µV , µT ) . There are several cases in which this assumption will be satisfied . It is trivially satisfied if the training and validation sets are the same as µT = µV . As a consequence of the Kantorovich duality formula ( Villani ( 2008 ) , Remark 6.5 ) , it is also satisfied if the function y 7→ ∇xf ( y , x ) is a G-Lipschitz function , uniformly for all x . Consider the following example : Example 2.4 . Suppose that g : Rq × Rd → R is a smooth function . Let h : Rd → R be the function that applies the hyperbolic tangent function to each of its components : h ( x ) = ( tanh ( x1 ) , . . . , tanh ( xd ) ) , and define f ( y , x ) = g ( y , h ( x ) ) . Further suppose that the training data are bounded : ‖y‖ ≤ J for all y ∈ YT . Then Assumption 2.3 is satisfied because y 7→ ∇xf ( x , y ) is a Lipschitz function with G = sup‖y‖≤J , ‖x‖≤√d ‖ ∂ 2g ∂x∂y ( y , x ) ‖ . In our analyses the notion of success is that an algorithm generates an approximate stationary point : Definition 2.5 . A point x ∈ Rd is an -approximate stationary point of f if ‖∇f ( x ) ‖2 ≤ . We measure the complexity of algorithms according to how many function value and gradient queries they make . Formally , an IFO ) is defined as follows ( Agarwal & Bottou , 2015 ) : Definition 2.6 . An IFO takes a parameter x and an input y and returns the pair ( f ( y , x ) , ∇xf ( y , x ) ) . In Appendix A.1 , we briefly recall the notion of filtration , stopping times , and other concepts from stochastic processes that will be used in our analyses . We also refer readers to Williams ( 1991 ) for more details . 3 BIASED SGD Algorithm 1 SGD with early stopping 1 : input : Initial point x1 ∈ Rd 2 : t = 1 3 : while ‖∇fV ( xt ) ‖2 > do 4 : for n = t to t+m− 1 do 5 : xn+1 = xn − ηhn 6 : end 7 : t = t+m 8 : end 9 : return xt This section details our analysis of SGD with early stopping shown on the right in Algorithm 1 . Starting from an initial point x1 , at each iteration , the parameter is updated with an approximate gradient hn using a step-size η . The gradient norm of the validation function is evaluated every m iterations , and the algorithm ends when the norm decreases below a threshold . We assume that the update direction ht is a sum of two components , vt and ∆t , that represent an unbiased gradient estimate and an error term , respectively : ht = vt + ∆t . ( 3 ) Let { Ft } t≥0 be a filtration such that x1 is F0-measurable , and for all t > 1 , the variables vt , ∆t are Ft-measurable . Our assumptions on the vt are as follows . Assumption 3.1 . For any t ≥ 1 , it holds that E [ vt −∇fT ( xt ) | Ft−1 ] = 0 , ( 4 ) E [ ‖vt −∇fT ( xt ) ‖2 | Ft−1 ] ≤ σ2v . ( 5 ) Assumption 3.1 states that the update directions vt are valid approximations to the gradient∇fT ( xt ) , and it also bounds the error in the approximation . For the variables ∆t we assume the following : Assumption 3.2 . There is a sequence of random variables V1 , V2 , . . . , and U1 , U2 , . . . such that for all t ≥ 1 the variables Vt and Ut are Ft-measurable , ‖∆t‖2 ≤ Vt , and the Vt satisfy the following Lyapunov-type inequality : For constants α ∈ [ 0 , 1 ) and β ≥ 0 , V1 ≤ β , ( 6 ) ∀t ≥ 2 , Vt ≤ αVt−1 + Ut−1 , ( 7 ) ∀t ≥ 1 , E [ Ut | Ft−1 ] ≤ β . ( 8 ) Assumption 3.2 models a scenario where the gradient error dynamics is a combination of contracting and expanding behaviors . Contraction shrinks the error and is represented by a factor α . External noise , represented by the Ut terms , prevents the error from vanishing completely . Note that the assumption would be satisfied in the unbiased case by simply setting Vt = 0 . We can now state our result on the expected number of iterations for SGD with early stopping : Proposition 3.3 . Let { xt } t≥1 be as in Algorithm 1 . Let Assumptions 2.1 , 2.2 , 2.3 , 3.1 , and 3.2 hold . For > 0 , let τ ( ) be the stopping time τ ( ) = inf { n ≥ 1 | n ≡ 1 ( modm ) and ‖∇fV ( xn ) ‖2 ≤ } . Suppose that η ≤ 1L and − 2Lmησ2v − 2mβ/ ( 1− α ) −G2d1 ( µV , µT ) 2 > 0 . Then E [ τ ( ) ] ≤ ηG 2d1 ( µV , µT ) 2 + 2 ( fT ( x1 ) − f∗ ) + η /m+ 2ηβ/ ( 1− α ) η /m− 2Lη2σ2v − 2ηβ/ ( 1− α ) − ηG2d1 ( µV , µT ) 2/m . ( 9 ) Furthermore , it holds with probability 1 that ‖∇fT ( xτ ( ) ) ‖2 ≤ 2 + 2G2d1 ( µV , µT ) 2 . This result can be strengthened by assuming a coupling between the step-size and the expansion bound β , then the result can be strengthened as demonstrated in the next corollary . Corollary 3.4 . Let Assumptions 2.1 , 2.2 , 2.3 , 3.1 , and 3.2 hold . In the context of Proposition 3.3 , let the constant β be of the form β = ηR for some R ≥ 0 , and suppose that > G2d1 ( µV , µT ) 2 . Let c ∈ ( 0 , 1 ) and let the step-size be η = c ·min { 1 L , −G2d1 ( µV , µT ) 2 m ( 2Lσ2v + 2R/ ( 1− α ) ) } . ( 10 ) Then E [ τ ( ) ] = O ( m2 ( 1 +R/ ( 1− α ) ) ( 1− c ) c ( −G2d1 ( µV , µT ) 2 ) 2 ) ( 11 ) See the appendix for the complete formula , including lower order terms . From this corollary we see that when β is proportional to the step-size η , the complexity bound is of the order O ( 1/ 2 ) . As we shall see below , this coupling assumption is satisfied for Stacked SGD . For the case of using SGD to minimize a finite sum using unbiased , we obtain the following : Corollary 3.5 . Let Assumptions 2.1 , 2.2 , and 2.3 hold . Suppose each gradient estimate is obtained by selecting a data point yt ∈ YT uniformly at random and setting vt = ∇xf ( yt , xt ) and that there is a σ2v ≥ 0 such that ∀ y′ ∈ YT , x ∈ Rd , ∥∥∥∇xf ( y′ , x ) − 1nT ∑y∈YT ∇xf ( y , x ) ∥∥∥2 ≤ σ2v . If the step-sizes are defined according to Equation equation 20 using c = 1/2 then the expected number of IFO calls used by SGD before reaching an -approximate stationary point is E [ IFO ( ) ] = O ( mnV +m 2 ( −G2d1 ( µV , µT ) 2 ) 2 + nV ) See the appendix for the complete formula , including lower-order terms . Note that when d ( µV , µT ) is on the order of √ , this result states that the expected IFO complexity is O ( 1/ ( 2 ) ) . This can be compared with the RSG algorithm , where O ( 1/ ( 2 ) ) iterations are sufficient for the expected ( squared ) norm of the gradient at a random iterate to be at most ( Cor . 2.2 in Ghadimi & Lan ( 2013 ) ) . 4 STACKED SGD Cluster 1 Cluster 2 Cluster 3 Cluster 4 In this section , we introduce SSGD for decentralized optimization , which involves a distributed system with two types of nodes : workers and communicators . The communication pattern is shown on the right for a hypothetical system consisting of 16 workers and 4 communicators . Workers ( circles ) are grouped into clusters , each containing a communicator ( triangles ) . Algorithm 2 depicts the steps of SSGD . For every m epochs , the gradient of the validation function is computed at the mean of the communication node parameters . The algorithm stops when the norm of this gradient drops below a threshold . This step is naturally carried out by one of the communication nodes because they can store the average computed during Line 9 of the algorithm . For workers , each iteration begins on Line 6 with one step of SGD . Then , on Line 7 , an averaging step is performed to partially synchronize the model parameters within the local cluster . The steps for a communication node begin in Line 9 with an averaging step , among the other communication nodes from each cluster . On Line 10 , there is a partial synchronization with the worker nodes in the cluster . The key to the algorithm ’ s efficiency is that the averaging among communication nodes can happen in parallel with the gradient descent step as it occurs within the cluster . In the naive approach to parallelizing SGD , all nodes must block after each iteration for synchronization of their parameters . We assume there are M ≥ 1 clusters , each containing K ≥ 1 computation nodes and 1 communication node . Thus , there are M ( K + 1 ) total nodes . Given a worker node 1 ≤ i ≤ KM , we let Algorithm 2 SSGD with early stopping 1 : input : Node id i , initial parameters xi1 ( for workers ) , or x̂ i 1 ( for communicators ) 2 : t = 1 3 : while ∥∥∥∇fV ( 1M ∑Mj=1 x̂jt ) ∥∥∥2 > do 4 : for n = t to t+m− 1 do 5 : if node i is a computation node then 6 : xi n+ 12 = xin − ηvin 7 : xin+1 = 1 K+1 ( x̂ c ( i ) n+ 12 + ∑K j=1 x j n+ 12 ) 8 : else ( node i is a communication node ) 9 : x̂i n+ 12 = 1M ∑M j=1 x̂ j n 10 : x̂in+1 = 1 K+1 ( x̂i n+ 12 + ∑ j∈c−1 ( i ) x j n+ 12 ) 11 : end if 12 : end 13 : t = t+m 14 : end 15 : return x̂t+ 12 4 8 16 32 64 128 256 Number of nodes 104 105 D at a p er se co n d SGD SSGD 4 8 16 32 64 128 256 Number of nodes 0 1 2 3 f T ( x t ) SGD SSGD SSGD results for a climate modeling task . Top : throughput vs number of nodes .. Bottom : comparison of the loss of the algorithms . c ( i ) ∈ { 1 , . . . , M } denote the index of the communication node of the group containing worker i . For analysis , we define the filtration { Ft } t≥0 as follows : F0 = σ ( { xi1 ∣∣ 1 ≤ i ≤ KM } ∪ { x̂i1 ∣∣ 1 ≤ i ≤M } ) , ∀t ≥ 1 , Ft = σ ( { xi1 , v i n ∣∣ 1 ≤ n ≤ t , 1 ≤ i ≤ KM } ∪ { x̂i1 ∣∣ 1 ≤ i ≤M } ) . We assume that the gradient estimates used in SSGD are unbiased and have bounded variance : Assumption 4.1 . For any t ≥ 1 and 1 ≤ i ≤ KM , E [ vit −∇fT ( xit ) | Ft−1 ] = 0 , ( 12 ) E [ ∥∥vit −∇fT ( xit ) ∥∥2 | Ft−1 ] ≤ σ2v . ( 13 ) The first step in our analysis is a bound on the dispersion of the parameters across the system . Proposition 4.2 . Let Assumption 2.1 , 2.2 , and 4.1 hold , and let the variables x̂it be as defined in Line 6 of Algorithm 2 . Suppose the step-size satisfies η < 1/ ( 2LK ) . Define the variables V1 , U1 , V2 , U2 , . . . and the constants α , β as follows : Vt = L2 M2 M∑ i=1 M∑ j=1 ∥∥∥x̂it − x̂jt∥∥∥2 , ( 14a ) Ut = ηKL ( K + 5/8 ) 1 M2 M∑ i=1 M∑ j=1 ∥∥∥∥∥ ( 1 K ∑ k∈c−1 ( i ) vkt −∇fT ( x̂it ) ) − ( 1 K ∑ k∈c−1 ( j ) vkt −∇fT ( x̂jt ) ) ∥∥∥∥∥ 2 ( 14b ) α = ( K + 3/4 ) 2 ( K + 1 ) 2 , ( 14c ) β = η · 4Lσ 2 v ( K + 5/8 ) . ( 14d ) Then , for all t ≥ 1 , it holds that Vt+1 ≤ αVt + Ut and E [ Ut | Ft−1 ] ≤ β . In the preceding definitions , Vt represents the dispersion of the parameter values across different nodes . An averaging step tends to reduce the dispersion by a factor of α , while the independent gradient updates at each node may increase parameter dispersion by an amount β . This result allows us to model SSGD as a form of biased SGD , leading to the following : Proposition 4.3 . Let Assumptions 2.1 , 2.2 , 2.3 , and 4.1 hold . Assume that the initial parameters at every node are equal : xi1 = x j 1 , for all 1 ≤ i , j ≤ KM and x̂i1 = x̂j1 for 1 ≤ i , j ≤ M . For some c < 12K , suppose that the step-size η is η = c ·min { 1 L , −G2d1 ( µV , µT ) 2 m ( 2Lσ2v/K + 32Lσ 2 v ( K + 1 ) / ( K + 5/8 ) ) } . ( 15 ) Let x̂t be the average of the communicator states at time t : x̂t = 1M ∑M i=1 x̂ i t. For > 0 , define τ ( ) to be the stopping time τ ( ) = inf { n ≥ 1 | n ≡ 1 ( mod m ) and ‖∇fV ( x̂n ) ‖2 ≤ } . Then , E [ τ ( ) ] = O ( m2 ( 1− c ) c ( −G2d1 ( µV , µT ) 2 ) 2 ) . Refer to the Appendix for the complete formula , including lower-order terms . This leads to a bound on the expected IFO complexity to minimize a finite sum using SSGD : Corollary 4.4 . Let Assumptions 2.1 , 2.2 , and 2.3 hold . Suppose each gradient estimate is obtained by selecting a data point yjt ∈ YT uniformly at random and setting vjt = ∇xf ( yjt , xjt ) and that there is a σ2 ≥ 0 , such that ∀ y′ ∈ YT , x ∈ Rd , ∥∥∥∇xf ( y′ , x ) − 1nT ∑y∈YT ∇xf ( y , x ) ∥∥∥2 ≤ σ2v . If the step-sizes are defined as in Equation equation 15 with c = 1/ ( 4K ) , then the expected number of IFO calls used by SSGD before reaching an -approximate stationary point is E [ IFO ( ) ] = O ( mK ( nV +mK ) ( −G2d1 ( µV , µT ) 2 ) 2 + nV ) . Experimental Result SSGD has been implemented to train a neural network model as part of research into spatio-temporal data analysis for climate research . The model is LSTNet , a neural network architecture that includes a convolutional neural network to extract short-term local dependency patterns from spatial variables and a recurrent neural network component to discover long-term patterns from time series trends Lai et al . ( 2018 ) . LSTNet is trained for the prediction of solar radiation from past sensor measurements . Fig . 4 shows results from the experiment . The upper plot compares the throughput of SGD and SSGD as the number of nodes increases . We see that the performance of SGD degrades after 64 nodes , while SSGD maintains near-linear scalability . The lower plot shows that although SSGD involves biased gradients , it does not sacrifice accuracy , and yields similar prediction errors compared to SGD . See Appendix G.1 for more details about the experimental methodology .

Summary: This paper proposes an optimization approach in which the optimizer computes the gradient on a given function yet uses another to decide a stopping time. Conceptually those functions are empirical errors on train and validation folds in the most common setting, although the authors seem to use other settings later in the paper to consider decentralized optimization schemes. The authors introduce a bound on the Wasserstein distance between the train and validation distributions in their analysis which plays a crucial role in their results. The authors use these results to motivate variants of existing optimization algorithms.