ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computations

∑n i=1 fi ( x ) . To the best of our knowledge , in this noncon- vex finite-sum regime , all existing variance-reduced methods , including SARAH , SVRG , SAGA and their variants , need to compute the full gradient over all n data samples at the initial point x0 , and then periodically compute the full gradient once every few iterations ( for SVRG , SARAH and their variants ) . Note that SVRG , SAGA and their variants typically achieve weaker convergence results than variants of SARAH : n2/3/ 2 vs. n1/2/ 2 . Thus we focus on the variant of SARAH . The proposed ZeroSARAH and its distributed variant D-ZeroSARAH are the first variance-reduced algorithms which do not require any full gradient computations , not even for the initial point . Moreover , for both standard and distributed settings , we show that ZeroSARAH and D-ZeroSARAH obtain new state-of-the-art convergence results , which can improve the previous best-known result ( given by e.g. , SPIDER , SARAH , and PAGE ) in certain regimes . Avoiding any full gradient computations ( which are time-consuming steps ) is important in many applications as the number of data samples n usually is very large . Especially in the distributed setting , periodic computation of full gradient over all data samples needs to periodically synchronize all clients/devices/machines , which may be impossible or unaffordable . Thus , we expect that ZeroSARAH/D-ZeroSARAH will have a practical impact in distributed and federated learning where full device participation is impractical . 1 INTRODUCTION . Nonconvex optimization is ubiquitous across many domains of machine learning ( Jain & Kar , 2017 ) , especially in training deep neural networks . In this paper , we consider the nonconvex finite-sum problems of the form min x∈Rd { f ( x ) : = 1 n n∑ i=1 fi ( x ) } , ( 1 ) where f : Rd → R is a differentiable and possibly nonconvex function . Problem ( 1 ) captures the standard empirical risk minimization problems in machine learning ( Shalev-Shwartz & Ben-David , 2014 ) . There are n data samples and fi denotes the loss associated with i-th data sample . We assume the functions fi : Rd → R for all i ∈ [ n ] : = { 1 , 2 , . . . , n } are also differentiable and possibly nonconvex functions . Beyond the standard/centralized problem ( 1 ) , we further consider the distributed/federated nonconvex problems : min x∈Rd { f ( x ) : = 1 n n∑ i=1 fi ( x ) } , fi ( x ) : = 1 m m∑ j=1 fi , j ( x ) , ( 2 ) where n denotes the number of clients/devices/machines , fi denotes the loss associated with m data samples stored on client i , and all functions are differentiable and can be nonconvex . Avoiding any full gradient computations is important especially in this distributed setting ( 2 ) , periodic computation of full gradient over all data samples needs to periodically synchronize all clients , which may be impossible or very hard to achieve . There has been extensive research in designing first-order ( gradient-type ) methods for solving centralized/distributed nonconvex problems ( 1 ) and ( 2 ) such as SGD , SVRG , SAGA , SCSG , SARAH and their variants , e.g. , ( Ghadimi & Lan , 2013 ; Ghadimi et al. , 2016 ; Allen-Zhu & Hazan , 2016 ; Reddi et al. , 2016 ; Lei et al. , 2017 ; Li & Li , 2018 ; Zhou et al. , 2018 ; Fang et al. , 2018 ; Wang et al. , 2018 ; Ge et al. , 2019 ; Pham et al. , 2019 ; Li , 2019 ; Li & Richtárik , 2020 ; Horváth et al. , 2020 ; Li et al. , 2021 ) . Note that SVRG and SAGA variants typically achieve weaker convergence results than SARAH variants , i.e. , n2/3/ 2 vs. √ n/ 2 . Thus the current best convergence results are achieved by SARAH variants such as SPIDER ( Fang et al. , 2018 ) , SARAH ( Pham et al. , 2019 ) and PAGE ( Li et al. , 2021 ; Li , 2021 ) . However , all of these variance-reduced algorithms ( no matter based on SVRG , SAGA or SARAH ) require full gradient computations ( i.e. , compute ∇f ( x ) = 1n ∑n i=1∇fi ( x ) ) without assuming additional assumptions except standard L-smoothness assumptions . We would like to point out that under an additional bounded variance assumption ( e.g. , Ei [ ‖∇fi ( x ) −∇f ( x ) ‖2 ] ≤ σ2 , ∀x ∈ Rd ) , some of them ( such as SCSG ( Lei et al. , 2017 ) , SVRG+ ( Li & Li , 2018 ) , PAGE ( Li et al. , 2021 ) ) may avoid full gradient computations by using a large minibatch of stochastic gradients instead ( usually the minibatch size is O ( σ2/ 2 ) ) . Clearly , there exist some drawbacks : i ) σ2 usually is not known ; ii ) if the target error is very small ( defined as E [ ‖∇f ( x̂ ) ‖2 ] ≤ 2 in Definition 1 ) or σ is very large , then the minibatch size O ( σ2/ 2 ) is still very large for replacing full gradient computations . In this paper , we only consider algorithms under the standard L-smoothness assumptions , without assuming any other additional assumptions ( such as bounded variance assumption mentioned above ) . Thus , all existing variance-reduced methods , including SARAH , SVRG , SAGA and their variants , need to compute the full gradient over all n data samples at the initial point x0 , and then periodically compute the full gradient once every few iterations ( for SVRG , SARAH and their variants ) . However , full gradient computations are time-consuming steps in many applications as the number of data samples n usually is very large . Especially in the distributed setting , periodic computation of full gradient needs to periodically synchronize all clients/devices , which usually is impractical . Motivated by this , we focus on designing new algorithms which do not require any full gradient computations for solving standard and distributed nonconvex problems ( 1 ) – ( 2 ) . 2 OUR CONTRIBUTIONS . In this paper , we propose the first variance-reduced algorithm ZeroSARAH ( and also its distributed variant D-ZeroSARAH ) without computing any full gradients for solving both standard and distributed nonconvex finite-sum problems ( 1 ) – ( 2 ) . Moreover , ZeroSARAH and Distributed D-ZeroSARAH can obtain new state-of-the-art convergence results which improve previous bestknown results ( given by e.g. , SPIDER , SARAH and PAGE ) in certain regimes ( see Tables 1–2 for the comparison with previous algorithms ) . ZeroSARAH is formally described in Algorithm 2 , which is a variant of SARAH ( Nguyen et al. , 2017 ) . See Section 4 for more details and comparisons between ZeroSARAH and SARAH . Then , D-ZeroSARAH is formally described in Algorithm 3 of Section 5 , which is a distributed variant of our ZeroSARAH . Now , we highlight the following results achieved by ZeroSARAH and D-ZeroSARAH : • ZeroSARAH and D-ZeroSARAH are the first variance-reduced algorithms which do not require any full gradient computations , not even for the initial point ( see Algorithms 2–3 or Tables 1–2 ) . Avoiding any full gradient computations is important in many applications as the number of data samples n usually is very large . Especially in the distributed setting , periodic computation of full gradient over all data samples stored in all clients/devices may be impossible or very hard to achieve . We expect that ZeroSARAH/D-ZeroSARAH will have a practical impact in distributed and federated learning where full device participation is impractical . • Moreover , ZeroSARAH can recover the previous best-known convergence result O ( n + √ nL∆0 2 ) ( see Table 1 or Corollary 1 ) , and also provide new state-of-the-art convergence results without any full gradient computations ( see Table 1 or Corollary 2 ) which can improve the previous best result in certain regimes . • Besides , for the distributed nonconvex setting ( 2 ) , the distributed D-ZeroSARAH ( Algorithm 3 ) enjoys similar benefits as our ZeroSARAH , i.e. , D-ZeroSARAH does not need to periodically synchronize all n clients to compute any full gradients , and also provides new state-of-the-art convergence results . See Table 2 and Section 5 for more details . • Finally , the experimental results in Section 6 show that ZeroSARAH is slightly better than the previous state-of-the-art SARAH . However , we should point out that ZeroSARAH does not compute any full gradients while SARAH needs to periodically compute the full gradients for every l iterations ( here l = √ n ) . Thus the experiments validate our theoretical results ( can be slightly better than SARAH ( see Table 1 ) ) and confirm the practical superiority of ZeroSARAH ( avoid any full gradient computations ) . Similar experimental results of D-ZeroSARAH for the distributed setting are provided in Appendix A.2 . 3 PRELIMINARIES . Notation : Let [ n ] denote the set { 1 , 2 , · · · , n } and ‖ · ‖ denote the Euclidean norm for a vector and the spectral norm for a matrix . Let 〈u , v〉 denote the inner product of two vectors u and v. We use O ( · ) and Ω ( · ) to hide the absolute constant , and Õ ( · ) to hide the logarithmic factor . We will write ∆0 : = f ( x0 ) − f∗ , f∗ : = minx∈Rd f ( x ) , G0 : = 1n ∑n i=1 ‖∇fi ( x0 ) ‖2 , ∆̂0 : = f ( x0 ) − f̂∗ , f̂∗ : = 1n ∑n i=1 minx∈Rd fi ( x ) and G ′ 0 : = 1 nm ∑n , m i , j=1,1 ‖∇fi , j ( x0 ) ‖2 . Definition 1 A point x̂ is called an -approximate solution for nonconvex problems ( 1 ) and ( 2 ) if E [ ‖∇f ( x̂ ) ‖2 ] ≤ 2 . To show the convergence results , we assume the following standard smoothness assumption for nonconvex problems ( 1 ) . Assumption 1 ( L-smoothness ) A function fi : Rd → R is L-smooth if ∃L > 0 , such that ‖∇fi ( x ) −∇fi ( y ) ‖ ≤ L‖x− y‖ , ∀x , y ∈ Rd . ( 3 ) It is easy to see that f ( x ) = 1n ∑n i=1 fi ( x ) is also L-smooth under Assumption 1 . We can also relax Assumption 1 by defining Li-smoothness for each fi . Then if we further define the average L2 : = 1n ∑n i=1 L 2 i , we know that f ( x ) = 1 n ∑n i=1 fi ( x ) is also L-smooth . Here we use the same L just for simple representation . For the distributed nonconvex problems ( 2 ) , we use the following Assumption 2 instead of Assumption 1 . Similarly , we can also relax it by defining Li , j-smoothness for different fi , j . Here we use the same L just for simple representation . Assumption 2 ( L-smoothness ) A function fi , j : Rd → R is L-smooth if ∃L > 0 , such that ‖∇fi , j ( x ) −∇fi , j ( y ) ‖ ≤ L‖x− y‖ , ∀x , y ∈ Rd . ( 4 ) Algorithm 1 SARAH ( Nguyen et al. , 2017 ; Pham et al. , 2019 ) Input : initial point x0 , epoch length l , stepsize η , minibatch size b 1 : x̃ = x0 2 : for s = 0 , 1 , 2 , . . . do 3 : x0 = x̃ 4 : v0 = ∇f ( x0 ) = 1n n∑ i=1 ∇fi ( x0 ) // compute the full gradient once for every l iterations 5 : x1 = x0 − ηv0 6 : for k = 1 , 2 , . . . , l do 7 : Randomly sample a minibatch data samples Ib with |Ib| = b 8 : vk = 1b ∑ i∈Ib ( ∇fi ( xk ) −∇fi ( xk−1 ) ) + vk−1 9 : xk+1 = xk − ηvk 10 : end for 11 : x̃ randomly chosen from { xk } k∈ [ l ] or x̃ = xl+1 12 : end for Algorithm 2 SARAH without full gradient computations ( ZeroSARAH ) Input : initial point x0 , stepsize { ηk } , minibatch size { bk } , parameter { λk } 1 : x−1 = x0 2 : v−1 = 0 , y−11 = y −1 2 = · · · = y−1n = 0 // no full gradient computation 3 : for k = 0 , 1 , 2 , . . . do 4 : Randomly sample a minibatch data samples Ikb with |Ikb | = bk 5 : vk = 1bk ∑ i∈Ikb ( ∇fi ( xk ) −∇fi ( xk−1 ) ) + ( 1− λk ) vk−1 + λk ( 1 bk ∑ i∈Ikb ( ∇fi ( xk−1 ) − yk−1i ) + 1n n∑ j=1 yk−1j ) // no full gradient computations for vks 6 : xk+1 = xk − ηkvk 7 : yki = { ∇fi ( xk ) for i ∈ Ikb yk−1i for i /∈ Ikb // the update of { yki } directly follows from the stochastic gradients computed in Line 5 8 : end for 4 ZeroSARAH ALGORITHM AND ITS CONVERGENCE RESULTS In this section , we consider the standard/centralized nonconvex problems ( 1 ) . The distributed setting ( 2 ) is considered in the following Section 5 . 4.1 ZeroSARAH ALGORITHM We first describe the proposed ZeroSARAH in Algorithm 2 , which is a variant of SARAH ( Nguyen et al. , 2017 ) . To better compare with SARAH and ZeroSARAH , we also recall the original SARAH in Algorithm 1 . Now , we highlight some points for the difference between SARAH and our ZeroSARAH : • SARAH requires the full gradient computations for every epoch ( see Line 4 of Algorithm 1 ) . However , ZeroSARAH combines the past gradient estimator vk−1 with another estimator to avoid periodically computing the full gradient . See the difference between Line 8 of Algorithm 1 and Line 5 of Algorithm 2 ( also highlighted with blue color ) . • The gradient estimator vk in ZeroSARAH ( Line 5 of Algorithm 2 ) does not require more stochastic gradient computations compared with vk in SARAH ( Line 8 of Algorithm 1 ) if the minibatch size bk = b . • The new gradient estimator vk of ZeroSARAH also leads to simpler algorithmic structure , i.e. , single-loop in ZeroSARAH vs. double-loop in SARAH . •Moreover , the difference of gradient estimator vk also leads to different results in expectation , i.e. , 1 ) for SARAH : Ek [ vk −∇f ( xk ) ] = vk−1 −∇f ( xk−1 ) ; 2 ) for ZeroSARAH : Ek [ vk −∇f ( xk ) ] = ( 1− λk ) ( vk−1 −∇f ( xk−1 ) ) . 4.2 CONVERGENCE RESULTS FOR ZeroSARAH Now , we present the main convergence theorem ( Theorem 1 ) of ZeroSARAH ( Algorithm 2 ) for solving nonconvex finite-sum problems ( 1 ) . Subsequently , we formulate two corollaries which present the detailed convergence results by specifying the choice of parameters . In particular , we list the results of these two Corollaries 1–2 in Table 1 for comparing with convergence results of previous works . Theorem 1 Suppose that Assumption 1 holds . Choose stepsize ηk ≤ 1 L ( 1+ √ Mk+1 ) for any k ≥ 0 , where Mk+1 : = 2λk+1bk+1 + 8λk+1n 2 b3k+1 . Moreover , let λ0 = 1 , γ0 ≥ η02λ1 and α0 ≥ 2nλ1η0 b21 . Then the following equation holds for ZeroSARAH ( Algorithm 2 ) for solving problem ( 1 ) , for any iteration K ≥ 0 : E [ ‖∇f ( x̂K ) ‖2 ] ≤ 2∆0∑K−1 k=0 ηk + ( n− b0 ) ( 4γ0 + 2α0b0 ) G0 nb0 ∑K−1 k=0 ηk . ( 5 ) Remark : Note that we can upper bound both terms on the right-hand side of ( 5 ) . It means that there is no convergence neighborhood of ZeroSARAH and hence , ZeroSARAH can find an -approximate solution for any > 0 . In the following , we provide two detailed convergence results in Corollaries 1 and 2 by specifying two kinds of parameter settings . Note that the algorithm computes full gradient in iteration k if the minibatch bk = n. Our convergence results show that without computing any full gradients actually does not hurt the convergence performance of algorithms ( see Table 1 ) . In particular , we note that the second term of ( 5 ) will be deleted if we choose minibatch size b0 = n for the initial point x0 ( see Corollary 1 for more details ) . Here Corollary 1 only needs to compute the full gradient once for the initialization , and does not compute any full gradients later ( i.e. , bk ≡ √ n for all k > 0 ) . Also note that even if we choose b0 < n , we can also upper bound the second term of ( 5 ) . It means that ZeroSARAH can find an -approximate solution without computing any full gradients even for the initial point , i.e. , minibatch size bk < n for all iterations k ≥ 0 . For instance , we choose bk ≡ √ n for all k ≥ 0 in Corollary 2 , i.e. , ZeroSARAH never computes any full gradients even for the initial point . Corollary 1 Suppose that Assumption 1 holds . Choose stepsize ηk ≤ 1 ( 1+√8 ) L for any k ≥ 0 , minibatch size bk ≡ √ n and parameter λk = bk2n for any k ≥ 1 . Moreover , let b0 = n and λ0 = 1 . Then ZeroSARAH ( Algorithm 2 ) can find an -approximate solution for problem ( 1 ) such that E [ ‖∇f ( x̂K ) ‖2 ] ≤ 2 and the number of stochastic gradient computations can be bounded by # grad : = K−1∑ k=0 bk ≤ n+ 2 ( 1 + √ 8 ) √ nL∆0 2 = O ( n+ √ nL∆0 2 ) . Remark : In Corollary 1 , ZeroSARAH only computes the full gradient ∇f ( x0 ) = 1n ∑n i=1∇fi ( x0 ) once for the initial point x0 , i.e. , minibatch size b0 = n , and then bk ≡ √ n for all iterations k ≥ 1 in Algorithm 2 . In the following Corollary 2 , we show that ZeroSARAH without computing any full gradients even for the initial point does not hurt its convergence performance . Corollary 2 Suppose that Assumption 1 holds . Choose stepsize ηk ≤ 1 ( 1+√8 ) L for any k ≥ 0 , minibatch size bk ≡ √ n for any k ≥ 0 , and parameter λ0 = 1 and λk = bk2n for any k ≥ 1 . Then ZeroSARAH ( Algorithm 2 ) can find an -approximate solution for problem ( 1 ) such that E [ ‖∇f ( x̂K ) ‖2 ] ≤ 2 and the number of stochastic gradient computations can be bounded by # grad = O ( √ n ( L∆0 +G0 ) 2 ) . Note that G0 can be bounded by G0 ≤ 2L∆̂0 via L-smoothness Assumption 1 , then we also have # grad = O ( √ n ( L∆0 + L∆̂0 ) 2 ) . Remark : In Corollary 2 , ZeroSARAH never computes any full gradients even for the initial point , i.e. , minibatch size bk ≡ √ n for all iterations k ≥ 0 in Algorithm 2 . If we consider L , ∆0 , G0 or ∆̂0 as constant values then the stochastic gradient complexity in Corollary 2 is # grad = O ( √ n 2 ) , i.e. , full gradient computations do not appear in ZeroSARAH ( Algorithm 2 ) and the term ‘ n ’ also does not appear in its convergence result . Also note that the parameter settings ( i.e. , { ηk } , { bk } and { λk } in Algorithm 2 ) of Corollaries 1 and 2 are exactly the same except for b0 = n ( in Corollary 1 ) and b0 = √ n ( in Corollary 2 ) . Moreover , the parameter settings ( i.e. , { ηk } , { bk } and { λk } ) for Corollaries 1 and 2 only require the values of L and n , which is the same as all previous algorithms . If one further allows other values , e.g. , , G0 or ∆̂0 , for setting the initial b0 , then the gradient complexity can be further improved ( see Appendix D for more details ) . 5 D-ZeroSARAH ALGORITHM AND ITS CONVERGENCE RESULTS Now , we consider the distributed nonconvex problems ( 2 ) , i.e. , minx∈Rd { f ( x ) : = 1n ∑n i=1 fi ( x ) } with fi ( x ) : = 1m ∑m j=1 fi , j ( x ) , where n denotes the number of clients/devices/machines , fi denotes the loss associated with m data samples stored on client i . 5.1 D-ZeroSARAH ALGORITHM To solve distributed nonconvex problems ( 2 ) , we propose a distributed variant of ZeroSARAH ( called D-ZeroSARAH ) and describe it in Algorithm 3 . Same as our ZeroSARAH , D-ZeroSARAH also does not need to compute any full gradients at all . Avoiding any full gradient computations is important especially in this distributed setting , periodic computation of full gradient across all n clients may be impossible or unaffordable . Thus , we expect the proposed D-ZeroSARAH ( Algorithm 3 ) will have a practical impact in distributed and federated learning where full device participation is impractical . 5.2 CONVERGENCE RESULTS FOR D-ZeroSARAH Similar to ZeroSARAH in Section 4.2 , we also first present the main convergence theorem ( Theorem 2 ) of D-ZeroSARAH ( Algorithm 3 ) for solving distributed nonconvex problems ( 2 ) . Subsequently , we formulate two corollaries which present the detailed convergence results by specifying the choice of parameters . In particular , we list the results of these two Corollaries 3–4 in Table 2 for comparing with convergence results of previous works . Note that here we use the smoothness Assumption 2 instead of Assumption 1 for this distributed setting ( 2 ) . Theorem 2 Suppose that Assumption 2 holds . Choose stepsize ηk ≤ 1 L ( 1+ √ Wk+1 ) for any k ≥ 0 , whereWk+1 : = 2λk+1sk+1bk+1 + 8λk+1n 2m2 s3k+1b 3 k+1 . Moreover , let λ0 = 1 and θ0 : = nm ( nm−1 ) λ1 + 4nmλ1s0b0 s21b 2 1 . Then the following equation holds for D-ZeroSARAH ( Algorithm 3 ) for solving distributed problem ( 2 ) , for any iteration K ≥ 0 : E [ ‖∇f ( x̂K ) ‖2 ] ≤ 2∆0∑K−1 k=0 ηk + ( nm− s0b0 ) η0θ0G′0 nms0b0 ∑K−1 k=0 ηk . ( 6 ) Algorithm 3 Distributed ZeroSARAH ( D-ZeroSARAH ) Input : initial point x0 , parameters { ηk } , { sk } , { bk } , { λk } 1 : x−1 = x0 2 : v−1 = 0 , y−11 = y −1 2 = · · · = y−1n = 0 // no full gradient computation 3 : for k = 0 , 1 , 2 , . . . do 4 : Randomly sample a subset of clients Sk from n clients with size |Sk| = sk 5 : for each client i ∈ Sk do 6 : Sample the data minibatch Ikbi ( with size |I k bi | = bk ) from the m data samples in client i 7 : Compute its local minibatch gradient information : gki , curr = 1 bk ∑ j∈Ikbi ∇fi , j ( xk ) , gki , prev = 1bk ∑ j∈Ikbi ∇fi , j ( xk−1 ) , yki , prev = 1bk ∑ j∈Ikbi yk−1i , j yki , j = { ∇fi , j ( xk ) for j ∈ Ikbi yk−1i , j for j /∈ Ikbi , yki = 1 m m∑ j=1 yki , j 8 : end for 9 : vk = 1sk ∑ i∈Sk ( gki , curr − gki , prev ) + ( 1− λk ) vk−1 + λk 1sk ∑ i∈Sk ( gki , prev − yki , prev ) + λky k−1 // no full gradient computations for vks 10 : xk+1 = xk − ηkvk 11 : yk = 1n n∑ i=1 yki // here y k i = y k−1 i for client i /∈ S k 12 : end for Corollary 3 Suppose that Assumption 2 holds . Choose stepsize ηk ≤ 1 ( 1+√8 ) L for any k ≥ 0 , clients subset size sk ≡ √ n , minibatch size bk ≡ √ m and parameter λk = skbk2nm for any k ≥ 1 . Moreover , let s0 = n , b0 = m , and λ0 = 1 . Then D-ZeroSARAH ( Algorithm 3 ) can find an -approximate solution for distributed problem ( 2 ) such that E [ ‖∇f ( x̂K ) ‖2 ] ≤ 2 and the number of stochastic gradient computations for each client can be bounded by # grad = O ( m+ √ m n L∆0 2 ) . Corollary 4 Suppose that Assumption 2 holds . Choose stepsize ηk ≤ 1 ( 1+√8 ) L for any k ≥ 0 , clients subset size sk ≡ √ n and minibatch size bk ≡ √ m for any k ≥ 0 , and parameter λ0 = 1 and λk = skbk 2nm for any k ≥ 1 . Then D-ZeroSARAH ( Algorithm 3 ) can find an -approximate solution for distributed problem ( 2 ) such that E [ ‖∇f ( x̂K ) ‖2 ] ≤ 2 and the number of stochastic gradient computations for each client can be bounded by # grad = O ( √ m n L∆0 +G ′ 0 2 ) . Remark : Similar discussions and remarks of Theorem 1 and Corollaries 1–2 for ZeroSARAH in Section 4.2 also hold for the results of D-ZeroSARAH ( i.e. , Theorem 2 and Corollaries 3–4 ) .

Summary: The authors propose a variance-reduced algorithm ZeroSARAH for solving the nonconvex finite-sum problems. They also propose a distributed variant D-ZeroSARAH for the corresponding distributed framework. They provide the state-of-the-art convergence results for these methods using standard assumptions. The authors claim that both methods does not require computing full gradients, which gives a major advantage when the number of samples $n$ is large. They further validate ZeroSARAH using nonconvex linear regression and binary classification for neural networks.