AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods

1 INTRODUCTION . We consider the unconstrained finite-sum optimization problem minw∈Rd [ P ( w ) def = 1n ∑n i=1fi ( w ) ] . ( 1 ) This problem is prevalent in machine learning tasks where w corresponds to the model parameters , fi ( w ) represents the loss on the training point i , and the goal is to minimize the average loss P ( w ) across the training points . In machine learning applications , ( 1 ) is often considered the loss function of Empirical Risk Minimization ( ERM ) problems . For instance , given a classification or regression problem , fi can be defined as logistic regression or least square by ( xi , yi ) where xi is a feature representation and yi is a label . Throughout the paper , we assume that each function fi , i ∈ [ n ] def= { 1 , ... , n } , is smooth and convex , and there exists an optimal solution w∗ of ( 1 ) . 1.1 MAIN CONTRIBUTIONS . We propose AI-SARAH , a practical variant of stochastic recursive gradient methods ( Nguyen et al. , 2017 ) to solve ( 1 ) . This practical algorithm explores and adapts to local geometry . It is adaptive at full scale yet requires zero effort of tuning hyper-parameters . The extensive numerical experiments demonstrate that our tune-free and fully adaptive algorithm is capable of delivering a consistently competitive performance on various datasets , when comparing with SARAH , SARAH+ and other state-of-the-art first-order method , all equipped with fine-tuned hyper-parameters ( which are selected from ≈ 5 , 000 runs for each problem ) . This work provides a foundation on studying adaptivity ( of stochastic recursive gradient methods ) and demonstrates that a truly adaptive stochastic recursive algorithm can be developed in practice . 1.2 RELATED WORK . Stochastic gradient descent ( SGD ) ( Robbins & Monro , 1951 ; Nemirovski & Yudin , 1983 ; ShalevShwartz et al. , 2007 ; Nemirovski et al. , 2009 ; Gower et al. , 2019 ) is the workhorse for training supervised machine learning problems that have the generic form ( 1 ) . In its generic form , SGD defines the new iterate by subtracting a multiple of a stochastic gradient g ( wt ) from the current iterate wt . That is , wt+1 = wt − αtg ( wt ) . In most algorithms , g ( w ) is an unbiased estimator of the gradient ( i.e. , a stochastic gradient ) , E [ g ( w ) ] = ∇P ( w ) , ∀w ∈ Rd . However , in several algorithms ( including the ones from this paper ) , g ( w ) could be a biased estimator , and convergence guarantees can still be well obtained . Adaptive step-size selection . The main parameter to guarantee the convergence of SGD is the step-size . In recent years , several ways of selecting the step-size have been proposed . For example , an analysis of SGD with constant step-size ( αt = α ) or decreasing step-size has been proposed in Moulines & Bach ( 2011 ) ; Ghadimi & Lan ( 2013 ) ; Needell et al . ( 2016 ) ; Nguyen et al . ( 2018 ) ; Bottou et al . ( 2018 ) ; Gower et al . ( 2019 ; 2020b ) under different assumptions on the properties of problem ( 1 ) . More recently , adaptive / parameter-free methods ( Duchi et al. , 2011 ; Kingma & Ba , 2015 ; Bengio , 2015 ; Li & Orabona , 2018 ; Vaswani et al. , 2019 ; Liu et al. , 2019a ; Ward et al. , 2019 ; Loizou et al. , 2020 ) that adapt the step-size as the algorithms progress have become popular and are particularly beneficial when training deep neural networks . Normally , in these algorithms , the step-size does not depend on parameters that might be unknown in practical scenarios , like the smoothness parameter or the strongly convex parameter . Random vector g ( wt ) and variance reduced methods . One of the most remarkable algorithmic breakthroughs in recent years was the development of variance-reduced stochastic gradient algorithms for solving finite-sum optimization problems . These algorithms , by reducing the variance of the stochastic gradients , are able to guarantee convergence to the exact solution of the optimization problem with faster convergence than classical SGD . In the past decade , many efficient variancereduced methods have been proposed . Some popular examples of variance reduced algorithms are SAG ( Schmidt et al. , 2017 ) , SAGA ( Defazio et al. , 2014 ) , SVRG ( Johnson & Zhang , 2013 ) and SARAH ( Nguyen et al. , 2017 ) . For more examples of variance reduced methods , see Defazio ( 2016 ) ; Konečný et al . ( 2016 ) ; Gower et al . ( 2020a ) ; Khaled et al . ( 2020 ) ; Horváth et al . ( 2020 ) . Among the variance reduced methods , SARAH is of our interest in this work . Like the popular SVRG , SARAH algorithm is composed of two nested loops . In each outer loop k ≥ 1 , the gradient estimate v0 = ∇P ( wk−1 ) is set to be the full gradient . Subsequently , in the inner loop , at t ≥ 1 , a biased estimator vt is used and defined recursively as vt = ∇fi ( wt ) −∇fi ( wt−1 ) + vt−1 , ( 2 ) where i ∈ [ n ] is a random sample selected at t. A common characteristic of the popular variance reduced methods is that the step-size α in their update rule wt+1 = wt − αvt is constant ( or diminishing with predetermined rules ) and that depends on the characteristics of problem ( 1 ) . An exception to this rule are the variance reduced methods with Barzilai-Borwein step size , named BB-SVRG and BB-SARAH proposed in Tan et al . ( 2016 ) and Li & Giannakis ( 2019 ) respectively . These methods allow to use Barzilai-Borwein ( BB ) step size rule to update the step-size once in every epoch ; for more examples , see Li et al . ( 2020 ) ; Yang et al . ( 2021 ) . There are also methods proposing approach of using local Lipschitz smoothness to derive an adaptive step-size ( Liu et al. , 2019b ) with additional tunable parameters or leveraging BB step-size with averaging schemes to automatically determine the inner loop size ( Li et al. , 2020 ) . However , these methods do not fully take advantage of the local geometry , and a truly adaptive algorithm : adjusting step-size at every ( inner ) iteration and eliminating need of tuning any hyper-parameters , is yet to be developed in the stochastic variance reduced framework . This is exactly the main contribution of this work , as we mentioned in previous section . 2 MOTIVATION . With our primary focus on the design of a stochastic recursive algorithm with adaptive step-size , we discuss our motivation in this chapter . A standard approach of tuning the step-size involves the painstaking grid search on a wide range of candidates . While more sophisticated methods can design a tuning plan , they often struggle for efficiency and/or require a considerable amount of computing resources . More importantly , tuning step-size requires knowledge that is not readily available at a starting point w0 ∈ Rd , and choices of step-size could be heavily influenced by the curvature provided ∇2P ( w0 ) . What if a step-size has to be small due to a `` sharp '' curvature initially , which becomes `` flat '' afterwards ? To see this is indeed the case for many machine learning problems , let us consider logistic regression for a binary classification problem , i.e. , fi ( w ) = log ( 1 + exp ( −yixTi w ) ) + λ2 ‖w‖ 2 , where xi ∈ Rd is a feature vector , yi ∈ { −1 , +1 } is a ground truth , and the ERM problem is in the form of ( 1 ) . It is easy to derive the local curvature of P ( w ) , defined by its Hessian in the form ∇2P ( w ) = 1n ∑n i=1 exp ( −yixTi w ) [ 1+exp ( −yixTi w ) ] 2︸ ︷︷ ︸ si ( w ) xix T i + λI . ( 3 ) Given that a ( 1+a ) 2 ≤ 0.25 for any a ≥ 0 , one can immediately obtain the global bound on Hessian , i.e . ∀w ∈ Rd we have ∇2P ( w ) 14 1 n ∑n i=1 xix T i + λI . Consequently , the parameter of global Lipschitz smoothness is L = 14λmax ( 1 n ∑n i=1 xix T i ) + λ . It is well known that , with a constant step-size less than ( or equal to ) 1L , a convergence is guaranteed by many algorithms . However , suppose the algorithm starts at a random w0 ( or at 0 ∈ Rd ) , this bound can be very tight . With more progress being made on approaching an optimal solution ( or reducing the training error ) , it is likely that , for many training samples , −yixTi wt 0 . An immediate implication is that si ( wt ) defined in ( 3 ) becomes smaller and hence the local curvature will be smaller as well . It suggests that , although a large initial step-size could lead to divergence , with more progress made by the algorithm , the parameter of local Lipschitz smoothness tends to be smaller and a larger step-size can be used . That being said , such a dynamic step-size can not be well defined in the beginning , and a fully adaptive approach needs to be developed . For illustration , we present the inspiring results of an experiment on real-sim dataset1 with ` 2-regularized logistic regression . Figures 1 ( a ) and 1 ( b ) compare the performance of classical SARAH with AI-SARAH in terms of the evolution of the optimality gap and the squared norm of recursive gradient . As is clear from the figure , AI-SARAH displays a significantly faster convergence per effective pass2 . Now , let us discuss why this could happen . The distribution of si as shown in Figured 1 ( d ) indicates that : initially , all si ’ s are concentrated at 0.25 ; the median continues to reduce within a few effective passes on the training samples ; eventually , it stabilizes somewhere below 0.05 . Correspondingly , as presented in Figure 1 ( c ) , AI-SARAH starts with a conservative step-size dominated by the global Lipschitz smoothness , i.e. , 1/λmax ( ∇2P ( w0 ) ) ( red dots ) ; however , within 5 effective passes , the moving average ( magenta dash ) and upper-bound ( blue line ) of the step-size start surpassing the red dots , and eventually stablize above the conservative step-size . For classical SARAH , we configure the algorithm with different values of the fixed step-size , i.e. , { 2−2 , 2−1 , ... , 24 } , and notice that 25 leads to a divergence . On the other hand , AI-SARAH starts with a small step-size , yet achieves a faster convergence per effective pass with an eventual ( moving average ) step-size larger than 25 . 1The dataset is available at https : //www.csie.ntu.edu.tw/~cjlin/libsvmtools/ datasets/ 2The effective pass is defined as a complete pass on the training dataset . Each data sample is selected once per effective pass on average .

Summary: In this paper, the authors proposed a SARAH-type variance reduced gradient method that adaptively and automatically selects the stepsize.  In each step, the authors propose to do an approximate line-search of the currently sampled function $f_{i_t}$ (or $f_{S_t}$) to estimate the local Lipschitz constant of the sampled function. To estimate the Lipschitz constant of the true objective function (the summation), the authors propose to do an exponential moving average of the estimated local Lipschitz constants.   Extensive experiments are done in solving convex problems, and the comparison with other state-of-the-art first-order methods is also carefully presented. The authors also presented some theoretical guarantees of the algorithm with certain modifications.