AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods

1 INTRODUCTION . We consider the unconstrained finite-sum optimization problem minw∈Rd [ P ( w ) def = 1n ∑n i=1fi ( w ) ] . ( 1 ) This problem is prevalent in machine learning tasks where w corresponds to the model parameters , fi ( w ) represents the loss on the training point i , and the goal is to minimize the average loss P ( w ) across the training points . In machine learning applications , ( 1 ) is often considered the loss function of Empirical Risk Minimization ( ERM ) problems . For instance , given a classification or regression problem , fi can be defined as logistic regression or least square by ( xi , yi ) where xi is a feature representation and yi is a label . Throughout the paper , we assume that each function fi , i ∈ [ n ] def= { 1 , ... , n } , is smooth and convex , and there exists an optimal solution w∗ of ( 1 ) . 1.1 MAIN CONTRIBUTIONS . We propose AI-SARAH , a practical variant of stochastic recursive gradient methods ( Nguyen et al. , 2017 ) to solve ( 1 ) . This practical algorithm explores and adapts to local geometry . It is adaptive at full scale yet requires zero effort of tuning hyper-parameters . The extensive numerical experiments demonstrate that our tune-free and fully adaptive algorithm is capable of delivering a consistently competitive performance on various datasets , when comparing with SARAH , SARAH+ and other state-of-the-art first-order method , all equipped with fine-tuned hyper-parameters ( which are selected from ≈ 5 , 000 runs for each problem ) . This work provides a foundation on studying adaptivity ( of stochastic recursive gradient methods ) and demonstrates that a truly adaptive stochastic recursive algorithm can be developed in practice . 1.2 RELATED WORK . Stochastic gradient descent ( SGD ) ( Robbins & Monro , 1951 ; Nemirovski & Yudin , 1983 ; ShalevShwartz et al. , 2007 ; Nemirovski et al. , 2009 ; Gower et al. , 2019 ) is the workhorse for training supervised machine learning problems that have the generic form ( 1 ) . In its generic form , SGD defines the new iterate by subtracting a multiple of a stochastic gradient g ( wt ) from the current iterate wt . That is , wt+1 = wt − αtg ( wt ) . In most algorithms , g ( w ) is an unbiased estimator of the gradient ( i.e. , a stochastic gradient ) , E [ g ( w ) ] = ∇P ( w ) , ∀w ∈ Rd . However , in several algorithms ( including the ones from this paper ) , g ( w ) could be a biased estimator , and convergence guarantees can still be well obtained . Adaptive step-size selection . The main parameter to guarantee the convergence of SGD is the step-size . In recent years , several ways of selecting the step-size have been proposed . For example , an analysis of SGD with constant step-size ( αt = α ) or decreasing step-size has been proposed in Moulines & Bach ( 2011 ) ; Ghadimi & Lan ( 2013 ) ; Needell et al . ( 2016 ) ; Nguyen et al . ( 2018 ) ; Bottou et al . ( 2018 ) ; Gower et al . ( 2019 ; 2020b ) under different assumptions on the properties of problem ( 1 ) . More recently , adaptive / parameter-free methods ( Duchi et al. , 2011 ; Kingma & Ba , 2015 ; Bengio , 2015 ; Li & Orabona , 2018 ; Vaswani et al. , 2019 ; Liu et al. , 2019a ; Ward et al. , 2019 ; Loizou et al. , 2020 ) that adapt the step-size as the algorithms progress have become popular and are particularly beneficial when training deep neural networks . Normally , in these algorithms , the step-size does not depend on parameters that might be unknown in practical scenarios , like the smoothness parameter or the strongly convex parameter . Random vector g ( wt ) and variance reduced methods . One of the most remarkable algorithmic breakthroughs in recent years was the development of variance-reduced stochastic gradient algorithms for solving finite-sum optimization problems . These algorithms , by reducing the variance of the stochastic gradients , are able to guarantee convergence to the exact solution of the optimization problem with faster convergence than classical SGD . In the past decade , many efficient variancereduced methods have been proposed . Some popular examples of variance reduced algorithms are SAG ( Schmidt et al. , 2017 ) , SAGA ( Defazio et al. , 2014 ) , SVRG ( Johnson & Zhang , 2013 ) and SARAH ( Nguyen et al. , 2017 ) . For more examples of variance reduced methods , see Defazio ( 2016 ) ; Konečný et al . ( 2016 ) ; Gower et al . ( 2020a ) ; Khaled et al . ( 2020 ) ; Horváth et al . ( 2020 ) . Among the variance reduced methods , SARAH is of our interest in this work . Like the popular SVRG , SARAH algorithm is composed of two nested loops . In each outer loop k ≥ 1 , the gradient estimate v0 = ∇P ( wk−1 ) is set to be the full gradient . Subsequently , in the inner loop , at t ≥ 1 , a biased estimator vt is used and defined recursively as vt = ∇fi ( wt ) −∇fi ( wt−1 ) + vt−1 , ( 2 ) where i ∈ [ n ] is a random sample selected at t. A common characteristic of the popular variance reduced methods is that the step-size α in their update rule wt+1 = wt − αvt is constant ( or diminishing with predetermined rules ) and that depends on the characteristics of problem ( 1 ) . An exception to this rule are the variance reduced methods with Barzilai-Borwein step size , named BB-SVRG and BB-SARAH proposed in Tan et al . ( 2016 ) and Li & Giannakis ( 2019 ) respectively . These methods allow to use Barzilai-Borwein ( BB ) step size rule to update the step-size once in every epoch ; for more examples , see Li et al . ( 2020 ) ; Yang et al . ( 2021 ) . There are also methods proposing approach of using local Lipschitz smoothness to derive an adaptive step-size ( Liu et al. , 2019b ) with additional tunable parameters or leveraging BB step-size with averaging schemes to automatically determine the inner loop size ( Li et al. , 2020 ) . However , these methods do not fully take advantage of the local geometry , and a truly adaptive algorithm : adjusting step-size at every ( inner ) iteration and eliminating need of tuning any hyper-parameters , is yet to be developed in the stochastic variance reduced framework . This is exactly the main contribution of this work , as we mentioned in previous section . 2 MOTIVATION . With our primary focus on the design of a stochastic recursive algorithm with adaptive step-size , we discuss our motivation in this chapter . A standard approach of tuning the step-size involves the painstaking grid search on a wide range of candidates . While more sophisticated methods can design a tuning plan , they often struggle for efficiency and/or require a considerable amount of computing resources . More importantly , tuning step-size requires knowledge that is not readily available at a starting point w0 ∈ Rd , and choices of step-size could be heavily influenced by the curvature provided ∇2P ( w0 ) . What if a step-size has to be small due to a `` sharp '' curvature initially , which becomes `` flat '' afterwards ? To see this is indeed the case for many machine learning problems , let us consider logistic regression for a binary classification problem , i.e. , fi ( w ) = log ( 1 + exp ( −yixTi w ) ) + λ2 ‖w‖ 2 , where xi ∈ Rd is a feature vector , yi ∈ { −1 , +1 } is a ground truth , and the ERM problem is in the form of ( 1 ) . It is easy to derive the local curvature of P ( w ) , defined by its Hessian in the form ∇2P ( w ) = 1n ∑n i=1 exp ( −yixTi w ) [ 1+exp ( −yixTi w ) ] 2︸ ︷︷ ︸ si ( w ) xix T i + λI . ( 3 ) Given that a ( 1+a ) 2 ≤ 0.25 for any a ≥ 0 , one can immediately obtain the global bound on Hessian , i.e . ∀w ∈ Rd we have ∇2P ( w ) 14 1 n ∑n i=1 xix T i + λI . Consequently , the parameter of global Lipschitz smoothness is L = 14λmax ( 1 n ∑n i=1 xix T i ) + λ . It is well known that , with a constant step-size less than ( or equal to ) 1L , a convergence is guaranteed by many algorithms . However , suppose the algorithm starts at a random w0 ( or at 0 ∈ Rd ) , this bound can be very tight . With more progress being made on approaching an optimal solution ( or reducing the training error ) , it is likely that , for many training samples , −yixTi wt 0 . An immediate implication is that si ( wt ) defined in ( 3 ) becomes smaller and hence the local curvature will be smaller as well . It suggests that , although a large initial step-size could lead to divergence , with more progress made by the algorithm , the parameter of local Lipschitz smoothness tends to be smaller and a larger step-size can be used . That being said , such a dynamic step-size can not be well defined in the beginning , and a fully adaptive approach needs to be developed . For illustration , we present the inspiring results of an experiment on real-sim dataset1 with ` 2-regularized logistic regression . Figures 1 ( a ) and 1 ( b ) compare the performance of classical SARAH with AI-SARAH in terms of the evolution of the optimality gap and the squared norm of recursive gradient . As is clear from the figure , AI-SARAH displays a significantly faster convergence per effective pass2 . Now , let us discuss why this could happen . The distribution of si as shown in Figured 1 ( d ) indicates that : initially , all si ’ s are concentrated at 0.25 ; the median continues to reduce within a few effective passes on the training samples ; eventually , it stabilizes somewhere below 0.05 . Correspondingly , as presented in Figure 1 ( c ) , AI-SARAH starts with a conservative step-size dominated by the global Lipschitz smoothness , i.e. , 1/λmax ( ∇2P ( w0 ) ) ( red dots ) ; however , within 5 effective passes , the moving average ( magenta dash ) and upper-bound ( blue line ) of the step-size start surpassing the red dots , and eventually stablize above the conservative step-size . For classical SARAH , we configure the algorithm with different values of the fixed step-size , i.e. , { 2−2 , 2−1 , ... , 24 } , and notice that 25 leads to a divergence . On the other hand , AI-SARAH starts with a small step-size , yet achieves a faster convergence per effective pass with an eventual ( moving average ) step-size larger than 25 . 1The dataset is available at https : //www.csie.ntu.edu.tw/~cjlin/libsvmtools/ datasets/ 2The effective pass is defined as a complete pass on the training dataset . Each data sample is selected once per effective pass on average .

Summary: This paper proposes a new variance reduced algorithm for solving smooth and strongly convex optimization. The authors build on SARAH and propose AI-SARAH which aims to estimate the local Lipschitz smoothness parameters on the fly. The authors provide theoretical analysis for a modified version of AI-SARAH and provide extensive practical experiments for AI-SARAH.