Accelerated Variance Reduced Stochastic Extragradient Method for Sparse Machine Learning Problems

1 INTRODUCTION . In this paper , we mainly consider the following composite convex optimization problem : min x∈Rd { P ( x ) def = F ( x ) +R ( x ) = 1 n n∑ i=1 fi ( x ) +R ( x ) } ( 1 ) where F ( x ) : Rd→R is the average of smooth convex component functions fi ( x ) , and R ( x ) is a relatively simple convex function ( but may not be differentiable ) . In this paper , we use ‖·‖ to denote the standard Euclidean norm , and ‖·‖1 to denote the ` 1-norm . Moreover , we use P∗ to denote the real optimal value of P ( · ) , and P̂∗ to denote the optimal value obtained by algorithms . This form of optimization problems often appears in machine learning , signal processing , data science , statistics and operations research , and has a wide range of applications such as regularized empirical risk minimization ( ERM ) , sparse coding for image and video recovery , and representation learning for object recognition . Specifically , for a collection of given training examples { ( a1 , b1 ) , ... , ( an , bn ) } , where ai ∈ Rd , bi ∈ R ( i = 1 , 2 , ... , n ) and ai is a feature vector , while bi is the desired response . When fi ( x ) = 12 ( a T i x−bi ) 2 , we can obtain the ridge regression problem by setting R ( x ) = λ2 ‖x‖ 2 . We also get the Lasso or Elastic-Net problems by setting R ( x ) =λ‖x‖1 or R ( x ) = λ22 ‖x‖ 2+λ1‖x‖1 , respectively . Moreover , if we set fi ( x ) = log ( 1+exp ( −bixTai ) ) , we will get the regularized logistic regression problem . 1.1 RECENT RESEARCH PROGRESS . The proximal gradient descent ( PGD ) method is a standard and effective method for Problem ( 1 ) , and can achieve linear convergence for strongly convex problems . Its accelerated algorithms , e.g. , accelerated proximal gradient ( APG ) ( Tseng ( 2008 ) ; Beck & Teboulle ( 2009 ) ) , attain the convergence rate ofO ( 1/T 2 ) for non-strongly convex problems , where T denotes the number of iterations . In recent years , stochastic gradient descent ( SGD ) has been successfully applied to many large-scale learning problems , such as training for deep networks and linear prediction ( Tong ( 2004 ) ) , because of its significantly lower per-iteration complexity than deterministic methods , i.e. , O ( d ) vs. O ( nd ) . Besides , many tricks for SGD have also been proposed , such as Loshchilov & Hutter ( 2016 ) . However , the variance of the stochastic gradient may be large due to random sampling ( Johnson & Tong ( 2013 ) ) , which leads that the algorithm requires a gradually reduced step size , thus it will converge slow . Even under the strongly convex condition , SGD only achieves a sub-linear convergence rate O ( 1/T ) . Recently , many SGD methods with variance reduction have been proposed . For the case of R ( x ) = 0 , Roux et al . ( 2012 ) developed a stochastic average gradient descent ( SAG ) method , which is a randomized variant of the incremental aggregated gradient method proposed by Blatt et al . ( 2007 ) . Then stochastic variance reduced gradient ( SVRG ) ( Johnson & Tong ( 2013 ) ) was proposed , and has been widely introduced into various subsequent optimization algorithms , due to its lower storage space ( i.e. , O ( d ) ) than that of SAG ( i.e. , O ( nd ) ) . SVRG reduced the variance effectively by changing the estimation of stochastic gradients . The introduction of a snapshot point x̃ mainly has the effect of correcting the direction of gradient descent , and reduces the variance . Later , Konečný & Richtárik ( 2013 ) proposed the semi-stochastic gradient descent methods as well as their mini-batch version ( Konečný et al . ( 2014 ) ) . And their asynchronous distributed variant ( Ruiliang et al . ( 2016 ) ) is also been proposed later . More recently , Lin & Tong ( 2014 ) proposed the Prox-SVRG method , which introduced the proximal operator , and then applied the idea of SVRG to solve the non-smooth optimization problems . However , Prox-SVRG can only be used to solve the strongly convex optimization problems . In order to solve the non-strongly convex problems , Zeyuan & Yuan ( 2016 ) proposed the SVRG++ algorithm . Besides , to accelerate the algorithm and reducing the complexity , by combining the main ideas of APG and Prox-SVRG , Nitanda ( 2014 ) proposed an accelerated variance reduction proximal stochastic gradient descent ( Acc-Prox-SVRG ) method , which can effectively reduce the complexity of the algorithm compared to the two basic algorithms . Very recently , Zeyuan ( 2017 ) developed a novel Katyusha algorithm which introduced the Katyusha momentum to accelerate the algorithm . With the development of parallel and distributed computing which can effectively reduce computing time and improve performance , Ryu & Wotao ( 2017 ) came up with an algorithm called Proximal Proximal Gradient , which combined the proximal gradient method and ADMM ( Gabay & Mercier ( 1976 ) ) . Furthermore , it is easy to implement in parallel and distributed environments because of its innovative algorithm structure . 1.2 OUR MAIN CONTRIBUTIONS . We find that due to the introduction of proximal operator , there is a gap between P̂∗ and P∗ , and its theoretical derivation can be seen in Appendix A . To address this issue , Nguyen et al . ( 2017 ) proposed the idea of extragradient which can be seen as a guide during the process , and introduced it into the optimization problems . Intuitively , this additional iteration allows us to examine the geometry of the problem and consider its curvature information , which is one of the most important bottlenecks for first order methods . By using the idea of extragradient , we can get a better result in each inner-iteration . Therefore , the idea of extragradient is our main motivation . In this paper , we propose a novel algorithm for solving non-smooth optimization problems . The main contributions of this paper are summarized as follows . • In order to improve the result of the gap between P̂∗ and P∗ , and achieve fast convergence , a novel algorithm , which combines the idea of extragradient , Prox-SVRG and the trick of momentum acceleration , is proposed , called accelerated variance reduced stochastic extragradient descent ( AVR-SExtraGD ) . •We provide the convergence analysis of our algorithm , which shows that AVR-SExtraGD achieves linear convergence for strongly convex problems , and the convergence condition in the non-strongly convex case is also given . According to the convergence rate , we can know that AVR-SExtraGD has the same excellent result as the best-known algorithms , such as Katyusha . • Finally , we show by experiments that the performance of AVR-SExtraGD ( as well as VRSExtraGD , which is the basic algorithm of AVR-SExtraGD ) is obviously better than the popular algorithm , Prox-SVRG , which confirms the advantage of extragradient . For the widely used accelerated algorithm , Katyusha , the performance of our algorithm is still improved . 2 RELATED WORK . 2.1 BASIC ASSUMPTIONS . We first make the following assumptions to solve the problem ( 1 ) : Assumption 1 ( Smoothness ) . The convex function F ( · ) is L-smooth , i.e. , there exists a constant L > 0 such that for any x , y∈Rd , ‖∇F ( x ) −∇F ( y ) ‖≤L‖x−y‖ . Assumption 2 ( Lower Semi-continuity ) . The regularization function R ( · ) is a lower semicontinuous function , i.e. , ∀x0∈Rd , lim inf x→x0 R ( x ) ≥R ( x0 ) . But it is not necessarily differentiable or continuous .. Assumption 3 ( Strong Convexity ) . In Problem ( 1 ) , the functionR ( · ) is µ-strongly convex , i.e. , there exists a constant µ > 0 such that for all x , y∈Rd , it holds that R ( x ) ≥ R ( y ) + 〈G , x− y〉+ µ 2 ‖x− y‖2 , ( 2 ) where G∈∂R ( y ) which is the set of sub-gradient of R ( · ) at y . 2.2 PROX-SVRG AND EXTRAGRADIENT DESCENT METHODS . An effective method for solving Problem ( 1 ) is Prox-SVRG which improved Prox-FG ( Lions & Mercier ( 1979 ) ) and Prox-SG ( Langford et al . ( 2009 ) ) by introducing the stochastic gradient and combining the idea of SVRG , respectively . For strongly convex problems , Prox-SVRG can reach linear convergence with a constant step size , and its main update rules are ∇̃fik ( xk−1 ) = ∇fik ( xk−1 ) −∇fik ( x̃ ) +∇F ( x̃ ) ; xk = ProxRη ( xk−1 − η∇̃fik ( xk−1 ) ) , ( 3 ) where x̃ is the snapshot point used in SVRG , ∇̃fik ( xk−1 ) is the variance reduced stochastic gradient estimator , and ProxRη ( · ) is the proximal operator . Although Prox-SVRG can converge fast , because of proximal operator , the final solution has the deviation , which makes the solution inaccurate , thus Prox-SVRG still needs to be further improved , which is our important motivation . The extragradient method was first proposed by Korpelevič ( 1976 ) . It is a classical method for solving variational inequality problems , and it generates an estimation sequence by using two projection gradients in each iteration . By combining this idea with some first-order descent methods , Nguyen et al . ( 2017 ) proposed an extended extragradient method ( EEG ) which can effectively solve the problem ( 1 ) , and can also solve relatively more general problems as follows : min x∈Rd { P ( x ) def = F ( x ) +R ( x ) } where F ( x ) is not necessarily composed by multiple functions fi ( x ) . Unlike the classical extragradient method , EEG uses proximal gradient instead of orthogonal projection in each iteration . The main update rules of EEG are yk = ProxRsk ( xk − sk∇F ( xk ) ) ; xk+1 = Prox R αk ( xk − αk∇F ( yk ) ) , where sk and αk are two step sizes . From the update rules of EEG , we can see that in each iteration , EEG needs to calculate two gradients , which will definitely slow down the algorithm . Therefore , the algorithm needs to be further accelerated by an efficient technique . 2.3 MOMENTUM ACCELERATION AND MIG . Firstly , we introduce the momentum acceleration technique whose main update rules are vdwt = βvdwt−1 + ( 1− β ) dwt ; wt = wt−1 − αvdwt , where dw is the gradient of the objective function at w , β is a parameter , and α is a step size . The update rules take not only the gradient of the current position , but also the gradient of the past position into account , which makes the final descent direction of wt after using momentum Algorithm 1 AVR-SExtraGD Input : Initial vector x0 , the number of epochs S , the number of iterations m per epoch , the step sizes η1 , η2 , momentum parameter β , and the set K. Initialize : x̃0 = x10 = x0 , ρ = 1+ηµ . 1 : for s = 1 , 2 , . . . , S do 2 : Compute∇F ( x̃s−1 ) ; 3 : βs = β ( SC ) or βs = 2s+4 ( non-SC ) ; 4 : for k = 1 , 2 , . . . , m do 5 : Pick ik uniformly at random from { 1 , ... , n } ; 6 : if k ∈ K then 7 : xsk−1/2 = Prox R η1 ( xsk−1 − η1∇̃fik ( βsxsk−1 + ( 1−βs ) x̃s−1 ) ) ; 8 : xsk = Prox R η2 ( xsk−1/2 − η2∇̃fik ( βsx s k−1/2 + ( 1−βs ) x̃ s−1 ) ) ; 9 : else 10 : xsk = Prox R η1 ( x s k−1 − η1∇̃fik ( βsxsk−1 + ( 1−βs ) x̃s−1 ) ) ; 11 : end if 12 : end for 13 : x̃s = βs ( ∑m k=1 ρ k−1 ) −1 ∑m k=1 ρ k−1 x s k−1/2+x s k 2 + ( 1−βs ) x̃ s−1 ( SC ) or x̃s = βsm ∑m k=1 xsk−1/2+x s k 2 + ( 1−βs ) x̃ s−1 ( non-SC ) ; 14 : xs+10 = x s m ; 15 : end for Output : x̃S . reduce the oscillation of descent , thus this method can effectively accelerate the convergence of the algorithm . According to the Nesterov ’ s momentum , lots of accelerated algorithms were proposed , such as APG and Acc-Prox-SVRG . Later , Zeyuan ( 2017 ) proposed Katyusha to further accelerate the algorithm , and MiG ( Kaiwen et al . ( 2018 ) ) was proposed to simplify the structure of Katyusha , and the momentum acceleration of MiG is embodied in each iteration as follows : ysk−1 = βsx s k−1 + ( 1− βs ) x̃s−1 . Moreover , it is easy to get that the oracle complexity of MiG is less than that of Prox-SVRG and APG , which means that MiG can effectively accelerate the original Prox-SVRG algorithm . Therefore , we can also use this acceleration technique to accelerate our algorithm and address the issue of slow convergence due to the calculations of two different gradients .

Summary: The paper proposes an optimization method for solving unconstrained convex optimization problems where the objective function consists of a sum of several smooth components f_i and a (not necessarily smooth) convex function R. The proposed method AVR-SExtraGD is a stochastic descent method building on the previous algorithms Prox-SVRG (Lin 2014) and Katyusha (Zeyuan 2017). The previous Prox-SVRG method using a proximal operator is explained to converge fast but leads to inaccurate final solutions, while the Katyusha method is an algorithm based on  momentum acceleration. The current paper builds on these two approaches and applies the momentum acceleration technique in a stochastic extragradient descent framework to achieve fast convergence.