Accelerated Variance Reduced Stochastic Extragradient Method for Sparse Machine Learning Problems

1 INTRODUCTION . In this paper , we mainly consider the following composite convex optimization problem : min x∈Rd { P ( x ) def = F ( x ) +R ( x ) = 1 n n∑ i=1 fi ( x ) +R ( x ) } ( 1 ) where F ( x ) : Rd→R is the average of smooth convex component functions fi ( x ) , and R ( x ) is a relatively simple convex function ( but may not be differentiable ) . In this paper , we use ‖·‖ to denote the standard Euclidean norm , and ‖·‖1 to denote the ` 1-norm . Moreover , we use P∗ to denote the real optimal value of P ( · ) , and P̂∗ to denote the optimal value obtained by algorithms . This form of optimization problems often appears in machine learning , signal processing , data science , statistics and operations research , and has a wide range of applications such as regularized empirical risk minimization ( ERM ) , sparse coding for image and video recovery , and representation learning for object recognition . Specifically , for a collection of given training examples { ( a1 , b1 ) , ... , ( an , bn ) } , where ai ∈ Rd , bi ∈ R ( i = 1 , 2 , ... , n ) and ai is a feature vector , while bi is the desired response . When fi ( x ) = 12 ( a T i x−bi ) 2 , we can obtain the ridge regression problem by setting R ( x ) = λ2 ‖x‖ 2 . We also get the Lasso or Elastic-Net problems by setting R ( x ) =λ‖x‖1 or R ( x ) = λ22 ‖x‖ 2+λ1‖x‖1 , respectively . Moreover , if we set fi ( x ) = log ( 1+exp ( −bixTai ) ) , we will get the regularized logistic regression problem . 1.1 RECENT RESEARCH PROGRESS . The proximal gradient descent ( PGD ) method is a standard and effective method for Problem ( 1 ) , and can achieve linear convergence for strongly convex problems . Its accelerated algorithms , e.g. , accelerated proximal gradient ( APG ) ( Tseng ( 2008 ) ; Beck & Teboulle ( 2009 ) ) , attain the convergence rate ofO ( 1/T 2 ) for non-strongly convex problems , where T denotes the number of iterations . In recent years , stochastic gradient descent ( SGD ) has been successfully applied to many large-scale learning problems , such as training for deep networks and linear prediction ( Tong ( 2004 ) ) , because of its significantly lower per-iteration complexity than deterministic methods , i.e. , O ( d ) vs. O ( nd ) . Besides , many tricks for SGD have also been proposed , such as Loshchilov & Hutter ( 2016 ) . However , the variance of the stochastic gradient may be large due to random sampling ( Johnson & Tong ( 2013 ) ) , which leads that the algorithm requires a gradually reduced step size , thus it will converge slow . Even under the strongly convex condition , SGD only achieves a sub-linear convergence rate O ( 1/T ) . Recently , many SGD methods with variance reduction have been proposed . For the case of R ( x ) = 0 , Roux et al . ( 2012 ) developed a stochastic average gradient descent ( SAG ) method , which is a randomized variant of the incremental aggregated gradient method proposed by Blatt et al . ( 2007 ) . Then stochastic variance reduced gradient ( SVRG ) ( Johnson & Tong ( 2013 ) ) was proposed , and has been widely introduced into various subsequent optimization algorithms , due to its lower storage space ( i.e. , O ( d ) ) than that of SAG ( i.e. , O ( nd ) ) . SVRG reduced the variance effectively by changing the estimation of stochastic gradients . The introduction of a snapshot point x̃ mainly has the effect of correcting the direction of gradient descent , and reduces the variance . Later , Konečný & Richtárik ( 2013 ) proposed the semi-stochastic gradient descent methods as well as their mini-batch version ( Konečný et al . ( 2014 ) ) . And their asynchronous distributed variant ( Ruiliang et al . ( 2016 ) ) is also been proposed later . More recently , Lin & Tong ( 2014 ) proposed the Prox-SVRG method , which introduced the proximal operator , and then applied the idea of SVRG to solve the non-smooth optimization problems . However , Prox-SVRG can only be used to solve the strongly convex optimization problems . In order to solve the non-strongly convex problems , Zeyuan & Yuan ( 2016 ) proposed the SVRG++ algorithm . Besides , to accelerate the algorithm and reducing the complexity , by combining the main ideas of APG and Prox-SVRG , Nitanda ( 2014 ) proposed an accelerated variance reduction proximal stochastic gradient descent ( Acc-Prox-SVRG ) method , which can effectively reduce the complexity of the algorithm compared to the two basic algorithms . Very recently , Zeyuan ( 2017 ) developed a novel Katyusha algorithm which introduced the Katyusha momentum to accelerate the algorithm . With the development of parallel and distributed computing which can effectively reduce computing time and improve performance , Ryu & Wotao ( 2017 ) came up with an algorithm called Proximal Proximal Gradient , which combined the proximal gradient method and ADMM ( Gabay & Mercier ( 1976 ) ) . Furthermore , it is easy to implement in parallel and distributed environments because of its innovative algorithm structure . 1.2 OUR MAIN CONTRIBUTIONS . We find that due to the introduction of proximal operator , there is a gap between P̂∗ and P∗ , and its theoretical derivation can be seen in Appendix A . To address this issue , Nguyen et al . ( 2017 ) proposed the idea of extragradient which can be seen as a guide during the process , and introduced it into the optimization problems . Intuitively , this additional iteration allows us to examine the geometry of the problem and consider its curvature information , which is one of the most important bottlenecks for first order methods . By using the idea of extragradient , we can get a better result in each inner-iteration . Therefore , the idea of extragradient is our main motivation . In this paper , we propose a novel algorithm for solving non-smooth optimization problems . The main contributions of this paper are summarized as follows . • In order to improve the result of the gap between P̂∗ and P∗ , and achieve fast convergence , a novel algorithm , which combines the idea of extragradient , Prox-SVRG and the trick of momentum acceleration , is proposed , called accelerated variance reduced stochastic extragradient descent ( AVR-SExtraGD ) . •We provide the convergence analysis of our algorithm , which shows that AVR-SExtraGD achieves linear convergence for strongly convex problems , and the convergence condition in the non-strongly convex case is also given . According to the convergence rate , we can know that AVR-SExtraGD has the same excellent result as the best-known algorithms , such as Katyusha . • Finally , we show by experiments that the performance of AVR-SExtraGD ( as well as VRSExtraGD , which is the basic algorithm of AVR-SExtraGD ) is obviously better than the popular algorithm , Prox-SVRG , which confirms the advantage of extragradient . For the widely used accelerated algorithm , Katyusha , the performance of our algorithm is still improved . 2 RELATED WORK . 2.1 BASIC ASSUMPTIONS . We first make the following assumptions to solve the problem ( 1 ) : Assumption 1 ( Smoothness ) . The convex function F ( · ) is L-smooth , i.e. , there exists a constant L > 0 such that for any x , y∈Rd , ‖∇F ( x ) −∇F ( y ) ‖≤L‖x−y‖ . Assumption 2 ( Lower Semi-continuity ) . The regularization function R ( · ) is a lower semicontinuous function , i.e. , ∀x0∈Rd , lim inf x→x0 R ( x ) ≥R ( x0 ) . But it is not necessarily differentiable or continuous .. Assumption 3 ( Strong Convexity ) . In Problem ( 1 ) , the functionR ( · ) is µ-strongly convex , i.e. , there exists a constant µ > 0 such that for all x , y∈Rd , it holds that R ( x ) ≥ R ( y ) + 〈G , x− y〉+ µ 2 ‖x− y‖2 , ( 2 ) where G∈∂R ( y ) which is the set of sub-gradient of R ( · ) at y . 2.2 PROX-SVRG AND EXTRAGRADIENT DESCENT METHODS . An effective method for solving Problem ( 1 ) is Prox-SVRG which improved Prox-FG ( Lions & Mercier ( 1979 ) ) and Prox-SG ( Langford et al . ( 2009 ) ) by introducing the stochastic gradient and combining the idea of SVRG , respectively . For strongly convex problems , Prox-SVRG can reach linear convergence with a constant step size , and its main update rules are ∇̃fik ( xk−1 ) = ∇fik ( xk−1 ) −∇fik ( x̃ ) +∇F ( x̃ ) ; xk = ProxRη ( xk−1 − η∇̃fik ( xk−1 ) ) , ( 3 ) where x̃ is the snapshot point used in SVRG , ∇̃fik ( xk−1 ) is the variance reduced stochastic gradient estimator , and ProxRη ( · ) is the proximal operator . Although Prox-SVRG can converge fast , because of proximal operator , the final solution has the deviation , which makes the solution inaccurate , thus Prox-SVRG still needs to be further improved , which is our important motivation . The extragradient method was first proposed by Korpelevič ( 1976 ) . It is a classical method for solving variational inequality problems , and it generates an estimation sequence by using two projection gradients in each iteration . By combining this idea with some first-order descent methods , Nguyen et al . ( 2017 ) proposed an extended extragradient method ( EEG ) which can effectively solve the problem ( 1 ) , and can also solve relatively more general problems as follows : min x∈Rd { P ( x ) def = F ( x ) +R ( x ) } where F ( x ) is not necessarily composed by multiple functions fi ( x ) . Unlike the classical extragradient method , EEG uses proximal gradient instead of orthogonal projection in each iteration . The main update rules of EEG are yk = ProxRsk ( xk − sk∇F ( xk ) ) ; xk+1 = Prox R αk ( xk − αk∇F ( yk ) ) , where sk and αk are two step sizes . From the update rules of EEG , we can see that in each iteration , EEG needs to calculate two gradients , which will definitely slow down the algorithm . Therefore , the algorithm needs to be further accelerated by an efficient technique . 2.3 MOMENTUM ACCELERATION AND MIG . Firstly , we introduce the momentum acceleration technique whose main update rules are vdwt = βvdwt−1 + ( 1− β ) dwt ; wt = wt−1 − αvdwt , where dw is the gradient of the objective function at w , β is a parameter , and α is a step size . The update rules take not only the gradient of the current position , but also the gradient of the past position into account , which makes the final descent direction of wt after using momentum Algorithm 1 AVR-SExtraGD Input : Initial vector x0 , the number of epochs S , the number of iterations m per epoch , the step sizes η1 , η2 , momentum parameter β , and the set K. Initialize : x̃0 = x10 = x0 , ρ = 1+ηµ . 1 : for s = 1 , 2 , . . . , S do 2 : Compute∇F ( x̃s−1 ) ; 3 : βs = β ( SC ) or βs = 2s+4 ( non-SC ) ; 4 : for k = 1 , 2 , . . . , m do 5 : Pick ik uniformly at random from { 1 , ... , n } ; 6 : if k ∈ K then 7 : xsk−1/2 = Prox R η1 ( xsk−1 − η1∇̃fik ( βsxsk−1 + ( 1−βs ) x̃s−1 ) ) ; 8 : xsk = Prox R η2 ( xsk−1/2 − η2∇̃fik ( βsx s k−1/2 + ( 1−βs ) x̃ s−1 ) ) ; 9 : else 10 : xsk = Prox R η1 ( x s k−1 − η1∇̃fik ( βsxsk−1 + ( 1−βs ) x̃s−1 ) ) ; 11 : end if 12 : end for 13 : x̃s = βs ( ∑m k=1 ρ k−1 ) −1 ∑m k=1 ρ k−1 x s k−1/2+x s k 2 + ( 1−βs ) x̃ s−1 ( SC ) or x̃s = βsm ∑m k=1 xsk−1/2+x s k 2 + ( 1−βs ) x̃ s−1 ( non-SC ) ; 14 : xs+10 = x s m ; 15 : end for Output : x̃S . reduce the oscillation of descent , thus this method can effectively accelerate the convergence of the algorithm . According to the Nesterov ’ s momentum , lots of accelerated algorithms were proposed , such as APG and Acc-Prox-SVRG . Later , Zeyuan ( 2017 ) proposed Katyusha to further accelerate the algorithm , and MiG ( Kaiwen et al . ( 2018 ) ) was proposed to simplify the structure of Katyusha , and the momentum acceleration of MiG is embodied in each iteration as follows : ysk−1 = βsx s k−1 + ( 1− βs ) x̃s−1 . Moreover , it is easy to get that the oracle complexity of MiG is less than that of Prox-SVRG and APG , which means that MiG can effectively accelerate the original Prox-SVRG algorithm . Therefore , we can also use this acceleration technique to accelerate our algorithm and address the issue of slow convergence due to the calculations of two different gradients .

Summary: This is an optimization algorithm paper, using the idea of "extragradient" and proposing to combine acceleration with proximal gradient descent-type algorithms (Prox-SVRG). Their proposed algorithm, i.e., accelerated variance reduced stochastic extra gradient descent, combines the advantages of Prox-SVRG and momentum acceleration techniques. The authors prove the convergence rate and oracle complexity of their algorithm for strongly convex and non-strongly convex problems. Their experiments on face recognition show improvement on top of Prox-SVRG as well Katyusha. They also propose an asynchronous variant of their algorithm and show that it outperforms other asynchronous baselines.