Stochastically Controlled Compositional Gradient for the Composition problem

We consider composition problems of the form 1n ∑n i=1 Fi ( 1 n ∑n j=1Gj ( x ) ) . Composition optimization arises in many important machine learning applications : reinforcement learning , variance-aware learning , nonlinear embedding , and many others . Both gradient descent and stochastic gradient descent are straightforward solution , but both require to compute 1n ∑n j=1Gj ( x ) in each single iteration , which is inefficient-especially when n is large . Therefore , with the aim of significantly reducing the query complexity of such problems , we designed a stochastically controlled compositional gradient algorithm that incorporates two kinds of variance reduction techniques , and works in both strongly convex and non-convex settings . The strategy is also accompanied by a mini-batch version of the proposed method that improves query complexity with respect to the size of the mini-batch . Comprehensive experiments demonstrate the superiority of the proposed method over existing methods . 1 INTRODUCTION . In this paper , we study the following composition minimization problem , min x∈RN { f ( x ) def = F ( G ( x ) ) def = 1 n n∑ i=1 Fi ( 1 n n∑ j=1 Gj ( x ) ) } , ( 1.1 ) where f : RN → R is differentiable and possibly non-convex , each Fi : RM → R is a smooth function , each Gi : RN → RM is a mapping function , both the numbers of Fi ’ s and Gj ’ s are assumed to be n for simplicity We call G ( x ) : = 1n ∑n j=1Gj ( x ) the inner function , and F ( w ) : = 1 n ∑n i=1 Fi ( w ) the outer function . Many machine learning problems can be cast as composition problems that include two finite-sum structures : reinforcement learning ( Sutton et al. , 1998 ; Wang et al. , 2017 ; Liu et al. , 2016 ) , variance-averse learning ( Lian et al. , 2017 ) , and nonlinear embedding ( Hinton & Roweis , 2003 ; Dikmen et al. , 2015 ) . In particular , • ( reinforcement learning ) The S × S system of Bellman equations Wang et al . ( 2017 ) can be written as minx∈RS ‖E [ B ] x− E [ b ] ‖2 , where E [ B ] = I − γPπ , γ ∈ ( 0 , 1 ) is a discount factor , Pπ is the transition probability under policy π , and E [ b ] is the expected state transition reward . This is one of key problems in reinforcement learning for evaluating the value of a policy π . • ( risk-averse learning ) The risk-averse learning Lian et al . ( 2017 ) aims to maximize the expected return while control the variance ( or risk ) in the meantime : minx − Ea [ h ( x ; a ) ] + λVara [ h ( x ; a ) ] , where h ( x ; a ) is the loss function including a random variable a , λ > 0 is a regularization parameter . • ( nonlinear embedding ) Stochastic nonlinear embedding Hinton & Roweis ( 2003 ) aims to map a group of points from a high dimensional space to a low dimensional space by minimizing the KL divergence . It is a non-convex composition optimization minx ∑ t KL ( p·|t ‖ q·|t ) : = ∑ t ∑ i pi|t log pi|t qi|t , ( 1.2 ) where pi|t and qi|t are the conditional probabilities w.r.p . { zi } ni=1 and { xi } ni=1 , pi|t = d ( zt , zi ) ∑ j 6=t d ( zt , zj ) , qi|t = d ( xt , xi ) ∑ j 6=t d ( xt , xj ) , where d ( · , · ) is the dissimilar distance function between two samples . To solve the composition optimization including the finite-sum structure in ( 1.1 ) , two most straightforward approaches are the gradient descent ( GD ) and the stochastic gradient descent ( SGD ) . However , it is extremely expensive to scan all the inner functions ( for both SGD and GD ) as well as all the outer functions ( for GD ) in each iteration . However , note that , unlike solving common stochastic optimization problems , randomly sampling one inner function and one outer function does not give an unbiased estimate for the true gradient ; that is , Ei∼ [ n ] , j∼ [ n ] [ ( ∂Gj ( x ) ) T∂Fi ( G̃ ( x ) ) ] 6= ∇f ( x ) , where G̃ ( x ) is the estimation of G ( x ) . The key to solving this composition objective is how to estimate the value of G ( xk ) and its Jacobian with high accuracy using only a few samples in each iteration . Recently , many stochastic optimization methods solving the composition problem have been developed , such as the stochastic gradient based method and the variance-reduction based method . For example , stochastic compositional gradient descent ( SCGD ) ( Wang et al. , 2017 ; Liu et al. , 2016 ) estimates the inner function G ( x ) by using an iterative weighted average of the past values of G ( x ) , and then performs the stochastic quasi-gradient iteration . The advantage of this method is that convergence rate does not depend on n ; however , it queries more samples to the desired point . Another set of approaches is based on variance reduction – for instance , compositional stochastic variance reduction gradient ( Compositional-SVRG ) ( Lian et al. , 2017 ) estimates the inner function G ( x ) and the gradient of function f ( x ) by using the variance reduction technique ; however , the derived linear convergence rate is related to n. Motivated by a few recent works ( Lei & Jordan , 2017 ; Lei et al. , 2017 ; Allen-Zhu , 2017 ) that focus on the stochastically controlled gradient , we were inspired to look for a way to improve the query complexity and reduce the dependence on n to solve the composition optimization in ( 1.1 ) . Hence , this paper presents a novel and more efficient method named stochastically controlled compositional gradient ( SCCG ) for solving composition problems involving a two-finite-sum structure . The result is improved query complexity over existing approaches . Further , all results in this paper can be easily extended to cases where the number of Fi and the number of Gj are different . The main contributions of this article are summarized below . • We provide a stochastically controlled function to estimate the inner function G ( x ) . Inspired by stochastically controlled stochastic gradient ( SCSG ) ( Lei & Jordan , 2017 ) that estimates the gradient , G ( x ) can also be estimated by using a snapshot x̃s , in which G ( x̃s ) is not computed directly , but is estimated through a random subset from [ n ] . This is the first time that a stochastically controlled function has been incorporated into the process of estimating the inner function . We have also analyzed how the size of the subset might influence the query complexity for both strongly convex and non-convex functions . • We provide a stochastically controlled compositional gradient to estimate the∇f ( x ) . However , there are two potential situations that could be encountered in the estimation process that can impede convergence . First , the expectation of the gradient is no longer an unbiased estimation ; and , second , the gradient of f ( x̃s ) at the snapshot is formed by two random subsets , which are used for the functions Fi and Gj respectively . Moreover , the biased gradient bring more difficulty in proving the convergence , which are greatly different from those encountered in ( Lei & Jordan , 2017 ; Lian et al. , 2017 ; Lei et al. , 2017 ) . To address these scenarios , we have identified a bound on the size of the subsets that are used to estimate the gradient . The details of the analysis can be referred to Section 3.1 and 3.2 . • A mini-batch version of the proposed algorithm is also provided for both strongly convex and non-convex functions . The corresponding query complexities are improved according to the size of the mini-batch . More information can be referred to Section 3.3 . 1.1 RESULTS . Following the classical benchmark for a general problem , the composition algorithm is also to find a point x satisfying f ( x ) − f ( x∗ ) ≤ for a convex function , where x∗ is the optimal point in the strongly convex function , and ‖∇f ( x ) ‖2 ≤ for a non-convex function , respectively . The elegance of a composition algorithm is evaluated based on its query complexity , defined as the number of queries in a given sampling oracle that are needed to compute the gradient . Here , we give the query complexities of the composition problem in Table 1 , which offers an insightful comparison to other algorithms . Strongly convex function The query complexity for the strongly convex function is O ( ( min { n , 1/ ( µ2 ) } + L2f/µ2min { n , 1/µ2 } ) log ( 1/ ) ) . The result is the general form for the strongly convex composition and is equal to or better than the query complexity in ( Lian et al. , 2017 ) and ( Liu et al. , 2017a ) . Non-convex function The query complexity is O ( min { 1/ 9/5 , n4/5/ } ) , which is better than the result in ( Liu et al. , 2016 ) and comparable to the result in ( Liu et al. , 2017b ) . Mini-batch 1 For the mini-batch version , the query complexity can be improved to some extent , that is O ( ( min { n , 1/ ( µ2 ) } + L2f/ ( bµ2 ) min { n , 1/µ2 } ) log ( 1/ ) ) . and O ( min { 1/ 9/5 , n4/5/ } /b1/5 ) for strongly convex and non-convex functions , respectively , which are better than mini-batch variance reduced stochastic compositional proximal gradient method ( VRSC-PG ) ( Huo et al. , 2018 ) and mini-batch Composite SAGA ( C-SAGA ) ( Yuan et al. , 2019 ) when b=1/µ and min { n , 1/ } 2/3 for strongly convex and non-convex functions , respectively . 1.2 RELATED WORK . As the amount of data we have at our disposal grows , stochastic optimization has become a popular technique in the realm of machine and deep learning , particularly for optimizing finite-sum functions . The typical algorithms for solving such problems include stochastic gradient descent ( Ghadimi & Lan , 2016 ) , SVRG ( Johnson & Zhang , 2013 ; Reddi et al. , 2016 ) , stochastic dual coordinate ascent ( SDCA ) ( Shalev-Shwartz & Zhang , 2014 ; 2013 ) and the accelerated Nesterov ’ s method ( Nesterov , 2013 ) , accelerated randomized proximal coordinate ( APCG ) ( Lin et al. , 2014 ; 2015 ) and Katyusha method ( Allen-Zhu , 2017 ) . The standard procedure for optimizing a problem with a finite-sum structure is to randomly select one or a block of components to estimate the gradient . However , knowing that the estimated gradient usually has a large variance , the gradient of the function is estimated from a snapshot to appropriately reduce the variance – in other words , the procedure includes a variance reduction mechanism . Composition optimization problems can also be solved with the above algorithms , but the twofinite-sum structures in composition problems mean that when the gradient of the inner function is estimated directly , the query complexity can substantially increase . Recently , Wang et al . ( 2017 ) proposed a method based on first-order SCGD to overcome this issue where the variable and the inner function are updated alternately in two steps . The method has a query complexity ofO ( −7/2 ) for a general function andO ( −5/4 ) for a strongly convex function . Liu et al . ( 2016 ) employed Nes- 1b denotes the size of the mini-batch , can be obtained through the η ≤ 1 from Theorem 1 and Theorem 2. terov ’ s method to accelerate the composition problem , reachingO ( −5/4 ) andO ( −9/4 ) for strongly convex and non-convex functions , respectively . Ghadimi et al . ( 2018 ) proposed a nested averaged stochastic approximation method to find an approximate stationary point within the problem , resulting in a sample complexity of O ( 1/ 2 ) . However , these methods estimate the inner function using an iterative weighted average of the past function . The other stream of solutions focuses on variance reduction technology . For instance , Lian et al . ( 2017 ) initially applied the SVRG-based method to estimate the inner function G ( x ) and the gradient of the function f ( x ) , which yields a linear convergence rate . Subsequently , Liu et al . ( 2017a ) and Devraj & Chen ( 2019 ) applied a dual-based method to composition problem , which also yields a linear convergence rate . Devraj & Chen ( 2019 ) also applied the stochastic variance reduced primal-dual algorithms to composition problem . Yu & Huang ( 2017 ) turned to an ADMM-based Boyd ( 2011 ) method and provided an analysis of convex functions that do not rely on Lipschitz smoothness . Moreover , Liu et al . ( 2017b ) went a step further and considered non-convex functions , analyzing the query complexity with both inner and outer functions of different sizes . Lin et al . ( 2018 ) considered non-smooth convex composition functions , offering an incremental first-order oracle complexity analysis . Zhang & Xiao ( 2019 ) and Huo et al . ( 2018 ) also provided an randomized incremental gradient method for the composition problem including regularization . Many recent articles have discussed variance reduction methods that estimate the gradient from a random subset rather than through direct computation . Lei & Jordan ( 2017 ) , for example , proposed an SCSG method for a convex finite-sum function . They then applied it to a non-convex problem in ( Lei et al. , 2017 ) by using less than a single pass to compute the gradient at the snapshot point . Furthermore , Allen-Zhu ( 2017 ) proposed the Natasha1.5 algorithm , in which the gradient for each epoch is based on a random subset . Moreover , the objective function has the regularization term . Liu et al . ( 2018 ) applied an SCSG based method to the zeroth-order optimization problems with the finite-sum function . Recently , Yuan et al . ( 2019 ) applied the stochastic recursive gradient descent method to the composition problem . The rest of paper is organized as follows : in Section 2 , we give preliminaries used for analyzing the proposed algorithm . Section 3 presents the SCSG-based method for the strongly convex and non-convex composition problem and the corresponding mini-batch version . In Section 4 , we give the experimental results . We conclude our paper in Section 5 .

Summary: This paper proposes a new method for empirical composition problems to which the vanilla SGD is not applicable because it has a finite-sum structure inside non-linear loss functions. A proposed method (named SCCG) is a combination of stochastic compositional gradient descent (SCGD) and stochastically controlled stochastic gradient (SCSG). In a theoretical analysis part, a linear convergence rate and a sub-linear convergence rate are derived under the strong convex and non-convex settings, respectively. In experiments, the superior performance of the method to competitors is verified on both strongly convex and non-convex problems.