Greedy-GQ with Variance Reduction: Finite-time Analysis and Improved Complexity

1 INTRODUCTION . In reinforcement learning ( RL ) , an agent interacts with a stochastic environment following a certain policy and receives some reward , and it aims to learn an optimal policy that yields the maximum accumulated reward Sutton & Barto ( 2018 ) . In particular , many RL algorithms have been developed to learn the optimal control policy , and they have been widely applied to various practical applications such as finance , robotics , computer games and recommendation systems Mnih et al . ( 2015 ; 2016 ) ; Silver et al . ( 2016 ) ; Kober et al . ( 2013 ) . Conventional RL algorithms such as Q-learning Watkins & Dayan ( 1992 ) and SARSA Rummery & Niranjan ( 1994 ) have been well studied and their convergence is guaranteed in the tabular setting . However , it is known that these algorithms may diverge in the popular off-policy setting under linear function approximation Baird ( 1995 ) ; Gordon ( 1996 ) . To address this issue , the two time-scale Greedy-GQ algorithm was developed in Maei et al . ( 2010 ) for learning the optimal policy . This algorithm extends the efficient gradient temporal difference ( GTD ) algorithms for policy evaluation Sutton et al . ( 2009b ) to policy optimization . In particular , the asymptotic convergence of GreedyGQ to a stationary point has been established in Maei et al . ( 2010 ) . More recently , Wang & Zou ( 2020 ) studied the finite-time convergence of Greedy-GQ under linear function approximation and Markovian sampling , and it is shown that the algorithm achieves an -stationary point of the objective function with a sample complexity in the order of O ( −3 ) . Such an undesirable high sample complexity is caused by the large variance induced by the Markovian samples queried from the dynamic environment . Therefore , we want to ask the following question . • Q1 : Can we develop a variance reduction scheme for the two time-scale Greedy-GQ algorithm ? In fact , in the existing literature , many recent work proposed to apply the variance reduction techniques developed in the stochastic optimization literature to reduce the variance of various TD learning algorithms for policy evaluation , e.g. , Du et al . ( 2017 ) ; Peng et al . ( 2019 ) ; Korda & La ( 2015 ) ; Xu et al . ( 2020 ) . Some other work applied variance reduction techniques to Q-learning algorithms , e.g. , Wainwright ( 2019 ) ; Jia et al . ( 2020 ) . Hence , it is much desired to develop a variance-reduced Greedy-GQ algorithm for optimal control . In particular , as many of the existing variance-reduced RL algorithms have been shown to achieve an improved sample complexity under variance reduction , it is natural to ask the following fundamental question . • Q2 : Can variance-reduced Greedy-GQ achieve an improved sample complexity under Markovian sampling ? In this paper , we provide affirmative answers to these fundamental questions . Specifically , we develop a two time-scale variance reduction scheme for the Greedy-GQ algorithm by leveraging the SVRG scheme Johnson & Zhang ( 2013 ) . Moreover , under linear function approximation and Markovian sampling , we prove that the proposed variance-reduced Greedy-GQ algorithm achieves an -stationary point with an improved sample complexity O ( −2 ) . We summarize our technical contributions as follows . 1.1 OUR CONTRIBUTIONS . We develop a variance-reduced Greedy-GQ ( VR-Greedy-GQ ) algorithm for optimal control in reinforcement learning . Specifically , the algorithm leverages the SVRG variance reduction scheme Johnson & Zhang ( 2013 ) to construct variance-reduced stochastic updates for updating the parameters in both time-scales . We study the finite-time convergence of VR-Greedy-GQ under linear function approximation and Markovian sampling in the off-policy setting . Specifically , we show that VR-Greedy-GQ achieves an -stationary point of the objective function J ( i.e. , ‖∇J ( θ ) ‖2 ≤ ) with a sample complexity in the order of O ( −2 ) . Such a complexity result improves that of the original Greedy-GQ by a significant factor of O ( −1 ) Wang & Zou ( 2020 ) . In particular , our analysis shows that the bias error caused by the Markovian sampling and the variance error of the stochastic updates are in the order ofO ( M−1 ) , O ( ηθM−1 ) , respectively , where ηθ is the learning rate and M corresponds to the batch size of the SVRG reference batch update . This shows that the proposed variance reduction scheme can significantly reduce the bias and variance errors of the original Greedy-GQ update ( by a factor of M ) and lead to an improved overall sample complexity . The analysis logic of VR-Greedy-GQ partly follows that of the conventional SVRG , but requires substantial new technical developments . Specifically , we must address the following challenges . First , VR-Greedy-GQ involves two time-scale variance-reduced updates that are correlated with each other . Such an extension of the SVRG scheme to the two time-scale updates is novel and requires new technical developments . Specifically , we need to develop tight variance bounds for the two time-scale updates under Markovian sampling . Second , unlike the convex objective functions of the conventional GTD type of algorithms , the objective function of VR-Greedy-GQ is generally non-convex due to the non-stationary target policy . Hence , we need to develop new techniques to characterize the per-iteration optimization progress towards a stationary point under nonconvexity . In particular , to analyze the two time-scale variance reduction updates of the algorithm , we introduce a ‘ fine-tuned ’ Lyapunov function of the form Rmt = J ( θ ( m ) t ) + ct‖θ ( m ) t − θ̃ ( m ) ‖2 , where the parameter ct is fine-tuned to cancel other additional quadratic terms ‖θ ( m ) t − θ̃ ( m ) ‖2 that are implicitly involved in the tracking error terms . The design of this special Lyapunov function is critical to establish the formal convergence of the algorithm . With these technical developments , we are able to establish an improved finite-time convergence rate and sample complexity for VR-Greedy-GQ . 1.2 RELATED WORK . Q-learning and SARSA with function approximation . The asymptotic convergence of Q-learning and SARSA under linear function approximation were established in Melo et al . ( 2008 ) ; Perkins & Precup ( 2003 ) , and their finite-time analysis were developed in Zou et al . ( 2019 ) ; Chen et al . ( 2019 ) . However , these algorithms may diverge in off-policy training Baird ( 1995 ) . Also , recent works focused on the Markovian setting . Various analysis techniques have been developed to analyze the finite-time convergence of TD/Q-learning under Markovian samples . Specifically , Wang et al . ( 2020 ) developed a multi-step Lyapunov analysis for addressing the biasedness of the stochastic approximation in Q-learning . Srikant & Ying ( 2019 ) developed a drift analysis to the linear stochastic approximation problem . Besides the linear function approximation , the finite-time analysis of Q-learning under neural network function approximation is developed in Xu & Gu ( 2019 ) . GTD algorithms . The GTD2 and TDC algorithms were developed for off-policy TD learning . Their asymptotic convergence was proved in Sutton et al . ( 2009a ; b ) ; Yu ( 2017 ) , and their finite-time analysis were developed recently in Dalal et al . ( 2018 ) ; Wang et al . ( 2017 ) ; Liu et al . ( 2015 ) ; Gupta et al . ( 2019 ) ; Xu et al . ( 2019 ) . The Greedy-GQ algorithm is an extension of these algorithms to optimal control and involves nonlinear updates . RL with variance reduction : Variance reduction techniques have been applied to various RL algorithms . In TD learning , Du et al . ( 2017 ) reformulate the MSPBE problem as a convex-concave saddle-point optimization problem and applied SVRG Johnson & Zhang ( 2013 ) and SAGA Defazio et al . ( 2014 ) to primal-dual batch gradient algorithm . In Korda & La ( 2015 ) , the variance-reduced TD algorithm was introduced for solving the MSPBE problem , and later Xu et al . ( 2020 ) provided a correct non-asymptotic analysis for this algorithm over Markovian samples . Recently , some other works applied the SVRG , SARAH Nguyen et al . ( 2017 ) and SPIDER Fang et al . ( 2018 ) variance reduction techniques to develop variance-reduced Q-learning algorithms , e.g. , Wainwright ( 2019 ) ; Jia et al . ( 2020 ) . In these works , TD or TDC algorithms are in the form of linear stochastic approximation , and Q-learning has only a single time-scale update . As a comparison , our VR-Greedy-GQ takes nonlinear two time-scale updates to optimization a nonconvex MSPBE . 2 PRELIMINARIES : POLICY OPTIMIZATION AND GREEDY-GQ . In this section , we review some preliminaries of reinforcement learning and recap the Greedy-GQ algorithm under linear function approximation . 2.1 POLICY OPTIMIZATION IN REINFORCEMENT LEARNING . In reinforcement learning , an agent takes actions to interact with the environment via a Markov Decision Process ( MDP ) . Specifically , an MDP is specified by the tuple ( S , A , P , r , γ ) , where S and A respectively correspond to the state and action spaces that include finite elements , r : S×A×S → [ 0 , +∞ ) denotes a reward function and γ ∈ ( 0 , 1 ) is the associated reward discount factor . At any time t , assume that the agent is in the state st ∈ S and takes a certain action at ∈ A following a stationary policy π , i.e. , at ∼ π ( ·|st ) . Then , at the subsequent time t + 1 , the current state of the agent transfers to a new state st+1 according to the transition kernel P ( ·|st , at ) . At the same time , the agent receives a reward rt = r ( st , at , st+1 ) from the environment for this action-state transition . To evaluate the quality of a given policy π , we often use the action-state value function Qπ : S ×A → R that accumulates the discounted rewards as follows : Qπ ( s , a ) = Es′∼P ( ·|s , a ) [ r ( s , a , s′ ) + γV π ( s′ ) ] , where V π ( s ) is the state value function defined as V π ( s ) = E [ ∑∞ t=0 γ trt|s0 = s ] . In particu- lar , define the Bellman operator Tπ such that TπQ ( s , a ) = Es′ , a′ [ r ( s , a , s′ ) + γQ ( s′ , a′ ) ] for any Q ( s , a ) , where a′ ∼ π ( ·|s′ ) . Then , Qπ ( s , a ) is a fixed point of Tπ , i.e. , TπQπ ( s , a ) = Qπ ( s , a ) , ∀s , a . ( 1 ) The goal of policy optimization is to learn the optimal policy π∗ that maximizes the expected total reward E [ ∑∞ t=0 γ trt|s0 = s ] for any initial state s ∈ S , and this is equivalent to learn the optimal value function Q∗ ( s , a ) = supπ Q π ( s , a ) , ∀s , a . In particular , Q∗ is a fixed point of the Bellman operator T that is defined as TQ ( s , a ) = Es′∼P ( ·|s , a ) [ r ( s , a , s′ ) + γmaxb∈AQ ( s′ , b ) ] .

Summary: This paper combines a widely used variance reduction technique SVRG with the greedy-GQ. It provides a finite-time analysis of the proposed algorithm  in the off-policy and Markovian sampling setting (convergence to the stationary point) and improves the sample complexity from the order $\epsilon^{-3}$ to $\epsilon^{-2}$ comparing with the vanilla greedy GQ. Interestingly, the analysis shows that the biase error caused by the Markovian sampling and the variance error of the stochastic gradient are reduced by the $M$, where M is the batch size of the batch gradient in SVRG. At last, it verifies the theoretical claim by two toy examples.