Greedy-GQ with Variance Reduction: Finite-time Analysis and Improved Complexity

1 INTRODUCTION . In reinforcement learning ( RL ) , an agent interacts with a stochastic environment following a certain policy and receives some reward , and it aims to learn an optimal policy that yields the maximum accumulated reward Sutton & Barto ( 2018 ) . In particular , many RL algorithms have been developed to learn the optimal control policy , and they have been widely applied to various practical applications such as finance , robotics , computer games and recommendation systems Mnih et al . ( 2015 ; 2016 ) ; Silver et al . ( 2016 ) ; Kober et al . ( 2013 ) . Conventional RL algorithms such as Q-learning Watkins & Dayan ( 1992 ) and SARSA Rummery & Niranjan ( 1994 ) have been well studied and their convergence is guaranteed in the tabular setting . However , it is known that these algorithms may diverge in the popular off-policy setting under linear function approximation Baird ( 1995 ) ; Gordon ( 1996 ) . To address this issue , the two time-scale Greedy-GQ algorithm was developed in Maei et al . ( 2010 ) for learning the optimal policy . This algorithm extends the efficient gradient temporal difference ( GTD ) algorithms for policy evaluation Sutton et al . ( 2009b ) to policy optimization . In particular , the asymptotic convergence of GreedyGQ to a stationary point has been established in Maei et al . ( 2010 ) . More recently , Wang & Zou ( 2020 ) studied the finite-time convergence of Greedy-GQ under linear function approximation and Markovian sampling , and it is shown that the algorithm achieves an -stationary point of the objective function with a sample complexity in the order of O ( −3 ) . Such an undesirable high sample complexity is caused by the large variance induced by the Markovian samples queried from the dynamic environment . Therefore , we want to ask the following question . • Q1 : Can we develop a variance reduction scheme for the two time-scale Greedy-GQ algorithm ? In fact , in the existing literature , many recent work proposed to apply the variance reduction techniques developed in the stochastic optimization literature to reduce the variance of various TD learning algorithms for policy evaluation , e.g. , Du et al . ( 2017 ) ; Peng et al . ( 2019 ) ; Korda & La ( 2015 ) ; Xu et al . ( 2020 ) . Some other work applied variance reduction techniques to Q-learning algorithms , e.g. , Wainwright ( 2019 ) ; Jia et al . ( 2020 ) . Hence , it is much desired to develop a variance-reduced Greedy-GQ algorithm for optimal control . In particular , as many of the existing variance-reduced RL algorithms have been shown to achieve an improved sample complexity under variance reduction , it is natural to ask the following fundamental question . • Q2 : Can variance-reduced Greedy-GQ achieve an improved sample complexity under Markovian sampling ? In this paper , we provide affirmative answers to these fundamental questions . Specifically , we develop a two time-scale variance reduction scheme for the Greedy-GQ algorithm by leveraging the SVRG scheme Johnson & Zhang ( 2013 ) . Moreover , under linear function approximation and Markovian sampling , we prove that the proposed variance-reduced Greedy-GQ algorithm achieves an -stationary point with an improved sample complexity O ( −2 ) . We summarize our technical contributions as follows . 1.1 OUR CONTRIBUTIONS . We develop a variance-reduced Greedy-GQ ( VR-Greedy-GQ ) algorithm for optimal control in reinforcement learning . Specifically , the algorithm leverages the SVRG variance reduction scheme Johnson & Zhang ( 2013 ) to construct variance-reduced stochastic updates for updating the parameters in both time-scales . We study the finite-time convergence of VR-Greedy-GQ under linear function approximation and Markovian sampling in the off-policy setting . Specifically , we show that VR-Greedy-GQ achieves an -stationary point of the objective function J ( i.e. , ‖∇J ( θ ) ‖2 ≤ ) with a sample complexity in the order of O ( −2 ) . Such a complexity result improves that of the original Greedy-GQ by a significant factor of O ( −1 ) Wang & Zou ( 2020 ) . In particular , our analysis shows that the bias error caused by the Markovian sampling and the variance error of the stochastic updates are in the order ofO ( M−1 ) , O ( ηθM−1 ) , respectively , where ηθ is the learning rate and M corresponds to the batch size of the SVRG reference batch update . This shows that the proposed variance reduction scheme can significantly reduce the bias and variance errors of the original Greedy-GQ update ( by a factor of M ) and lead to an improved overall sample complexity . The analysis logic of VR-Greedy-GQ partly follows that of the conventional SVRG , but requires substantial new technical developments . Specifically , we must address the following challenges . First , VR-Greedy-GQ involves two time-scale variance-reduced updates that are correlated with each other . Such an extension of the SVRG scheme to the two time-scale updates is novel and requires new technical developments . Specifically , we need to develop tight variance bounds for the two time-scale updates under Markovian sampling . Second , unlike the convex objective functions of the conventional GTD type of algorithms , the objective function of VR-Greedy-GQ is generally non-convex due to the non-stationary target policy . Hence , we need to develop new techniques to characterize the per-iteration optimization progress towards a stationary point under nonconvexity . In particular , to analyze the two time-scale variance reduction updates of the algorithm , we introduce a ‘ fine-tuned ’ Lyapunov function of the form Rmt = J ( θ ( m ) t ) + ct‖θ ( m ) t − θ̃ ( m ) ‖2 , where the parameter ct is fine-tuned to cancel other additional quadratic terms ‖θ ( m ) t − θ̃ ( m ) ‖2 that are implicitly involved in the tracking error terms . The design of this special Lyapunov function is critical to establish the formal convergence of the algorithm . With these technical developments , we are able to establish an improved finite-time convergence rate and sample complexity for VR-Greedy-GQ . 1.2 RELATED WORK . Q-learning and SARSA with function approximation . The asymptotic convergence of Q-learning and SARSA under linear function approximation were established in Melo et al . ( 2008 ) ; Perkins & Precup ( 2003 ) , and their finite-time analysis were developed in Zou et al . ( 2019 ) ; Chen et al . ( 2019 ) . However , these algorithms may diverge in off-policy training Baird ( 1995 ) . Also , recent works focused on the Markovian setting . Various analysis techniques have been developed to analyze the finite-time convergence of TD/Q-learning under Markovian samples . Specifically , Wang et al . ( 2020 ) developed a multi-step Lyapunov analysis for addressing the biasedness of the stochastic approximation in Q-learning . Srikant & Ying ( 2019 ) developed a drift analysis to the linear stochastic approximation problem . Besides the linear function approximation , the finite-time analysis of Q-learning under neural network function approximation is developed in Xu & Gu ( 2019 ) . GTD algorithms . The GTD2 and TDC algorithms were developed for off-policy TD learning . Their asymptotic convergence was proved in Sutton et al . ( 2009a ; b ) ; Yu ( 2017 ) , and their finite-time analysis were developed recently in Dalal et al . ( 2018 ) ; Wang et al . ( 2017 ) ; Liu et al . ( 2015 ) ; Gupta et al . ( 2019 ) ; Xu et al . ( 2019 ) . The Greedy-GQ algorithm is an extension of these algorithms to optimal control and involves nonlinear updates . RL with variance reduction : Variance reduction techniques have been applied to various RL algorithms . In TD learning , Du et al . ( 2017 ) reformulate the MSPBE problem as a convex-concave saddle-point optimization problem and applied SVRG Johnson & Zhang ( 2013 ) and SAGA Defazio et al . ( 2014 ) to primal-dual batch gradient algorithm . In Korda & La ( 2015 ) , the variance-reduced TD algorithm was introduced for solving the MSPBE problem , and later Xu et al . ( 2020 ) provided a correct non-asymptotic analysis for this algorithm over Markovian samples . Recently , some other works applied the SVRG , SARAH Nguyen et al . ( 2017 ) and SPIDER Fang et al . ( 2018 ) variance reduction techniques to develop variance-reduced Q-learning algorithms , e.g. , Wainwright ( 2019 ) ; Jia et al . ( 2020 ) . In these works , TD or TDC algorithms are in the form of linear stochastic approximation , and Q-learning has only a single time-scale update . As a comparison , our VR-Greedy-GQ takes nonlinear two time-scale updates to optimization a nonconvex MSPBE . 2 PRELIMINARIES : POLICY OPTIMIZATION AND GREEDY-GQ . In this section , we review some preliminaries of reinforcement learning and recap the Greedy-GQ algorithm under linear function approximation . 2.1 POLICY OPTIMIZATION IN REINFORCEMENT LEARNING . In reinforcement learning , an agent takes actions to interact with the environment via a Markov Decision Process ( MDP ) . Specifically , an MDP is specified by the tuple ( S , A , P , r , γ ) , where S and A respectively correspond to the state and action spaces that include finite elements , r : S×A×S → [ 0 , +∞ ) denotes a reward function and γ ∈ ( 0 , 1 ) is the associated reward discount factor . At any time t , assume that the agent is in the state st ∈ S and takes a certain action at ∈ A following a stationary policy π , i.e. , at ∼ π ( ·|st ) . Then , at the subsequent time t + 1 , the current state of the agent transfers to a new state st+1 according to the transition kernel P ( ·|st , at ) . At the same time , the agent receives a reward rt = r ( st , at , st+1 ) from the environment for this action-state transition . To evaluate the quality of a given policy π , we often use the action-state value function Qπ : S ×A → R that accumulates the discounted rewards as follows : Qπ ( s , a ) = Es′∼P ( ·|s , a ) [ r ( s , a , s′ ) + γV π ( s′ ) ] , where V π ( s ) is the state value function defined as V π ( s ) = E [ ∑∞ t=0 γ trt|s0 = s ] . In particu- lar , define the Bellman operator Tπ such that TπQ ( s , a ) = Es′ , a′ [ r ( s , a , s′ ) + γQ ( s′ , a′ ) ] for any Q ( s , a ) , where a′ ∼ π ( ·|s′ ) . Then , Qπ ( s , a ) is a fixed point of Tπ , i.e. , TπQπ ( s , a ) = Qπ ( s , a ) , ∀s , a . ( 1 ) The goal of policy optimization is to learn the optimal policy π∗ that maximizes the expected total reward E [ ∑∞ t=0 γ trt|s0 = s ] for any initial state s ∈ S , and this is equivalent to learn the optimal value function Q∗ ( s , a ) = supπ Q π ( s , a ) , ∀s , a . In particular , Q∗ is a fixed point of the Bellman operator T that is defined as TQ ( s , a ) = Es′∼P ( ·|s , a ) [ r ( s , a , s′ ) + γmaxb∈AQ ( s′ , b ) ] .

Summary: Greedy-GQ is an RL algorithm for a control problem that extends on GTD, which is a prediction algorithm. While Greedy-GQ asymptotically converges to a stationary point, it does so with high sample complexity. The authors reduce the variance of Greedy-GQ by incorporating SVRG variance reduction scheme to both the time-scale update of the algorithm. The main contribution of the paper is in showing that the variance reduced Greedy-GQ algorithm achieves a sample complexity that is an order of magnitude less than the vanilla Greedy-GQ.