Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPs

Q-learning is a popular Reinforcement Learning ( RL ) algorithm which is widely used in practice with function approximation ( Mnih et al. , 2015 ) . In contrast , existing theoretical results are pessimistic about Q-learning . For example , ( Baird , 1995 ) shows that Q-learning does not converge even with linear function approximation for linear MDPs . Furthermore , even for tabular MDPs with synchronous updates , Q-learning was shown to have sub-optimal sample complexity ( Li et al. , 2021 ; Azar et al. , 2013 ) . The goal of this work is to bridge the gap between practical success of Q-learning and the relatively pessimistic theoretical results . The starting point of our work is the observation that in practice , Q-learning is used with two important modifications : ( i ) training with two networks , called online network and target network simultaneously ( online target learning , or OTL ) , and ( ii ) experience replay ( ER ) ( Mnih et al. , 2015 ) . While they have been observed to play a significant role in the practical success of Q-learning , a thorough theoretical understanding of how these two modifications improve the convergence behavior of Q-learning has been missing in literature . By carefully combining the Q-learning with OTL and reverse experience replay ( RER ) ( a form of experience replay ) , we present novel methods Q-Rex and Q-RexDaRe ( Q-Rex+ data reuse ) . We show that Q-Rex efficiently finds the optimal policy for linear MDPs and provide non-asymptotic bounds on sample complexity – the first such result for a Q-learning method for Linear MDPs under standard assumptions . Furthermore , we demonstrate that Q-RexDaRe in fact achieves near optimal sample complexity in the tabular setting , improving upon the existing results for vanilla Q-learning . 1 INTRODUCTION . Reinforcement Learning ( RL ) has been shown to be highly successful for a variety of practical problems in the realm of long term decision making ( Mnih et al. , 2015 ) . Several classical works have studied RL methods like TD-learning , Q-learning , SARSA and their variants for many decades ( Sutton & Barto , 2018 ; Bertsekas , 2011 ; Borkar & Meyn , 2000 ; Sutton , 1988 ; Tsitsiklis & Van Roy , 1997 ; Watkins & Dayan , 1992 ; Watkins , 1989 ) but the guarantees are mostly asymptotic and therefore do not sufficiently answer important questions that are relevant to practitioners who struggle with constraints on the number of data points and the computation power . Recent works provide non-asymptotic results for a variety of important settings ( Kearns & Singh , 1999 ; Even-Dar et al. , 2003 ; Beck & Srikant , 2012 ; Qu & Wierman , 2020 ; Ghavamzadeh et al. , 2011 ; Bhandari et al. , 2018 ; Chen et al. , 2020 ; 2019 ; Dalal et al. , 2018a ; b ; Doan et al. , 2020 ; Gupta et al. , 2019 ; Srikant & Ying , 2019 ; Weng et al. , 2020 ; Xu & Gu , 2020 ; Yang & Wang , 2019 ; Zou et al. , 2019 ) . Despite a large body of work , several aspects of fundamental methods like Q-learning ( Watkins & Dayan , 1992 ) are still ill-understood . Q-learning ’ s simplicity and the ability to learn from off-policy data makes it attractive to the practitioner . However , theoretical analyses show that even with linear function approximation and when the approximation is exact , Q-learning can fail to converge even in simple examples ( Baird , 1995 ; Boyan & Moore , 1995 ; Tsitsiklis & Van Roy , 1996 ) . Furthermore , even in the simple case of tabular RL with synchronous updates , Q-learning is known to have suboptimal sample complexity ( Wainwright , 2019a ; Li et al. , 2021 ) . Despite the negative results , Q-learning has been deployed with tremendous success in practice . The practitioners , however , use Q-learning with “ heuristic ” modifications like experience replay ( ER ) and online target learning ( OTL ) . ER is used to alleviate the issue that the samples obtained in an episode might be highly dependent on each other whereas OTL helps stabilize the Q iteration . Mnih et al . ( 2015 ) conducted extensive experiments to show that both these techniques , along with neural function approximation , are essential for the success of Q-learning . But , existing analyses for ER with Q-learning either require stringent assumptions ( Carvalho et al. , 2020 ) to ensure convergence to a good Q value , or assume that ER provides i.i.d . samples which might not hold in practice ( Fan et al. , 2020 ; Carvalho et al. , 2020 ) . In this paper , we attempt to bridge the gap between theory and practice , by rigorously investigating how Q-learning performs with these practical heuristics . To this end , we introduce two model free algorithms : Q-Rex and its sample efficient variant Q-RexDaRe that combine the standard Q-learning with OTL and reverse experience replay ( RER ) . RER is a form of ER which was recently introduced to unravel spurious correlations present while learning form Markovian data in the context of system identification ( Rotinov , 2019 ; Jain et al. , 2021b ) . We show that OTL stabilizes the Q value by essentially serving as a variance reduction technique and RER unravels the spurious correlations present in the off-policy Markovian data to remove inherent biases introduced in vanilla Q learning . Inclusion of these simple heuristics has surprisingly far-reaching consequences . Firstly , this allows us to show that unlike vanilla Q-learning , Q-Rex finds the optimal policy for linear MDPs and allows us to derive non-asymptotic sample complexity bounds . In the tabular setting , Q-Rex even with asynchronous data is able to match the best known bounds for Q-learning with synchronous data . Furthermore , we extend Q-Rex to obtain a new method Q-RexDaRethat reuses old samples and admits nearly optimal sample complexity for recovering the optimal Q-function in tabular setting . Previously , only Q-learning methods with explicit variance-reduction techniques ( not popular in practice ) ( Wainwright , 2019b ; Li et al. , 2020b ) or model based methods ( Agarwal et al. , 2020 ; Li et al. , 2020a ) were known to achieve such sample complexity bound . Our experiments show that when the algorithmic parameters are chosen carefully , Q-Rex and its variants outperform both vanilla Q-learning and OTL+ER+Q-learning with the same parameters ( see Appendix A ) . To summarize , in this work , we study Q-learning with practical heuristics like ER and OTL , and propose two concrete methods Q-Rex and Q-RexDaRe based on OTL and reverse experience replay – a modification of the standard ER used in practice . We show that Q-Rex is able to find the optimal policy for linear MDPs with strong sample complexity bound which is the first such result for Q-learning . We also show that Q-RexDaRe obtains nearly optimal sample complexity for the simpler tabular setting despite not using any explicit variance reduction technique . See Table 1 for an comparison of our guarantees against the state-of-the-results for linear MDPs and tabular setting . Organization We review related works in next subsection . In Section 2 we develop the MDP problem which we seek to solve and present our algorithm , Q-Rex in Section 3 . The main theoretical results are presented in Section 4 . We present a brief overview of the analysis in Section 5 and present our experiments in Section A . Most of the formal proofs are relegated to the appendix . 1.1 RELATED WORKS . Tabular Q-learning Tabular MDPs are the most basic examples of MDPs where the state space ( S ) and the action space ( A ) are both finite and the Q-values are represented by assigning a unique co-ordinate to each state-action pair . This setting has been well studied over the last few decades and convergence guarantees have been derived in both asymptotic and non-asymptotic regime for popular model-free and model-based algorithms . Azar et al . ( 2013 ) shows that the minimax lower bounds on the sample complexity of obtaining the optimal Q-function up-to error is |S||A| ( 1−γ ) 3 2 , where γ is the discount factor . Near sample-optimal estimation is achieved by several model-based algorithms ( Agarwal et al. , 2020 ; Li et al. , 2020a ) and model-free algorithms like variance reduced Q-learning ( Wainwright , 2019b ; Li et al. , 2020b ) . ( Li et al. , 2021 ) also shows that vanilla Q-learning with standard step sizes , even in the synchronous data setting – where transitions corresponding to each state action pair are sampled independently at each step – suffers from a sample complexity of |S||A| ( 1−γ ) 4 2 and the best known bounds in the asynchronous setting – where data is derived from a Markovian trajectory and only one Q value is updated in each step – is |S||A| ( 1−γ ) 5 2 . These results seem unsatisfactory since γ ∼ 0.99 ( or even 0.999 ) in most practical applications . In contrast , our algorithm Q-Rex with asynchronous data has a sample complexity that matches Q-learning bound with synchronous data and its data-efficient variant Q-RexDaRe has near minimax optimal sample complexity ( see Table 1 ) . For details on model based algorithms , and previous works with suboptimal guarantees we refer to ( Agarwal et al. , 2020 ; Li et al. , 2020b ) . Q-learning with Linear Function Approximation Even though tabular Q-learning is fairly well understood , it is intractable in most practical RL problems due to large size of the state space S. Therefore , Q-learning is deployed with function approximation . Linear function approximation is the simplest such case where the Q-function is approximated with a linear function of the ‘ feature embedding ’ associated with each state-action pair . However , Q-learning can be shown to diverge even in the simplest cases as was first noticed in ( Baird , 1995 ) , which also introduced residual gradient methods which converged rather slowly but provably . We will only discuss recent works closest to our work and refer the reader to ( Carvalho et al. , 2020 ; Jin et al. , 2020 ; Yang & Wang , 2019 ) for a full survey of various works in this direction . Yang & Wang ( 2019 ) consider MDPs with approximate linear function representation - which is more general than the assumptions in this work but require additional assumptions like finite stateaction space and existence of known anchor subsets which might not hold in practice . Our results on the other hand hold with standard assumptions , with asynchronous updates and can handle infinite state-action spaces ( see Theorem 1 ) . Similarly , Chen et al . ( 2019 ) consider Q-learning with linear function approximation for finite state-action spaces , which need not be exact . But the result requires a rather restrictive assumption that the offline policy is close to optimal policy . In contrast , we consider the less general but well-studied case of linear MDPs and provide global convergence without restrictive assumptions on the behaviour policy . Under the most general conditions Maei et al . ( 2010 ) present the Greedy-GQ algorithm which converges to a point asymptotically instead of diverging . Similar results are obtained by Carvalho et al . ( 2020 ) for Coupled Q-learning , a 2-timescale variant of Q-learning which uses a version of OTL and ER1 . This algorithm experimentally resolves the popular counter-examples provided by ( Tsitsiklis & Van Roy , 1996 ; Baird , 1995 ) . Carvalho et al . ( 2020 , Theorem 2 ) provides value function guarantees for the point to which the algorithm converges ( albeit without sample complexity guarantees ) . However , the assumptions for this result are very stringent and even in the case of tabular Q-learning , the method might not converge to the optimal policy . Experience Replay and Reverse Experience Replay Reinforcement learning involves learning on-the-go with Markovian data , which are highly correlated . Iterative learning algorithms like Qlearning can sometimes get coupled to the Markov chain resulting in sub-optimal convergence . Experience replay ( ER ) was introduced in order to mitigate this drawback ( Lin , 1992 ) – here a large FIFO buffer of a fixed size stores the streaming data and the learning algorithm samples a data point uniformly at random from this buffer at each step . This makes the samples look roughly i.i.d . due 1The version of ER used in Carvalho et al . ( 2020 ) makes the setting completely synchronous as opposed to the asynchronous setting considered by us . to mixing , thus breaking the harmful correlations . Reverse experience replay ( RER ) is a form of experience replay which stores a buffer just like ER but processes the data points in the reverse order as stored in the buffer . This was introduced in entirely different contexts by ( Rotinov , 2019 ; Jain et al. , 2021b ; a ) . In the context of this work , we note that reverse order traversal endows a supermartingale structure which yields the strong concentration result in Theorem 4 , which is not possible with forward order traversal . Yet another way to look at RER is through the lens of Dynamic programming ( Bertsekas , 2011 ) – where the value function is evaluated backwards starting from time T to time 1 . Similarly , RER bootstraps Q values to the future Q values instead of the past Q values . Online Target Learning OTL ( Mnih et al. , 2015 ) maintains two different Q-values ( called online Q-value and target Q-value ) where the target Q-value is held constant for some time and only the online Q-value is updated by ‘ bootstrapping ’ to the target . After a number of such iterations , the target Q-value is set to the current online Q value . OTL thus attempts to mitigate the destabilizing effects of bootstrapping by removing the ‘ moving target ’ . This technique has been noted to allow for an unbiased estimation of the bellman operator ( Fan et al. , 2020 ) and when trained with large batch sizes is similar to the well known neural fitted Q-iteration ( Riedmiller , 2005 ) .

Summary: This paper studies the convergence of Q-learning with two popular empirical heuristics (a) online target learning and (b) experience replay. The analysis in this paper is established upon the off-policy setting along with fast-mixing and minimum reachability assumptions. Main convergence results are provided for the linear MDP where all relevant quantities (transition, reward, and Q functions) can be represented as linear functions. The authors propose two variations of Q-learning, namely, Q-Rex and Q-RexDaRe. Convergence results are also given for two algorithms with the sample-optimality guarantee for Q-RexDaRe.