Double Q-learning: New Analysis and Sharper Finite-time Bound

Double Q-learning ( Hasselt , 2010 ) has gained significant success in practice due to its effectiveness in overcoming the overestimation issue of Q-learning . However , theoretical understanding of double Q-learning is rather limited and the only existing finite-time analysis was recently established in Xiong et al . ( 2020 ) under a polynomial learning rate . This paper analyzes the more challenging case with a rescaled linear/constant learning rate for which the previous method does not appear to be applicable . We develop new analytical tools that achieve an orderlevel better finite-time convergence rate than the previously established result . Specifically , we show that synchronous double Q-learning attains an -accurate global optimum with a time complexity of Ω ( lnD ( 1−γ ) 7 2 ) , and the asynchronous algorithm attains a time complexity of Ω̃ ( L ( 1−γ ) 7 2 ) , where D is the cardinality of the state-action space , γ is the discount factor , and L is a parameter related to the sampling strategy for asynchronous double Q-learning . These results improve the order-level dependence of the convergence rate on all major parameters ( , 1 − γ , D , L ) provided in Xiong et al . ( 2020 ) . The new analysis in this paper presents a more direct and succinct approach for characterizing the finite-time convergence rate of double Q-learning . 1 INTRODUCTION . Double Q-learning proposed in Hasselt ( 2010 ) is a widely used model-free reinforcement learning ( RL ) algorithm in practice for searching for an optimal policy ( Zhang et al. , 2018a ; b ; Hessel et al. , 2018 ) . Compared to the vanilla Q-learning proposed in Watkins & Dayan ( 1992 ) , double Q-learning uses two Q-estimators with their roles randomly selected at each iteration , respectively for estimating the maximum Q-function value and updating the Q-function . In this way , the overestimation of the action-value function in vanilla Q-learning can be effectively mitigated , especially when the reward is random or prone to errors ( Hasselt , 2010 ; Hasselt et al. , 2016 ; Xiong et al. , 2020 ) . Moreover , double Q-learning has been shown to have the desired performance in both finite state-action setting ( Hasselt , 2010 ) and infinite setting ( Hasselt et al. , 2016 ) where it successfully improved the performance of deep Q-network ( DQN ) , and thus inspired many variants ( Zhang et al. , 2017 ; Abed-alguni & Ottom , 2018 ) subsequently . In parallel to its empirical success in practice , the theoretical convergence properties of double Qlearning has also been explored . Its asymptotic convergence was first established in Hasselt ( 2010 ) . The asymptotic mean-square error for double Q-learning was studied in Weng et al . ( 2020c ) under the assumption that the algorithm converges to a unique optimal policy . Furthermore , in Xiong et al . ( 2020 ) , the finite-time convergence rate has been established for double Q-learning with a polynomial learning rate α = 1/tω , ω ∈ ( 0 , 1 ) . Under such a choice for the learning rate , they showed that double Q-learning attains an -accurate optimal Q-function at a time complexity approaching to but never reaching Ω ( 1 2 ) at the cost of an asymptotically large exponent on 1 1−γ . However , a polynomial learning rate typically does not offer the best possible convergence rate , as having been shown for RL algorithms that a so-called rescaled linear learning rate ( with a form of αt = ab+ct ) and a constant learning rate achieve a better convergence rate ( Bhandari et al. , 2018 ; Wainwright , 2019a ; b ; Chen et al. , 2020 ; Qu & Wierman , 2020 ) . Therefore , a natural question arises as follows : Can a rescaled linear learning rate or a constant learning rate improve the convergence rate of double Q-learning order-wisely ? If yes , does it also improve the dependence of the convergence rate on other important parameters of the Markov decision process ( MDP ) such as the discount factor and the cardinality of the state and action spaces ? The answer to the above question does not follow immediately from Xiong et al . ( 2020 ) , because the finite-time analysis framework in Xiong et al . ( 2020 ) does not handle such learning rates to yield a desirable result . This paper develops a novel analysis approach and provides affirmative answers to the above question . 1.1 OUR CONTRIBUTIONS . This paper establishes sharper finite-time bounds for double Q-learning with a rescaled linear/constant learning rate , which are orderwisely better than the existing bounds in Xiong et al . ( 2020 ) . We devise a different analysis approach from that in Xiong et al . ( 2020 ) , which is more capable of handling variants of double Q-learning . • For synchronous double Q-learning , where all state-action pairs are visited at each iteration , we apply a rescaled linear learning rate αt = 33+ ( 1−γ ) t and show that the algorithm can attain an -accurate global optimum with a time complexity of Ω ( lnD ( 1−γ ) 7 2 ) , where γ is the discount factor and D = |S||A| is the cardinality of the finite state-action space . As a comparison , for the dominated regime ( with relatively small γ ) , our result attains an -accurate optimal Qfunction with a time complexity Ω ( 1 2 ) , whereas the result in Xiong et al . ( 2020 ) ( see Table 1 ) does not exactly reach Ω ( 1 2 ) and its approaching to such an order ( η : = 1 − ω → 0 ) is at an additional cost of an asymptotically large exponent on 11−γ . For 1− γ dominated regime , our result improves on that in Xiong et al . ( 2020 ) ( which has been optimized in the dependence on 1− γ in Table 1 ) by O ( ( ln 11−γ ) 7 ) . • For asynchronous double Q-learning , where only one state-action pair is visited at each iteration , we obtain a time complexity of Ω̃ ( L ( 1−γ ) 7 2 ) , where L is a parameter related to the sampling strategy in Assumption 1 . As illustrated in Table 1 , our result improves upon that in Xiong et al . ( 2020 ) order-wisely in terms of its dependence on and 1− γ as well as on L by at least O ( L5 ) . Our analysis takes a different approach from that in Xiong et al . ( 2020 ) in order to handle the rescaled linear/constant learning rate . More specifically , to deal with a pair of nested stochastic approximation ( SA ) recursions , we directly establish the dependence bound of the error dynamics ( of the outer SA ) between the Q-estimator and the global optimum on the error propagation ( of the inner SA ) between the two Q-estimators . Then we develop a bound on the inner SA , integrate it into that on the outer SA as a noise term , and establish the final convergence bound . This is a very different yet more direct approach than that in Xiong et al . ( 2020 ) , the latter of which captures the blockwise convergence by constructing two complicated block-wisely decreasing bounds for the two SAs . The sharpness of the bound also requires careful selection of the rescaled learning rates and proper usage of their properties . 1.2 RELATED WORK . Theory on double Q-learning : Double Q-learning was proposed and proved to converge asymptotically in Hasselt ( 2010 ) . In Weng et al . ( 2020c ) , the authors explored the properties of mean-square errors for double Q-learning both in the tabular case and with linear function approximation , under the assumption that a unique optimal policy exists and the algorithm can converge . The most relevant work to this paper is Xiong et al . ( 2020 ) , which established the first finite-time convergence rate for tabular double Q-learning with a polynomial learning rate . This paper provides sharper finite-time convergence bounds for double Q-learning , which requires a different analysis approach . Tabular Q-learning and convergence under various learning rates : Proposed in Watkins & Dayan ( 1992 ) under finite state-action space , Q-learning has aroused great interest in its theoretical study . Its asymptotic convergence has been established in Tsitsiklis ( 1994 ) ; Jaakkola et al . ( 1994 ) ; Borkar & Meyn ( 2000 ) ; Melo ( 2001 ) ; Lee & He ( 2019 ) by requiring the learning rates to satisfy∑∞ t=0 αt = ∞ and ∑∞ t=0 α 2 t < ∞ . Another line of research focuses on the finite-time analysis of Q-learning under different choices of the learning rates . Szepesvári ( 1998 ) captured the first convergence rate of Q-learning using a linear learning rate ( i.e. , αt = 1t ) . Under similar learning rates , Even-Dar & Mansour ( 2003 ) provided finite-time results for both synchronous and asynchronous Q-learning with a convergence rate being exponentially slow as a function of 11−γ . Another popular choice is the polynomial learning rate which has been studied for synchronous Q-learning in Wainwright ( 2019b ) and for both synchronous/asynchronous Q-learning in Even-Dar & Mansour ( 2003 ) . With this learning rate , however , the convergence rate still has a gap with the lower bound of O ( 1√ T ) ( Azar et al. , 2013 ) . To handle this , a more sophisticated rescaled linear learning rate was introduced for synchronous Q-learning ( Wainwright , 2019b ; Chen et al. , 2020 ) and asynchronous Q-learning ( Qu & Wierman , 2020 ) , and thus yields a better convergence rate . The finite-time bounds for Q-learning were also given with constant stepsizes ( Beck & Srikant , 2012 ; Chen et al. , 2020 ; Li et al. , 2020 ) . In this paper , we focus on the rescaled linear/constant learning rate and obtain sharper finite-time bounds for double Q-learning . Q-learning with function approximation : When the state-action space is considerably large or even infinite , the Q-function is usually approximated by a class of parameterized functions . In such a case , Q-learning has been shown not to converge in general ( Baird , 1995 ) . Strong assumptions are typically needed to establish the convergence of Q-learning with linear function approximation ( Bertsekas & Tsitsiklis , 1996 ; Melo et al. , 2008 ; Zou et al. , 2019 ; Chen et al. , 2019 ; Du et al. , 2019 ; Yang & Wang , 2019 ; Jia et al. , 2019 ; Weng et al. , 2020a ; b ) or neural network approximation ( Cai et al. , 2019 ; Xu & Gu , 2019 ) . The convergence analysis of double Q-learning with function approximation raises new technical challenges and can be an interesting topic for future study . 2 PRELIMINARIES ON DOUBLE Q-LEARNING . We consider a Markov decision process ( MDP ) over a finite state space S and a finite action space A with the total cardinality given by D : = |S||A| . The transition kernel of the MDP is given by P : S × A × S → [ 0 , 1 ] denoted as P ( ·|s , a ) . We denote the random reward function at time t as Rt : S × A × S 7→ [ 0 , Rmax ] , with E [ Rt ( s , a , s′ ) ] = Rs ′ sa . A policy π : = π ( ·|s ) captures the conditional probability distribution over the action space given state s ∈ S. For a policy π , we define Q-function Qπ ∈ R|S|×|A| as Qπ ( s , a ) : =E [ ∞∑ t=1 γtRt ( st , at , s ′ t ) ∣∣∣s1 = s , a1 = a ] , ( 1 ) where γ ∈ ( 0 , 1 ) is the discount factor , at ∼ π ( ·|st ) , and s′t ∼ P ( ·|st , at ) . Both vanilla Q-learning ( Watkins & Dayan , 1992 ) and double Q-learning ( Hasselt , 2010 ) aim to find the optimal Q-function Q∗ which is the unique fixed point of the Bellman operator T ( Bertsekas & Tsitsiklis , 1996 ) given by T Q ( s , a ) = Es′∼P ( ·|s , a ) [ Rs ′ sa + γmax a′∈A Q ( s′ , a′ ) ] . ( 2 ) Note that the Bellman operator T is γ-contractive which satisfies ‖T Q− T Q′‖ ≤ γ ‖Q−Q′‖ under the supremum norm ‖Q‖ : = maxs , a |Q ( s , a ) | . The idea of double Q-learning is to keep two Q-tables ( i.e. , Q-function estimators ) QA and QB , and randomly choose one Q-table to update at each iteration based on the Bellman operator computed from the other Q-table . We next describe synchronous and asynchronous double Q-learning algorithms in more detail . Synchronous double Q-learning : Let { βt } t≥1 be a sequence of i.i.d . Bernoulli random variables satisfying P ( βt = 0 ) = P ( βt = 1 ) = 0.5 . At each time t , βt = 0 indicates that QB is updated , and otherwise QA is updated . The update at time t ≥ 1 can be written in a compact form as , { QAt+1 ( s , a ) = ( 1− αtβt ) QAt ( s , a ) + αtβt ( Rt ( s , a , s ′ ) + γQBt ( s ′ , a∗ ) ) , QBt+1 ( s , a ) = ( 1− αt ( 1− βt ) ) QBt ( s , a ) + αt ( 1− βt ) ( Rt ( s , a , s ′ ) + γQAt ( s ′ , b∗ ) ) , ( 3 ) for all ( s , a ) ∈ S × A , where s′ is sampled independently for each ( s , a ) by s′ ∼ P ( ·|s , a ) , a∗ = arg maxa∈AQ A ( s′ , a ) , b∗ = arg maxa∈AQ B ( s′ , a ) and αt is the learning rate . Note that the rewards for both updates of QAt+1 and Q B t+1 are the same copy of Rt . Asynchronous double Q-learning : Different from synchronous double Q-learning , at each iteration the asynchronous version samples only one state-action pair to update the chosen Q-estimator . That is , at time t , only the chosen Q-estimator and its value at the sampled state-action pair ( st , at ) will be updated . We model this by introducing an indicator function τt ( s , a ) = 1 { ( st , at ) = ( s , a ) } . Then the update at time t ≥ 1 of asynchronous double Q-learning can be written compactly as { QAt+1 ( s , a ) = ( 1− αtτt ( s , a ) βt ) QAt ( s , a ) + αtτt ( s , a ) βt ( Rt + γQ B t ( s ′ , a∗ ) ) , QBt+1 ( s , a ) = ( 1− αtτt ( s , a ) ( 1− βt ) ) QBt ( s , a ) + αtτt ( s , a ) ( 1− βt ) ( Rt + γQ A t ( s ′ , b∗ ) ) , ( 4 ) for all ( s , a ) ∈ S ×A , where Rt is evaluated as Rt ( s , a , s′ ) . In the above update rules ( 3 ) and ( 4 ) , at each iteration only one of the two Q-tables is randomly chosen to be updated . This chosen Q-table generates a greedy optimal action , and the other Qtable is used for estimating the corresponding Bellman operator ( or evaluating the greedy action ) for updating the chosen table . Specifically , if QA is chosen to be updated , we use QA to obtain the optimal action a∗ and then estimate the corresponding Bellman operator using QB to update QA . As shown in Hasselt ( 2010 ) , E [ QB ( s′ , a∗ ) ] is likely smaller than Emaxa [ QA ( s′ , a ) ] , where the expectation is taken over the randomness of the reward for the same ( s , a , s′ ) tuple . Such a two-estimator framework adopted by double Q-learning can effectively reduce the overestimation . Without loss of generality , we assume that QA and QB are initialized with the same value ( usually both all-zero tables in practice ) . For both synchronous and asynchronous double Q-learning , it has been shown in Xiong et al . ( 2020 ) that either Q-estimator is uniformly bounded by Rmax1−γ throughout the learning process . Specifically , for either i ∈ { A , B } , we have ∥∥Qit∥∥ ≤ Rmax1−γ and ∥∥Qit −Q∗∥∥ ≤ 2Rmax 1−γ : = Vmax for all t ≥ 1 . This boundedness property will be useful in our finite-time analysis .

Summary: This paper studies the convergence rate of double Q learning under the tabular setting. Both the synchronous and asynchronous double Q learning algorithms are studied and analyzed. The technical novelty of this paper seems limited -- possibly a direct combination of [Wainwright 2019a] and [Xiong et al 2020]. Moreover, the scope of this work seems also limited -- it only consider the tabular version of double Q-learning, with either a generative model (synchronous) or i.i.d. sampling (asynchronous) sampling models, which is never used in practice. This seems to makes this work only appealing to theorists. However, in terms of the theory, the results for double Q learning is pessimist -- the rate is much worse than standard Q learning. However, practical implementations of double Q learning has demonstrated its advantage in terms of correcting the over-estimation bias. Without numerical results or additional theory, such a gap makes one ponder whether the rates in this paper are tight.