Double Q-learning: New Analysis and Sharper Finite-time Bound

Double Q-learning ( Hasselt , 2010 ) has gained significant success in practice due to its effectiveness in overcoming the overestimation issue of Q-learning . However , theoretical understanding of double Q-learning is rather limited and the only existing finite-time analysis was recently established in Xiong et al . ( 2020 ) under a polynomial learning rate . This paper analyzes the more challenging case with a rescaled linear/constant learning rate for which the previous method does not appear to be applicable . We develop new analytical tools that achieve an orderlevel better finite-time convergence rate than the previously established result . Specifically , we show that synchronous double Q-learning attains an -accurate global optimum with a time complexity of Ω ( lnD ( 1−γ ) 7 2 ) , and the asynchronous algorithm attains a time complexity of Ω̃ ( L ( 1−γ ) 7 2 ) , where D is the cardinality of the state-action space , γ is the discount factor , and L is a parameter related to the sampling strategy for asynchronous double Q-learning . These results improve the order-level dependence of the convergence rate on all major parameters ( , 1 − γ , D , L ) provided in Xiong et al . ( 2020 ) . The new analysis in this paper presents a more direct and succinct approach for characterizing the finite-time convergence rate of double Q-learning . 1 INTRODUCTION . Double Q-learning proposed in Hasselt ( 2010 ) is a widely used model-free reinforcement learning ( RL ) algorithm in practice for searching for an optimal policy ( Zhang et al. , 2018a ; b ; Hessel et al. , 2018 ) . Compared to the vanilla Q-learning proposed in Watkins & Dayan ( 1992 ) , double Q-learning uses two Q-estimators with their roles randomly selected at each iteration , respectively for estimating the maximum Q-function value and updating the Q-function . In this way , the overestimation of the action-value function in vanilla Q-learning can be effectively mitigated , especially when the reward is random or prone to errors ( Hasselt , 2010 ; Hasselt et al. , 2016 ; Xiong et al. , 2020 ) . Moreover , double Q-learning has been shown to have the desired performance in both finite state-action setting ( Hasselt , 2010 ) and infinite setting ( Hasselt et al. , 2016 ) where it successfully improved the performance of deep Q-network ( DQN ) , and thus inspired many variants ( Zhang et al. , 2017 ; Abed-alguni & Ottom , 2018 ) subsequently . In parallel to its empirical success in practice , the theoretical convergence properties of double Qlearning has also been explored . Its asymptotic convergence was first established in Hasselt ( 2010 ) . The asymptotic mean-square error for double Q-learning was studied in Weng et al . ( 2020c ) under the assumption that the algorithm converges to a unique optimal policy . Furthermore , in Xiong et al . ( 2020 ) , the finite-time convergence rate has been established for double Q-learning with a polynomial learning rate α = 1/tω , ω ∈ ( 0 , 1 ) . Under such a choice for the learning rate , they showed that double Q-learning attains an -accurate optimal Q-function at a time complexity approaching to but never reaching Ω ( 1 2 ) at the cost of an asymptotically large exponent on 1 1−γ . However , a polynomial learning rate typically does not offer the best possible convergence rate , as having been shown for RL algorithms that a so-called rescaled linear learning rate ( with a form of αt = ab+ct ) and a constant learning rate achieve a better convergence rate ( Bhandari et al. , 2018 ; Wainwright , 2019a ; b ; Chen et al. , 2020 ; Qu & Wierman , 2020 ) . Therefore , a natural question arises as follows : Can a rescaled linear learning rate or a constant learning rate improve the convergence rate of double Q-learning order-wisely ? If yes , does it also improve the dependence of the convergence rate on other important parameters of the Markov decision process ( MDP ) such as the discount factor and the cardinality of the state and action spaces ? The answer to the above question does not follow immediately from Xiong et al . ( 2020 ) , because the finite-time analysis framework in Xiong et al . ( 2020 ) does not handle such learning rates to yield a desirable result . This paper develops a novel analysis approach and provides affirmative answers to the above question . 1.1 OUR CONTRIBUTIONS . This paper establishes sharper finite-time bounds for double Q-learning with a rescaled linear/constant learning rate , which are orderwisely better than the existing bounds in Xiong et al . ( 2020 ) . We devise a different analysis approach from that in Xiong et al . ( 2020 ) , which is more capable of handling variants of double Q-learning . • For synchronous double Q-learning , where all state-action pairs are visited at each iteration , we apply a rescaled linear learning rate αt = 33+ ( 1−γ ) t and show that the algorithm can attain an -accurate global optimum with a time complexity of Ω ( lnD ( 1−γ ) 7 2 ) , where γ is the discount factor and D = |S||A| is the cardinality of the finite state-action space . As a comparison , for the dominated regime ( with relatively small γ ) , our result attains an -accurate optimal Qfunction with a time complexity Ω ( 1 2 ) , whereas the result in Xiong et al . ( 2020 ) ( see Table 1 ) does not exactly reach Ω ( 1 2 ) and its approaching to such an order ( η : = 1 − ω → 0 ) is at an additional cost of an asymptotically large exponent on 11−γ . For 1− γ dominated regime , our result improves on that in Xiong et al . ( 2020 ) ( which has been optimized in the dependence on 1− γ in Table 1 ) by O ( ( ln 11−γ ) 7 ) . • For asynchronous double Q-learning , where only one state-action pair is visited at each iteration , we obtain a time complexity of Ω̃ ( L ( 1−γ ) 7 2 ) , where L is a parameter related to the sampling strategy in Assumption 1 . As illustrated in Table 1 , our result improves upon that in Xiong et al . ( 2020 ) order-wisely in terms of its dependence on and 1− γ as well as on L by at least O ( L5 ) . Our analysis takes a different approach from that in Xiong et al . ( 2020 ) in order to handle the rescaled linear/constant learning rate . More specifically , to deal with a pair of nested stochastic approximation ( SA ) recursions , we directly establish the dependence bound of the error dynamics ( of the outer SA ) between the Q-estimator and the global optimum on the error propagation ( of the inner SA ) between the two Q-estimators . Then we develop a bound on the inner SA , integrate it into that on the outer SA as a noise term , and establish the final convergence bound . This is a very different yet more direct approach than that in Xiong et al . ( 2020 ) , the latter of which captures the blockwise convergence by constructing two complicated block-wisely decreasing bounds for the two SAs . The sharpness of the bound also requires careful selection of the rescaled learning rates and proper usage of their properties . 1.2 RELATED WORK . Theory on double Q-learning : Double Q-learning was proposed and proved to converge asymptotically in Hasselt ( 2010 ) . In Weng et al . ( 2020c ) , the authors explored the properties of mean-square errors for double Q-learning both in the tabular case and with linear function approximation , under the assumption that a unique optimal policy exists and the algorithm can converge . The most relevant work to this paper is Xiong et al . ( 2020 ) , which established the first finite-time convergence rate for tabular double Q-learning with a polynomial learning rate . This paper provides sharper finite-time convergence bounds for double Q-learning , which requires a different analysis approach . Tabular Q-learning and convergence under various learning rates : Proposed in Watkins & Dayan ( 1992 ) under finite state-action space , Q-learning has aroused great interest in its theoretical study . Its asymptotic convergence has been established in Tsitsiklis ( 1994 ) ; Jaakkola et al . ( 1994 ) ; Borkar & Meyn ( 2000 ) ; Melo ( 2001 ) ; Lee & He ( 2019 ) by requiring the learning rates to satisfy∑∞ t=0 αt = ∞ and ∑∞ t=0 α 2 t < ∞ . Another line of research focuses on the finite-time analysis of Q-learning under different choices of the learning rates . Szepesvári ( 1998 ) captured the first convergence rate of Q-learning using a linear learning rate ( i.e. , αt = 1t ) . Under similar learning rates , Even-Dar & Mansour ( 2003 ) provided finite-time results for both synchronous and asynchronous Q-learning with a convergence rate being exponentially slow as a function of 11−γ . Another popular choice is the polynomial learning rate which has been studied for synchronous Q-learning in Wainwright ( 2019b ) and for both synchronous/asynchronous Q-learning in Even-Dar & Mansour ( 2003 ) . With this learning rate , however , the convergence rate still has a gap with the lower bound of O ( 1√ T ) ( Azar et al. , 2013 ) . To handle this , a more sophisticated rescaled linear learning rate was introduced for synchronous Q-learning ( Wainwright , 2019b ; Chen et al. , 2020 ) and asynchronous Q-learning ( Qu & Wierman , 2020 ) , and thus yields a better convergence rate . The finite-time bounds for Q-learning were also given with constant stepsizes ( Beck & Srikant , 2012 ; Chen et al. , 2020 ; Li et al. , 2020 ) . In this paper , we focus on the rescaled linear/constant learning rate and obtain sharper finite-time bounds for double Q-learning . Q-learning with function approximation : When the state-action space is considerably large or even infinite , the Q-function is usually approximated by a class of parameterized functions . In such a case , Q-learning has been shown not to converge in general ( Baird , 1995 ) . Strong assumptions are typically needed to establish the convergence of Q-learning with linear function approximation ( Bertsekas & Tsitsiklis , 1996 ; Melo et al. , 2008 ; Zou et al. , 2019 ; Chen et al. , 2019 ; Du et al. , 2019 ; Yang & Wang , 2019 ; Jia et al. , 2019 ; Weng et al. , 2020a ; b ) or neural network approximation ( Cai et al. , 2019 ; Xu & Gu , 2019 ) . The convergence analysis of double Q-learning with function approximation raises new technical challenges and can be an interesting topic for future study . 2 PRELIMINARIES ON DOUBLE Q-LEARNING . We consider a Markov decision process ( MDP ) over a finite state space S and a finite action space A with the total cardinality given by D : = |S||A| . The transition kernel of the MDP is given by P : S × A × S → [ 0 , 1 ] denoted as P ( ·|s , a ) . We denote the random reward function at time t as Rt : S × A × S 7→ [ 0 , Rmax ] , with E [ Rt ( s , a , s′ ) ] = Rs ′ sa . A policy π : = π ( ·|s ) captures the conditional probability distribution over the action space given state s ∈ S. For a policy π , we define Q-function Qπ ∈ R|S|×|A| as Qπ ( s , a ) : =E [ ∞∑ t=1 γtRt ( st , at , s ′ t ) ∣∣∣s1 = s , a1 = a ] , ( 1 ) where γ ∈ ( 0 , 1 ) is the discount factor , at ∼ π ( ·|st ) , and s′t ∼ P ( ·|st , at ) . Both vanilla Q-learning ( Watkins & Dayan , 1992 ) and double Q-learning ( Hasselt , 2010 ) aim to find the optimal Q-function Q∗ which is the unique fixed point of the Bellman operator T ( Bertsekas & Tsitsiklis , 1996 ) given by T Q ( s , a ) = Es′∼P ( ·|s , a ) [ Rs ′ sa + γmax a′∈A Q ( s′ , a′ ) ] . ( 2 ) Note that the Bellman operator T is γ-contractive which satisfies ‖T Q− T Q′‖ ≤ γ ‖Q−Q′‖ under the supremum norm ‖Q‖ : = maxs , a |Q ( s , a ) | . The idea of double Q-learning is to keep two Q-tables ( i.e. , Q-function estimators ) QA and QB , and randomly choose one Q-table to update at each iteration based on the Bellman operator computed from the other Q-table . We next describe synchronous and asynchronous double Q-learning algorithms in more detail . Synchronous double Q-learning : Let { βt } t≥1 be a sequence of i.i.d . Bernoulli random variables satisfying P ( βt = 0 ) = P ( βt = 1 ) = 0.5 . At each time t , βt = 0 indicates that QB is updated , and otherwise QA is updated . The update at time t ≥ 1 can be written in a compact form as , { QAt+1 ( s , a ) = ( 1− αtβt ) QAt ( s , a ) + αtβt ( Rt ( s , a , s ′ ) + γQBt ( s ′ , a∗ ) ) , QBt+1 ( s , a ) = ( 1− αt ( 1− βt ) ) QBt ( s , a ) + αt ( 1− βt ) ( Rt ( s , a , s ′ ) + γQAt ( s ′ , b∗ ) ) , ( 3 ) for all ( s , a ) ∈ S × A , where s′ is sampled independently for each ( s , a ) by s′ ∼ P ( ·|s , a ) , a∗ = arg maxa∈AQ A ( s′ , a ) , b∗ = arg maxa∈AQ B ( s′ , a ) and αt is the learning rate . Note that the rewards for both updates of QAt+1 and Q B t+1 are the same copy of Rt . Asynchronous double Q-learning : Different from synchronous double Q-learning , at each iteration the asynchronous version samples only one state-action pair to update the chosen Q-estimator . That is , at time t , only the chosen Q-estimator and its value at the sampled state-action pair ( st , at ) will be updated . We model this by introducing an indicator function τt ( s , a ) = 1 { ( st , at ) = ( s , a ) } . Then the update at time t ≥ 1 of asynchronous double Q-learning can be written compactly as { QAt+1 ( s , a ) = ( 1− αtτt ( s , a ) βt ) QAt ( s , a ) + αtτt ( s , a ) βt ( Rt + γQ B t ( s ′ , a∗ ) ) , QBt+1 ( s , a ) = ( 1− αtτt ( s , a ) ( 1− βt ) ) QBt ( s , a ) + αtτt ( s , a ) ( 1− βt ) ( Rt + γQ A t ( s ′ , b∗ ) ) , ( 4 ) for all ( s , a ) ∈ S ×A , where Rt is evaluated as Rt ( s , a , s′ ) . In the above update rules ( 3 ) and ( 4 ) , at each iteration only one of the two Q-tables is randomly chosen to be updated . This chosen Q-table generates a greedy optimal action , and the other Qtable is used for estimating the corresponding Bellman operator ( or evaluating the greedy action ) for updating the chosen table . Specifically , if QA is chosen to be updated , we use QA to obtain the optimal action a∗ and then estimate the corresponding Bellman operator using QB to update QA . As shown in Hasselt ( 2010 ) , E [ QB ( s′ , a∗ ) ] is likely smaller than Emaxa [ QA ( s′ , a ) ] , where the expectation is taken over the randomness of the reward for the same ( s , a , s′ ) tuple . Such a two-estimator framework adopted by double Q-learning can effectively reduce the overestimation . Without loss of generality , we assume that QA and QB are initialized with the same value ( usually both all-zero tables in practice ) . For both synchronous and asynchronous double Q-learning , it has been shown in Xiong et al . ( 2020 ) that either Q-estimator is uniformly bounded by Rmax1−γ throughout the learning process . Specifically , for either i ∈ { A , B } , we have ∥∥Qit∥∥ ≤ Rmax1−γ and ∥∥Qit −Q∗∥∥ ≤ 2Rmax 1−γ : = Vmax for all t ≥ 1 . This boundedness property will be useful in our finite-time analysis .

Summary: This paper provides a sharper analysis for the finite time convergence rate of the double Q learning algorithm. The authors provides bounds for the synchronous and asynchronous settings and uses a more refined learning rate of $a/(b+t)$. It is shown that with such step size rule, a sharper convergence rate than (Xiong et al., 2020) can be obtained.