Faster Non-asymptotic Convergence for Double Q-learning

Abstract: Lin Zhao, Huaqing Xiong, Yingbin Liang

Reviews and author responses:

 I thank the authors for the response. I still lean towards acceptance and choose to keep my score the same. 

Thanks to the practical applicability of double Q-learning, this paper denoted to analyzing the finite-time convergence of double Q-learning algorithms with constant stepsizes. Building on existing analysis in Xiong etal'2020, the paper brings in some new analysis tools for nested stochastic approximation schemes and achieves faster non-asymptotic error bounds for double Q-learning. Some interesting observations are made on the practical usefulness of constant step in Q-learning. 
  Originality.
The paper builds heavily upon and compares against the work by Xiong etal'2020, but provides a faster finite-time convergence of nested SA schemes for which double Q-learning can be taken as an interesting example; and the topic is interesting, yet most of the algorithm and analysis follows from existing ones with several new techniques. However, the work Xiong etal'2020 establishes high-probability bound (1-\delta) on the finite-time convergence error, while this paper focuses on the 

Quality.
The quality of this paper looks reasonabe and the bounds appear tighter than existing ones. 

Clarity.
The paper is mostly clear, well-written, and easy to follow.

Significance.
The authors' effort in mproving finite-time error bounds of double Q-learning is appreciated. However, the paper is very similar to the work by Xiong et al'2020 in terms of both the content, algorithm, analysis, and results. The practical implication of the improved results is not fully demonstrated. Whether the results shed light on designing new Q-learning algorithms, or helps advise the efficient use of Q-learning algorithms? How general are the results?  Numerical simulations are also encouraged to validate the theoretical claims. 

-----------
I have seen the rebuttal and read the other reviews, and updated the score. But I am still not convinced by the explanation of the "typo" and the merits of the paper relative to existing works.  Strengths.

Improved high probability error bounds are derived. 
Interesting analysis involving some new tools. 

Major comments/limitations.

1. In general, function approximation is used, in particular nonlinear approximation? 
2. The paper missed comparison against a number of closely related works on two-timescale SA algorithms, Q-learning, and coupled Q-learning algorithms, especially the coupled Q-learning in [D. S. Carvalho, F. S. Melo, P. A. Santos'2020: A new convergent variant of Q-learning with linear function approximation].
3. It is not clear why in (5) (same for the asychronous setting), the noise term {εt}t is a sequence of i.i.d. random variables satisfying Eεt = E[TbtQ∗] − Q∗ = T Q∗ − Q∗ =0 ∈ RD. Then how the Markovian correlation between s and s' is handled? 
4. When and why the SA in (9) has only one unique fixed point? Under what conditions? Do the considered two double Q-learning algorithms obey these conditions?
5. What assumptions shall be made on the error \epsilon_t and bias term v_t for the general SA scheme to cover the two types of double Q-learning? These are important to be claimed as a general result.
6. No numerical simulations validate the claimed theoretical improvement vs the bound provided by Xiong etal'2020.

 **Q1:** What algorithmic ideas are fundamentally new here? It is well-known in stochastic approximation that constant step-sizes can result in faster improvement but to looser limiting neighborhoods of convergence. In that sense, none of the results here are particularly surprising or innovative here.

**A:** We would like to clarify that the focus of this paper is not to design new Q-learning variants, but rather to theoretically characterize shaper bounds on the convergence rate of the vanilla Double-Q learning. Specifically, our convergence rate bound is tight (i.e., order-wise optimal) in terms of its dependence on the major parameters: the learning error $\epsilon$ and the dimension $D$, which matches the lower bound in (Azar et al., 2013). However, the bound in (Xiong et al., 2020) does not have such optimality. Moreover, our result reveals that double Q-learning is as good as vanilla Q-learning in terms of sample efficiency (i.e., their sample complexities scale at the same rate with respect to $\epsilon$ and $D$). Again, the results in (Xiong et al., 2020) don’t suggest this. We believe that these are useful contributions to the understanding of double-Q learning. As a matter of fact, the NeurIPS community does appreciate this type of theoretical studies. Many works on the finite-time analysis of tabular Q-learning have been published in past NeurIPS.

Further, our analysis technique is also novel compared to those in (Xiong et al., 2020), which of course is the reason that we obtain sharper results. These new analysis tools are also contributions to the community.



**Q2:** The results here are restricted to tabular settings only, which limits their applicability/usefulness to preclude large-scale MDPs common in practice in technology today.

**A:** We clarify that “tabular” RL algorithms have always been an active research topic in the ML and NeurIPS community, because understanding these algorithms is a necessary and useful step to further understand function approximation cases. Many studies on “tabular” Q-learning have been published in the past and in NeurIPS last year. For example, the recent study by Xiong et al. (2020) on tabular double Q-learning was published as NeurIPS 2020 spotlight paper. 

However, Xiong et al. (2020) does not provide a sufficiently tight characterization of the convergence rate, which motivated this paper. We believe that our new understanding of the tabular double Q-learning and our new techniques will be useful contributions to the community. Our analysis techniques shall also be further generalized to function approximation cases (whereas Xiong et al. (2020)’s methods may not be easily extended)

**Q3:** There seem to be no experiments contrasting whether the improved learning rates are actually borne out in practice.

**A:** We have done a quick numerical experiment, which shows that a constant learning rate indeed improves the convergence of synchronous double Q-learning over the polynomial learning rate in (Xiong et al., 2020). This matches our theoretical result, which shows that using the constant learning rate provably yields order-level better sample efficiency bound compared to the polynomial learning rate in (Xiong et al., 2020). 

We have done another quick numerical experiment, and showed that the experimental convergence rate of double Q-learning under a constant learning rate matches our theoretical convergence rate in terms of the learning error $\epsilon$. 

We will include them in the revision. 

**Reference:**

(Azar et al., 2013) Mohammad Gheshlaghi Azar, Rémi Munos, Hilbert Kappen. Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model. Machine Learning, Springer Verlag, 2013


 **Q-a(Significance):** The paper is very similar to the work by (Xiong et al., 2020) in terms of both the content, algorithm, analysis, and results.

**A:** We clarify that this paper differs significantly from (Xiong et al., 2020) in both the analysis techniques used and the results obtained. 

(a) Technique-wise,  (Xiong et al., 2020) developed two block-wisely decreasing bounds (with increasing lengths) for the nested SAs, whereas our bounds are per iteration for synchronous sampling and per frame (each frame has the same length) for asynchronous sampling. All other proof techniques are also fundamentally different. In particular, new recursions, bounding inequalities, martingale characterizations, auxiliary Markov chains, etc,  are developed to establish the sharper results in this paper.

(b) Result-wise, by comparing with the lower bound in (Azar et al., 2013), our theoretical bound is tight (i.e., order-wise optimal) in terms of its dependence on the major parameters: learning error $\epsilon$ and dimension $D$, whereas (Xiong et al., 2020) didn’t achieve such optimality. 


**Q-b(Significance):** The practical implication of the improved results is not fully demonstrated. Whether the results shed light on designing new Q-learning algorithms, or help advise the efficient use of Q-learning algorithms? 

**A:** We have done a quick numerical experiment, which shows that a constant learning rate indeed improves the convergence of synchronous double Q-learning over the polynomial learning rate in (Xiong et al., 2020). This matches our theoretical result, which shows that using the constant learning rate provably yields order-level better sample efficiency bound compared to the polynomial learning rate in (Xiong et al., 2020). This suggests that the practical implementation of double Q-learning should adopt a constant learning rate to achieve fast convergence. 

**Q-c(Significance):** How general are the results? 

**A:** Our results are obtained under very mild conditions on MDP. We only assumed in the asynchronous case that the underlying Markov chain (induced by the Markov decision process and behavior policy) is uniformly ergodic. This assumption is widely used and studied in the literature.

**Q-d(Significance):** Numerical simulations are also encouraged to validate the theoretical claims.

**A:** As mentioned in our response to Q-b, We have done a quick numerical experiment, which shows that a constant learning rate indeed improves the convergence of synchronous double Q-learning over the polynomial learning rate in (Xiong et al., 2020). This matches our theoretical result, which shows that using the constant learning rate provably yields order-level better sample efficiency bound compared to the polynomial learning rate in (Xiong et al., 2020). 

We have done another quick numerical experiment, and showed that the experimental convergence rate of double Q-learning under a constant learning rate matches our theoretical convergence rate in terms of its dependence on the learning error $\epsilon$.

**Q1:** In general, function approximation is used, in particular nonlinear approximation?

**A:** Without assuming additional conditions, linear and nonlinear function approximations (such as forward neural networks) don’t have a convergence guarantee in general. This is the case even for vanilla Q-learning, see (Chen et al., 2021, Cai et al., 2019, Xu and Gu, 2020). Thus, in the literature, tabular cases are often studied first to gain more insights into the nature of the problem. Fundamental understanding of the tabular cases shall pave the way for analyzing the more challenging function approximation cases. In fact, our analysis techniques here can be further generalized to consider the function approximation cases (whereas the previous approach by Xiong et al. (2020) may not be easily extended).


**Q2:** The paper missed comparison against a number of closely related works on two-timescale SA algorithms, Q-learning, and coupled Q-learning algorithms, especially the coupled Q-learning in [D. S. Carvalho, F. S. Melo, P. A. Santos'2020: A new convergent variant of Q-learning with linear function approximation].

**A:** Thanks for recommending the references. We will mention them in the related work section in the revision. In the literature, the sample complexity for Q-learning algorithms under the tabular case and with function approximation are not directly comparable due to their different convergence criterions. Specifically, tabular Q-learning typically uses the optimality gap between Q-functions, whereas function approximation uses the optimality gap between function approximation parameters. 

**Q3:** It is not clear why in (5) (same for the asychronous setting), the noise term $\\{\varepsilon_{t}\\}_t$ is a sequence of i.i.d. random variables satisfying  $\mathbb{E}\varepsilon_t= \mathbb{E} [ \widehat{\mathcal{T}}_t Q^{\*}]-Q^{\*}=\mathcal{T}Q^{\*}-Q^{\*}=0\in\mathbb{R}^{D}$. Then how the Markovian correlation between $s$ and $s’$ is handled?

**A:** Thanks for pointing out the typo. The iid is only assumed for the synchronous sampling. In the asynchronous sampling, we instead assume the more challenging Markovian correlation, where the sampling is over a Markovian trajectory path.  To handle the Markovian correlation, we constructed a corresponding martingale sequence of $\varepsilon_t$ with respect to the filtration $\mathcal{F}_t$ defined in Eq. (7). We further bound the terms that involve $\varepsilon_t$ in Proposition 2 via a more general analysis in Lemma 7. Moreover, the coupling between $\varepsilon_t$ and $\beta_t$ were handled via a conditional concentration analysis, see Eqs. (48) and (49) in the proof of Proposition 2.

**Q4:** When and why the SA in (9) has only one unique fixed point? Under what conditions? Do the considered two double Q-learning algorithms obey these conditions?

**A:** In our paper, Eq. (9) is an inequality, which is an intermediate step derived in our proof of Corollary 1, and is not an SA. If the reviewer refers to Eqs. (4) and (6), here is our answer. We didn’t introduce a fixed point for the considered double Q-learning algorithm, but rather, we consider the fixed point of the Bellman operator defined in Eq.(1), denoted by $Q^{*}$ throughout the paper.

**Q5:** What assumptions shall be made on the error $\varepsilon_t$ and bias term $\nu_t$ for the general SA scheme to cover the two types of double Q-learning? These are important to be claimed as a general result.

**A:** The assumptions are not made directly on the error $\varepsilon_t$ and the bias term $\nu_t$. In fact, we do not think it is reasonable to require assumptions on these parameters, because they cannot be easily checked and guaranteed. Instead, the properties of these two errors follow from the standard assumptions on the MDP and the sampling approaches. 

**Q6:** No numerical simulations validate the claimed theoretical improvement vs the bound provided by Xiong et al'2020.

**A:** As mentioned in our response to Q-b, We have done a quick numerical experiment, which shows that a constant learning rate indeed improves the convergence of synchronous double Q-learning over the polynomial learning rate in (Xiong et al., 2020). This matches our theoretical result, which shows that using the constant learning rate provably yields order-level better sample efficiency bound compared to the polynomial learning rate in (Xiong et al., 2020). 

We have done another quick numerical experiment and showed that the experimental convergence rate of double Q-learning under a constant learning rate matches our theoretical convergence rate in terms of the dependence on the learning error $\epsilon$. 

**References:**

(Azar et al., 2013) Mohammad Gheshlaghi Azar, Rémi Munos, Hilbert Kappen. Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model. Machine Learning, Springer Verlag, 2013.

(Chen et al., 2021) Zaiwei Chen, Sheng Zhang, Thinh T. Doan, John-Paul Clarke, Siva Theja Maguluri. Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning, 2021

(Cai et al., 2019) Qi Cai, Zhuoran Yang, Jason D. Lee, Zhaoran Wang. Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima, arXiv:1905.10027, 2019.

(Xu and Gu, 2020) Pan Xu and Quanquan Gu. A Finite-Time Analysis of Q-Learning with Neural Network Function Approximation, in Proc. of the 37th International Conference on Machine Learning (ICML), 2020.


 Thanks again for recognizing our contributions!

 **Q:** As mentioned in the paper, the results are worse than the guarantees obtained for the vanilla Q-learning in terms of the discount factor $\gamma$ which is the major drawback of the work.

**A:** Thanks for pointing this out! In fact, there has not been a conclusive performance comparison between double Q-learning and vanilla Q-learning in numerical experiments in the literature. For example, a recent experimental study (Weng, et al, 2020) under a similar setting to this paper shows that when using the same step size, Q-learning has a faster rate of convergence than double Q-learning initially but suffers from a higher mean-squared error. This observation matches our theoretical analysis. On the other hand,  when double Q-learning uses twice the step size of Q-learning, their simulation shows that double Q-learning achieves a faster initial convergence rate, at the cost of a possibly worse mean-squared error than Q-learning. This topic is still up to the future investigation to have a solid conclusion. 

**Reference**

(Weng et al., 2020) Wentao Weng, Harsh Gupta, Niao He, Lei Ying, R. Srikant. The Mean-Squared Error of Double Q-Learning, NeurIPS, 2020.


Vanilla Q learning often suffers from the over estimation problem i.e, it might choose bad actions initially which gets reinforced later on. Double Q learning maintains two Q functions $Q^A$ and $Q^B$ which are used to update each other. This helps in the case of random rewards. However, the complicated coupled dynamics of $Q^A$ and $Q^B$ makes the analysis much harder. Even though there are many satisfactory analyses for vanilla Q learning, double Q learning is lacking in this front.  This paper attempts to fill in this gap by giving a non-asymptotic convergence analysis of DQL, which is much better than the previous analyses in terms of every parameter. 

In both the synchronous and asynchronous case (i.e all cases updated at once or updated from a stream of off-policy data respectively) they obtain a time complexity of $O(1/\epsilon^2(1-\gamma)^7)$ where $\epsilon$ is the target error and $\gamma$ is the discount factor. The proof technique mentioned in Section 3.3 considers the error propagation dynamics and analyzes the iterations using standard concentration inequalities.  I think this is an important problem with lots of recent interest. The paper is fairly well written and makes a good attempt at obtaining a finite time analysis of DQL. As mentioned in the paper, the results are worse than the guarantees obtained for the vanilla Q learning in terms of the discount factor $\gamma$ which is the major drawback of the work.

All in all, I expect that the techniques established here would be useful in further investigation of DQL and eventually show that it outperforms vanilla Q learning. Yes

The paper studied double Q-learning in RL and established finite time convergence analysis for both synchronous/asynchronous version of the algorithm. The complexity result improves the existing result in terms of the dependency on parameters such as $\epsilon, 1-\gamma, \cdots$.  This papers studied Double Q-learning, which is one of the most widely used technique in RL. The authors established the non-asymptotic convergence result of both synchronous/asynchronous Double Q-learning for tabular discounted MDP. In particular, the paper proved $O(\frac{1}{(1-\gamma)^7\epsilon^2})$ time complexity for both methods (ignoring other constants), improving upon the existing result that adopts a decaying step size. This is achieved by using more aggressive constants step size and a more refined analysis. It still remains unknown if Double Q-learning is provably faster than Q-learning, and this paper serves as an important step towards understanding Double Q-learning.

With the prevalence of Double Q-learning methods in various applications, it is of great importance to understand its theoretical guarantee. Overall this paper is solid and very well written.  See full review.

This work studies the Double Q Learning algorithm for reinforcement learning under tabular settings. The main technical contribution is a refined convergence to the optimal Q function with high probability under a specification of constant step-size. This improves upon previous results which use a polynomial learning rates. No experiments are given to contrast whether the performance gap translates to substantial gains in practice.  
The technical contribution of a novel convergence result for double Q learning is interesting, and clearly refines existing results on the performance of Double Q learning that require attenuating step-sizes. I agree that the results hinge upon some technical innovations illuminated on page 3, lines 68-75. However, I am not sure this alone is sufficient to warrant acceptance. That is because:

1) what algorithmic ideas are fundamentally new here? It is well-known in stochastic approximation that constant step-sizes can result in faster improvement but to looser limiting neighborhoods of convergence. In that sense, none of the results here are particularly surprising or innovative here.

2) The results here are restricted to tabular settings only, which limits their applicability/usefulness to preclude large-scale MDPs common in practice in technology today.

3) There seem to be no experiments contrasting whether the improved learning rates are actually borne  out in practice. 

Therefore, this work should really be evaluated on the technical novelty of the proofs alone. While the characterization of the error dynamics in 3.1 which gives rise to the main results is well-executed, I feel the lack of actual algorithmic innovation or experimental validation is difficult to overlook. Yes.

Meta-review: This paper received borderline reviews.  Some main issues were about lack of clarity, and of a lack of empirical validation of the theoretical benefits of the proposed constant step sizes.

I felt the rebuttal did help clarify some of the raised issues.  I would also encourage the authors to include the numerical experiments that they mentioned they ran in a future version of this paper.  Even if the authors are of the opinion that these do not strictly add new information and only validate the theory, it is important to consider a publication as primarily a means to communicate our findings to the larger research community, and such numerical experiments can help illustrate the main points, as well as augment the theory, if and when there are discrepancies.

For instance, I'd personally love to understand better whether the practical convergence rates match the current theory.  Is there an important 'gap' still between practical and theoretical convergence rates?  Would it be possible to do an even tighter analysis and find even-better bounds?  These are examples of questions that could help the reader better understand the significance of this work (not necessarily in terms of "does it work with neural nets?", but rather in terms of "what kind of further theoretical study does this enable?" or "what kind of research questions does this raise or answer?").

I think the paper has promise, but could benefit from another round of reviews, using the reviewers' comments to improve the work.  I therefore would like to strongly encourage the authors to consider how the paper could be made even better, and then resubmit (e.g., to a related venue).

Decision: accept