Title: On the Convergence of Single-Timescale Actor-Critic
Venue: NeurIPS 2025
Accepted: yes
Keywords: Policy Gradient

Reinforcement Learning

Actor-Critic Algorithms

Sample Complexity

Convergence
Abstract: We analyze the global convergence of the single-timescale actor-critic (AC) algorithm for the infinite-horizon discounted Markov Decision Processes (MDPs) with finite state spaces. To this end, we introduce an elegant analytical framework for handling complex, coupled recursions inherent in the algorithm. Leveraging this framework, we establish that the algorithm converges to an $\epsilon$-close \textbf{globally optimal} policy with a sample complexity of $ O(\epsilon^{-3}) $. This significantly improves upon the existing complexity of $O(\epsilon^{-2})$ to achieve $\epsilon$-close  \textbf{stationary policy}, which is equivalent to the complexity of $O(\epsilon^{-4})$ to achieve $\epsilon$-close  \textbf{globally optimal} policy using gradient domination lemma. Furthermore, we demonstrate that to achieve this improvement, the step sizes for both the actor and critic must decay as $ O(k^{-\frac{2}{3}}) $ with iteration $k$, diverging from the conventional $O(k^{-\frac{1}{2}}) $ rates commonly used in (non)convex optimization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the standard actor-critic algorithm with the step sizes for the policy updates and value function updates on the same timescale, and shows that the algorithm's last iterate converges with rate $\\mathcal{O}(1/k^{1/3})$ (measured in the function value space). This is the best complexity derived so far in the literature.\n\nSTRENGTHS AND WEAKNESSES:\nThe strength lies in the result itself. The convergence rate $\\mathcal{O}(1/k^{1/2})$ measured by gradient norm squared has been the best-known result and this paper establishes $\\mathcal{O}(1/k^{1/3})$ convergence in function value space, which is highly significant. I believe many researchers in the RL theory will find the result and analysis interesting.\n\nOn the flip side, the paper is badly written, not in terms of the language, but in delivering a clear message on what specific technique/analysis enables the improved rate. The authors The authors repeatedly emphasize the challenges arising from coupled actor and critic variables and from using single-loop single-timescale updates. I do not believe these are the actual challenges. As the authors themselves noted, Chen and Zhao [2024] analyzes the same single-loop single-timescale algorithm and establishes convergence. Moreover, in broader settings of stochastic approximation/optimization, under structures like non-convexity with Lipschitz gradients, PL condition, and strong convexity, many other papers consider a similar single-timescale algorithm and show that the algorithm achieves the optimal convergence rates. \n\nThe weakness in the analysis from Chen and Zhao [2024] is that they treat the policy optimization objective as a general smooth non-convex function without using the gradient domination structure (or one may plug in the inequality at the end). In my understanding, the core innovation of the current paper is to exploit this structure when bounding the iteration-wise drift of the actor. A more thorough discussion of this aspect should be the actual focus of the presentation.\n\nOverall, I believe that the results derived in the paper are very much worth being known to the community, but significant additional work is needed to improve the presentation and to help the audience understand the core contribution and technical innovation.\n\nReferences\n\n[1] Shen, H. and Chen, T., 2022. A single-timescale analysis for stochastic approximation with multiple coupled sequences. Advances in Neural Information Processing Systems, 35, pp.17415-17429.\n\n[2] Zeng, S. and Doan, T., 2024, June. Fast two-time-scale stochastic gradient method with applications in reinforcement learning. In The Thirty Seventh Annual Conference on Learning Theory (pp. 5166-5212). PMLR.\n\n[3] Huang, Y., Wu, Z., Ma, S. and Ling, Q., 2024. Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applications. arXiv e-prints, pp.arXiv-2410.\n\nQUESTIONS:\n1) Can the authors clarify if I correctly understand the innovation of the paper in my comments above (in Strengths & Weaknesses)? Specifically, the limitation in prior works like Chen and Zhao [2024] is that they disregard the gradient domination structure and treat the policy optimization objective as a standard smooth non-convex function, but the current paper leverages the structure in the analysis of the actor. From this perspective, the key novelty of the paper lies in Lemma 4.1. To my knowledge, Lemma 4.2 is a standard result (namely, showing the approximate iteration-wise critic convergence up to gradient norm of the actor). The fact that Lemma 4.2 is standard does not undermine the contribution of the paper. For clarity, I strongly encourage the authors to be precise and transparent in communicating to the audience which parts of the analysis are novel and which rely on established results. If this is indeed the case, the authors should consider substantially revising the paper to place more emphasis on Lemma 4.1 and its analysis in the main paper.\n\n2) The authors state \"Additionally, when this ODE tracking method is applied to the recursion for the exact gradient case, it produces better results compared to existing bounds, such as those in Mei et al. (2022).\" in line 52-54, but never establish anything along this line.\n\n3) In line 119-121, the authors make the comment \"Yuan et al. (2022) too establishes global convergence with sample complexity of $\\mathcal{O}(\\epsilon^{-3})$, however, it requires an additional structural assumption on the problem which is highly restrictive.\" The sentence is written in a way that makes the audience want to know the exact assumption. Please provide more detail here.\n\n4) In line 150-151, \"However, the return ... leading to the following result.\" -- the gradient domination lemma is not a direct consequence of $J$ being smooth.\n\n5) I suggest that the authors re-organize the appendix and add proper linkage between lemmas/propositions and their proofs, to make it easier to the audience to locate the proofs.\n\n6) Something nice about the result in this paper is that the convergence is established on the last iterate. If we follow the analysis in Chen and Zhao [2024] and plugs in the gradient domination condition at the end, the convergence in value function space will be on the best iterate (in addition to having an inferior rate). The authors should consider mentioning this additional advantage of their analysis.\n\nLIMITATIONS:\nI do not foresee any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the sample-complexity for the problem of finding a $\\varepsilon$-optimal policy for softmax-parameterised infinite-horizon discounted Markov decision processed. They focus on the sample-complexity’s dependence on $\\varepsilon$ and establish that a single-timescale actor-critic aglorithm achieves a sample-complexity of $O(\\varepsilon^{-3})$ improving upon the best known results of $O(\\varepsilon^{-4})$ from prior work. This is achieved by an analysis that directly bounds the sub-optimality gap (instead of convergence to stationary points) and combines recursions for the actor and the critic simultaneously (the Lyapunov term).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The problem considered is of practical relevance: the single-timescale version of actor-critic more closely resembles algorithms used in practice than the two-timescale version.\n- The sample-complexity of the single-timescale actor-critic algorithm is improved from $O(\\varepsilon^{-4})$ to $O(\\varepsilon^{-3})$.\n- The provided analysis and key idea is novel, fairly simple, insightful and general.\n- The work generally does a good job of presenting the setting and contextualising their results within prior work.\n\nWeaknesses:\n- The sample-complexity does not match the lower-bound whose dependence in $O(\\varepsilon^{-2})$. The dependence on quantities other than $\\varepsilon$ are ignored. In particular, the dependence on the effective horizon $1/(1-\\gamma)$ and the size of the state-action space $SA$, which are hidden within constants, appears like it could be quite poor. While I understand that it could be argued that until the optimal dependence on $\\varepsilon$ is achieved, the dependence on these other quantities is of secondary importance, having a discussion on the explicit dependence on these in the final bound would be helpful for the reader.\n- Presentation: the writing / clarity of presentation is below the standard expected. The appendix has quite a few repeated entries (one reference even appears three identical times), some references are missing their publication venues… etc. There are many typos and grammatical errors (e.g. l.70, l.78, l.81-84, l.266… etc). Table 4 referenced at l.242 does not exist or points to the wrong thing – I guess Table 2 was meant. At the top of page 7, $Q_k$ has not been formally defined - I guess it is the update in Algo 1 but then would be clearer to write in Algo 1 with this notation. In the Convergence Analysis section, it would be useful to have pointers to where the specific sub-results are proved in the appendix. Some of the discussions are a bit unclear / hard to follow. In particular, the discussion l.268-272 is not easy to understand.\n\nQUESTIONS:\n- See weaknesses for some suggestions on improving presentation. This paper has interesting results but suffers from the quality of the presentation. If the presentation of the paper was improved, it would have a higher score.\n- In the  discussion following Lemma 4.1, it is not clear to me why the last term “accounts for the variance in the stochastic update of the policy” since in the analysis this term arises from the smoothness of the value and not from the stochasticity of the update ?\n- You mention that you believe that a bias analysis may lead to the optimal $O(\\varepsilon^{-2})$ sample-complexity. Do you believe this is achievable without any modification to the algorithm ? Is there perhaps a slight adaptation of the algorithm that could improve your variance analysis ?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence of actor-critic algorithm, under the setting of infinite-horizon discounted MDPs. The paper considers single-timescale setting, where the actor and the critic are updated simultaneously. The paper develop a new analysis technique, which first modeling the evolution of the actor error and the critic error with a dynamic system. To bypass the difficulty incurred by the hard-to-estimate gradient norm term, the paper considers an Lyapunov term of the dynamic system. Analysis shows that the Lyapunov term is evolved according to a relative simple single-variable ODE. The solution to the ODE gives a sample complexity of $O(\\epsilon^{-3})$, which strictly improves the previous $O(\\epsilon^{-4})$ result.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper consider single-timescale actor-critic algorithm. Compared to two-timescale actor-critic, this setting is more practically relevant and hard-to-analysis.\n\n2. The result of this paper strictly improved upon the previous results, which is new to me.\n\n3. The technique introduced in this paper is also interesting to me. To overcome the difficulty in solving the dynamic system depicting the evolution of the errors, the paper consider the sum of actor and critic error and largely simplified the system. This technique might be inspiring to solve other similar problems.\n\nWeakness:\n\nI don't see any major weakness in this paper. Some part of the presentation are not clear enough. Here are some examples.\n\n1. The 2nd and 3rd row of Table 1 looks the same.\n\n2. In equation (3), the $z_k$ at the end of the first line seems redundant.\n\n3. The dimension of variables is obscure. Something like $A^k \\in \\mathbb{R}^{|S||A|}$ might make it clearer.\n\n4. Some references looks ill-formated. For example, it looks like that line 342-347 are referring to the same paper.\n\nQUESTIONS:\nHere are my questions:\n\n1. How does the analysis techniques compared to the analysis in previous works and why previous analysis cannot achieve the rate obtained in this paper?\n\n2. Is there a sample-complexity lower bound for this kind of actor-critic algorithm?\n\nLIMITATIONS:\nThe author discussed the limitations in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the global convergence of single-timescale actor-critic algorithms in reinforcement learning, for infinite-horizon, discounted Markov Decision Processes  with finite state and actions spaces. In a single-timescale setting, both the actor and critic are updated simultaneously using the same step-size schedule, which contrasts with the more commonly analyzed two-timescale methods that separate actor and critic updates across different learning rates or loops. This makes the analysis more challenging but brings it closer to the practically implemented. The paper achieves a sample complexity of $\\mathcal{O}(\\epsilon^{-3})$ which matches that of state of the art results but does so with less assumptions as compared to other works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n1. The paper proves that a single-timescale AC algorithm achieves an ϵ-optimal policy with a sample complexity of $\\epsilon^{-3}$. This improves upon the previous best-known  $\\epsilon^{-4}$ result for such algorithms.\n2. The paper achieves this under single timescale setup which makes the setup more realistic and applicable to real world applications.\n3. The paper does not require the extra memory requirements for works such as Gaur et. al 2024. Making it more efficient.\n4. Achieves global convergence using Lyapunov related analyses. Improving upon previous results which only got local convergence.\n\nWeaknesses.\n1. The paper is restricted to tabular setups. This means not applicable to environments such as robotics.\n2. In certain situations, using multiple samples to estimate the critic can be beneficial such as DDPG. A discussion of these would be good.\n3. The authors claim that the analysis may be extended to bilvel RL setup. More discussion on that would be good.\n\nQUESTIONS:\nMy questions are related to the weaknesses I pointed out. \n1. Is there any path to extend the analyses to infinite state/action spaces?\n2. What challenges do you anticipate if any in extending your analysis to bilevel RL setup?\n\nLIMITATIONS:\nLimitations are that the analysis is restricted to tabular state/action setup.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}