Title: Non-Asymptotic Guarantees for Average-Reward Q-Learning with Adaptive Stepsizes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Average-reward Q-learning

adaptive stepsizes

non-asymptotic analysis

non-Markovian stochastic approximation.
Abstract: This work presents the first finite-time analysis of average-reward $Q$-learning with an asynchronous implementation. A key feature of the algorithm we study is the use of adaptive stepsizes that act as local clocks for each state-action pair. We show that the mean-square error of this $Q$-learning algorithm, measured in the span seminorm, converges at a rate of $\smash{\tilde{\mathcal{O}}(1/k)}$. To establish this result, we demonstrate that adaptive stepsizes are necessary: without them, the algorithm fails to converge to the correct target. Moreover, adaptive stepsizes can be viewed as a form of implicit importance sampling that counteracts the effect of asynchronous updates.

Technically, the use of adaptive stepsizes causes each $Q$-learning update to depend on the full sample history, introducing strong correlations and making the algorithm a non-Markovian stochastic approximation (SA) scheme. Our approach to overcoming this challenge involves (1) a time-inhomogeneous Markovian reformulation of non-Markovian SA, and (2) a combination of almost-sure time-varying bounds, conditioning arguments, and Markov chain concentration inequalities to break the strong correlations between the adaptive stepsizes and the iterates.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delivers the first non-asymptotic analysis of average-reward asynchronous Q-learning. Due to the asynchronous sampling, the authors show the universal stepsizes cannot guarantee convergence. Motivated by the importance-sampling, the authors design an adaptive stepsize and establish an $O(1/k)$ convergence rate. To establish the convergence, the authors reformulate the non-Markovian update as a time-inhomogeneous Markovian SA and breaks the correlation between iterates and data through two main steps: 1. demonstrating that the span of $Q_k$ grows at most $O(\\log k)$ and 2. controlling the data sequence via conditioning arguments and appropriate Markov concentration bounds.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nS1. The paper delivers the first finite-time performance bound for average-reward asynchronous Q-learning.\n\nS2. It extends classical Lyapunov-based analyses for Markovian SA to settings with time-inhomogeneous Markovian noise—a contribution that, to my knowledge, is new to the literature. The technique appears broadly applicable to a wider class of SA algorithms with adaptive stepsizes.\n\nS3. The use of adaptive stepsizes derived through an importance-sampling argument is particularly interesting.\n\nS4. This is a theoretically solid work. The paper is relatively well-written. The proof seems correct, although I did not check line by line.\n\nWeaknesses:\n\nPlease see questions below.\n\nQUESTIONS:\nIt may be due to my lack of expertise in averaged-reward RL, I have some questions about the related work and the main theorem. I hope the following questions would help the authors improve the paper.\n\nQ1. Theorem 3.1 makes no assumptions on the initial point $Q_1$. Should the constant in the upper bound therefore depend on  $||Q_1||_{\\text{span}}$ (or a comparable measure of its magnitude)?\n\nQ2. This work focuses on the adaptive stepsize: $\\alpha_k(s,a) = \\alpha/(N_k(s,a)+h)$ and achieves a $O(1/k)$ convergence rate with some condition on $\\alpha$. In the Markovian-SA literature (e.g. [21]), one can obtain $O(1/k^\\xi)$ for $0<\\xi<1$ with a broader rule of stepsize $\\alpha_k = \\alpha/(k+h)^\\xi$ without imposing extra conditions on $\\alpha$. Would it be straightforward, or are there hidden technical obstacles, to extend your proof to the more robust choice, with no extra conditions on $\\alpha$?\n\nQ3. I understand that convergence in a semi-norm guarantees only convergence to a set, the paper would benefit from a discussion of how your techniques could be adapted—either through algorithm design or deeper analysis—to produce a consistent estimate of the true $Q^{*}$.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a variant of the Average-Reward Q-learning algorithm with an adaptive step-size based on the counters of each state-action pair. They provide a finite-time analysis for the last-iterate convergence in the span seminorm, based on a single trajectory of Markovian samples. Additionally, they demonstrate that standard Q-learning with a diminishing step-size might fail to converge to the same optimum $Q^*$. The derived convergence rate is optimal in terms of the accuracy level $\\varepsilon$ for $\\varepsilon$ small enough, resulting in sample complexity $O(\\varepsilon^{-2})$.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is well-structured and rather easy to follow. The paper suggests an MSE bounds for the last-iterate for average-reward Q-learning with a single Markovian data trajectory. These bounds are, to the best of my understanding, new. \n\n***Weaknesses***\n\nMain weaknesses, to my opinion, are limited theoretical novelty and overly strong theoretical assumptions. \n\nFirst, regarding the theoretical assumptions - theoretical guarantees, developed by the authors, rely on the Assumption 2.1 that the Bellman operator is a \\(\\beta\\)-contraction in the span seminorm. The authors suggest a sufficient condition (which is a total-variation contraction of the transition kernel in $1$ step), which is overly strong. This assumption, in fact, is even stronger compared to a unifrom geometric ergodicity of transition kernel.\n\nRegarding the theoretical novelty, it seems that significant parts of the analysis adapts recent papers on seminorm-contractive SA [21,22] in the current submission. The contribution would be much clearer if the authors highlight the new elements of the current submission from those which readily follows from the past stream of works.\n\nFinally, although the work is primarily theoretical, it lacks experimental validation of the results. Moreover, the literature review shows that while adaptive step-sizes have been explored in prior works on discounted Q-learning, no works have attempted to apply them in practice.\n\n***Minor points***\nLine 132: I do not understand where \"(to be discussed shortly)\" refers to\n\nQUESTIONS:\n1. Can Assumption 2.1 be relaxed, at least to cover the setting of uniformly geometrically ergodic chain (s_k,a_k,s_{k+1})?\n2. Assumption 3.1 requires a fixed behavioral policy with full support. In practice, behaviour policies are often learned, sometimes in a particular parametric family. How the derived bounds scale with the minimum visitation probability of a state-action pair $(s,a)$, that is, with $\\min_{s,a}\\pi(a|s)$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main theorem is the first to establish nonasymptotic convergence guarantees for the classical asynchronous Q-learning algorithm for average-reward RL. The algorithm deviates from usual Q-learning by using adaptive stepsizes, and the paper also presents formal arguments for why they are necessary for the process to converge to the correct limit, as well as demonstrating that they can be understood via a connection to importance sampling. Their use introduces the main technical challenge of the paper (by causing the underlying stochastic process to be non-Markovian).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper is very well-written and clear. The main result is a fundamental contribution with great significance, and also is the consequence of overcoming multiple conceptual and technical obstacles. At a high level, I think it is great that the paper attempts to study whether adaptive stepsizes are necessary, and I imagine that the related result and discussion may have even greater significance than the convergence analysis of Theorem 3.1. The connection to importance sampling is similarly an interesting and significant contribution. I also appreciate the comparison with discounted Q-learning.\n\nWeaknesses: \nAssumption 2.1 is a strong assumption which is not satisfied in many of the more general classes of MDPs and is crucial to the convergence guarantees of the algorithm.\n\nWhile Proposition 4.1 states that the fixed points of the synchronous and asynchronous fixed-point equations are disjoint, to completely justify the necessity of adaptive stepsizes it seems like we would instead need to understand the distance between the sets of fixed points? In principle if the distance is small (relative to the number of iterations $ k $) then it seems this would not be a fatal issue.\n\n(As discussed after Corollary 3.1.1) the sample complexity analysis appears suboptimal. Specifically, I do not view it as a weakness that the upper bound does not match the minimax lower bound (for a synchronous setting), since this would not be expected, both due to the more difficult setting and the lack of variance-reducing techniques. It is very worthwhile to understand the complexity of Q-learning despite its algorithmic suboptimality. However, it additionally seems that the analysis is very likely suboptimal. It might be interesting to have additional discussion of suboptimal aspects of the proof structure.\n\nWhile the algorithm accepts an arbitrary initialization $ Q_1$, the guarantees do not seem to depend on the distance between $ Q_1$ and some fixed point.\n\nQUESTIONS:\nWhat can be said about the distance between the two sets of fixed points described in Proposition 4.1? (For instance with a universal stepsize schedule such as $ \\alpha_k(s,a) = c/k $)?\n\nCould the authors provide any commentary on aspects of the proof of the main theorem which are suboptimal (in terms of leading to a suboptimal sample complexity)? \n\nCould the theorem statement be refined to provide a guarantee which depends on the fixed-point residual of the initialization $ Q_1 $ or its distance from a fixed-point ? (Would such a term replace some factors of $ 1/(1-\\beta)$ ?)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve the infinite-horizon, undiscounted MDP problem by proposing a provably optimal asynchronous updating Q-learning algorithm. The algorithm adapts the RVI Q-learning [Abounadi et al., 2001] by using an adaptive stepsize that is related to the history trajectory per state-action pair. In measuring the optimality of a policy in the infinite-horizon, undiscounted MDP environment, the authors reformulate the Bellman equation by adapting it to the Seminorm Bellman equation with a seminorm $p_{\\text{span}}$. Under the new proposed seminorm, the optimality Q-function $Q^\\star$ should satisfy the equation $p_{\\text{span}}(\\mathcal{H}(Q^\\star) - Q^\\star) = 0$, where $\\mathcal{H}(\\cdot)$ is a Bellman operator. Solving this equation is equivalent to finding a fixed point/set of fixed points in the space measured by $p_{\\text{span}}$.\nThe adaptive stepsize breaks the Markovian property of the trajectories and makes the standard analysis technique inapplicable.\nThe authors overcome this difficulty and derive the theoretical regret guarantee in the finite-time regime.\nThe authors also present the necessity of introducing an adaptive stepsize by showing that using a universal stepsize will lead the outcome to converge to the wrong target.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written with rigorous theory analysis and clear notations.\n\n2. This is the first work to analyze an asynchronous updating algorithm in the setting of an average reward infinite-horizon undiscounted MDP environment.\nThe introduction of adaptive stepsizes breaks the Markovian property of the trajectories, making the algorithm a non-Markovian stochastic approximation scheme. The authors propose innovative techniques to address this challenge by incorporating the empirical frequencies of state-action pairs in the trajectories of the stochastic process. \n\n3. For completeness, the authors justify the necessity of the adaptive stepsize by showing that using a universal stepsize like harmonic stepsize $\\alpha_k \\coloneqq \\alpha / (k + h)$.\n\nWeakness:\n\n1. The analysis relies on Assumption 2.1, which states that the Bellman operator is a $\\beta$-contraction mapping with respect to the span seminorm. While a sufficient condition is provided, the paper notes that analyzing Q-learning without such an assumption (i.e., under non-expansive operators) is an important direction for future research\n\n2. While the paper acknowledges $h$ as a \"tunable parameter\" in the adaptive stepsize formula, it does not offer practical guidelines or methods for its selection. This leaves a gap for practitioners trying to implement and optimize the algorithm.\n\nQUESTIONS:\n1. For Proposition 4.1, it claims that without choosing a proper $D$, the set of solutions to the asynchronous Bellman equation is disjoint from the solutions to the original Bellman equation measured in the seminorm $p_{\\text{span}}$. How to interpret the condition to ensure the two solution sets are identical: $D(s, a) = I/(|\\mathcal{S}| |\\mathcal{A} |)$? $D$ is supposed to be a diagonal matrix with diagonal entries $\\{ \\mu(s)\\pi(a \\mid s) \\}_{(s, a) \\in \\mathcal{S} \\times \\mathcal{A}}$, does that mean that policy $\\pi$ is a deterministic policy?\n\n2. Another question about Proposition 4.1 is that although the sets of solutions are disjoint with some choice of $D$, are those two sets close to each other?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}