Sharp asymptotic theory for Q-learning with \texttt{LD2Z} learning rate and its generalization
Authors: Soham Bonnerjee
Zhipeng Lou
Wei Biao Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Q-learning
Stochastic approximation
central limit theory
strong invariance principle
Primary area: learning theory

Abstract: Despite the sustained popularity of Q-learning as a practical tool for policy determination, a majority of relevant theoretical literature deals with either constant ($\eta_t\equiv \eta$) or polynomially decaying ($\eta_t = \eta t^{-\alpha}$) learning schedules. However, it is well known the these choices suffer from either persistent bias or prohibitively slow convergence. In contrast, the recently proposed linear decay to zero (\texttt{LD2Z}: $\eta_t=\eta(1-t/n)$) schedule has shown appreciable empirical performance, but its theoretical and statistical properties remain largely unexplored, especially in the Q-learning setting. We address this gap in the literature by first considering a general class of power-law decay to zero (\texttt{PD2Z}-$\nu$: $\eta_t=\eta(1-t/n)^{\nu}$). Proceeding step-by-step, we present a sharp non-asymptotic error bound for Q-learning with \texttt{PD2Z}-$\nu$ schedule, which then is used to derive a central limit theory for a new \textit{tail} Polyak-Ruppert averaging estimator. Finally, we also provide a novel time-uniform Gaussian approximation (also known as \textit{strong invariance principle}) for the partial sum process of Q-learning iterates, which facilitates bootstrap-based inference. All our theoretical results are complemented by extensive numerical experiments. Beyond being new theoretical and statistical contributions to the Q-learning literature, our results definitively establish that \texttt{LD2Z} and in general \texttt{PD2Z}-$\nu$ achieve a best-of-both-worlds property: they inherit the rapid decay from initialization (characteristic of constant step-sizes) while retaining the asymptotic convergence guarantees (characteristic of polynomially decaying schedules). This dual advantage explains the empirical success of \texttt{LD2Z} while providing practical guidelines for inference through our results.

SHARP ASYMPTOTIC THEORY FOR Q-LEARNING WITH LD2Z LEARNING RATE AND ITS GENERALIZATION
Soham Bomnerjee
Zhipeng Lou
Department of Statistics
Department of Mathematics
University of Chicago
University of California, San Diego 2 ouê wudducdddedd
sohambonner wwwwheicceeeeccco....oooooo
Wei Biao Wu Department of Statistics University of Chicago wbwu@uchi cago. edu edu
ABSTRACT
Despite the sustained popularity of Q-learning practical tool for policy deter- mination, a majority of relevant theoretical literature deals with either constant (n = 7)) polynomially decaying (7) nt learning schedules. However, it is well known that these choices suffer from either persistent bias or prohibitively slow convergence. In contrast, the recently proposed linear decay to zero (LD27: Ie. .7(1 - t/n)) schedule has shown appreciable empirical performance, but theoretical and statistical properties remain largely unexplored. especially in the Q-learning setting. We address this gap in the literature by first considering a general class of power-law decay to zero (PD2Z-V: TTt, = 7y( - t/n)"". Proceed- ing step-by-step, we present sharp con-asymptotic error bound for Q-learning with Z-2 schedule, which then is used to derive a central limit theory for a new tail Polyak-Rupperi averaging estimator. Finally, we also provide novel time-uniform Gaussian approximation (also known as strong invariance principle) for the partial sum process of Q-learnin iterates, which facilitates bootstrap- based inference. All our thcoretical results are complemented by extensive numerical experiments. Beyond being new theoretical and statistical contributions the Q-learning literature, our results definitively establish that LD2Z and in general ?D22-V achieve sest-of-both-worffe property: they inherit the rapid decay from initialization (characteristic of constant step-sizes) while retaining the asymptotic convergence guarantees (characteristic of polynomially decaying schedules). This dual advantage explains the empirical success of LD22 while providing practical guidelines for inference through our results.
INTRODUCTION
With the advent of generative AI models and its continuing ascent towards ubiquity, the of reinforcement learning (RL) to train multiple agents to undertake complex sequential decisions seamlessly, has occupied central role in modern learning theory. In that regard, Q-learning (Watkins et al., 1989; Watkins & Dayan, 1992; Sutton & Barto, 2018; Chi et al., 2025), represents classical, yet practically relevant model-free approach to estimate the optimal policy of Markov decision process (MDP). Research on the statistical properties of the Q-learning algorithm has been extensive; particular, treatment of asymptotic and non-asymptotic error bounds have ranged from techniques particular to synchronous Q-learning (Jaakkola et al., 1993; Tsitsiklis, 1994; Szepesvári, 1997; Shi et al., 2022), to the more modern lens of stochastic approximation (SA) algorithms (Chen et al., 2020b; Qu & Wierman, 2020; Chen al., 2021). Specifically, these latter works cast the Q-learning algorithm as an SA targeting the Bellman equation, and thereby. more general tools can be employed to derive finer theoretical results on these algorithms. This direction also has been, arguably, adequately explored with central limit theory, and functional central-limit wwt--liiimmtttttttett appearing in (Xie & Zhang, 2022; Li et al., 2023b;a; Panda et al., 2024). A special case of Q-learning with singleton action

---

space, is the Temporal -ifffreeee (TD) learning, for which Berry-Esseen theorems and subsequent Gaussian approximations and bootstrap strategies have been discussed (Wu et al., 2024b; 2025; Samsonov et al., 2025).
A very important, but often ignored aspect in these theoretical studies is the choice of step-sizes or learning rates. Indeed, it has become widely common in statistical inference literature analyze either the constant learning rates or the polynomially decaying learning rate. Such choices are not without their own advantages; the constant learning rate enjoys experimental evidence of much faster convergence, however a proof similar to Li et al. (2024b) shows that the Q-learning with constant learning rate will converge stationary distribution around the optimal Q": in other words, the asymptotic bias is non-neglig,,,,, and requires further jackknifing to ensure convergence, On the other hand, the polynomially decaying learning rate theoretically attractive; the aforementioned results establishing Gaussian approximations and other inferential results extensively use polynomially decaying learning rate. This choice has been guided by theory of stochastic gradient descent least since (Ruppert, 1988; Polyak & Juditsky, 1992), however its theoretical optimality often masks its excruciatingly slow convergence, as also observed by (Zhang & Xie, 2024). These criticisms have been echoed by the broad stochastic optimization community, leading to recent proposal of linearly decaying to zero (LD22) learning rate 7/,,, = 711 - -t/n) (Devlin et al., 2019; Touvron et al., 2023). Despite requirement of pre-specified number of schedules, this step-size choice achieves balance between the rapid initial dissipation of initialization effects provided by a constant learning rate and the asymptotic convergence guarantees of a polynomially decaying learning rate. this article, we establish a number of sharp asymptotic results for the Q-learning algorithm with this particular learning rate schedule. To the best of our knowledge, our results are the first-of-its-king theory using this step-size for Q-learning; the theoretical results and subsequent numerical exercise definitively showcases the effectiveness and superiority of this learning rate over the ones usually employed in theoretical analyses.
1.1 MAIN CONTRIBUTIONS
The paper develops comprehensive theoretical framework for Q-learning with power-law decay to zero (PD22-1) learning schedules. Our results advance the theoretical understanding of Q-learning and offer new insights into its statistical properties and practical performance. The main contributions are summarized below:
Non-asymptotic concentration inequality. Under standard regularity conditions, we derive explicit non-asymptotic bounds the p-th moments of the Q-learning iterates for any fixed p 2 2. In particular, our Le bounds can be summarized as follows.
Theorem 1.1 (Theorem 3.1, Informal). If Qn denotes the final Q-learning iterate with the PD2Z-v step-size, then it follows that
IQ Q' S exp(-en))Qo Q' 36
where Q" is the long term reward corresponding to the optimal policy n*.
These bounds serve as fundamental tools underpinning the empirical success of Q-learning with PD22-V schedules compared to their polynomially decaying counterparts (Section 3.1). In particular, the exponential decay from the initialization is empirically observed in Figure 1, further validating our theory.
We propose novel averaging scheme that aggregates batch of the most recent Q-learning iterates, referred to as the tail Polyak-Rupp Ruppert averaging estimator, and establish its asymptotic normality (Section 3.2). This is, the best our knowledge, novel contribution in stochastic approximation literature. For the PD2Z-v learning schedules, our simulation (in $5.4) also establishes the superiority of tail PR averaged estimator over the usual PR averaged ones.
Distribution theory.
Strong invariance principle. We establish strong invariance principles with covariance matching for the partial sum processes of Q-learning with both PD2 Z-1 and polynomially decaying learning schedules. This is accomplished via novel construction of the coupling Gaussian process, enabling a more refined probabilistic analysis of the stochastic dynamics (Section 4).
2

---

1.2 RELATED LITERATURE
I Linearly decaying-to-Pe to-7eeo (LD22)) learning-rate schedules have recently gained substantial traction in applications characterized by highly non-smooth or complex optimization landscapes, including state-space models (Touvron al.. 2023), large language models (Devlin et al., 2019; Liu al., 2019; Bergsma et al., 2025), and vision transformers (Wu et al., 2024a). A number of studies further advocate for the so-called "knee schedule" (Howard & Ruder, 2018; Hoffmann et al., 2022; Iyer et al., 2023; Defazio et al., 2023; Hägele et al., 2024; Bergsma et al., 2025), which employs an initial large learning rate (a "warm start") followed by LD22 phase. Despite their empirical popularity, the asymptotic properties of LD2Z schedules remain poorly understood-evar in relatively simple convex problems. To the best of our knowledge, Goldreich al. (2025) provides the first theoretical analysis of LD22 schedules in strongly convex stochastic gradient descent; but their results not directly applicable to Q-lcarning, and they only establish an L2 control of the terminal iterates Qu. This gap in theory presents significant obstacle to principled statistical inference and uncertainty quantification, motivating the need for a more systematic analysis.
1212
Irox 12 11
112
03
1000
200
200
2400
300
Iphondo
Figure 1: Comparison between polynomially decaying (7: = 0.05t -..65) LD22(7; = (.05(1-t/m)) and Constant (n. = 0.05) step-sizes
1.3 NOTATION
In this paper, we denote the set (1...., n by [n]. The d-dimensional Fuclidean space is IRd, with 15 the positive orthant. For a vector a E RR |a| denotes its Euclidean norm. The set of x n real matrices is denoted by _m and correspondingly. for M E RmXx, |M denotes its Frobenius norm. For random vector X E Rd, we denote ||X]| == v P.. We also denote in-probability convergence, and stochastic boundedness by op and Op respectively. The weak convergence is
denoted by %. We write on S bn if an < Cb2, for some constant C > 0, and an x b, bi On < C2bn for some constants C1, C2 > O.
PRELIMINARIES OF Q-LEARNING
Subsequently, consider discounted, infinite horizon Markov Decision Process (MDP) M (S,A, Y, PR). Here = {1...., SS} is the finite state space, A is the finite action space, and y e (0, 1) is the discount factor. For simplicity, we define D 1S Al. We use P S A (S)) to represent the probability transition kernel with P(s' [s, a) the probability of transiting to s' from
given state-action pair (s, a) S A. Let R: : S x A + [0, oo) stand for the random reward, i... R(s, a) is the immediate reward collected in state s E S when action a E A is taken. We represent the distribution P(s'|s, a) using quantile transformation: there exists a measurable function N(s, a, U), where U ~ Uniform( 0, 1). such that
PPN a, U) = 5') = ((sl)s a) for all 8,8 E S and a E A.
Similarly, we can write the reward function as R(s, a, 2)), where U ~ Uniform 1). Let π be a policy, meaning that for each s E S, (-1(s is probability distribution over actions A. Define
3

---

the expected long-term reward
(8, = 1F
Σ
R (St. (,) d0
1=0
Let Q" = (Qea
(x.a) ( where Q max; Q (s a) is the maximizer.
To estimate Q", the Q-function vector Q: RD is updated by (e.g., Watkins & Dayan (1992))
Qu Ne 1t- + Te Q Q0, = Qo:
(2.1)
where B, is the empirical Bellman operator given by
(B.Q)(s.a) R(s, Vi,) + max Q(N(ss, Q
(2.2)
Here U,, E Z, are i..d. Uniform(O, 1) random variables. With a slight abuse notations, define the matrix P E RD si with rows P(s.a).- = (P(s')s, IfII" E R D a projection matrix associated with given policy
, ... (-11S) then we define the Markov transition kernel H" = PII" E DxD
II diag {
3 Q-LEARNING DYNAMICS WITH LD2Z SCHEDULE AND BEYOND
Before introducing our key results on Q-learning with the LD22 schedule and its generalization, is crucial state the regularity conditions that guarantee the validity of the theoretical excursion. In particular, we require the following assumptions.
Assumption 3.1. It holds that E R(s, a)l" < oo for all (s, E x for some 2 2.
Assumption 3.2. There exist xt II and a positive constant L < oo such that for any function estimator Q E RD,, we have
11 -1a0--H'QQQ90 S LQ
Q" 1oos
where (8) = arg m Q(s,a) is the greedy policy w.r.f. Q.
Assumption 3.1 establishes a uniform control over the p-th moment of the reward function. In contrast, often the statistical literature on this topic imposes a severely restrictive condition of bounded reward, usually constrained in the interval [0. 1] or [- 1, 1] (Li et al., 2021; Shi et al., 2022; Panda al., 2024; Li et al., 2024a; Zhang & Xie, 2024: Chen, 2025). We also remark that Assumption 3.1 is objectively weaker than the corresponding bounded fourth moment assumption in Li et al. (2023b). the other hand, conditions of the type of Assumption 3.2 were first introduced
Puterman Brumelle (1979), and have since been employed in Q-learning literature (Li et al., 2023b; Xia et al., 2024) as a means to establish a local attraction basin around the optimal policy x* Interestingly. this also be derived from a mild margin condition, is described in Appendix $9. The corresponding versions of Assumptions 3.123.2 is pervasive in non-asymptotic analysis of SA algorithms (Ruppert, 1988; Polyak& & Juditsky, 1992; Borkar, 2023; Bottou et al., 2018: Chen al., 2020a; Zhu et al., 2023; Wei et al., 2023).
3.1 NON-ASYMPTOTIC ERROR BOUND
Before establishing inferential results involving LD2Z schedules, is crucial to ascertain their non- asymptotic convergence properties. On the other hand, it is conceivable to broaden our view to the class of learning schedules Tt,, = 77(1 t/n)",, , v > 0, which LD2Z but a special case with = 1. This perspective raises another pertinent question; due to the lack of previous theoretical justifications, it is somewhat unclear as to why the linear decay-to-zero less effective, in any sense, compared to some iteration- dependent choice of v. We address both the questions through our first result. For brevity, we subsequently refer to the schedule Th,, = 17(1 t/n)" as the Power-law decay to zero (abbreviated PD27-V).
4

---

Let the Bellman noise be given by
a) = (. (sQ))s
(3.1)
which, via (2.2) immediately implies that Z, are i..d. D-dimensional random vectors. Our first theorem is presented below.
Theorem 3.1. Consider the Q-learning iterates in (2.1). Suppose for some p Z 2, the Bellman
noise satisfies Op =- [P| < co. Then, with the PD2Z-V learning schedule with o > 0, 2 1/p satisfying
2(1 - 7) 7 2)- + 2(p - 1)
it holds that
||Q,, Q'Ipp Sexp ( cant(11 - n 1)) - QQI
v Ci Ci((,, U, 2) 2) Mt
t S n
11 --
+ +2/p- 10
(eam) 5+1
n n= (3.2)
C2((C3, U, 2,22 20041)
t > n
(can) 7tI
where 121-1f02 with = 2(1 - 7), (1 7) + 2(p 1)7?, and Ci Ci(c, C2(c,u,)) 2n
positive constants given by
C((,u,))== 20(x+) + 22 + 1)) and.
C2(c,u.)) == 4PP exp( 2
upt1 [v + 6-- 1)60-1) पंं (cn)
vp + 1 - r(
+
Theorem 3.1 proved Appendix $7.
Remark 3.1 (A sample complexity version of Theorem 3.1). Let N(e, 7, denotes the minimal number of samples required ensure 1Qn.. Q" 11 In the worst case, 0 S 1- from Theorem 3.1, we obtain the following iteration complexity:
1
Therefore,
N(e,,,e) 7,+) - 7 ( 71- log (1Qu (1Qu-11)
+ (1 (1
We note that for large vs the rate approximately matches that derived by Li et al. (2024a). The gap for finite value of example, we do not assume the rewards to be bounded, and therefore, are only constrained to work with finite p-th moments of the Bellman noise. In contrast, Li et al. (2024a) assumes the rewards [0, 1]. which makes the Bellman noise sequences bounded and allows them to use finer tools from subGaussian theory, such as Freedman's 's inequality (in contrast to the Burkholder's inequality which is sharp in absence of boundedness). It is conceivable that in presence of stricter assumption, the worst-case sample complexity can be further improved, but that is non-trivial.
also be explained by the much weaker assumption that work with. For
The non-asymptotic bound in (3.2) is convenient since it covers a general class of learning schedules with explicitly quantified bound. Crucial is also the two distinct regimes with two different rates. We pause for moment parse the bound carefully. In the transient regime with < n Cn =fr, the Le error decays with ar. In particular, for any choice of v > 0, T),, x as long as L S nc for any fixed constant E (0, 1). Therefore, in the early regime, the class of PD22-V learning schedules behave like a constant learning rate while decaying polynomially. The corresponding C2 error displays a diminishing bias, but this constant learning rate is a crucial key to its much faster convergence, pushing towards its convergence regime where
Cz क In this regime the Q-learning chain has converged with an error-rate n. 20041), enabling an early stopping at any steps in [n Cn - n).
The afore- mentioned fast decay, followed by stabilization in the latter phase, is exemplified empirically in Figure 1. For a more detailed insight into this early phase decay, it is instrumental to specify one immediate corollary to Theorem 3.1.
5

---

Corollary 3.2. Under the assumptions of Theorem 3.1, it follows that for all E [12].
1Qnn - Q0 S exp ( e - (
ws V xeiT)), Q-1 -
where Oeg.v hides constants pertaining to C3 and v. We note that at n, the right hand side is minimized at x log, log n.
Corollary 3.2 has some interesting connotations, which will discuss in successive remarks. To initiate our first discussion, is illuminating to recall the following well-known result for the often-used polynomially decaying learning schedules.
Theorem 3.3 (Chen et al. (2020b), Corollary 4.1.2; Li et al. (2023b), Theorem E.1), Consider the O-learning iterates in (2.1) with the polynomially decaying step-size nu x t to, a (1/2, 1). Then, follows that for all E [n].
QQ Q*1 Z exp(-cl'
+01+0
In light of Theorem 3.3, Corollary 3.2 sheds more light on the faster decay of the LD2Z and in general PD22-V schedules in the transient phase.
Remark 3.2. Assume > 0 is fixed. Note that, in particular, when n, i.e. at the final iterate, Q-lcarning with PD22-v schedule instructs that
|Qn, - Q- 2 (op(-4-'n)1Q. +*I+n"
The dominating decay rate in the convergence phase (the second term in the rates on the right) is similar in both PD2Z-V and polynomial decay schedules (n अाナा versus n -a/2;;; however, the effect of initial point is much less pronounced in the former, with exponential rate exp(-cl) of forgetting the initialization for all E [ni] This explains the fast initial convergence of this linearly decaying rate a neighborhood of Q", also seen in Figure In contrast, the polynomial step-size only achieves forgetfulness of exp(-an -pp--tt-oo This explains the competitive advantage of linearly decaying rate over its polynomial counterpart- an advantage that has also been recently studied in the empirical literature (Defazio et al., 2023; Bergsma et al., 2025). To the best of our knowledge, this is the first such theoretical exposition highlighting the benefits linear decay rate LD2Z and its generalization in the context of Q-learning, while building on the previous works of Goldreich et al. (2025) in the more general context of Stochastic Approximation algorithms.
Next, we explore another interesting assertion from Corollary 3.2 regarding the optimal choice и in the class of PD2Z-v learning schedules.
Remark 3.3. The optimal balances the fact that C2(c3, 2) increases with v, while Σоiry decreases with for large n E N. This trade-off yields the threshold P x log, log n, which grows extremely slowly with n, justifying fixed, iteration-independerr choices of in practice. This aligns with the empirical success of v = 1, motivating deeper statistical study under the assumption of constant In particular, to round off our discussion choices of v, state clean result Q-learning dynamics with LD2% 7 schedule.
Corollary 3.4. Under the assumptions of Theorem 3.1, for the LD22 learning schedule it follows for t E [ni],
O 1 n O
V
|Qt, Q'11p exp (-con2-'))(Q Q'I +
Vel 1 Ven
(3.3)
VT,
where 0()) hides constants depending on n and n7
Subsequently, we assume that v is fixed, and move towards sharper asymptotic result beyond L2 control.
3.2 TAIL POLYAK-RUPPERT AVERAGES AND CENTRAL LIMIT THEORY
As means of variance reduction and faster convergence, Polyak-Ruppert averaging (Ruppert, 1988; Polyak Juditsky, 1992) has a relatively long history of application policy evaluation (Bhandari al., 2018; Khamaru al., 2021), Q-learning (Li al., 2023a;b; 2024a) and Temporal Difference

---

(TD) learning (Mou et al., 2020; Samsonov al., 2024; 2025). However, L2 error-bounds reveal a crucial insight into whether usual Galyak-Rupper averaging would ensure asymptotic normality with these LD2Z and PD22-V schedules. Consider
1. Write
1Σ Qt. Σ Q 2 n
1 1 Qr,
1 1 vn Qun
n/2 = An + Bn + Cn. (3.4)
2
n/2
Observe that as long as t S n/2, it holds nu.n 2 20 approximation literature with constant step-size, alone achieve asymptotic Gaussianity. It is not yet clear if C, may achieve Gaussianity individually; at the very least, its 4p convergence Q' is guaranteed through argument similar Theorem 3.1. Therefore, unless one shows that the asymptotic distribution of B,, exactly cancels that of An. is conceivable that the error of "i= Qt, is in effect, much larger compared to Q". This theoretical insight can also be empirically validated (Figure 4). Therefore, it is arguably more prudent to investigate the inferential properties of the term Cn, which we refer to Tail Polyak- Polyak-Rupper Ruppert Averages.
2. Therefore, based on the intuition from stochastic do not expect An to converge to Q*, let
Theorem 3.5. For any constant > 0 and 2 1/p with p 2 2 same as in Assumption 3.1, let
1
Q
Q
cn PTT
(=n- [en 1- 1+1
Grant Assumptions 3.1 and 3.2 for the MDP. Further assume that Qo, Q E K where K is compact set. Then with the PD27-v learning rate for (2.1) with,
2(1 - 7)
< < (1- 2(p 1
there exists a positive definite matrix 2 0 independent of n, such that
[[P Q2, Q")) N(O, E).
(3.5)
Theorem 3.5 is proved in Appendix $7. We remark that an exact expression for Σ is highly intractable, nullifying any direct approach to estimate Σ. In $4 we indicate a direct bootstrap-based approach to perform valid inference.
4 STRONG INVARIANCE PRINCIPLE
Moving beyond the asymptotic normality of the Q-iterates, the primary goal of this section is to further deepen the understanding of their stochastic dynamics and to better characterize the asymptotic distributional approximation of the associated partial sum process by deriving powerful probabilistic tool known as the strong invariance principle. Due to space constraints, we include a broad discussion on the relevant literature in $8. Due to the non-stationary nature of the sequence Qu,1)121. its stochastic dynamics cannot be well captured by the standard Brownian process. Motivated by Bonnerjee et al. (2024), instead propose approximating the partial sum process of (Q...) by that of a non-stationary Gaussian process specifically designed for matching the covariance structure. Specifically, let Nı,..., ...,,. N,, E IR. be i.i.d. centered Gaussian random vectors with covariance matrix Cov(N,) Cov (Zz). Then, in light of (2.1) and the linear approximation
in (7.18), we define the Gaussian process (Yr)221 via Yo = 0 and
Y, = (I 17,,, T/.,,, G)Y, + N,, t 2
(4.1)
where G E RDD Throughout this section, we focus on the LD2% schedule.
Theorem 4.1. Grant Assumptions 3.1 and 3.2 for the MDP. Consider the learning rate PD2Z-V learning rate and grant the assumptions of Theorem 3.5. Then, for all sufficiently large n, there exists
probability space on which one can define random vectors Qi...., Qi such that (Qi,) 2 (Q..) ) and
[ - Q Q'-Yi) vi) = =p( +
max kn SiSn
1=
where Kn = and 0, 1/p are constants.
7

---

Remark 4.1. Theorem 4.1 provides the first strong Gaussian approximation for the partial sum process of Q-iterates with PD2Z-V schedule. In the context of Q-learning, only functional central limit theorem is established Li et al. (2023b) for the polynomially decaying step sizes. A similar time-uniform approximation can also be established for the polynomially decaying learning schedule, which may be of independent interest.
Theorem 4.2. Grant Assumptions 3.1 and 3.2 for the MDP. Consider the learning rate in ntt -tt in (2.1) for yj > 0, 3 E 1/p, 1), where is same as in Assumption 3.1. Then, there exists (No)=1 i.d.d. N(O,T) T) such that, with
Ye = (I G 1.N. Yo- 0, 1, - -
(4.2)
it holds that,
Σ Q Q' -Yi) op 10g//oog))) log SS
max
The key difference between the results of Theorems 4.1 and 4.2 is in the way partial sums are uniformly approximated. It is well-known that the polynomially decaying step-sizes offer attractive asymptotic properties; the optimality of Theorem 4.2, despite being new in the literature, is therefore not surprising. The strong approximation result is also classical in its expression, strongly echoing results such as Komlós et al. (1976). In fact, it can be argued that the approximation in Theorem 4.2 is much sharper than functional CLT approximation Li et al. (2023b). As toy example, consider the vanilla SGD setting, and suppose K = 1. Suppose F(0) = (0 /2) and V ((0, == = u g. In this setting, the Gaussian approximation analogous (4.2)
YG (I Itt, A)Y, + Tt,n 2r 7, ~ N(0 Var(E)). YO =0
(4.3)
Here A = (2 = 1. On the other hand, the vanilla SGD iterates can also be seen as Y,,- P (T - Tht 4)( - (4) + nt,nCe. Therefore, it can be seen that Vr,- - fu and Y. have exactly the same covariance structure, i.e. Cov(Y s.n. Y Cov(Ya,n; Y,,):; on the other hand, even in such a simplified setting, an approximation by Brownian motion, such as that by functional CLT, captures the covariance structure of the iterates (Yi #he21 only in an asymptotic sense. The Gaussian approximation Y. in (4.3) is particular example of govariance-matchin approximations, introduced by Bonnerjee et al. (2024)- but generalized to account for the particular non-stationarity imposed by Q-leaming iterates.
On the other hand, a strong approximation result for PD22-v schedule works on the tail partial sums, much akin to the tail PR-averaged central limit theory. Moreover, the range of the approximation is also limited between k, and n, which may mean n - Lvn to n for the particular case of LD22 schedule. Noticeably, despite the much faster decay from the initialization, for larger values of v, PD2Z-V can also maintain a time-uniform strong approximation for almost the entire range of its steps. Moreover, in polynomially decaying step-sizes, in aiming for the optimality of strong invariance principles, the choice of B s implies that the decay of Q; from the initialization Qo is O(1); there is practically or very slow decay, which results extremely slow convergence the asymptotic regime. In contrast, even when uniform Gaussian approximation is assured, the inherent properties of the PD2 Z-v schedules not affect convergence. Finally, no functional central limit theory is even known for these learning schedules.
Finally, we remark that as an immediate result of Theorem 4.1, 1, for p >>> 2,
n I( Q*)
n
sup #20
P max Ry, StSn
P max Rr, Sttn
Σ
Yi
+0.
(4.4)
Beyond theoretical interest, (4.4) hints at practical, bootstrap-based algorithms for unne-uniform inference. In particular, the estimation of covariance matrix of Q... especially for the PD2Z-V learning schedule, may be significantly non-trivial. However, estimation of I and H= can be essentially done using (2.2) and the fact that BQ" = Q* This hints at an easily implementable Gaussian bootstrap procedure by running multiple independent chains of Y, parallelly. Similar inferential procedures have been proposed in a time-series context in Wu & Zhao (2007), and also more recently in Bonnerjee et al. (2025) in local SGD setting.
8

---

5 SIMULATION RESULTS
In this section, we present some numerical experiments that empirically explore our theoretical results. In $5.2, we compare the performance of LD22 schedule with the polynomially decaying and the constant learning rates, well as the PD2Z-v learning rates with = 2,3. Moving on, In $5.3 we investigate the accuracy of our time-uniform approximations. We also provide some additional simulation studies involving the central limit theorem in Appendix $5.4.
5.1 SET-UP
For each of the experiments, we consider a 4 x gridworld with the slippery mechanism in Frozen- Lake (Zhang Xie, 2024), and four actions (lett/uprighh/down). The discount factor is taken as 0.1. There two special states, A and B, from which the agent and B", respectively. Once an action is chosen according to the behavior policy, the agent moves in the intended direction with probability 0.9, and with probability 0.05 each, it instead moves in one of the two perpendicular directions. If the agent attempts to move outside the grid, it remains in the same state and receives reward of 1. Otherwise, the reward depends on the current state, with r(A) = 10, r r(B) = 5, and r(s) = 0 for all + A, B.
only intend to move to A'
5.2 COMPARATIVE PERFORMANCE BETWEEN LEARNING RATES.
In these experiments, we consider Q-lerrning with initialization at O; since it's clearly evident in Figure that LD2Z massively outperforms the polynomially decaying step size, we focus on LD22 PD2Z-v and constant learning schedules. For the experiments in Figure 2 (Left), we fix 5000, and run 1000 many Monte-Carlo Q-learning chains. Subsequently, for each learning schedules S along with corresponding
considered, plot the mean eπror [Q.,. Qo for 1000 shaded bands indicating one standard deviation. On the other hand, for Figure 2 (Right), we run B 1000 many independent Q-learning chains for each of E {500, 100, 1500, 2000, 2500), and plot the mean error 1Qn, Q" against n, along with corresponding shaded bands.
Clearly the PD22-r/ learning schedules outperforms the constant learning rate, which maintains consistent bias having converged to a stationary distribution. On the other hand, increasing v seems have small effect reducing the error QQ... Q^ oo when the final iterate error Qu,n Q'1oas the performance similar across v E (1, 2,33. This hints at surprising stability across the PD2 Z-1 class, justifying the widespread use of LD2Z schedule.
However, if focus only
1022
5.80
0.30
032
Figure 2: Performance comparison between LD27, PD2Z-V with 2, and constant learning schedules.
5.3 EXPERIMENTS TIME-UNIFORM APPROXIMATIONS.
In this section, empirically investigate the time-uniform strong approximation results in Theorems
4.1 and 4.2. Working with the same 4 x gridworld setting with number iterations n = 5000 as in the previous section, in Figure 3 (Left),
consider the quantiles of maxk, Σr (Q Q" (o)
for the LD2Z step-size T), = 0.05(1 - t/n) and compare them with the corresponding quantiles of max:, St5n Ei Viloo. All the quantiles empirically calculated based on B 500 Monte

---

0153
0.71
Figure 4: Loo error comparison of PR-averaged and tail PR-averaged iterates.
Carlo repetitions. Similarly, Figure (Right) corresponds to the Gaussian approximation in Theorem 4.2 for the polynomially decaying learning rate 1/- 0.05t-0.5... In particular, Figure (Right) also contains the corresponding quantiles of the Brownian motion based approximation (Theorem 3. Li et al. (2023b)). Despite the ubiquity of functional central limit theory, the sub-optimality of such approximation terms of uniform approximation is evident. Together, these experiments establish the accuracy of the time-uniform approximations in $4, calling for their increased use in bootstrap procedures.
Oec appoodeatie
Quantio of
1.5
100
Quattin ot 1n -00-0000
ou -0"104
Figure 3: Q-Q plots of sup-norm distributions.
5.4 CENTRAL LIMIT THEORY IN PRACTICE.
This section is devoted to empirically validating the central limit theory established in $3.2. To that end, we first establish the efficacy the tail Polyak-Rupper averaged iterates (Q..) over the usual PR-averaged versions (denote by Q..) for LD2Z learning schedule. For n E (1000, 1500, ... 5000), we estimate [E[ Qr - [o] and [
Q*l over = 1000 Monte-Carlo repetitions. From the corresponding illustration in Figure 4, the superiority of Q, over Q, is clear.
DISCUSSION & LIMITATIONS
In this article, we develop asymptotic theory for the Q-learning with LD22 and the more general PD22-v learning schedules. Despite their increasing use in generative models, these learning schedules are yet be thoroughly explored in the theoretical literature of stochastic approximation algorithms. To the best of our knowledge, this work constitutes the first one to include systematic treatment of this step-size for Q-learning. Future extensions include the theory for the potential bootstrap algorithm and Berry-Esseen bounds properly quantify the central limit theory.
Moreover, as pointed out by a reviewer, LD22 step-size schedule is applicable primarily in offline reinforcement learning settings with pre-collected datasets, where the total sample size is known in advance. We acknowledge this as the main limitation of the LD2Z schedule when applied to Q-learning. However, methods allow for the case where is mis-specified. Let ng & n denote
10

---

the true sample size, while is used in the LD22 step-size schedule. Then, as long the mis- specification satisfies n - no I av ni for some constant E (0.1), 1), our asymptotic results remain valid. Generalizing LD22 and PD22-V to online RL set-up constitutes interesting direction, and warrants further research.
ETHICS STATEMENT
The research follows ethical guidelines. No human data or ethically sensitive content is involved. All potential limitations and justifications are adequately addressed. We do not anticipate any negative impacts, and as such the paper does not include a dedicated speculative discussion of broader societal impacts.
REPRODUCIBILITY STATEMENT
All the relevant reproducible codes and figures be found in the anonymous Github repository. All the theoretical results and assumptions are rigorously proved and validated in $7 and $8.
AUTHOR CONTRIBUTIONS
All the authors contributed equally to this research.
REFERENCES
Shane Bergsma, Nolan Simran Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hest- ness. Straight to zero: Why linearly decaying the learning rate to zero works best for Ilms. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: /openrvview net. forum?id- hrO1BgHsMI.
István Berkes, Weidong Liu, and Wei Biao Wu. momlós-major-tusndee approximation under dependence. The Annals of Probability, 42(2):794-817, 2014.
Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory, pp. 1691-1692. PMLR, 2018.
Soham Bonnerjee, Sayar Karmakar, and Wei Biao Wu. Gaussian approximation for nonstationary time series with optimal rate and explicit construction. Ann. Statist., 52(5):2233-2317, 2024. ISSN 0090-5364,2168 24-aos2436.
doi: 1..2214/2-aos24433 URL https: /ttp:/oooo//////// 1214/
Soham Bonnerjee, Sayar Karmakar, and Wei Biao Wu. Sharp gaussian approximations for decentral- ized federated learning. arXiv preprint mtx.xxv2555.0122.. 2025.
Vivek S. Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 48 Texts and Readings in Mathematics. Springer, Singapore; Hindustan Book Agency, New Delhi, second edition, 2023, ISBN -79-9981-9998276-99 9911777--999778-81199---87777779 URL https: doii / 1007 htt.://///....000700-----------0
doi: 10.10077978-981-999887777
Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learn- ing. SIAM Rev., 60(2):223-3111 2018. ISSN 0036-1445,10955722000 doi: 10. 10.1137/16M1000177
URL https:tpp//.m /oii www.ooogo/.......... 1137 111111177/611000011.1
Xi Chen, Jason D. Lee, Xin T. Tong, and Yichen Zhang, Statistical inference for model parameters in stochastic gradient descent. Ann. Statist., 48(1):251-273, 2020a. ISSN 0090-5364,216 10.1214/18-AOS//
doi: URL https: //t::///iii /ro/10../.........//./////11//// 1214/18-A0S1800 AOS1801.
Zaiwei Chen. Nom-asymptotic guarantees for average-reward q-learning with adaptive stepsizes. arXiv preprint arXiv:2504118743 2025.
11

---

Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, and Karthikeyan Shanmugam. Finite-sample analysis of contractive stochastic approximation using smooth convex envelopes. Advances in Neural Information Processing Systems, 33:8223-822 8234,, 2020b.
Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, and Karthikeyan Shanmugam. A Iyapunov theory for samite guarantees of asynchronous q-learning and td-learning variants. arXiv preprint arXiv:2102.01567 2021.
Yuejie Chi, Yuxin Chen, and Yuting Wei. Statistical and algorithmic foundations of reinforcement learning. arXiv preprini 244444500..1444,, 2025.
M Csörgö and Pal Révész. A method prove strassen type laws of invariance principle. 1. Zeitschrift für Wahrschhinnicbkkitssheeri und verwandte Gebiete, 31(4):255-255 1975a.
M Csörgö and Pál Révész. A new method to prove strassen type laws of invariance principle. ii. Zeitschrift für Wahrscheiilicckketssheeeee und verwandte Gebiete, 31(():261-20) 1975b.
Miklos Csörgo and Pál Révész. Strong approximations probability and statistics. Academic press, 2014.
Sándor Csörgö and Peter Hall. The komlós-major-tusnáá, approximations and their applications. Australian Journal of Statistics, 26(2):189-216 1984.
Jérôme Dedecker, Paul Doukhan, and Florence Merlevède. Rates of convergence in the strong invariance principle under projective criteria. Electron. J. Probab, 17(16)11-1,, 2012.
Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal linear decay learning rate schedules and further refinements. arXiv preprint arXiv:2310.07831, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- doi: 10.18653/V//199 11111/1199114..... URL
4186. Association for Computational Linguistics, 2019.
https: //oii wtw/ 11653/886/1/r 1-
Uwe Einmahl. A useful estimate in the multidimensional invariance principle. Probability theory and related fields, 76(1):81-101 1987.
P Erdös and M Kac. On certain limit theorems of the theory of probability. Bulletin of the American Mathematical Society, (29))222,, 1946.
Or Goldreich, Ziyang Wei, Soham Bonnerjee, Jiaqi Li, and Wei Biao Wu. Asymptotic theory of sgd with a general learning-rate Preprint, 2025.
Götze and A. Yu. Zaitsev. Bounds for the rate of strong approximation in the multidimensional invariance principle. Theory of Probability & Its Applications, 53(1):59-80, 80, 2009.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, and Martin Jaggi. Scaling laws and computc-optimal training beyond fixed training durations. In Amir Glober- sons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Newral Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, Decem- files/ https//apppe//2/2 Abstract Conference. html.
ber 10 15, 2024, 2024. URL http: /wwprs.pppppppp/aaaeeeeeeeeeeeep nips waips.cc/papeeee hwtppppeeeiileeeaeeeeeeeeeeeeeee
hash/ 8b9 200.10eeesee9e55a2222220aaeeeeee
CC Heyde and DJ Scott. Invariance principles for the law of the iterated logarithm for martingales and processes with stationary increments. The Annals of Probability, pp. 428-436, 1973.
12

---

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Au- relia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Ad- vances Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: /wtp:/aaapssssssaapppee nips ec wwtpptppppppppp.eteeeteeteppeetp pawppaappppa_eae
2022/hash cieefaafff55888703111116e66 Aabp::aattee-atttttt-------- wobotrracc-ooffeeeeecee htmll html.
http:/papr__eeeppeeeeeeeee files/
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Pupers, pp. 328-339. Association for Computational Linguistics, 2018. doi: 10.18653/11/18-10033 URL httph:ttp//// http:///aaaaathlloooooo .rgg /ptg/P115 8-1031/.
Nikhil Iyer, V. Thejas, Nipun Kwatra, Ramachandran Ramjee, and Muthian Sivathanu. Wide-minima density hypothesis and the explore-exploit learning rate schedule. J. Mach. Learn. Res., 24:
65:1-65:37, 2023. URL https:t/p/// //mmmmmmrr../.. wwtp:oorp/pppooo..o http///v///11112 20044/21-055 html.
Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence stochastic iterative dynamic programming algorithms. Advances in neural information processing systems, 6, 1993.
Sayar Karmakar and Wei Biao Wu. Optimal gaussian approximation for multiple time series. Statistica Sinica, 30(3): 199--117,, 2020.
Sayar Karmakar, Stefan Richter, and Wei Biao Wu. Simultancous inference for time-varying models. Journal of Econometrics, 227(2):408 428, 2022.
Koulik Khamaru, Ashwin Pananjady, Feng Ruan, Martin J. Wainwright, and Michael I. Jordan. Is temporal difference learning optimal? An instance-dependen analysis. SIAM J. Math. Data Sci., 3(4):1013-1040, 2021. ISSN 2577-0187. doi: 10.1137720M13311524 URL https:
//doi orriioorg/110 111133332223322222
János Komlós, Péter Major, and Gábor Tusnády. An approximation of partial sums of independent rv' rv'-s, and the sample df. i. Zeitschrift für Wahnscheinlickkeisshheori und verwandte Gebiete, 32: 111-131, 1975.
János Komlós, Péter Major, and Gábor Tusnády. An approximation of partial sums independent rv's, and the sample df. ii Zeilschrift für Wadrechhennicckeeissheeeee und verwandte Gebiete, 34: 33-58, 1976.
Sokbae Lee, Yuan Liao, Myung Hwan Seo, and Youngki Shin. Fast and robust online inference with stochastic gradient descent via random scaling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7381-7389, 2022.
Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Sample complexity of asynchronous q-learning: Sharper analysis and variance reduction. IEEE Transactions on Information Theory, 68(1):448-473 2021.
Gen Li, Changxiao Cai, Yuxin Chen, Yuting Wei, and Yuejie Chi. Is q-larningg minimax optimal? tight sample complexity analysis. Operations Research, 72(1):222-255 2024a.
Jiaqi Li. Zhipeng Lou, Stefan Richter, and Wei Biao Wu. The stochastic gradient descent from nonlinear time series perspective. Preprint, 2024b.
Xiang Li, Jiadong Liang, and Zhihua Zhang. Online statistical inference for nonlinear stochastic approximation with markovian data. arXiv preprint arXiv:2302.076990 2023a.
Xiang Li. Wenhao Yang, Jiadong Liang, Zhihua Zhang, and Michael I Jordan. A statistical analysis of ruppert averaged q-learning. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2261. PMLR, 2023b.
13

---

Weidong Liu and Zhengyan Lin. Strong approximation for class of stationary processes. Stochastic Processes and their Applications, 119(1):249-25)) 280, 2009.
Weidong Liu and Wei Biao Wu. Simultaneous nonparametric inference of time series. The Annals of Statistics, 38(4):2388-242) 2010.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 2019.
approach. arXiv preprint
CR Lu and Qi-Man Shao. Strong approximations for partial sums of weakly dependent random variables. Sci. Sinica Ser. A, 1987.
Florence Merlevède and Emmanuel Rio. Strong approximation of partial sums under dependence conditions with application to dynamical systems. Stochastic Processes and their applications, 122(():386-417, 2012.
Fabian Mies and Ansgar Steland. Sequential gaussian approximation for nonstationary time series in high dimensions. Bernoulli, 29(4) 2023.
Wenlong Mou, Chris Junchi Li, Martin Wainwright, Peter L Bartlett, and Michael I Jordan. On linear stochastic approximation: Fine Fine-grained polyak-ruppert and non-asymptotic concentration. In Conference on learning theory, pp. 2947-2997. PMLR, 2020.
Saunak Kumar Panda, Tong Li, Ruiqi Liu, and Yisha Xiang. Online statistical inference of constant averaee q-learning. First Reinforcement Learning Safety Workshop. 2024.
Boris T. Polyak and Anatoli B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4))838-855, 1992.
Martin Puterman and Shelby Brumelle. On the convergence of policy iteration in stationary
dynamic programming. Mathematics of Operations Research, 4(1)x60-69, 69, 1979.
Guannan Qu and Adam Wierman. Finite-time analysis of asynchronous stochastic approximation and g-learning. In Conference Learning Theory, pp. 3185-3205. PMLR, 2020.
David Ruppert. Efficient estimations from slowly convergent robbins-monn process. Technical Report, 1988.
Sergey Samsonov, Eric Moulines, Qi-Man Shao, Zhuo-Song Zhang, and Alexey Naumov. Gaussian approximation and multiplier bootstrap for polyak ruppert averaged linear stochastic approximation with applications to td learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Sergey Samsonov, Marina Sheshukova, Eric Moulines, and Alexey Naumov. Statistical inference for linear stochastic approximation with markovian noise. arXiv preprint arXiv:2505.1 1910005./9900 /9/02, 2025.
Qi-Man Shao. Strong approximation theorems for independent random variables and their applica- tions. Journal of multivariate analysis, 52(1):107-13, 1995.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International conference on machine learning, pp. 19967-20022 PMLR, 2022.
Volker Strassen. An invariance principle for the law of the iterated logarithm. Zeitschrift für Wadhscheennicckkiissteenn und verwandte Gebiete, 3(3):211-22) 1964.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, second edition, 2018. ISBN 678-0-262-03922-6.
Csaba Szepesvári. The asymptotic convergence-rate of qelaarning Advances in neural information processing systems, 10, 1997.
14

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXtv:2302.3397, 2023,
John N Tsitsiklis.
Asynchronous stochastic approximation and Q-learming, volume 16. Springer,
1994.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292 1992.
Christopher John Cornish Hellaby Watkins et al. Learning from delayed rewards. 1989.
Ian -hiut, David Arbour, Ritwik Sinha, Edward H Kennedy, and Aaditya Ramdas. Time- uniform central limit theory and asymptotic confidence sequences. The Annals of Statistics, 52(6): 2613-2640, 2024.
Ziyang Wei, Wanrong Zhu, and Wei Biao Wu. Weighted averaged stochastic gradient descent: Asymptotic normality and optimality, arXiv preprint arXiv:2307.069115 2023.
Shuang Wu, Guangjian Zhang, and Xuefeng Liu. SwinSOD: Salient object detection using swin- transformer. Image Vis. Comput., 146(105039):105039) June 2024a.
Wei Biao Wu. Strong invariance principles for dependent random variables. The Annals of Probability, pp. 2294-2320, 2007.
Wei Biao Wu and Zhibiao Zhao. Inference of trends in time series. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(3):391-410, 2007.
Wei Biao Wu and Zhou Zhou Gaussian approximations for non-stationary multiple time series. Statistica Sinica, pp. 1397-1413, 2011.
Weichen WU, Gen Li, Yuting Wei, and Alessandro Rinaldo. Statistical inference for temporal difference learning with linear function approximation. arXiv preprint arXiv:2400.16106, 2024b.
Weichen Wu, Yuting Wei, and Alessandro Rinaldo. Uncertainty quantification for markov chains with application to temporal difference learning. arXiv preprint arXiv:2502.1382) 2025.
Eric Xia, Koulik Khamaru, Martin J Wainwright, and Michael I Jordan. Instance-optimality in optimal value estimation: Adaptivity via variance-reduced q-learning. IEEE Transactions on Information Theory, 2024.
Chuhan Xie and Zhihua Zhang. A statistical online inference approach in averaged stochastic approximation. Advances in Neural Information Processing Systems, 35:8998- 90099,999,,,---00 2022.
Chuhan Xie, Kaicheng Jin, Jiadong Liang, and Zhihua Zhang. Asymptotic time-uniform inference for parameters in averaged stochastic approximation. arXiv preprint 5057, 2024.
Yixuan Zhang and Qiaomin Xie. Constant stepsize q-learnin:: Distributional convergence, bias and extrapolation. arXiv preprint arXiv:2401.1388, 2024.
Wanrong Zhu, Xi Chen, and Wei Biao Wu. Online covariance matrix estimation in stochastic gradient descent. J. Amer. Statist. Assoc 1 118(541):393 404, 2023. ISSN 0162-1459,1537-222
Wanrong Zhu, Zhipeng Lou, Ziyang Wei, and Wei Biao Wu. High confidence level inference is almost free using parallel stochastic optimization. arXiv preprint arXiv:2401.09346 2024.
15

---

7
APPENDIX A
In this section we collect the proofs of Theorems 3.1 and 3.5.
Proof of Theorem 3.1. Denote Ann == Qrn - Q' Then, it is immediate that
- A (1 - nt,n ( Qt--1n - Q*) + T/t, Qt,Q B(Q*))
= A + 11,n 11 + 7 (Me -( (11 - 11=1111 )0=-11))))))
(7.1)
where A, = - Mhnn G, and Min = (Pr - -P)(V - V*). From the definition greedy policy, it follows that =-1 1d ) < 0, where I and 2 are interpreted element-wise. Therefore, clearly
A S(T nn,n -H A Ve,n (Zx M,,,,,
which directly yields, via Proposition 4 of that
| 11 <(1 T/t, - + 2(p 1) +
<((1 mt,n - -)) 2(p 1)mi. 1 + np
with o = 2pp-1ppp) //0 Recursively, it holds that
A 1 Ag + Cp
Σ 1 n
(7.2)
8=1
where A² = I[ (1 - 71 + 1 02), where c1 = 2(1 - ((1-)),0 = -1--)² 12p-2ppp10 1 From
the choice of 77 satisfying 7/1 c/222
0, we can derive
1 x A
II (1- MgnnC,)) j=++1
for some small constant c3 (0, 1). In light of Σj=1 M1. 1/((1 n-1)², have AB
- -1n---n(1nnn---- Therefore, applying Lemma 11.1 the proof is completed.
Proof of Theorem 3.5. We consider deriving the Gaussian approximation through a series steps. In particular, proof strategy is to linearize the Q-learning iterates before applying suitable, off-the off-the- shelf central limit theory. The steps till linearization are not straightforward, especially in light of the complications arising out of PD2Z-1 learning rates. In particular, the non-linearity of the Bellman operator requires careful tempering. We provide the formal proof in the following. Throughout the proof, we let hi = n [cn=+I].
7.1 STEP
Let Q = Q", and define the oracle Q-learning iterates
Qi (1 n/,n Qi + n,, B, t 1.
(7.3)
Note that
Q - 1o < (1 - nt, ( - Qi + mmn B.. I
BiQi
< (1 M., (11))11)]]-11... Qi-1,nlo
< 1% 7) Qu Q*1
(7.4)
where for > 0, (c) = M7 (1 Mj,nc), and the second inequality in (7.4) follows from the contraction of Bellman operators (2.2). Elementary calculations show that Yg 7) S
16

---

for some 0, which implies, via (7.4), that
F[ Qn
Qii
S n (n kn)
Qtn
Qi -f,1
==kn
< n [[+1]]
exp( ct) dt
= O(n xلंي) almost surely.
(7.5)
Therefore, Step enables us to investigate the asymptotic properties of Q.
7.2 STEP II
Define the empirical version of P as
Pe((s,a),) = (1,, 14
(7.6)
In other words, P. E Px|S| is matrix with hot-hot-coded rows. Moreover, let
(s) max (s, and (*) max Q²(ssa))
(7.7)
with Vr, (V),n))) E BIS and V* likewise defined. Note that, P.V1-1.n = max Qt-1.n (N (s, U,), e'EA
and PV--1,n E[P,VV 1, F where Fi-1 is the a-field induced by the random variables (U,, V.)agt. Clearly, PV* = El ma Q" (N (s,a, (D),"")),, U ~ U(0, 1]. Observe that
B,Q BQ B,Q B²Q² Ze
(7.8)
-PPoVo-1.n V*) Ze
(7.9)
=2(Mux (H H" ) + 7H" (Qi-1, - -Qn) Zt;
(7.10)
where (7.8) follows from Z, = B BQ"; (7.9) is implied by (2.2), and (7.10) is obtained after defining Mtn (Pr (V, V*). Note that, in partieular 7, are mean-zero i.i.d. random variables, and (Man))11 a martingale difference sequence. Now, using B(Q*) = Q* and (7.8)-(7100), rewrite (7.3) as
1. = Qi,n Q (1 - ne, (Qi Q') Th,n (B, B(Q"))
A1 + T1 Ze + Mr.n (Me, + (H*? i.a H" )Q (7.11)
where Ar, = n,, G, = I - -HH² and Ao = Define another "sandwich" sequence as follows:
(1) i, A,A 1,n + nt Zi + M Meni 46" =0 0.
(7.12)
Following the property of optimal policy, it immediate that (H H== "=""Q.-.. 2 0, and hence,
(L) < Ann-
(7.13)
Moreover, it follows that
E|JA,
(E) [o] 5(1 17,,0 - -1) A 1-1
4-1, loo El (H H QQ¹ 1,ulo]]
(1)
<(1 77. (1 - A,,
A [o0] 71T 11"
(1)
) A,
(7.14)
5(1 Th.n 73)
A (1)
1-1. bo] LTT, ETI 12
(7.15)
Ms T Q., Q
6p
T m
n
As 1' S n si..
(7.16)
8=0
17

---

where (7.14) follows from noting that (H H 0 0; (7.15) follows from Assumption
3.2, and (7.16) involves an application of Theorem 3.1 and Lemma 11.1. Clearly, (7.16) produces
n शामन E A. A [ool O(n _लमा
n 26+ (An. A _ O. which implies that
(7.17)
7.3 STEP III
In this step, we will show that both A well-approximated by linear process. To that end, further define
Xt, = A, + 11 Zr. Xo = 0.
(7.18)
With this definition established, we can proceed to approximate A (L) (L)
by Xt,- Indeed, with A
X E R.P.
(2 n SD B = E|[(! - Me, - )) (2) + "I.
$ (1 m1 - -1) ( - @ + + 2E| 2 - V* 21
2
A
Ve-1-
A
Σ
S n 2,
(7.19)
B=k,+1
where the second equality uses the fact that Me,,. are martingale differences; the inequality in the third assertion involves (i) using that - is stochastic matrix to deduce nt... - -- 1 - n,., (1 - 7), and (ii) using that both P. and P are stochastic matrices to obtain P. - Pl S 2;
the final assertion invokes Theorem 3.1 and Lemma 11.1. Equation 7.19 immediately results
n PP A Xxbzl=
2OFTI
v E R O(n ....))
which, similar to (7.17) implies that
0(++1) (ALL)
Xx)
B O.
(7.20)
7.4
STEP IV
In light of (7.5), (7.17) and (8.3), the proof is complete if one derives a central limit theory of X, (n- kn Ei=k, Xt,- To that end, re-write
Σ X
no VenVen Σ s=1
Van
Σ
A 8.n
t=
t=sVkn
where At =[[}=0t1 Ajn.
We proceed step-by-step. Let Teg,n V hin Firstly, note that
Er (n 1)4
Σ n V Va S nFII Σ
Σ
A
/.r S noii
Non A s o On
11"
t=b,att
(7.21)
which establishes the Lindeberg condition that n 1) max, ns Va.nl = O(1). Now we shift focus showing that
W, = n Ms V. IVV. a. + E
18

---

for some Σ > O. Write
W, = (1 (1-1/n) -T Whr + Rn:
where
Rn =
(Cam - -omm-1) sn- Cennl((os,, Can-1) Con nen Van
(7.22)
The proof follows by showing that nR, is Cauchy sequence in Rdxd through an argument mimick- ing Lemma 11.1, and we omit the details for brevity. Finally, our conclusion follows from (7.21) via Lindeberg-Felle Feller central limit theory.
8 APPENDIX B: DISCUSSION ON STRONG APPROXIMATION OF Q-LEARNING ITERATES
Related Literature. The method of invariance principle was introduced by Erdös & Kac (1946) and has since been extensively studied, serving powerful tool for analyzing distributional properties in wide range of statistical inference problems (Csörgö Hall, 1984: Csörgo & Révész, 201 4). Applications include nonparametric simultancous inference (Liu & Wu, 2010; Karmakar et al., 2022), change-point detection and inference (Wu & Zhao, 2007), online statistical inference (Lee et al., 2022;
Zhu et al., 2024: Li et al., 2023b), and construction of time-unifome confidence sequences (Waudby- Smith et al., 2024; Xie et al., 2024).
For independent and identically distributed (i.i.d)) random variables, Strassen (1964) initialed the study of almost sure approximation for the partial sums by Wiener process, and was later refined by Csörgö & Révész (1975a) and Csörgö & Révész (1975b). The optimal strong approximation in this setting was established the celebrated work (Komlós et al., 1975; 1976). Specifically, let En, E R be i.i.d. centered random variables with Var(š1) r² and E| < co for some constant p 2. Then, for the sequence of partial sums {S.]_ where S. EE= E₁, there exists probability space which
define random variables E -.,-..; En with the partial sum process S = ビ Sg. f. 2 1, and a Brownian motion BB() such that (5 2 (S,)²= and
ST - BB()) = Oa. (n...("'' (P).
max Istsn
Extensions this result to multidimensional independent (but not necessarily identically distributed) random vectors has been developed by Einmahl (1987), Shao (1995), Götze & Zaitsev (2009), among others. Another line research, more relevant to the online learning where the outputs may exhibit temporal dependence, has focused on generalizing the above strong approximation dependent data; see, for example, Heyde & Scott (1973), Lu & Shao (1987), Wu (2007), Liu & Lin (2009), Dedecker al. (2012), Merlevède & Rio (2012), among others. A notable contribution in this direction was made by Berkes et al. (2014), who established the optimal strong approximation for a broad class of causal stationary sequence (E: Under mild regularity conditions, they proved that
max IS. B()) = (n.)), IstEn
(8.1)
where 2 = ELez Cov(Eo, (x) = lim Var(Sn)/n. stands for the long-run variance. This result implies that the process {a (B)) can preserve the second-order properties of {Sthe2 asymptotically.
=
However, in the context of Q-learning with time-varying step sizes, these results do not apply due to the nonstationary nature of the iterates (Qu.ahe1 defined in (2.1). Unfortunately, strong approximations for non-stationary data remain relatively underexplored. Some contributions include Wu & Zhou (2011), Karmakar & Wu (2020) and Mies & Steland (2023), which lead the following result: there exists Gaussian process (Gr)21 such that Cov(G,, G.) R Cov(S,, S.) and SS Gel = (r(Tn).
max I8t
(8.2)
Compared to B()) in (8.1), this more general {9.} can better capture the dependence structure of (St). as it allows potentially non-stationary increments (G. Ge-1)e21- However, until the recent
19

---

work of Bonnerjee al. (2024), it remained unclear how to explicitly construct such process with optimal convergence rate. They provided an optimal Gaussian approximation of the form (8.2) with optimal Tn // and an explicit construction of the coupling Gaussian process (Gi)- Motivated by this, one of the main objectives of this paper is to derive optimal Gaussian approximation for the coupling Gaussian process. is important to
Q-learning, including an explicit construction note that the dependence structure of (Qt.n 722 is significantly more complex than that considered in Bonnerjee et al. (2024), and thus their results are not directly applicable.
Now proceed to the proofs of the results in $4.
Proof of Theorem 4.1. From equations (7.4), (7.16) and (7.19) also follows that
71 Σ Q' X x,n) Op(1).
max SISD
(8.3)
Note that (7.18) can be cast into the following form:
X Σ A Zs.
(8.4)
where A.,. = -[]=stt Ay.. 8, O, and A == for 2 Moreover, using Theorem of Götze & Zaitsev (2009), on possibly enriched probability space, there exists N, i.dd. (0, such that
max Leten
(2, (o,-N,,1=0 N,)1 op(n
(8.5)
==1
one defines Y, as
Y1 (I nn - -I- 1) + ne,bbb.
then, for f 2 knn
Σ (X1 - Yi) = Σ
Nan Al 12, -8.)
1=t
1=t 8=1
Nsn A 8 1,0 -No)
==1 l=avt
Ma (2, + Σ Σ Ten Ae N.).
(8.6)
8=1+1
Let us tackle the terms in (8.6) one-by-one. In particular, similar treatment as Lemma 11.1 provides that for all [n]
max =I = 0(1).
max k St IssSt
1=t
Denote L n7 Et
1, Therefore, for the first term in (8.6), one obtains
t
max k, 515n
Σ
ns.n A 9- 1, (Zs -Na)lao
1=t
max k, St50
[T [T +
Σ
rlt 8.2
pd
max SESn
Σ (Z, N.)
00
2=2
(0 1 v42))
(8.7)
where the or assertion follows from (8.5), and the ni rate appears due to
r s,n
Pt 9-1,n
[1s, 1/s 1,0
The 1,m
A
e-1,n
17
7s.n
20

---

upon which invoke Lemma 11.1 along with t 2 kn and the elementary bound max, E.
Zam dh
< log n. The assertion for the second term follows from noting
na
Σ A
max max Te, Ry Sign S
S
max max Ta kn Stnn ISESE
after which we follow a similar argument as above. This completes the proof.
Proof of Theurem 4.2. We follow a proof similar to that of Theorem 4.1. Since the learning rates no longer depend on the number of iterations omit the from the subscript.
8.1
STEP
Similar to Step in Theorem 4.2, elementary calculations show that Y0 - -7) Sy exp(-ct! for some с 0, which implies, via (7.4), that
Σ(.. 29) S
Σ Q
-Q S (-" exp((ctt-0) 0(1) almost surely. (8.8)
max ISA
s=1
8.2 STEP II
In this case, follows that
BJJ,- - A (1) [o0] 5(1 - 7/c (1 - -)) - 4 E| -
H" 16
<(1 77c - - ( HJ 111110
<(1 (1 -)) 4 |0] + B
<1 n - -1))))))[/4 -- - A [00] + L
(8.9)
where (8.9) involves an application of Theorem E.2 of Li et al. (2023b). Clearly, in lieu of 3 (8.9) entails
1-1/p.
B[A
A (L) [c] O(n),
which produces
Σ A 11)) opp(11/).
(8.
Istšn
8.3 STEP III
In this step, we have,
E E155 12 S. 51 E[[ mi -
< (1 Tk( (1 7) 7) 1 +m 23
V*
$ (1 (1 - -1))) 5 + O
(8.11)
whereupon one invokes Theorem E.2 of Li et al. (2023b) to conclude E[|A,- = Om)). Equation (8.11) immediately results in
Σ (A (L) x = Oz(n' op(nn//)
max ISES 8=1
(8.12)
similar (8.10).
21

---

8.4
STEP IV
This step also follows similar to that of Theorem 4.1 by denoting Ban 11s Mij A and observing
max IstEn
E 10
max 24 ISt5
( 8,) = or(n"/Plogm) (8.13)
where
Ω = B1 + Σ Ba,t - B.-1,1500
#=2
and the log comes from Proposition of Bonnerjee al. (2025). Note that by construction,
(Xi)22 (Xt)121. The proof is concluded by combining (8.8), (8.10), (8.12) and (8.13).
APPENDIX C: DERIVATION OF ASSUMPTION 3.2
The key insight behind the Assumption 3.2 is ensuring that the optimal policy, and the optimal quality function is unique. In that regard, consider the following simple margin condition, that can be more illuminating in the Q-learning context.
Assumption 9.1. The greedy policy rs)) arg max, Q' (s. a) is unique for every state s, and satisfies
A = m=minnQ((( (s, ( (b)) - max ax)
Q' Q' a)) O.
Under Assumption 9.1, we derive Assumption 3.2. To that end, suppose IQ Q% S A/2. Then, by definition of the greedy policy, H = H,,, and hence, Assumption 3.2 is trivially satisfied. On the other hand, if Q - 10 > A/2, then from = 11 I 2 it follows
(H - H (Q - 2*) s 2|Q - --0-2%= S
Q Q7
A
10 ADDITIONAL EXPERIMENTS
In this section, we work with large discount factor 7/ 0.99. and consider the step-size choices polynomially decaying, LD2Z and PD2Z-V with v 2, 3. Firstly, we consider 20000 number of Q-learning iterations, and look special case of polynomially decaying step-size, viz. the 0.25/t. Based on B 500 Monte Carlo repetitions, we plot empirical estimates of [B[Q Q [x] against t E [n] for the LD22step-size 7}: = 0.25(1 t/n) 2, 3, and compare with empirical estimates of Q [x] for the linearly decaying step-size, where Q, 1-1 - Q denotes the running Polyak-Rupped average. can be seen in Figure 5 that as per our intuition and previous
linearly decaying step size
and PD22-1 step-size choices 7ı 0.25 t/nn) with
10-077 10-0"%x
202 stidet
202 -6220--00
Concoon
200
50
10000
15000
Figure 5: Comparison between different step-size choices.
results, neither the end-term nor the PR-avergaed iterates have converged even after 20000 iterations
22

---

for linearly decaying step-sizes; they will eventually converge, and will eventually obtain better asymptotic approximation error compared to LDTZ or PDTZ stepsize choices, but this asymptotic regime kicks in much, much later than is often realistically possible in many scenarios. We can also replicate corresponding versions of Figure 2 for ? = 0.99 with this particular setting, which we report below.
USD
10200
Figure 6: Performance comparison between LD22, PD2Z-V with 2, and constant learning schedules.
10.1 AFFECT OF LEARNING RATE CONSTANT
To further validate the efficacy of our learning rate schedules, we consider the effect of leading
constant y in the performance the Q-iterates. the following, we consider our x 4 gridworld
with discount 7 0.99, and the following step-sizes: polynomially decaying: 7) Λt 0.55; constant: 7 LD2 7: 7/- 17(1 1/n): PD22-V-2:
17(1 - 1/n)²; and PD2Z-v-3: Vp- = n - 1/n)².
We vary 1) E {0.1,...,, 0.9). For each choice of y and learning-rate, epsiodes averaged over 100 initial episodes (for the initial phase), and 1000 final episodes (for the asymptotic phase). The averaged total rewards
the Q-learning iterates for
T' = 20, 000 episodes, and report the sum of rewards
are further averaged over 500 Monte Carlo runs for stability.
Alpha Sweap: All Methods Comparison
Average Epioadd
taant 1ed (r))
Look B4d sex
0uo 1i0
than
63
86
38
Alpha (Leaming Rate)
Figure 7: Total sum of rewards on an average reward for initial phase and asymptotic phase for different learning rates and different n
In Figure 7, the solid lines correspond to the initial phase, and the dashed lines correspond to the asymptotic phase. It is clear that the polynomially decaying step-size least performing in the initial stage. Moreover, the fact that Q-learning constant learning rate does not converge, is also evident, as larger learning rate constant constant results increasing bias. In comparison, both the Zand PD2 PD27-vlearning
after 50000 episodes, its asymptotic phase hasn't 't kicked in. On the other hand,
23

---

rates maintain performance comparable to the constant learning rate in the initial phase, while providing convergence in the asymptotic stage.
10.2 ADDITIONAL. SIMULATIONS ON CENTRAL LIMIT THEORY
We investigate the asymptotic normality of Q.. For = 5000 and 10, 000, we compute Q Q*, and project them along 6 randomly chosen directions u E gd-1 For each random direction u, the empirical quantiles of n 2 (Qn Q*) generated based on B = 1000 Monte-Carlo repetitions visualized in a QQ-plot against the corresponding quantiles from standard normal distribution. The asymptotic normality is apparent from the QQ-plot being straight line. The accuracy of the
scaling n 1/4 also evident from the two QQ -plots, corresponding to n = 5000 and n = 10, 000, being virtually identical.
Sample quantles
0.05
Sample quantles
Sample quantiles
0.05
0.0
-0.05
-0.05
-0.10
-0.
Standard normal quantiles
Standard normal quactiles
Standard quantiles
0.10
Sample qucatiles
0.02
Sumple quzatiles
0.02
Sumple quentiless
0.05
0.00
0.00
-0.02
-0.04
-0.55
-0.01
Standard normal quontiles
Standard bormel quoctiles
Standord bormel quontiles
Figure 8: QQ-plots of u T (Q Q*) for randomly generated unit vectors and 5000.
0.10
0.15
0.2
Sample quanties
Sample quantiles
0.10
Sample quantiles
0.05
0.1
0.00
0.00
0.0
-0.1
-0.05
-0.10
-02
-0.15
Standard normal quantiles
Standard nommel quactiles
Standard nommel quantiles
0.04
0.10
0.04
Sample quantiee
Sample quanties
Sample quantiees
0.05
0.02
0.00
0.00
0.00
0.00
-0.04
-0.04
-0.10
-2
-2
Standard normal quantiles
Standard pormal quactiles
Standard normal quantiles
Figure 9: QQ-plots of 1/1/4u(Q Q-) for randomly generated unit vectors and 10000.
24

---

11 AUXILIARY RESULTS
In this section, we collect some key mathematical arguments that we have repeatedly used throughout proofs.
Lemma 11.1. Let A², [ (1 75,,nC) for some small e E (0, 1), with 77 17(1 )
7 > ne < and v 2 1. Then all p 2 1,1 [n], it holds that Ci(c
t 5 n
11 v+1
(en) T
C2(c,u,p)) in (1
t n
12 1+1
(en) ==1
where Ci v.) and V, are defined as in Theorem 3.1.
Proof of Lemma 11.1. Our proof proceeds through a series of steps by first establishing a uniform bound on Ar, and then carefully establishing control on E n on case- uuse-by-case basis. To
that end, let J [2) = (1 - u/n), u E [0, n). Observe that u + J (u)" is non-increasing function for any v 2 1. Therefore, for any s < E [n]. follows
Mi. 21 19 1 J(u)" du
mm (J () J(t I(++ 2 I
5),
j=++1
(11.1)
where the final inequality in (11.1) follows from the non-increasing property of J. Consequently, one can use (11.1) to derive that
A S exp( -3
C3
Σ min) S exp(-con -))
(11.2)
j=#+1
This completes the first step of our argument. Moving on, we use (11.2) to derive sharp upper bounds on E.=1 A2. This can be approached as follows.
Case 1. 1.21 n
(can) 5t1 nI. In this case, we proceed:
Σ non S rp Σ ( expl-cs, mm
((s+1)+t--Ti 1)
v +1
(n exep -33l -" = pppn" up
v ++ ((n - 8 1) (n 1)tt1))
n-1
1PP exp ( (ca) -
= 1f'n vp
((k 1) - (n- - - 1)
シ 1
S 18 a4
(u + 1)88 exp ( - C3l n
(+1 (n t 1) nu
+1
S 1144/p aA exp(
C37
(n - -- 1)u41
(n + 1) expl ccsl v +
v ++ du
1/ *+1 1
n2
< 2 vp exp
20+1
(1 + 1)(0-1)) (10--1)5
(cal)) WF
up +
nn पन (up+1) (11.3)
1/ +
< 4 exp(
2u+1
(( 1)(0-1)
पंन (ca7)
veth PFI
r
up +
)n -
(11.4)
+
where in (11.3) we have invoked n t Case 2: L. L.5. n
न.
(csn) μ+1
(can)
First observe that,
25

---

A S pp
Σ (xs)><<p((~3)TT) 11 - s))
1-8
np?
J[t k) exp( cc33J 1)(k)
1 / exp(-csmT(t+
(I(t)
(11.5)
exp( canJ + 1)9) / (I(t) + " exp(-cs) 1) du (11.6)
can (t + 1)
np
n
20p-1
[(t)P² [
(c)m exp( -
de)
exp( canJ (t 1)9)
(11.7)
2up-1
[[()]T (L)m)
(11.8)
exp( expp(c33000 C3nJ 1)
where (11.5) follows from noting [[ k) I()) + #; (11.6) derives from an application of Lemma 11.2; (11.7) is obtained by the elementary inequality (x + y)a < 2x" + y²) for 2 1. Finally, in (11.8), I'(-) denotes the Gamma function. The two terms in (11.8) following the leading constants are particularly interesting, the first term increases with f, and the second term decays with t. The interplay between these two terms will naturally lead to two regions which the rates will be controlled case-by-case
Now, recall that in this particular regime, it is immediate that J 2 2n-1 T(t) Moreover, since n 2, it follows that in this regime, J[t + 1) 2 (t)/22
is sufficiently large such that Therefore,
2
(can)
J(L) + (1)))(++)))))) 1) J() 2T + 1)),
which, when plugged in (11.8), implies that
20p-1
Σ 4 S 1p
(()))(1+2-PT(0)) 1))
exp(-can 1)1)
21( + +2 + 1))
ni
p-1
(11.9)
where in the final inequality we have used ca7) < 1 to deduce
(= (t 1)2
2
2
Finally, (11.4) and (11.9) completes the proof.
Lemma 11.2. Let f : + RR, be non-decreasing function and let K. >> O be a constant such that
Em f(n) expeeep((ppp < oc. Then
Σ f(n) exp(-kn) expl -k) 10 [(u) exp(-ku) du.
20
Proof. Since f is non-decreasing, hence for every N.
(() exp(-km) exp(-xi -)) -- ((m) 1m n+t
du 5 1 c R) Jo
f(u) expeeexxxx du.
26


Reviews:
[{"review_id": "iO0VSwfx17", "reviewer": "Reviewer_632X", "summary": "The paper studies sample complexity of Q-learning under the power-law decay to zero (PD2Z) step-size including LD2Z. The authors prove a convergence rate result using the PD2Z step-size, which has a regime of fast decay and stabilizing phase. The authors also provide a result for tail Polyak Ruppert averages that converges to normal distribution of which the superioity over standard PR is suppported by experimental result. Lastly, an approximation result on the partial sum of the iterates is provided.", "questions": "1. The quantile transformaiton using the uniform random vairable $U$ is not clear. $R$ is defined as a function on $S\\times A$ but the authors use $R(s,a,U)$.\n\n2. Can we derive Assumption 3.2 from the definitions?\n\n3. From Figure 1, the performance of constant step-size and LD2Z step-size does not seem to differ much.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper studies the first result of Q-learning using PD2Z step-size. It inherits the fast convergence benefits of a constant step-size with the diminishing error properties of using a polynomial step-size.\n\n2. The paper provides interesting properties not only convergence rate but also tail PR result and strong invariance property.", "weaknesses": "1. Importance of studying convergence rate in Q-learning : The literature of Q-learning focused on the sample complexity on $S,A$ or $\\frac{1}{1-\\gamma}$ dependency, which is lack of here.\n\n2.. The discount factor $\\gamma=0.1$ seems to be too small. It differs from usual choice of $\\gamma=0.99$.\n\n3.. Figure 4 seems to be an important result for tail PR case but it is deferred to Appendix.\n\n4.. The authors study a result for Q-learning with PD2Z but the unique challenge in applying the particular step-size to Q-learning compared to optimization literature is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DovaxKolRu", "reviewer": "Reviewer_HuHH", "summary": "The paper develops sharp theory for Q‑learning under a power‑law decay‑to‑zero stepsize schedule. This work then establishes explicit non‑asymptotic error bounds, a central‑limit theorem for a Polyak–Ruppert averaging estimator, and a time‑uniform strong invariance principle. Experiments on a 4×4 FrozenLake‑style gridworld corroborate the theory. This work provides the first non‑asymptotic analysis of LD2Z for Q‑learning.", "questions": "comments \n\n1. line 054: li et al is missing a year.\n2. line 241: aforementioned, decay followed by --> decay is followed by", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of this work lies in providing the first non asymptomatic analysis of Q-learning with PD2Z step-size schedules. The analysis is clear, sound, and explains why LD2Z achieves both fast early convergence and asymptotic stability.", "weaknesses": "The main weakness of this work lies in its empirical evaluation, while this work is mainly concerned with learning theory, tabular Q-learning is simply enough to run on any modern laptop (or colab). Ideally, using the publicly available code to generate figure 6.9 from the book by Sutton and Barto 2018 with your learning rate scheduler would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kvt792CrQr", "reviewer": "Reviewer_WVUn", "summary": "This paper studied the theoretical properties of tabular synchronous Q-learning with linear decay to zero (LD2Z) and more general power-law decay to zero (PD2Z) step sizes.\nThey first provided non-asymptotic L^p error bound, which showcases that, comparing to polynomial decaying step sizes, the error bound of PD2Z could rapidly decay from initialization (an advantage of constant step sizes). Numerical experiments were provided to support the theoretical results. The paper also showed the asymptotic normality and the stronger time-uniform Gaussian approximation of the tail Polyak-Ruppert averaging of their algorithms.", "questions": "see weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and clearly presents the significance of the research and its contributions. \nSpecifically, the step size of LD2Z (or more general PD2Z) is more practical in real-world applications, such as in generative models like LLMs. However, theoretical analysis of Q-learning algorithms with this type of step size has remained underdeveloped, and this paper uses rigorous theoretical results to demonstrate that the LD2Z step size strategy can simultaneously retain the advantages of constant step sizes (fast initial convergence) and polynomial decay step sizes (asymptotic normality of the tail average).\nAdditionally, the paper provides a strong approximation result that is stronger than the functional CLT-like results reported in previous works.\nExtensive numerical experiments further validate the theories proposed in this study.\nOverall, this represents a very solid theoretical work.", "weaknesses": "1. Goldreich et al. (2025) have provided the L2 non-asymptotic error bounds and asymptotic properties of the LD2Z step size for Stochastic Gradient Descent (SGD) under the strong convexity condition. It is recommended that the authors clarify the differences and innovations in the proof techniques of this paper compared to previous works.\n\n2. To my understanding, although the tail-averaged Q-learning with LD2Z step size exhibits asymptotic normality, its asymptotic variance is presumably not asymptotically optimal—in contrast, the asymptotic variance under polynomial decay step sizes is optimal. It would be beneficial if the authors could illustrate, either through theoretical analysis or experimental results, the discrepancy in asymptotic variance between the two step size strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03c4RbrRsC", "reviewer": "Reviewer_vcVR", "summary": "This paper develops a sharp asymptotic theory for Q-learning with the LZ2D step size schedule.\n\nThe authors establish several asymptotic results, including almost sure convergence, functional central limit theorems, and last-iterate convergence rates. \n\nThe paper highlights that the proposed LZ2D step size achieves improved convergence behavior compared to classical polynomial or constant step-size schemes. \n\nEmpirical results demonstrate the performance of LZ2D on synthetic Q-learning experiments, supporting its claimed theoretical benefits.", "questions": "Please see the Weaknesses part. I will increase my point if the weaknesses are addressed well.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Comprehensive theoretical contributions.**\n\n    The paper provides a complete asymptotic characterization — including almost sure convergence, convergence rates for the last iterate, and uniform approximation (Section 4). This set of results is new for Q-learning with the particular step size and strengthens the theoretical foundation of this step size choice.\n\n2. **Clear theoretical presentation.** \n\n    The main theorems are well organized and connected logically. The assumptions are explicitly stated, and the paper maintains consistency with the established stochastic approximation framework, which aids readability despite the technical depth.\n\n3. **Empirical validation.** \n\n    Figures 1 and 3 compare different step-size rules and suggest that the proposed LZ2D schedule achieves faster or more stable convergence. The figures are clearly presented and help visualize the claimed benefits.", "weaknesses": "1. **LZ2D makes an online algorithm effectively offline.**\n\n    The LZ2D step-size schedule $\\eta_{t,n}$ depends explicitly on the total sample size $n$, making it unsuitable for online learning. This fundamentally alters the nature of Q-learning, which is originally designed for streaming data.\n    * If new samples continue arriving after the prescribed $n$, it is unclear how to update $\\eta_{t, n}$.\n    * The paper should discuss whether asymptotic results remain valid if $n$ grows or is mis-specified.\n    * This issue is critical for any claim of practical superiority. \n\n    From this perspective, LZ2D transforms Q-learning from an online algorithm to an offline one, which undermines its applicability in reinforcement learning.\n\n2. **Trade-off between last-iterate convergence and asymptotic normality is missing.** \n\n    The paper claims both fast last-iterate convergence and asymptotic efficiency, but these are known to be mutually exclusive under standard stochastic approximation theory.\n    * As shown in ROOT-SGD (Li, Mou, Wainwright, Jordan, 2020), the last iterate typically converges at rate $O(1/T)$, while asymptotic normality applies to the averaged iterate with $\\sqrt{T}$ scaling.\n    * In this paper, Theorem 3.5 shows that for PD2Z-$\\nu$ step sizes, the scaling becomes $n^{\\nu/(2(\\nu+1))}$, slower than $\\sqrt{n}$, except in the limit $\\nu \\to \\infty$. This trade-off is not acknowledged. The current presentation gives the impression that LZ2D dominates all prior step-size rules, which is mathematically misleading.\n\n       **Reference:** Li C J, Mou W, Wainwright M, et al. Root-sgd: Sharp nonasymptotics and asymptotic efficiency in a single algorithm[C]//Conference on Learning Theory. PMLR, 2022: 909-981.\n\n3. **Experimental comparisons are incomplete and possibly biased.**\n\n    * Figure 1 compares LZ2D with polynomial and constant step sizes but does not include linearly decaying step sizes ($\\eta_t \\propto 1/t$), which have theoretical guarantees for Q-learning (Li et al., Operations Research, 2024).\n    * The paper also omits analysis of averaged iterates, which are standard when polynomial decay is used.\n    * The hyperparameters (e.g., exponent $\\alpha$ in polynomial decay) are not reported. Without tuning details, the apparent dominance of LZ2D may reflect unfair parameter choices rather than intrinsic superiority. Overall, the empirical claims in Figure 1 and 3 appear “too good to be true” without further clarification or ablation.\n\n        **Reference:** Li G, Cai C, Chen Y, et al. Is Q-learning minimax optimal? a tight sample complexity analysis[J]. Operations Research, 2024, 72(1): 222-236.\n\n4. **Unexplained discrepancy in Brownian motion approximation (Figure 3).**\n\n     The right panel of Figure 3 shows a large performance gap between the LZ2D method and the Brownian motion benchmark, despite both satisfying the same functional CLT.\n    * The paper does not explain why the LZ2D trajectory aligns so much better.\n    * This may be due to slow convergence of remainder terms, but no quantitative discussion is given.\n    * Additionally, there is an inconsistency: Line 455 defines the measurement as $\\max_{k_n \\le t \\le n} |\\sum_{l=t}^n (Q_l^c - Q^*)|{\\infty}$, while Figure 3 uses $\\max_{1 \\le t \\le n}$. Which definition is correct, and does LZ2D introduce the $k_n$ term while the Brownian benchmark does not? Clarifying this is important, since it affects the interpretation of the approximation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)