Title: Statistical inference for Linear Stochastic Approximation with Markovian Noise
Venue: NeurIPS 2025
Accepted: yes
Keywords: Markovian LSA

Polyak-Ruppert averaging

Berry-Esseen type bounds

Bootstrap methods

Asymptotic variance estimation
Abstract: In this paper we derive non-asymptotic Berry–Esseen bounds for Polyak–Ruppert averaged iterates of the Linear Stochastic Approximation (LSA) algorithm driven by the Markovian noise. Our analysis yields $O(n^{-1/4})$ convergence rates to the Gaussian limit in the Kolmogorov distance. We further establish the non-asymptotic validity of a multiplier block bootstrap procedure for constructing the confidence intervals, guaranteeing consistent inference under Markovian sampling. Our work provides the first non-asymptotic guarantees on the rate of convergence of bootstrap-based confidence intervals for stochastic approximation with Markov noise. Moreover, we recover the classical rate of order $\mathcal{O}(n^{-1/8})$ up to logarithmic factors for estimating the asymptotic variance of the iterates of the LSA algorithm.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives Berry–Esseen bounds for the averaged iterates of linear stochastic approximation under ergodic Markovian noise. Moreover, the authors obtain non-asymptotic guarantees for the multiplier subsample bootstrap estimator, including explicit error rates for estimating the asymptotic covariance matrix and for the coverage probabilities of $\\theta^*$.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1. To the best of my knowledge, the Berry–Esseen bounds the authors derive—expressed in Kolmogorov distance—for averaged linear stochastic-approximation iterates under ergodic Markovian noise are novel. The analysis also makes particularly effective use of the Poisson decomposition for Markov chains.\n\nS2. This is a theoretically solid work. The proof seems correct, although I did not check line by line.\n\nWeaknesses:\n\nPlease see questions below.\n\nQUESTIONS:\nI hope the following questions would help the authors improve the paper.\n\nQ1. This paper appears to extend the results of [67] by relaxing the i.i.d. noise assumption to ergodic Markovian noise. It would help readers if the authors could clarify (i) the main technical challenges in moving from the i.i.d. to the Markovian setting and (ii) the key distinctions between this work and [67].\n\nQ2. On line 222 the authors state that [81] derives a Berry–Esseen bound for TD learning under Markov noise with the similar rate $\\tilde{\\mathcal{O}}(1/n^{1/4})$. Could you elaborate on why the techniques in [81] do not carry over directly to the general linear stochastic approximation with Markovian noise?\n\nQ3. In the introduction the authors note two standard approaches for constructing confidence intervals based on asymptotic normality—plug-in and batch-mean estimators. In the i.i.d. setting, [80] employs a plug-in estimator and attains an \\$\\mathcal{O}(n^{-1/3})\\$ approximation rate for \\$\\theta^\\ast\\$ in TD learning, outperforming the \\$\\mathcal{O}(n^{-1/4})\\$ rate typically achieved by batch-mean estimators for general linear SA. In this paper, the authors adopt a batch-mean method and derives an \\$\\mathcal{O}(n^{-1/10})\\$ rate for linear SA with Markovian data. Could you clarify (i) why plug-in methods were not considered for the Markovian setting and (ii) whether the \\$\\mathcal{O}(n^{-1/10})\\$ rate is believed to be optimal?\n\nQ4. In this paper, the results are derived for projected iterates. Could the authors clarify whether analogous guarantees would hold for the unprojected algorithm, and what specific technical obstacles would need to be overcome to establish such an extension?\n\nQ5. The manuscript introduces many definitions—see, for instance, Lines 182 and 266—without offering the intuition behind them. As a result, readers may struggle to recall what each term represents and to follow the authors’ line of reasoning.\n\nQ6. Throughout the paper, the time index alternates between $k$ and $n$. For clarity and consistency, it would be preferable to choose a single symbol and use it uniformly.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper establishes finite-sample inference for Polyak–Ruppert averaged linear stochastic approximation under uniformly geometrically ergodic Markov noise by (1) proving a non-asymptotic Berry–Esseen bound of order $O(n^{-1/4})$ for one-dimensional projections of $\\sqrt{n}(\\bar\\theta_n - \\theta^*)$, and (2) developing a multiplier block bootstrap with $O(n^{-1/10})$ finite-sample coverage error (up to logs). The analysis, based on Poisson-equation decompositions, martingale Berry–Esseen techniques, and overlapping batch means for variance estimation, is illustrated in the context of temporal-difference learning with linear function approximation.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\n* The paper is very well organized, with thorough background and motivation, a comprehensive related works section, clear motivations before each theorem, and insightful discussions after each result, all of which greatly aid reader comprehension.\n* Provides the first non-asymptotic coverage error bounds for bootstrap methods under dependent (Markovian) SA iterates, which is valuable for online learning and RL applications.\n* The theoretical analysis is rigorous and solid, combining Poisson equation techniques for Markov chains, martingale Berry–Esseen bounds, and batch means/multiplier block bootstrap analysis under dependence, with sound logic and careful handling of technical conditions.\n\n# Weaknesses:\n\nPlease note that the following weaknesses do not undermine the core contributions, but addressing them would make the paper even stronger:\n\n* Since the paper’s central focus is on analysis under Markovian noise, the unsubstantiated claim that “many practical SA applications involve dependent noise, often forming a Markov chain” (line 21) leaves readers without the concrete motivation needed to appreciate why handling this specific dependency structure is both relevant and necessary.\n* The experimental validation in Appendix G is limited to synthetic and relatively small-scale examples, so it is unclear how the proposed theoretical bounds and bootstrap intervals hold up on larger or real-world problems.\n* The related works section (Section 2) reviews many results under i.i.d. noise but does not explain how the theoretical tools and proof techniques must change in the Markovian setting, making it hard to see what new challenges the paper overcomes.\n\nQUESTIONS:\nThe questions follow the order of the weaknesses listed above.\n\n1. Could the authors clarify how Markovian noise actually arises in real-world SA applications? In particular, could they provide concrete instances to substantiate the statement “many practical SA applications involve dependent noise, often forming a Markov chain” (line 21)? Addressing this would help improve the **clarity** and **significance** scores.\n2. Could the authors provide empirical mean-squared error decay curves (for example, log-log plots of\n   $\\mathbb{E}\\|\\bar\\theta_n - \\theta^*\\|^2$ versus $n$) on a nontrivial benchmark and compare the observed slope to the $O(n^{-1/4})$ rate predicted by Theorem 1? This would directly assess whether the theory captures real-world behavior. Addressing this would help improve the **significance** score.\n\n3. Although the related works section (Section 2) discusses many results under i.i.d. noise, could the authors comment on the key differences between the i.i.d. and Markovian noise settings, particularly regarding the theoretical tools and techniques required for analysis? Addressing this would help improve the **significance** score.\n\nLIMITATIONS:\nYes, the authors acknowledge potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the statistical inference properties of the projected Polyak-Ruppert averaged iterates in Linear Stochastic Approximation (LSA) algorithms under a Markovian noise model. The authors establish non-asymptotic bounds for the convergence of the averaged iterates to a Gaussian distribution in the Kolmogorov distance, with a rate of $O(n^{-1/4})$. They also validate the multiplier subsample bootstrap (MSB) procedure and demonstrate its consistency, allowing the probabilities for the actual parameter $\\theta^\\star$ to be approximated at a rate of $O(n^{-1/10})$. Furthermore, the paper recovers an $O(n^{-1/8})$ rate (up to logarithmic factors) for estimating the asymptotic variance of the iterates. The theoretical results are finally applied to the TD learning setting, which becomes a specific instance of LSA under appropriate assumptions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper provides clear and rigorous theoretical contributions with non-asymptotic bounds for both approximation and bootstrap procedures, an area that has received less attention for Markovian settings compared to IID.\n- The Kolmogorov convergence rate of $O(n^{-1/4})$ seems to be novel and tighter than previous results, such as the Wasserstein bound of $O(n^{-1/6})$ in [72], which is $O(n^{-1/12})$ under Kolmogorov distance.\n- The authors carefully utilize the Poisson equation/decomposition and martingale CLTs for their analysis. The derivation is detailed and appears to be technically interesting.\n- The block bootstrap method is rigorously justified and analyzed non-asymptotically, which is novel within the LSA setting with Markov noise.\n\nWeaknesses:\n- The analysis is limited to one-dimensional projections of the estimation error. While the authors acknowledge this, extending the results to multivariate bounds remains an open question. Additionally, the results of [74] apply in a $d$-dimensional setting.\n- Assumptions such as those related to step size and tuning may be restrictive and impact the significance of the results. It is advisable to include a dedicated section discussing these limitations and comparing them with relevant work.\n- The bootstrap bound of $O(n^{-1/10})$, although novel, does not provide guarantees regarding its tightness or optimality. Could the authors offer insights into the best possible upper bound that might be achieved?\n- The results are primarily theoretical. Although the TD learning application is mentioned, there are few empirical plots or data tables throughout the entire submission (including supplementary materials) to demonstrate empirical convergence rates and algorithm performance. Some technical notations and assumptions may also hinder accessibility for non-expert audiences.\n\nQUESTIONS:\nThe strengths and weaknesses section already provides some suggestions and questions. Here are additional questions:\n- Remark 1 states that \"Corollary 1 predicts the optimal error of ...\" Could the authors clarify what \"optimal\" refers to?\n- On page 6, it is claimed that the essential part of the analysis \"can be generalized to the $d$-dimensional case.\" Given this, what are the main challenges in extending the full result to multi-dimensional settings?\n- Are there technical tools used in the paper that are novel and worth highlighting? Or is the technical contribution mostly about applying existing results and their variants with a new proof strategy?\n\nLIMITATIONS:\nThis is a theoretical paper and does not directly present societal concerns. As mentioned earlier, it is recommended to include a dedicated section that discusses the limitations of the assumptions (for practical applications) and compares them with relevant work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of Linear Stochastic Approximation (LSA) where\nthe noise is Markovian. In this setting, the authors focus on establishing\nnon-asymptotic Berry-Essen bounds for estimators that employ Polyak-Ruppert\naveraging. Over the course of the paper, the authors establish $O(n^{-1/4})$\nrates of convergence to the limiting Gaussian distribution w.r.t. the\nKolmogorov distance.\n\nMoreover, the paper studies the block bootstrap for estimating the variance\nof their estimator $\\overline{\\theta}_n$ of interest, showing it is consistent under the\nassumption of Markovian noise. Under mild assumptions, the paper recommends\na bootstrap growth rate $b_n$ and stepsize growth rate $\\alpha_k$ which\nyield a $O(n^{-1/8})$ convergence rate w.r.t. the Kolmogorov distance for\nany projected version of $\\sqrt{n}(\\overline{\\theta}_n - \\theta^*)$.\n\nThe paper concludes by offering a concrete application of their theory for\nthe temporal-difference learning algorithm applied to Markov decision\nprocesses.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nOriginality:\n\n[+] The approach taken by the paper appears to borrow from many different\nprevious works, e.g., by using the Poisson decomposition of Markov chains to\nhelp derive its non-asymptotic bounds. While the results accord with other\nworks, the proof strategy appears to be unique.\n\n[+] There is accompanying theory on the block bootstrap that can be used\nto estimate the non-asymptotic variance of the desired parameter\n$\\theta$. This proof appears to be novel and the first to do so in the LSA\nsetting with Markov noise.\n\nQuality:\n\n[+] The paper is very thorough and takes great care in defining all\nnotation, salient assumptions and related work. While I have not checked\neverything in the appendix, the paper seems sensible and does not contain\nany obvious flaws.\n\nClarity:\n\n[+] The paper is very clear and detail oriented. Most proofs are pushed to\nthe appendix so that the core ideas can be discussed in the main portion of\nthe text. While there is quite a bit of notation, the general ideas in the\npaper are well conveyed.\n\nSignificance:\n\n[+] The core significance of this paper is mostly\ntheoretical. Its convergence rate of asymptotic variance matches other works\nbut is more general as these techniques apply to Markovian noise rather than\nsimply independent noise samples.\n\n[+] The paper specifies a specific learning rate and bootstrap growth rate\nin the case of the bootstrap. Thus while theoretical, this provides some\npractical input for those practicing with this technique.\n\nWeaknesses:\n\nSignificance:\n\n[-] As discussed in the paper, in order to obtain these non-asymptotic error\nbounds, one must know the final value $n$ in advance. While this is not a\nhuge deal, it does impose some structure on the algorithms for which these\nbounds can be derived.\n\n[-] The work does not improve upon previous bounds; that being said, it does\ngeneralize the previously shown bounds to a non-asymptotic regime with\nMarkovian noise, which does appear novel.\n\nQUESTIONS:\n[Q1] Most of the random quantites studied in the paper are scalars, as they\nusually are the dot product between the estimator and some random vector\n$u$. Can the results here be generalized to the multivariate domain,\neschewing the need to choose these fixed vectors $u$ in advance?\n\nLIMITATIONS:\nNo, the authors should explain whether there is any negative societal impact from doing this theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of deriving central limit theorems and bootstrap consistency results for iterates in linear stochastic approximation (LSA) problems under Markovian noise.   They derive a Berry-Esseen type result for Polyak–Ruppert averaged iterates projected onto the unit sphere with a fixed vector. The convergence rate $n^{-1/4}$ matches the sharpest available bounds in the literature for TD learning.  They also establish rate of convergence for a Gaussian block-multiplier bootstrap procedure.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proof of the central limit theorem uses interesting technical tools related to Poisson approximation of Markov chains, and the resulting rate matches other results derived for similar problems. \n- The authors derive a result involving the bootstrap for LSA problems, which appears to be the first such result in the literature. \n- The bootstrap result involves an interesting approximation argument that handles the fact that the iterates are not themselves a Markov chain. \n- The paper is well-written overall and clear.   \n\nWeaknesses\n- Results involving a fixed projection are weaker than those involving the convex distance in $\\mathbb{R}^d$.  The closely related work of Wu, Wei, and Rinaldo (2025) derive such a result, and attain the same rate of convergence.  Moreover, the above result appears to allow $d$ to grow, whereas the arguments in the current paper appear to require $d$ to be fixed.     \n- The rate of convergence for the bootstrap is slightly disappointing.  The authors give some intuition as to why the rate of convergence is slower. It appears to arise from choosing tuning parameters to balance the rate of convergence for the central limit theorem with variance estimation. It also appears to be related to the choice of target variance.\n\nQUESTIONS:\n- If you generalize your approach to avoid a fixed projection vector $u$, would the rate of convergence remain the same?  I expect that this would be the case, but if the authors could explain the main technical difficulty with considering the $d$-dimensional case, it would be appreciated. \n-  Along similar lines, would be possible to consider a growing $d$ regime with appropriate moment bounds?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}