Title: Gaussian Approximation and Multiplier Bootstrap for Polyak-Ruppert Averaged Linear Stochastic Approximation with Applications to TD Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Linear Stochastic Approximation

Normal Approximation

Bootstrap Validity
Abstract: In this paper, we obtain the Berry–Esseen bound for multivariate normal approximation for the Polyak-Ruppert averaged iterates of the linear stochastic approximation (LSA) algorithm with decreasing step size. Moreover, we prove the non-asymptotic validity of the confidence intervals for parameter estimation with LSA based on multiplier bootstrap. This procedure updates the LSA estimate together with a set of randomly perturbed LSA estimates upon the arrival of subsequent observations. We illustrate our findings in the setting of temporal difference learning with linear function approximation.

Gaussian Approximation and Multiplier Bootstrap for Polyak-Ruppert Averaged Linear Stochastic Approximation with Applications to TD Learning

Abstract

In this paper, we obtain the Berry-Esseen bound for multivariate normal approximation for the Polyak-Ruppert averaged iterates of the linear stochastic approximation (LSA) algorithm with decreasing step size. Moreover, we prove the non-asymptotic validity of the confidence intervals for parameter estimation with LSA based on multiplier bootstrap. This procedure updates the LSA estimate together with a set of randomly perturbed LSA estimates upon the arrival of subsequent observations. We illustrate our findings in the setting of temporal difference learning with linear function approximation.

1 Introduction

Stochastic approximation (SA) methods are a central component for solving various optimization problems that arise in machine learning [32, 26], empirical risk minimization [72] and reinforcement learning [42, 67]. There is a vast number of contributions in the literature, which cover both asymptotic [48, 53] and non-asymptotic [45, 15, 36] properties of the SA estimates. The primarily important property among the asymptotic ones of the SA estimates is their asymptotic normality [53], which is important due to its role in constructing (asymptotic) confidence intervals and hypothesis testing [71]. However, a natural question of the rate of convergence in the appropriate central limit theorems (CLT) is not well addressed in literature even in the relatively simple setting of the linear stochastic approximation (LSA) [21], [34], [9].

Alternatively, confidence sets for SA algorithms can be constructed in a non-asymptotic manner based on concentration inequalities [4]. These bounds are often regarded as loose [60], yielding suboptimal performance of the statistical procedures based on the latter estimates [28]. In contrast, for statistical inference procedures based on independent and identically distributed (i.i.d.) observations, such as M -estimators [71], there is a machinery of non-parametric methods for constructing confidence sets with the bootstrap [19, 58]. This approach is accompanied with theoretical guarantees, showing the non-asymptotic validity of the bootstrap-based confidence intervals for parameters in linear regression [64] and statistical tests [12]. Extending theoretical guarantees to a non-classical situation with online learning algorithms encounters serious problems, essentially related to the problem of obtaining rate of convergence in the corresponding CLTs. At the same time, many phenomena arising in the analysis of nonlinear SA algorithms already appear in the analysis of LSA problems.

The LSA procedure aims to find an approximate solution for the linear system ¯ A θ glyph[star] = ¯ b with a unique solution θ glyph[star] based on a sequence of observations { ( A ( Z k ) , b ( Z k )) } k ∈ N . Here A : Z → R d × d and b : Z → R d are measurable functions and ( Z k ) k ∈ N is a sequence of noise variables taking values in some measurable space ( Z , Z ) with a distribution π satisfying E [ A ( Z k )] = ¯ A and E [ b ( Z k )] = ¯ b . We focus on the setting of independent and identically distributed (i.i.d.) observations { Z k } k ∈ N . With a sequence of decreasing step sizes ( α k ) k ∈ N and the starting point θ 0 ∈ R d , we consider the estimates { ¯ θ n } n ∈ N given by

Here, we have fixed the size of the burn-in period (see, e.g., [16, 44]) to n 0 = n . Provided that n is large enough, the burn-in size affects only a constant factor in the subsequent bounds. The sequence { θ k } k ∈ N corresponds to the standard LSA iterates, while { ¯ θ n } n ∈ N corresponds to the Polyak-Ruppert (PR) averaged iterates [59, 53]. It is known that ¯ θ n is asymptotically normal with a minimax-optimal covariance matrix (see [53] and [23] for discussion). Specifically, under appropriate technical conditions on the step sizes { α k } and noisy observations { A ( Z k ) } , it holds that

where Σ ∞ is the asymptotic covariance matrix defined later in Section 3.1. There is a long list of contributions to the non-asymptotic analysis of ¯ θ n , particularly [43] and [16], which study moment and Bernstein-type concentration bounds for √ n ( ¯ θ n -θ glyph[star] ) . Unfortunately, such bounds do not imply Berry-Esseen type inequalities for √ n ( ¯ θ n -θ glyph[star] ) , that is, they do not allow us to control the quantity

where Conv( R d ) refers to the set of convex sets in R d . While the Berry-Esseen bounds are a popular subject of study in probability theory, starting from the classical work [20], most results are obtained for sums of random variables or martingale difference sequences [52, 8]. We can only mention a few results for SA algorithms, see Section 2 for more details. This paper aims to provide the latter bounds for the specific setting of the LSA procedure. Our primary contribution is twofold:

We establish a BerryEsseen bound for accuracy of normal approximation of the distribution of Polyak-Ruppert averaged LSA iterates with a polynomially decreasing step size. Our results suggest that the best rate of normal approximation, in the sense of (2), is of order n -1 / 4 up to logarithmic factors in n , where n denotes the number of samples. Interestingly, this rate is achieved with an aggressive step size, α k = c 0 / √ k . Our proof technique follows the Berry-Esseen bounds for nonlinear statistics provided in [63].

We provide non-asymptotic confidence bounds for the distribution of the PR-averaged statistic √ n ( ¯ θ n -θ glyph[star] ) using the multiplier bootstrap procedure. In particular, our bounds imply that the quantiles of the exact distribution of √ n ( ¯ θ n -θ glyph[star] ) can be approximated at a rate of n -1 / 4 , where n is the number of samples used in the procedure, provided that n is sufficiently large (see A4 for exact conditions). To the best of our knowledge, this is the first non-asymptotic bound on the accuracy of bootstrap approximation in SA algorithms. We apply the proposed methodology to the temporal difference learning (TD) algorithm for policy evaluation in reinforcement learning.

The rest of the paper is organized as follows. In Section 2, we provide a literature review on the non-asymptotic analysis of the LSA algorithm and bootstrap methods. Next, in Section 3, we analyze the convergence rate of Polyak-Ruppert averaged LSA iterates to the normal distribution. In Section 4, we discuss the multiplier bootstrap approach for LSA and establish bounds on the accuracy of approximating the quantiles of the true distribution. Finally, we apply our findings to TD learning and present numerical illustrations in Section 5.

glyph[negationslash]

Notations. For matrix A ∈ R d × d we denote by ‖ A ‖ its operator norm. For symmetric matrix Q = Q glyph[latticetop] glyph[follows] 0 , Q ∈ R d × d and x ∈ R d we define the corresponding norm ‖ x ‖ Q = √ x glyph[latticetop] Qx , and define the respective matrix Q -norm of the matrix B ∈ R d × d by ‖ B ‖ Q = sup x =0 ‖ Bx ‖ Q / ‖ x ‖ Q . For sequences a n and b n , we write a n glyph[lessorsimilar] b n if there exist a constant c > 0 such that a n ≤ cb n for c > 0 . For simplicity we state the main results of the paper up to constant factors.

2 Related works

Among contributions to the analysis of the LSA algorithm, we should mention the papers [53, 34, 9, 6]. These works investigate the asymptotic properties of the LSA estimates (such as asymptotic normality and almost sure convergence) under i.i.d. and Markov noise. Non-asymptotic results for the LSA and PR-averaged LSA estimates were obtained in [55, 47, 7, 35, 44], where MSE bounds were established, and in [43, 17, 16], which provided high-probability error bounds. The latter results enable the construction of Bernstein-type confidence intervals for the error ¯ θ n -θ glyph[star] . Unfortunately, the corresponding bounds typically depend on unknown problem properties of (1), related to the design matrix ¯ A and the noise variables A ( Z k ) , b ( Z k ) . For this reason, applying these error bounds in practice is complicated. Furthermore, concentration bounds for the LSA error [43, 17, 16] do not imply convergence rates of the rescaled error √ n ( ¯ θ n -θ glyph[star] ) to the normal distribution in Wasserstein or Kolmogorov distance. Non-asymptotic convergence rates were previously studied in [2] using the Stein method, but the resulting rate corresponds to a smoothed Wasserstein distance. Recent work [65] investigates convergence rates to the normal distribution in Wasserstein distance for LSA with Markovian observations. Both papers yield bounds that are less tight with respect to their dependence on trajectory length n than those presented in the present work, see a detailed comparison after Theorem 2.

A popular method for constructing confidence intervals in the context of parametric estimation is based on the bootstrap approach ([19]). Its analysis has attracted many contributions, in particular a series of papers [12] and [13] that validate a bootstrap procedure for a test based on the maximum of a large number of statistics. Their study shows a close relationship between bootstrap validity results, Gaussian comparison and anticoncentration bounds for rectangular sets. The papers [64] and [27] investigate the applicability of likelihood-based statistics for finite samples and large parameter dimensions under possible model misspecification. The important step in proving bootstrap validity is again based on Gaussian comparison and anticoncentration bounds, but now for spherical sets. The bootstrap procedure for spectral projectors of covariance matrices is discussed in [46] and [31]. The authors follow the same steps to prove the validity of the bootstrap.

Extending the classical bootstrap approach to online learning algorithms is a challenge. For example, the iterates { θ k } k ∈ N determined by (1) are not necessarily stored in memory, which makes the classical bootstrap inapplicable. This problem can be solved by performing randomly perturbed updates of the online procedure, as proposed in [22] for the iterates of the Stochastic Gradient Descent (SGD) algorithm. The authors in [56] used the same procedure for the case of Markov noise and policy evaluation algorithms in reinforcement learning, but in both papers the authors only consider the asymptotic validity. In our paper we use the same multiplier bootstrap approach (see Section 4), but we provide an explicit error bound for the bootstrap approximation of the distribution of the statistics √ n ( ¯ θ n -θ glyph[star] ) .

In addition to the bootstrap approach, one can also use the pivotal statistics [37, 40, 41] or various estimates of the asymptotic covariance matrix [73] to construct the confidence intervals for θ glyph[star] . The latter approach can be based on the plug-in estimators [39], batch mean estimators [11] or in combination with the multiplier bootstrap approach [74]. However, the theoretical guarantees for mentioned methods remain purely asymptotic.

3 Accuracy of normal approximation for LSA

We first study the rate of normal approximation for the tail-averaged LSA procedure. When there is no risk of ambiguity, we use simply the notations A k = A ( Z k ) and b k = b ( Z k ) . Starting from the definition (1), we get with elementary transformations that

where we have set ε k = ε ( Z k ) with

Here the random variable ε ( Z k ) can be viewed as a noise, measured at the optimal point θ glyph[star] . We now assume the following technical conditions:

A1. Sequence { Z k } k ∈ N is a sequence of i.i.d. random variables defined on a probability space (Ω , F , P ) with distribution π .

A2. ∫ Z A ( z )d π ( z ) = ¯ A and ∫ Z b ( z )d π ( z ) = ¯ b , with the matrix -¯ A being Hurwitz. Moreover, ‖ ε ‖ ∞ = sup z ∈ Z ‖ ε ( z ) ‖ < + ∞ , and the mapping z → A ( z ) is bounded, that is,

Moreover, for the noise covariance matrix

it holds that its smallest eigenvalue is bounded away from 0 , that is,

It is possible to change (4) to the moment-type bound as it was previously considered in [43] and [16], see the detailed discussion after Theorem 2. The fact that the matrix -¯ A is Hurwitz implies that the linear system ¯ A θ = ¯ b has a unique solution θ glyph[star] . Moreover, this fact is sufficient to show that the matrix I -α ¯ A is a contraction in an appropriate matrix Q -norm for small enough α > 0 . Precisely, the following result holds:

Proposition 1. Let -¯ A be a Hurwitz matrix. Then for any P = P glyph[latticetop] glyph[follows] I , there exists a unique matrix Q = Q glyph[latticetop] glyph[follows] I , satisfying the Lyapunov equation ¯ A glyph[latticetop] Q + Q ¯ A = P . Moreover, setting

where κ Q = λ max ( Q ) /λ min ( Q ) , it holds for any α ∈ [0 , α ∞ ] that αa ≤ 1 / 2 , and

The proof of Proposition 1 is provided in Appendix D.1. Note that it is possible to set P = I as in [18], yet it is possible that other choices of P could be more beneficial for particular applications. Now consider an assumption on the step sizes α k and number of observations n :

A3. The step sizes { α k } k ∈ N has a form α k = c 0 /k γ , where γ ∈ [1 / 2; 1) and c 0 ∈ (0; α ∞ ∧ a ∧ (1 -γ )] . Moreover, we assume that n ≥ d , and

The main aim of lower bounding n is to ensure that the number of observations is large enough in order that the LSA error related to the choice of initial condition θ 0 -θ glyph[star] becomes small.

3.1 Central limit theorem for Polyak-Ruppert averaged LSA iterates.

It is known that the assumptions A1-A3 guarantee that the CLT applies to the iterates of ¯ θ n , namely,

where the asymptotic covariance matrix Σ ∞ has a form

and Σ ε is defined in (5). This result can be found for example in [53] and [23]. We are interested in the Berry-Esseen type bound for the rate of convergence in (10), that is, we aim to bound ρ Conv n defined in (2) w.r.t. the available sample size n . We control ρ Conv n using a method from [63] based on randomized multivariate concentration inequality. Below we briefly state its setting and required definitions. Let X 1 , . . . , X n be independent random variables taking values in X and T = T ( X 1 , . . . , X n ) be a general d -dimensional statistics such that T = W + D , where

ξ glyph[lscript] = h glyph[lscript] ( X glyph[lscript] ) and h glyph[lscript] : X → R d is a Borel measurable function. Here the statistics D can be non-linear and is treated as an error term, which is "small" compared to W in an appropriate sense. Assume that E [ ξ glyph[lscript] ] = 0 and ∑ n glyph[lscript] =1 E [ ξ glyph[lscript] ξ glyph[latticetop] glyph[lscript] ] = I d . Let Υ = Υ n = ∑ n glyph[lscript] =1 E [ ‖ ξ glyph[lscript] ‖ 3 ] . Then, with η ∼ N (0 , I d ) ,

where D ( glyph[lscript] ) = D ( X 1 , . . . , X glyph[lscript] -1 , X ′ glyph[lscript] , X glyph[lscript] +1 , . . . , X n ) and X ′ glyph[lscript] is an independent copy of X glyph[lscript] . This result is due to [63, Theorem 2.1]. One can modify the bound (13) for the setting when ∑ n glyph[lscript] =1 E [ ξ glyph[lscript] ξ glyph[latticetop] glyph[lscript] ] = Σ glyph[follows] 0 . This result due to [63, Corollary 2.3]. Following the construction (12), we set T = √ n ¯ A ( ¯ θ n -θ glyph[star] ) and consider it as a nonlinear statistic of i.i.d. random variables Z 1 , . . . , Z 2 n , which drive the LSA dynamics (1). We can exactly represent T as a sum of linear ( W ) and non-linear parts ( D ), where

The proof of this result can be bound in Proposition 3. To obtain a bound for the approximation accuracy in (2) using the bound (13), we need to upper bound E 1 / 2 [ ‖ D ( Z 1 , . . . , Z 2 n ) ‖ 2 ] and E [ ‖ D -D ( i ) ‖ ] . The first result below provides a second moment bound on D :

Theorem 1. Assume A1, A2, and A3. Then we obtain the following error bound:

where glyph[lessorsimilar] stands for inequality up to an absolute constant, and ∆ 1 = ∆ 1 ( n, a, C A , c 0 ) is a polynomial function defined in Appendix A.3, eq. (29) .

The proof of Theorem 1 is provided in Appendix A.3. Now it remains to upper bound the term E [ ‖ D -D ( i ) ‖ ] , which is done in Appendix B.1 using the synchronous coupling methods [10]. Combining these bounds, we obtain the following theorem:

Theorem 2. Assume A1, A2, and A3. Then the following bound holds:

where ∆ 2 = ∆ 2 ( n, a, C A , TrΣ ε , c 0 ) is a polynomial function defined in (35) , and constants C 1 , C 2 , depending upon a, C A , κ Q , TrΣ ε , c 0 , are defined in Appendix B, eq. (36) .

The proof of Theorem 2 is provided in Appendix B. Note that the assumption A2 requires that ε ( Z 1 ) is almost sure bounded. It is a strong assumption, but it can be partially relaxed. Following the stability of matrix products technique, used in [17, Proposition 3], it is possible to consider the setting when the random variable ‖ ˜ A ( Z 1 ) ‖ has only finite number of moments. In particular, we expect that assuming finite third moment of ‖ ˜ A ( Z 1 ) ‖ and ‖ ε ( Z 1 ) ‖ is sufficient to obtain a counterpart to Theorem 1. However, this generalization requires non-trivial technical work on generalizing the stability of matrix products result (see Corollary 4 in Appendix D ).

Note that the bound of Theorem 2 predicts the optimal error of normal approximation for PolyakRuppert averaged estimates of order n -1 / 4 , which is achieved with the aggressive step size α k = c 0 / √ k , that is, when setting γ = 1 / 2 in (14). In this case we obtain the optimized bound

where C 3 = C 3 ( a, C A , κ Q , TrΣ ε , ‖ ε ‖ ∞ ) is provided in (36).

Discussion. Our proof technique of Theorem 2 reveals an interesting feature: fastest rate of convergence in the convex distance ρ Conv n corresponds to the learning rate schedule that admits the fastest decay of the second-order term in the MSE bound for remainder statistics D (see Theorem 1). Results similar to the one of Theorem 2 have been recently obtained in the literature in [65] and [2]. The author in [65] considers the LSA problem specified to the temporal-difference learning (see Section 5) with Markov noise and obtains convergence rate in Wasserstein distance of order n -1 / 4 , which corresponds to the "optimal" step size schedule α k = c 0 /k 3 / 4 . Using the bound of [49, eq. (3)] (see also section 2 in [57]), this result yield a suboptimal bound of order n -1 / 8 for the convex distance ρ Conv n . Such an upper bound may be loose for some classes of distributions, but it is not clear if in particular setting of LSA the bound of [65] could imply scaling of order n -1 / 4 for ρ Conv n . At the same time, in case of X 1 , . . . , X n forming a Markov chain in (12) there is no available counterpart of the bound (13). Generalizing (13) is an interesting research direction that would allow to obtain a counterpart of Theorem 2 in case of Markovian dynamics. Similarly, the result of [2] holds for much stronger metrics, which controls the convergence of moments of twice differentiable functions. We provide additional details about connections between this metric and ρ Conv n in Appendix B.2. At the same time, the authors in [2] cover the non-linear setting of PR-averaged iterates of stochastic gradient descent algorithm under strong convexity.

Remark 1. The leading (with respect to n ) terms of the bound from Theorem 1 have an implicit dependence on the problem dimension d due to the presence of λ min . Yet the result of Theorem 1 can be improved in a sense of dependence in dimension if one is interested not in the rates of convergence for √ n ( ¯ θ n -θ glyph[star] ) , but in the projected iterated √ n Π glyph[latticetop] ( ¯ θ n -θ glyph[star] ) for some Π ∈ R d × m , m ≤ d . If this is the case, one may apply (13) for the class Conv m = Conv( R m ) of convex sets in R m and obtain, setting step size α k = c 0 / √ k , and Σ (Π) ε = ΠΣ ε Π glyph[latticetop] , that

and the constant C 4 = C 4 ( a, C A , κ Q , TrΣ (Π) ε , ‖ ε ‖ ∞ ) is provided in (36) .

Remark 2. Results similar to Theorem 1 can be obtained not only for the Polyak-Ruppert averaged estimator ¯ θ n , but also for the last iterate θ n . In particular, it is known (see e.g. [23]), that the last iterate error θ n -θ glyph[star] is also asymptotically normal:

where the covariance matrix Σ last is different from Σ ∞ . In such a case Σ last can be found as a solution to appropriate Lyapunov equation, see [23]. Then, we expect that it is possible to use the perturbation-expansion approach from [1] together with randomized concentration inequalities [63] (see (13) ), in order to obtain the Berry-Esseen bound

We leave the detailed derivation for future work.

4 Multiplier bootstrap for LSA

In order to perform statistical inference with the Polyak-Ruppert estimator ¯ θ n , we propose an online bootstrap resampling procedure, which recursively updates the LSA estimate as well as a large number of randomly perturbed LSA estimates, upon the arrival of each data point. The suggested procedure follows the one outlined in [22]. It has the following advantages: it does not rely on the asymptotic distribution of the error √ n ( ¯ θ n -θ glyph[star] ) , does not require to know the moments of √ n ( ¯ θ n -θ glyph[star] ) or its asymptotic covariance matrix Σ ∞ , and does not involve any data splitting.

We state the suggested procedure as follows. Let W 2 n = { W glyph[lscript] } 1 ≤ glyph[lscript] ≤ 2 n be a set of i.i.d. random variables, independent of Z 2 n = { Z glyph[lscript] } 1 ≤ glyph[lscript] ≤ 2 n , with E [ W 1 ] = 1 and Var[ W 1 ] = 1 . We write, respectively, P b = P ( ·|Z 2 n ) and E b = E ( ·|Z 2 n ) for the corresponding conditional probability and expectation. In parallel with procedure (1) that generates { θ k } 1 ≤ k ≤ 2 n and ¯ θ n , we generate M independent samples ( w glyph[lscript] n , . . . , w glyph[lscript] 2 n ) , 1 ≤ glyph[lscript] ≤ M distributed as W 2 n , and recursively update M randomly perturbed LSA estimates, that is,

We use a short notation ¯ θ b n for ¯ θ b , 1 n . The key idea of the procedure (16) is that the "Bootstrap-world" distribution (that is, the one conditional on Z 2 n ) of the perturbed samples √ n ( ¯ θ b n -¯ θ n ) is close to the distribution of the quantity of interest, that is, √ n ( ¯ θ n -θ glyph[star] ) . Precisely, the main result of this section will show that the quantity

is small. Although an analytic expression for P b ( √ n ( ¯ θ b n -¯ θ n ) ∈ B ) is not available, one can approximate it from numerical simulations according to (16) by generating sufficiently large number M of perturbed trajectories. Standard arguments, see e.g. [62, Section 5.1] suggest that the accuracy of Monte-Carlo approximation is of order M -1 / 2 . To analyze the suggested procedure, we shall impose an additional assumption on the trajectory length n :

A4. Assumption A3 holds with γ = 1 / 2 , and c 0 ≤ 1 / (C 2 A κ Q e) . Moreover, setting

it holds that

Moreover, we assume that for λ min defined in (6) it holds that

Note that the new bound (19) simply states that √ n/ log 2 ( n ) is sufficiently large, since h ( n ) scales as log 2 n . We discuss the assumption A4 in more details in the proof scheme. Now we formulate the main result of this section. We analyze only the setting of polynomially decaying step size with γ = 1 / 2 , since decay rate of (17) essentially depends on the approximation rate of Theorem 2, with the fastest rate achieved when γ = 1 / 2 . For other learning rates the decay rate of right-hand side in Theorem 3 will be slower. For simplicity, we do not trace the dependence of the bound below on the parameter c 0 .

Theorem 3. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then with P - probability at least 1 -6 /n it holds that

where ∆ 3 = ∆ 3 ( n, a, C A , ‖ ε ‖ ∞ ) is a polynomial function defined in Appendix C, eq. (46) .

The proof of Theorem 3 is based on the Gaussian approximation performed both in the "real" world and bootstrap world together with an appropriate Gaussian comparison inequality. The main steps of the proof are illustrated by the following scheme:

In the above scheme we have denoted by Σ b ε = n -1 ∑ 2 n -1 glyph[lscript] = n ε glyph[lscript] ε glyph[latticetop] glyph[lscript] the sample covariance matrix approximating Σ ε . Gaussian approximation for the true distribution of √ n ¯ A ( ¯ θ n -θ glyph[star] ) follows from Theorem 2. Proof of Gaussian approximation in the Bootstrap world Theorem 4 is also based on the inequality (13), but is more complicated and involves the expansion analysis of the LSA error from [1]. This technique allows to separate the LSA error into different scales with respect to the step sizes { α k } , see Appendix C.4 for details. However, this technique requires to impose additional

assumption A4 - eq. (19). Proof of the Gaussian comparison part of Theorem 5 is based on Pinsker's inequality and matrix Bernstein inequality. The latter result requires that n is large enough to ensure that minimal eigenvalue of Σ b ε is close to λ min , justifying the assumption A4 - eq. (20). Detailed proof if provided in Appendix C.

Discussion. We emphasize that the Gaussian approximation result of Theorem 2 (with Bootstrap world generalization in Theorem 4) is a key result to prove the above bootstrap validity. This argument was missing in the earlier works studying confidence intervals for stochastic optimization algorithms [11, 73, 74], where the authors considered procedures to estimate Σ ∞ in (11). They combine nonasymptotic bounds on the accuracy of recovering Σ ∞ with only asymptotic validity of the resulting confidence intervals. We expect that our proof technique for Theorem 2 can be used to provide similar non-asymptotic validity results for outlined approaches for constructing confidence intervals based on the estimation of the asymptotic covariance matrix.

Corollary 1. (Set of Euclidean balls or ellipsoids) Suppose that we are interested in estimating quantile of a given order α ∈ (0 , 1) and some matrix B ∈ R d × d , that is, the quantity

We define its counterpart in the Bootstrap world, t b α , as

Note that t b α is defines with respect to the bootstrap measure, therefore, it depends on the data Z 2 n . This bootstrap critical value t b α is applied in the Bootstrap world to build the confidence set

Theorem 3 justifies this construction and evaluate the coverage probability of the true value θ glyph[star] by this set. It states that

with the error of order n -1 / 4 in the right-hand side. Although an analytic expression for t b α is not available, one can approximate it by generating a large number M of independent samples of W n and computing from them the empirical distribution function of √ n ‖ B ( ¯ θ b n -¯ θ n ) ‖ , following (16) .

Remark 3. A natural question that arises after Theorem 3 is whether it is possible to prove similar bounds for the iterates of first-order stochastic optimization algorithms. There are several MSE bounds for corresponding algorithms with explicit dependence on the step size α k ; see, for example, [45, 5]. Therefore, we expect that it is possible to obtain a counterpart to Theorem 2. At the same time, for general first-order stochastic optimization algorithms, unlike LSA, there are no counterparts to the precise error expansions of [1]. Thus, proving the counterpart of Theorem 3 in this setting is more challenging. Similarly, we emphasize that generalizations of the procedure (16) to cases where { Z k } k ∈ N are dependent, for example, form a Markov chain, are complicated. The approach of [63] is not directly applicable in this setting, and appropriate generalization of (13) is a separate and challenging research direction.

5 Applications to the TD learning and numerical results

We illustrate our findings for the setting of temporal difference (TD) learning algorithm [66, 67] for policy evaluation in RL. Non-asymptotic error bounds for this algorithm attracted lot of contributions [43, 16, 30, 51, 38]. At the same time, confidence intervals for TD were studied in [22, 56] only in terms of their asymptotic validity. In the TD algorithm we consider a discounted MDP (Markov Decision Process) given by a tuple ( S , A , P , r, γ ) . Here S and A stand for state and action spaces, and γ ∈ (0 , 1) is a discount factor. Assume that S is a complete metric space with metric d S and Borel σ -algebra B ( S ) . P stands for the transition kernel P( B | s, a ) , which determines the probability of moving from state s to a set B ∈ B ( S ) when action a is performed. Reward function r : S × A → [0 , 1] is assumed to be deterministic. Policy π ( ·| s ) is the distribution over action space A corresponding to agent's action preferences in state s ∈ S . We aim to estimate value function

where a k ∼ π ( ·| s k ) , and s k +1 ∼ P( ·| s k , a k ) for any k ∈ N . Define the transition kernel under π ,

which corresponds to the 1 -step transition probability from state s to a set B ∈ B ( S ) . The state space S here can be arbitrary. It is a common option to consider the linear function approximation for V π ( s ) , defined for s ∈ S , θ ∈ R d , and a feature mapping ϕ : S → R d as V π θ ( s ) = ϕ glyph[latticetop] ( s ) θ . Here d is the dimension of feature space. Our goal is to find a parameter θ glyph[star] which is defined as a unique solution to the projected Bellman equation, see [70]. We denote by µ the invariant distribution over the state space S induced by P π ( ·| s ) in (21). We define the design matrix Σ ϕ as

Consider the following assumptions on the generative mechanism and on the feature mapping ϕ ( · ) : TD1. Tuples ( s, a, s ′ ) are generated i.i.d.with s ∼ µ , a ∼ π ( ·| s ) , s ′ ∼ P( ·| s, a ) .

TD2. Matrix Σ ϕ is non-degenerate with the minimal eigenvalue λ min (Σ ϕ ) > 0 . Moreover, the feature mapping ϕ ( · ) satisfies sup s ∈S ‖ ϕ ( s ) ‖ ≤ 1 .

In the setting of linear function approximation the estimation of V π ( s ) reduces to estimating θ glyph[star] ∈ R d , which can be done via the LSA procedure. Here, the k -th step randomness is given by the tuple Z k = ( s k , a k , s ′ k ) . Then, the corresponding LSA update can be written as

where A k and b k are given, respectively, by

We provide the expressions for the corresponding system matrix ¯ A = E [ A k ] and the right-hand side ¯ b in Appendix E. We verify that assumption A2 holds and, furthermore, we provide a tighter counterpart to the result of Proposition 1. This result closely follows [51] and [61].

Proposition 2. Let { θ } k ∈ N be a sequence of TD updates generated by (23) under TD 1 and TD 2. Then this update scheme satisfies assumption A2 with

moreover, one can check that ‖ I -α ¯ A ‖ 2 ≤ 1 -αa with

that is, Proposition 1 holds with Q = I .

Proof of Proposition 2 is provided in Appendix E. Since all the assumptions in A2 are fulfilled, we can verify tightness of the bound Theorem 2 for different learning rate schedules α k in (23).

Numerical results. Efficiency of the multiplier bootstrap approach (16) to the problems of constructing confidence sets in online algorithms has been demonstrated in the works [22] and [56]. We aim to illustrate the tightness of our bounds for normal approximation outlined in Theorem 2 in the setting of TD learning with linear function approximation. To this end, we consider the classical Garnet problem [3], in the simplified version proposed by [25]. This problem is characterized by the number of states N s , number of actions a , and branching factor b (i.e. the number of neighbors of each state in the MDP). We set these values to N s = 10 , a = 2 and b = 3 , and aim to evaluate the value function of the randomly generated policy π ( ·| s ) . Details on the way the policy π is set can be found in Appendix F. We consider the problem of policy evaluation in this MDP using the TD learning algorithm with identity feature mapping, that is, φ ( s ) = e s (that is, s -th coordinate vector) for s ∈ { 1 , . . . , N s } . We run the procedure (23) with the learning rates α k = c 0 /k γ and different powers γ ∈ { 0 . 5 , 0 . 65 , 0 . 7 } . For each of the experiments we aim to estimate the supremum

η ∼ N (0 , I N s ) , and show that this supremum scales as n -1 / 4 when γ = 1 / 2 and admits slower decay for other powers of γ . We approximate true probability P ( ‖ Σ 1 / 2 ∞ η ‖ ≤ x ) by the corresponding empirical probabilities based on sample of size M glyph[greatermuch] n . Second, for n ∈ { 1600 , . . . , 1638400 } , where next sample size is twice larger than the previous one, we generate N = 6553600 trajectories of TD algorithm and approximate the distribution of √ n ‖ ¯ θ n -θ glyph[star] ‖ based on the corresponding empirical distribution. We report our results in Figure 1, showing that the smallest values of ∆ n correspond to the step size schedule γ = 1 / 2 , moreover, the decay rate n -1 / 4 seems to be tight, otherwise one should expect further decay of ∆ n n 1 / 4 . Additional simulations are provided in Appendix F.

10 4

10 5

10 6

(a)

1

2

3

4

5

6

7

PR error,

= 0.50

PR error,

= 0.65

PR error,

= 0.70

10 4

10 5

10 6

(b)

0.0

0.2

0.4

0.6

0.8

n

,

= 0.50

n

,

= 0.65

n

,

= 0.70

10 4

10 5

10 6

(c)

10

1

10 0

n

1/4

n

,

= 0.50

n

1/4

n

,

= 0.65

n

1/4

n

,

= 0.70

6 Conclusion

In this paper, we have established, to the best of our knowledge, the first fully non-asymptotic confidence bounds for parameter estimation in the LSA algorithm using the multiplier bootstrap. This result is based on a novel Berry-Esseen bound for the Polyak-Ruppert averaged LSA iterates, which is of independent interest. Our paper suggests several interesting directions for further research. First, our Berry-Esseen bounds are obtained using the randomized concentration inequality [63], and it would be valuable to generalize this approach to the setting of Markov chains. Second, it is natural to extend our results to the first-order gradient methods, both for stochastic optimization and variational inequalities. Third, it becomes possible to prove the fully non-asymptotic validity of confidence intervals obtained with plug-in techniques or other estimators of the asymptotic covariance matrix of ¯ θ n . These could then be compared with the multiplier bootstrap confidence intervals in terms of their dependence on problem dimension d and other instance-dependent quantities.

References

Rafik Aguech, Eric Moulines, and Pierre Priouret. On a perturbation approach for the analysis of stochastic tracking algorithms. SIAM Journal on Control and Optimization , 39(3):872-899, 2000.

Andreas Anastasiou, Krishnakumar Balasubramanian, and Murat A. Erdogdu. Normal approximation for stochastic gradient descent via non-asymptotic rates of martingale CLT. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 115-137. PMLR, 25-28 Jun 2019.

TW Archibald, KIM McKinnon, and LC Thomas. On the generation of markov decision processes. Journal of the Operational Research Society , 46(3):354-361, 1995.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

F. Bach and E. Moulines. Non-strongly-convex smooth stochastic approximation with convergence rate o(1/n). In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

A. Benveniste, M. Métivier, and P. Priouret. Adaptive algorithms and stochastic approximations , volume 22. Springer Science & Business Media, 2012.

J. Bhandari, D. Russo, and R. Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference On Learning Theory , pages 1691-1692, 2018.

E. Bolthausen. Exact Convergence Rates in Some Martingale Central Limit Theorems. The Annals of Probability , 10(3):672 - 688, 1982.

Vivek S Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint . Cambridge University Press, 2008.

Nicolas Brosse, Alain Durmus, and Eric Moulines. The promises and pitfalls of stochastic gradient langevin dynamics. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Xi Chen, Jason D. Lee, Xin T. Tong, and Yichen Zhang. Statistical inference for model parameters in stochastic gradient descent. The Annals of Statistics , 48(1):251 - 273, 2020.

Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors. Ann. Statist. , 41(6):2786-2819, 2013.

Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Central limit theorems and bootstrap in high dimensions. Ann. Probab. , 45(4):2309-2352, 2017.

G. Dalal, Balazs Szorenyi, and G. Thoppe. A tale of two-timescale reinforcement learning with the tightest finite-time bound. arXiv preprint arXiv:1911.09157 , 2019.

John C Duchi, Alekh Agarwal, Mikael Johansson, and Michael I Jordan. Ergodic mirror descent. SIAM Journal on Optimization , 22(4):1549-1578, 2012.

Alain Durmus, Eric Moulines, Alexey Naumov, and Sergey Samsonov. Finite-time highprobability bounds for Polyak-Ruppert averaged iterates of linear stochastic approximation. Mathematics of Operations Research , 2024.

Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov, Kevin Scaman, and Hoi-To Wai. Tight high probability bounds for linear stochastic approximation with fixed stepsize. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. S. Liang, J. W. Vaughan, and Y. Dauphin, editors, Advances in Neural Information Processing Systems , volume 34, pages 30063-30074. Curran Associates, Inc., 2021.

Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov, and Hoi-To Wai. On the stability of random matrix product with markovian noise: Application to linear stochastic approximation and td learning. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 1711-1752. PMLR, 15-19 Aug 2021.

Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution , pages 569-593. Springer, 1992.

Carl-Gustav Esseen. Fourier analysis of distribution functions. A mathematical study of the Laplace-Gaussian law. Acta Mathematica , 77(none):1 - 125, 1945.

E. Eweda and O. Macchi. Quadratic mean and almost-sure convergence of unbounded stochastic approximation algorithms with correlated observations. Ann. Inst. H. Poincaré Sect. B (N.S.) , 19(3):235-255, 1983.

Yixin Fang, Jinfeng Xu, and Lei Yang. Online bootstrap confidence intervals for the stochastic gradient descent estimator. Journal of Machine Learning Research , 19(78):1-21, 2018.

G. Fort. Central limit theorems for stochastic approximation with controlled Markov chain dynamics. ESAIM: PS , 19:60-80, 2015.

Robert E Gaunt and Siqi Li. Bounding Kolmogorov distances through Wasserstein and related integral probability metrics. Journal of Mathematical Analysis and Applications , 522(1):126985, 2023.

Matthieu Geist, Bruno Scherrer, et al. Off-policy learning with eligibility traces: a survey. J. Mach. Learn. Res. , 15(1):289-333, 2014.

Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, Cambridge, MA, USA, 2016. http://www.deeplearningbook.org .

Friedrich Götze, Alexey Naumov, Vladimir Spokoiny, and Vladimir Ulyanov. Large ball probabilities, Gaussian comparison and anti-concentration. Bernoulli , 25(4A):2538-2563, 2019.

Botao Hao, Yasin Abbasi Yadkori, Zheng Wen, and Guang Cheng. Bootstrapping upper confidence bound. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

De Huang, Jonathan Niles-Weed, Joel A Tropp, and Rachel Ward. Matrix concentration for products. Foundations of Computational Mathematics , pages 1-33, 2021.

Dongyan Huo, Yudong Chen, and Qiaomin Xie. Bias and extrapolation in markovian linear stochastic approximation with constant stepsizes. In Abstract Proceedings of the 2023 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems , pages 81-82, 2023.

Moritz Jirak and Martin Wahl. Quantitative limit theorems and bootstrap approximations for empirical spectral projectors. Probability Theory and Related Fields , 190(1):119-177, 2024.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

PS Kostenetskiy, RA Chulkevich, and VI Kozyrev. Hpc resources of the higher school of economics. In Journal of Physics: Conference Series , volume 1740, page 012050. IOP Publishing, 2021.

Harold Kushner and G George Yin. Stochastic approximation and recursive algorithms and applications , volume 35. Springer Science & Business Media, 2003.

C. Lakshminarayanan and C. Szepesvari. Linear stochastic approximation: How far does constant step-size and iterate averaging go? In International Conference on Artificial Intelligence and Statistics , pages 1347-1355, 2018.

Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming , 133(1-2):365-397, 2012.

Sokbae Lee, Yuan Liao, Myung Hwan Seo, and Youngki Shin. Fast inference for quantile regression with tens of millions of observations. Journal of Econometrics , page 105673, 2024.

Gen Li, Weichen Wu, Yuejie Chi, Cong Ma, Alessandro Rinaldo, and Yuting Wei. Highprobability sample complexities for policy evaluation with linear function approximation. IEEE Transactions on Information Theory , 70(8):5969-5999, 2024.

Xiang Li, Jiadong Liang, Xiangyu Chang, and Zhihua Zhang. Statistical estimation and online inference via local sgd. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1613-1661. PMLR, 02-05 Jul 2022.

Xiang Li, Jiadong Liang, and Zhihua Zhang. Online statistical inference for nonlinear stochastic approximation with Markovian data. arXiv preprint arXiv:2302.07690 , 2023.

Xiang Li, Wenhao Yang, Jiadong Liang, Zhihua Zhang, and Michael I Jordan. A statistical analysis of Polyak-Ruppert averaged Q-learning. In International Conference on Artificial Intelligence and Statistics , pages 2207-2261. PMLR, 2023.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

Wenlong Mou, Chris Junchi Li, Martin J Wainwright, Peter L Bartlett, and Michael I Jordan. On linear stochastic approximation: Fine-grained Polyak-Ruppert and non-asymptotic concentration. In Conference on Learning Theory , pages 2947-2997. PMLR, 2020.

Wenlong Mou, Ashwin Pananjady, Martin J Wainwright, and Peter L Bartlett. Optimal and instance-dependent guarantees for markovian linear stochastic approximation. Mathematical Statistics and Learning , 7(1):41-153, 2024.

Eric Moulines and Francis Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in neural information processing systems , 24:451-459, 2011.

Alexey Naumov, Vladimir Spokoiny, and Vladimir Ulyanov. Bootstrap confidence sets for spectral projectors of sample covariance. Probab. Theory Related Fields , 174(3-4):1091-1132, 2019.

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4):1574-1609, 2009.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.

Ivan Nourdin, Giovanni Peccati, and Xiaochuan Yang. Multivariate normal approximation on the wiener space: new bounds in the convex distance. Journal of Theoretical Probability , 35(3):2020-2037, 2022.

A. Osekowski. Sharp Martingale and Semimartingale Inequalities . Monografie Matematyczne 72. Birkhäuser Basel, 1 edition, 2012.

Gandharv Patil, LA Prashanth, Dheeraj Nagaraj, and Doina Precup. Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation. In International Conference on Artificial Intelligence and Statistics , pages 5438-5448. PMLR, 2023.

V. Petrov. Sums of Independent Random Variables . Ergebnisse der Mathematik und ihrer Grenzgebiete. 2. Folge. Springer Berlin Heidelberg, 1975.

Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization , 30(4):838-855, 1992.

A. S. Poznyak. Advanced Mathematical Tools for Automatic Control Engineers: Deterministic Techniques . Elsevier, Oxford, 2008.

Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1571-1578, 2012.

Pratik Ramprasad, Yuantong Li, Zhuoran Yang, Zhaoran Wang, Will Wei Sun, and Guang Cheng. Online bootstrap inference for policy evaluation in reinforcement learning. J. Amer. Statist. Assoc. , 118(544):2901-2914, 2023.

Nathan Ross. Fundamentals of Stein's method. Probability Surveys , 8(none):210 - 293, 2011.

Donald B Rubin. The bayesian bootstrap. The annals of statistics , pages 130-134, 1981.

David Ruppert. Efficient estimations from a slowly convergent robbins-monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.

Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

Sergey Samsonov, Daniil Tiapkin, Alexey Naumov, and Eric Moulines. Improved HighProbability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory , volume 247 of Proceedings of Machine Learning Research , pages 4511-4547. PMLR, 30 Jun-03 Jul 2024.

Jun Shao. Mathematical statistics . Springer Science & Business Media, 2003.

Qi-Man Shao and Zhuo-Song Zhang. Berry-Esseen bounds for multivariate nonlinear statistics with applications to M-estimators and stochastic gradient descent algorithms. Bernoulli , 28(3):1548-1576, 2022.

Vladimir Spokoiny and Mayya Zhilova. Bootstrap confidence sets under model misspecification. The Annals of Statistics , 43(6):2653 - 2675, 2015.

R Srikant. Rates of convergence in the central limit theorem for markov chains, with an application to TD learning. arXiv preprint arXiv:2401.15719 , 2024.

R. S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3(1):9-44, 1988.

R. S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018.

Joel A. Tropp. Freedman's inequality for matrix martingales. Electron. Commun. Probab. , 16:262-270, 2011.

Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

J. N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control , 42(5):674-690, May 1997.

A. W. Van Der Vaart and J. A. Wellner. Weak convergence and empirical processes . Springer Series in Statistics, 1996.

Vladimir Vapnik. The nature of statistical learning theory . Springer science & business media, 2013.

Xi Chen Wanrong Zhu and Wei Biao Wu. Online Covariance Matrix Estimation in Stochastic Gradient Descent. Journal of the American Statistical Association , 118(541):393-404, 2023.

Yanjie Zhong, Todd Kuffner, and Soumendra Lahiri. Online Bootstrap Inference with Nonconvex Stochastic Gradient Descent Estimator. arXiv preprint arXiv:2306.02205 , 2023.

Vladimir Mikhailovich Zolotarev. Probability metrics. Theory of Probability & Its Applications , 28(2):278-302, 1984.

A Proofs for accuracy of normal approximation

A.1 Expansion of the error of LSA equipped with the Polyak-Ruppert averaging

Proposition 3. The following expansion holds:

Proof. We use the recurrence (3) and rewrite it as

The previous equation implies, after algebraic manipulation and division by α k , that

Taking average for k from n +1 to 2 n and multiplying by √ n , we obtain (25).

A.2 Bounding the error of the LSA algorithm last iterate

We begin with of technical lemma on the behavior of the last iterate θ k of the LSA procedure given in (1). We aim to show that E 1 /p [ ‖ θ k -θ glyph[star] ‖ p ] scales as √ α k , provided that k is large enough. This result is classical and appears in a number of papers, e.g. [7, 14, 43, 17]. We provide the proof here for completeness. Our analysis of the bootstrap procedure and the last iterate error of LSA procedure is based on the error expansion technique from [1], see also [16]. Namely, to perform the expansion, we decompose the LSA iterates θ k defined in (1) into a transient and fluctuation terms:

where we have defined the quantities

setting

The dependence of Γ m : k upon the stepsizes ( α j ) is implicit in (27). Here the quantity ˜ θ ( tr ) k is the transient component of the error, which determines the rate at which the initial error θ 0 -θ glyph[star] is forgotten. The term ˜ θ ( fl ) k corresponds to the fluctuation component of the error and is determined by the oscillations of the last iterate θ k around θ glyph[star] .

Proposition 4. Assume A1, A2, and A3. Then for any k ≥ n , where n satisfies (9) , it holds for 2 ≤ p ≤ log n 2 , that

Proof. Expanding the decomposition (26), we obtain that

and we bound both terms separately. Since the sample size n satisfies (9), we get applying Corollary 4 (see equation (71)), that for 2 ≤ p ≤ log n 2 it holds

Now we proceed with the second term in (28). Applying Burholder's inequality [50, Theorem 8.6] and Lemma 3 with b = a/ 4 , we obtain that

Corollary 2. Under assumptions of Proposition 4, it holds that

where we have defined

Proof. We first note that Lemma 1 implies, setting δ = 1 /n 2 , that for every fixed k ∈ [ n ; 2 n -1] ,

Application of the union bound concludes the proof.

We conclude this part with a simple consequence of Markov's inequality.

Lemma 1. Fix δ ∈ (0 , 1 / e 2 ) and let Y be a positive random variable, such that

for any 2 ≤ p ≤ log (1 /δ ) . Then it holds with probability at least 1 -δ , that

Proof. Applying Markov's inequality, for any t ≥ 0 we get that

Now we set p = log (1 /δ ) , t = e C 1 +e C 2 log (1 /δ ) , and aim to check that

Taking logarithms from both sides, the latter inequality is equivalent to

which turns into exact equality.

A.3 Proof of Theorem 1

We first define explicitly the remainder term outlined in the statement of Theorem 1:

Proof. Since both terms in the right-hand side of the error bound of Proposition 4 scales linearly with √ κ Q , for simplicity we do not trace it in the subsequent bounds (i.e. assume κ Q = 1 ), and then keep the required scaling with κ Q only in the final bounds. The decomposition (25) is a key element of our proof and allows to treat different error sources D 1 -D 4 separately. For the last iterate we have, using Proposition 4, that

Thus, using that ∑ n k =1 α k ≥ c 0 ( n 1 -γ -1) 1 -γ and c 0 ≤ 1 -γ , we obtain that

Now we proceed with D 3 . Since it is a sum of a martingale-difference sequence w.r.t. F k = σ ( Z glyph[lscript] , glyph[lscript] ≤ k ) , we get using Proposition 4, that

where we additionally used that S 1 glyph[lessorsimilar] 1 /a due to Lemma 3. Now it remains to bound the term D 4 from the representation (25). Using Minkowski's inequality and Proposition 4, we get that

Here in (a) we additionally used that k γ -( k -1) γ glyph[lessorsimilar] k 1 -γ together with Lemma 3. Combining the estimates above yields the result of Theorem 1.

We conclude this section with some technical lemmas.

Lemma 2 (Lemma 24 in [18]) . Let b > 0 and ( α k ) k ≥ 0 be a non-increasing sequence such that α 0 ≤ 1 /b . Then

Proof. The proof of this statement is given in [18], we provide it here for completeness. Let us denote u j : n +1 = ∏ n +1 l = j (1 -α l b ) . Then, for j ∈ { 1 , . . . , n +1 } , u j +1: n +1 -u j : n +1 = bα j u j +1: n +1 . Hence,

and the statement follows.

Lemma 3 (Modified Lemma 25 in [18]) . Let b > 0 and let α glyph[lscript] = c 0 /glyph[lscript] γ , γ ∈ [1 / 2; 1) , such that c 0 ≤ 1 /b . Then for any n satisfying

and any k ≥ n , it holds that

Proof. From elementary algebra, we obtain that

where we used the fact that (1 + x ) γ ≤ 1 + γx for γ ∈ [1 / 2; 1) and x ∈ [0 , 1] . Hence,

Thus we obtain that, since k ≥ n ,

In the last identity we used the fact that, since n satisfies (30), it holds for glyph[lscript] ≥ n/ 2 that

We will now prove that for j ≤ n -1 , it holds

For j ≥ n/ 2 , the bound (33) directly follows from (32). We now turn to the proof of (33) for j ≤ glyph[ceilingleft] n/ 2 glyph[ceilingright] . Note first that

On the other hand, we get

Comparing the above bounds, to ensure that (33) holds, it is enough to check that

Note that (34) is guaranteed by (30). Using that e -x ≤ 1 -x/ 2 for x ∈ [0 , 1] , we obtain that

where the last inequality follows from Lemma 2.

B Proof of Theorem 2

We first define explicitly the remainder term outlined in the statement of Theorem 2:

and constants C 1 , C 2 , C 3 , C 4 from Theorem 2, optimized bound (15), and Remark 1, respectively:

To complete the proof we only need to combine (13) with the bounds of Theorem 1. Note that we apply (13) with

Thus, for Υ n defined in (13) we have

Applying the Cauchy-Schwartz inequality, we get

Now it remains to bound the last term in (13). Using the Cauchy-Schwartz inequality and Lemma 4, we obtain that

and the statement follows from [63, Corollary 2.3].

B.1 Proof of auxiliary lemmas for Theorem 2.

Our proof of Theorem 2 is based on the key lemma below, which allows us to bound E 1 / 2 [ ‖ D -D ( i ) ‖ 2 ] for i ∈ { n +1 , . . . , 2 n } .

Lemma 4. Assume A1, A2, and A3. Then

Proof. Since both terms in the right-hand side of the error bound of Proposition 4 scales linearly with √ κ Q , for simplicity we do not trace it in the subsequent bounds (i.e. assume κ Q = 1 ), and then keep the required scaling with κ Q only in the final bounds. Consider the sequences of noise variables

which differ only in position i , n +1 ≤ i ≤ 2 n , with Z ′ i being an independent copy of Z i . Consider the associated SA processes

glyph[negationslash]

where Y k = Z k for k = i and Y i = Z ′ i . From the above representations we easily observe that θ k = θ ( i ) k for k < i , moreover,

where ε i = ε ( Z i ) and ε ′ i = ε ( Z ′ i ) . Representation (38) implies, together with Proposition 4 and c 0 ≤ a , that

Moreover, for any j > i one observes, expanding (37), that

We use the above representations to estimate E 1 / 2 [ ‖ D -D ( i ) ‖ 2 ] . Using Minkowski's inequality,

and bound the respective differences separately. Recall that here D 1 -D 4 are defined in (25), and D ( i ) 1 -D ( i ) 4 are their respective counterparts with Z i substituted with Z ′ i . First we note that the term D 1 = D ( i ) 1 for any n +1 ≤ i ≤ 2 n . Next, using (40) and (39), we get

In the inequality (a) above we additionally used the stability of matrix product introduced from Corollary 4. Summing the above inequality for i = n +1 to 2 n and applying Lemma 2, we get

Now we proceed with the difference D 3 -D ( i ) 3 . Using (25), we get

The expression above is a sum of martingale-difference terms w.r.t. filtration F ′ k = σ ( Z ′ i , Z glyph[lscript] , glyph[lscript] ≤ k ) . Hence, we get, using (40) and Proposition 4, that

Using now the bound (39), we obtain that

In the above formula in (a) we additionally used that, since α i a ≤ 1 / 2 ,

Hence, combining everything in (43), and using additionally that α i ≤ a , we get

Summing the above inequality for i = n +1 to 2 n , and using that α k = c 0 /k γ , we get

and, hence, using again α i ≤ a , we get

where for the last identity we used the fact that α k = c 0 /k γ , and (44). It remains to upper bound the difference D 4 -D ( i ) 4 . Note first that, proceeding as in (31), we get

Using now the definition of D 4 in (25), we have that

Hence, using the bound (39) and taking sum for i = n +1 to 2 n ,

Changing now the summation order, we obtain that

Hence, combining the above bounds, we get

It remains now to combine (42), (43), and (45) in (41) and use that c 0 ≤ a

B.2 Relations between ρ Conv n and integral probability metrics

In this section we closely follow the exposition outlined in [24]. Consider two R d -valued random variables X and Y . Then the integral probability metric [75], associated with the class of functions H = { h : R d → R , E [ | h ( X ) | ] < ∞ , E [ | h ( Y ) | ] < ∞} , is defined as

Different choices of H induce different metrics, in particular, we mention the following:

glyph[negationslash]

where Conv( R d ) refers to the set of convex sets in R d , ‖ h ‖ Lip = sup x = y ‖ h ( x ) -h ( y ) ‖ ‖ x -y ‖ , and the quantity | h | j is defined as

In other words, for m ∈ N , the class H [ m ] corresponds to the functions with bounded derivatives up to the ( m -1) -th order. The class H K induces the Kolmogorov distance between distributions [75], class H Conv induces the metric ρ Conv n defined in (2), which is the main object of studies in the current paper. Class H W induces the celebrated Wasserstein distance, and classes H [ m ] induce smoothed Wasserstein distances. We will denote the respective metrics by d K , ρ Conv n , d W , and d [ m ] , respectively. Then, obviously,

for any random vectors X and Y . Other relations are more involved. When Y is a multivariate normal vector, it is known (see e.g. [49]) that

.

where the constant C in the above inequality depends on the covariance matrix of vector Y . This inequality justifies comparison of our bounds of Theorem 2 with the result of [65]. The authors in [2] considered integral probability metric d [2] and obtained rate of convergence

where Y ∼ N (0 , Σ ∞ ) , and C 1 in the above inequality stands for a constant depending upon problem dimension d and other instance-dependent parameters from A2. Applying the result of [24, Proposition 2.6] yields

Thus, the result of [2] implies rate of convergence of √ n ( ¯ θ n -θ glyph[star] ) to normal law N (0 , Σ ∞ ) of order n -1 / 6 in a sense of Kolmogorov distance d K . Our result of Theorem 2 implies the respective rate of order n -1 / 4 . At the same time, it is not clear if ρ Conv n can be directly related to d [2] .

C Bootstrap validity proof

C.1 Proof of Theorem 3

We first define explicitly the remainder term ∆ 3 outlined in the statement of Theorem 3, that is,

In the above bounds we do not trace the precise dependence on the constant c 0 from the definition of the step size. We now define the following sets, with the convention α glyph[lscript] = c 0 / √ glyph[lscript] :

Then, due to Corollary 2, we have that P (Ω 1 ) ≥ 1 -1 n . Similarly, due to Corollary 5, P (Ω 2 ) ≥ 1 -1 n . The bounds on P (Ω 3 ) and P (Ω 4 ) follows from Lemma 5 and Lemma 6, respectively. Similarly, Proposition 7 implies that P (Ω 5 ) ≥ 1 -1 n . Hence, based on the sets above, we can construct

,

such that P (Ω 0 ) ≥ 1 -6 n . All further on, we restrict ourselves to the event Ω 0 . Restricting to this event, we obtain that, with Minkowski's inequality,

where we set ξ b ∼ N (0 , ¯ A -1 Σ b ε ¯ A -glyph[latticetop] ) , Σ b ε = n -1 ∑ n glyph[lscript] =1 ε glyph[lscript] ε glyph[latticetop] glyph[lscript] , and ξ ∼ N (0 , Σ ∞ ) , where Σ ∞ = ¯ A -1 Σ ε ¯ A -glyph[latticetop] . Now we control the first supremum using Theorem 4, second one using Theorem 5, and third with Theorem 2.

Lemma 5. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then

Proof. The proof follows directly from the matrix Bernstein inequality, e.g. [69]. We note that

and

Lemma 6. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then

and let F k,l +1 = σ { Z j , glyph[lscript] + 1 ≤ j ≤ k } , glyph[lscript] + 1 ≤ k ≤ 2 n . Then E [ X k |F k -1 ,glyph[lscript] +1 ] = 0 . Let S glyph[lscript] = ∑ n k = glyph[lscript] +1 X k . Note that on Ω 2 , quadratic variation of S glyph[lscript] can be controlled as

Furthermore, on Ω 2

It remains to apply the Freedman inequality for matrix-values martingales [68] and use the union bound over glyph[lscript] ∈ [ n, 2 n -1] .

Lemma 7. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then

Proof. Denote

Proof. We first fix h ∈ [1; n ] , m ∈ [ n, 2 n -h ] , and consider the random variable

Then we control its variance as

moreover, ‖ ( A glyph[lscript] -¯ A )( A glyph[lscript] -¯ A ) glyph[latticetop] ‖ ≤ C 2 A . Applying now the matrix Bernstein inequality [69], we obtain that with probability at least 1 -1 /n 3 , we have

In the last line here we used that d ≤ n . Rest of the proof follows by taking union bound over h and m together with ‖ B ‖ 2 Q ≤ κ Q ‖ B ‖ 2 valid for any matrix B ∈ R d × d .

Lemma 8. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then

Proof. It is easy to check that ‖ ε glyph[lscript] ε glyph[latticetop] glyph[lscript] -Σ ε ‖ ≤ ‖ Σ ε ‖ + ‖ ε ‖ 2 ∞ . Moreover,

It remains to apply the matrix Bernstein inequality together with the bound n ≥ d from A3.

C.2 Rate of Gaussian approximation in the bootstrap world

The main result of this section is the following theorem.

Theorem 4. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then, conditionally on the event Ω 0 , the following error bound holds:

where ξ b ∼ N (0 , ¯ A -1 Σ b ε ¯ A -glyph[latticetop] ) and Σ b ε = n -1 ∑ n glyph[lscript] =1 ε glyph[lscript] ε glyph[latticetop] glyph[lscript] .

Proof. Since both terms in the right-hand side of the error bound of Proposition 4 scales linearly with √ κ Q , for simplicity we do not trace it in the subsequent bounds (i.e. assume κ Q = 1 ), and then keep the required scaling with κ Q only in the final bounds. Recall first that the quantities θ b k and θ k are defined in (16). We start from the following decomposition:

Taking average for k from n +1 to 2 n , we get after multiplying by √ n that

The formula (48) resembles the key representation T b := n A θ b -θ ) = W b + D b , where

√ ¯ ( ¯ n ¯ n D b = D b 1 + . . . + D b 5 , (49)

and D b 1 -D b 5 are defined in (48). Now we aim to apply the result of [63]:

where ξ glyph[lscript] = 1 √ n ( w glyph[lscript] -1) ε glyph[lscript] . We finish the proof by the application of the formula (50). In order to bound the quantities E b ‖ D b ‖ 2 and E b ‖ D ( b ,i ) ‖ 2 , we apply the respective results of Proposition 5 and Proposition 6, respectively. Namely, applying the Cauchy-Schwartz inequality together with Proposition 5, we get that on the event Ω 0 it holds

Similarly, applying Minkowski's inequality and Proposition 6, we obtain that

Now it remains to combine the bounds above in (50).

Proposition 5. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then, conditionally on the event Ω 0 , the following error bound holds:

where glyph[lessorsimilar] stands for inequality up to an absolute constant.

Proof of Proposition 5 is provided below in Appendix C.5. The lemma below is a direct counterpart of Lemma 4.

Proposition 6. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then, conditionally on the event Ω 0 , the following error bound holds:

Proof of Proposition 6 is provided below in Appendix C.6.

Lemma 9. For any k ≥ n on the set Ω 0 the following inequality holds:

Proof. A direct application of Lemma 11 with L = 0 yields that

Now to complete the proof it remains to notice that C A ≥ a .

Lemma 10. For any matrix-valued sequences ( U n ) n ∈ N , ( V n ) n ∈ N and for any M ∈ N , it holds that:

C.3 Gaussian comparison inequality

Theorem 5. Assume A1 and A2. Then on the set Ω 3

Proof. We will use the following inequality

Applying (51) we obtain

It remains to apply definition of Ω 3 .

C.4 Auxiliary technical results.

For the analysis of the difference term θ b k -θ k we use the perturbation expansion technique introduced in [1], see also [16]. Within this approach, we represent the fluctuation component of the error ˜ θ ( fl ) n defined in (26) as

where the latter terms are defined by the following pair of recursions

Moreover, it is known that for L ≥ 1 the term H (0) n can be further decomposed as follows:

Here the terms J ( glyph[lscript] ) n and H ( glyph[lscript] ) n are given by the following recurrences:

The expansion depth L here controls the desired approximation accuracy. Informally, one can show that E 1 /p [ ‖ J ( glyph[lscript] ) n ‖ p ] glyph[lessorsimilar] α ( glyph[lscript] +1) / 2 n , and similarly E 1 /p [ ‖ H ( glyph[lscript] ) n ‖ p ] glyph[lessorsimilar] α ( glyph[lscript] +1) / 2 n . Using the outlined expansion, we prove the following lemma:

Lemma 11. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then for any k ≥ n and L ∈ N the following decomposition holds:

where

and the quantities ˜ ε glyph[lscript] are defined as

Moreover, on the event Ω 0 ,

Proof. We start from the decomposition

Expanding the recurrence above till k = n , and using the fact that θ b n = θ n , we get running the recurrence (58), that

Hence, proceeding as in (52), we obtain the representation

It is easy to check that J ( b , 0) k + H ( b , 0) k = θ b k -θ k . Similarly, with further expansion of H ( b , 0) k along the lines of (53), we arrive at the decomposition (54). Since w k for k = n +1 , . . . , 2 n are i.i.d., we get using the definition of the events Ω 1 and Ω 2 , that on the event Ω 0 :

where for the last bound we have additionally used that α glyph[lscript] log 2 n/a ≤ 1 for glyph[lscript] ≥ n . The latter bound is guaranteed by A4. Hence, using the bound (59) together with the definition of J b , 0 k , we obtain that

where we additionally used the fact that k ∈ [ n ; 2 n ] and n satisfies A4. Assume now that the bound on J b ,j -1 k has a form

Then, using the martingale property of J b ,j k , we write that

and thus the bound (56) is proved. Moreover, using (55) and Minkowski's inequality, we obtain that

and (57) follows.

C.5 Proof of Proposition 5

Recall that the quantity D b is defined in (49). Since θ b n = θ n , we conclude that D b 1 = 0 . To estimate other terms we will use the main error decomposition outlined in Lemma 11, that is, the expansion

applied with different L ≥ 0 . To bound D b 2 we take L = 0 and obtain

To estimate D b 3 we note that

where we have set, respectively,

It follows from Lemma 9 that on the event Ω 0 it holds

Moreover, on the set Ω 0 it holds (since Ω 0 ⊆ Ω 1 ), that

Combining the above bounds, we get

Now we proceed with the term D b 4 . Applying Minkowski's inequality, we get

It remains to upper bound the term D b 5 . Using the decomposition, suggested by Lemma 11 with L = 2 , we get that

Here we have to consider expansion until H b , 2 , since dealing with the latter term (outlined as D b 5 , 4 in the above expansion) is possible only with Minkowski's inequality. Now we consider the summands D b 5 , 1 -D b 5 , 4 separately. Consider first the term D b 5 , 1 . Changing the summation order, we obtain

Then on the event Ω 0 we get, since Ω 0 ⊆ Ω 4 , and using that n satisfies A4,

Combining the above bound together with the one provided by (59), we obtain that

Similarly, for the term D b 5 , 2 we get, changing the order of summation, that

Hence, using the bound (60) together with (56), we get

We proceed with D b 5 , 3 . We change the summation order and proceed exactly as with D b 5 , 2 . Indeed,

and

It remains to upper bound D b 5 , 4 . Proceeding as above, we change the summation order, and obtain

Applying Minkowski's inequality, we get

Now the result follows from the representation (49) and combinations of the above bounds for D b 1 -D b 5 .

C.6 Proof of Proposition 6

Consider the sequences of weights

which differs only in position i , n +1 ≤ i ≤ 2 n , with w ′ i being an independent copy of w i . Consider the associated SA processes

glyph[negationslash]

where w ( i ) k = w k for k = i and w ( i ) i = w ′ i . Respective random variables D b and D ( b ,i ) are based on the first and second sequences from (61), respectively, and are constructed according to the equation (48). From the above representations we easily observe that θ b k = θ ( b ,i ) k for k < i , moreover,

where ε i = ε ( Z i ) and ε ′ i = ε ( Z ′ i ) . From the above representation we get, applying Lemma 9 and (59), that

where for the last line we have additionally assumed that α i glyph[lessorsimilar] a for i ≥ n . Moreover, for any j > i one observes, expanding (62), that

Thus, similarly to (41), we obtain that

and bound the respective differences separately. By the construction of the process above, we note that D b 1 = D ( b ,i ) 1 . Proceeding further, and using the equation (48), we obtain that

Thus, taking sum for i from n +1 to 2 n , and applying Lemma 2, we get that

Here in the last line above we used a particular form α k = c 0 / √ k , and relied on the bound

which is guaranteed by the lower bound on the trajectory length n of the form

The latter condition is guaranteed by A4. Now we proceed with D b 3 -D ( b ,i ) 3 . Using its definition in (48), we get

Since the latter term is a martingale-difference, we obtain that

Hence we obtain, using (63) together with the definition of Ω 1 , that

Considering the latter term in the sum, we obtain

Thus, summing the equations (65) for i from n +1 to 2 n , we obtain that

Using now the definition of D b 4 in (48) and Minkowski's inequality, we write

Thus, taking sum for i from n +1 to 2 n , we get

Similarly, with the definition of D b 5 in (48), we write

Now we bound the terms T 2 and T 3 separately. Indeed, for the term T 2 we get, applying the definition of the set Ω 4 , that

In the above bounds we have used that α glyph[lscript] ≤ 1 a log n . For the term T 3 we get, applying Lemma 10, that for any vector v ∈ R d ,

From the above representation we obtain, using the definition of the set Ω 4 , that

Combining the above bounds, we obtain that

where we have additionally used that α i ≤ a/ C 2 A . Thus, using the upper bound (63), we obtain that

Now it remains to combine the bounds outlined above in (64), (66), (67), and (68), and the statement follows.

D Proof of stability of random matrix product

D.1 Proof of Proposition 1

The fact that there exists a unique matrix Q , such that the following Lyapunov equation holds:

follows directly from [54, Lemma 9 . 1 , p. 140]. In order to show the second part of the statement, we note that for any non-zero vector x ∈ R d , we have where we set

and used the fact that α ≤ α ∞ , where α ∞ is defined in (7).

D.2 Proofs for auxiliary results on products of random matrix

In order to bound the moment E [ ‖ θ k -θ ‖ ] , we first prove a stability results on the products of random matrices Γ m : k arising in the LSA recursion. Towards this aim we first introduce some notations and definitions. For a matrix B ∈ R d × d we denote by ( σ ( B )) d

For q ≥ 1 , the Shatten q -norm of B is denoted by ‖ B ‖ q = { ∑ glyph[lscript] =1 σ glyph[lscript] ( B ) } . For q, p ≥ 1 and a random matrix X we write ‖ X ‖ q,p = { E [ ‖ X ‖ p q ] } 1 /p . Our proof technique is based on the stability results arising in [29], see also [16].

glyph[star] p glyph[lscript] glyph[lscript] =1 its singular values. d q 1 /q

Lemma 12 (Proposition 15 in [16]) . Let { Y glyph[lscript] } glyph[lscript] ∈ N be an independent sequence and P be a positive definite matrix. Assume that for each glyph[lscript] ∈ N there exist m glyph[lscript] ∈ (0 , 1) and σ glyph[lscript] > 0 such that ‖ E [ Y glyph[lscript] ] ‖ 2 P ≤ 1 -m glyph[lscript] and ‖ Y glyph[lscript] -E [ Y glyph[lscript] ] ‖ P ≤ σ glyph[lscript] almost surely. Define Z k = ∏ k glyph[lscript] =0 Y glyph[lscript] = Y k Z k -1 , for k ≥ 1 and starting from Z 0 . Then, for any 2 ≤ q ≤ p and k ≥ 1 ,

where we recall that κ P = λ -1 min ( P ) λ max ( P ) .

Now we aim to bound Γ m : k defined in (27) using Lemma 12. We identify the latter with ∏ k glyph[lscript] = m Y glyph[lscript] , where Y glyph[lscript] = I -α glyph[lscript] A glyph[lscript] , glyph[lscript] ≥ 1 , and Y 0 = I . Applying the bound (8), we get ‖ E [ Y glyph[lscript] ] ‖ 2 Q = ‖ I -α glyph[lscript] ¯ A ‖ 2 Q ≤ 1 -aα glyph[lscript] . Further, assumption A2 implies that almost surely,

Therefore, (70) holds with m glyph[lscript] = aα glyph[lscript] and σ glyph[lscript] = b Q α glyph[lscript] . As ‖ I ‖ p = d 1 /p , we obtain the following corollary.

Corollary 3. Assume A1 and A2. Then, for any α glyph[lscript] ∈ [0 , α ∞ ] , 2 ≤ q ≤ p , and 1 ≤ m ≤ k , it holds

where α ∞ was defined in (7) , and b Q = √ κ Q C A .

Corollary 4. Assume A1, A2, and A3. Then for any 2 ≤ q ≤ log n , and any k ≥ n , 1 ≤ m ≤ k , it holds that

where α ∞ is defined in (7) . Moreover,

Proof. We first apply the result of Corollary 3. Indeed, for k ≥ n , and any 2 ≤ q ≤ p , it holds, setting b Q = √ κ Q C A , that

Note that, setting p = log n , and provided that n satisfies (9), we easily obtain that, for glyph[lscript] ≥ n/ 2 ,

Hence, for m ≥ n/ 2 , we have

and the statement follows. Suppose now that m<n/ 2 . In such a case we have, applying (73), that

and we need to bound the first term in the product. We first consider α glyph[lscript] = c 0 glyph[lscript] -1 / 2 , and use the inequalities

and

Thus, it is enough to satisfy the constraint

Since m<n/ 2 , it is enough to ensure that

or, equivalently, which is granted by A3. Combining the above bounds in (74), we obtain that the lemma's statement (71) holds for the step size α glyph[lscript] = c 0 /glyph[lscript] 1 / 2 . Similarly, for α glyph[lscript] = c 0 /glyph[lscript] γ with γ ∈ (1 / 2; 1) , we get for m ≥ n/ 2 that

since the relation (73) holds. Similarly, for m<n/ 2 , the desired upper bound would follow from the inequality

together with an inequality and

The latter inequality can be re-written as

which is also granted by A3. Combining the above inequalities implies that (71) holds for α glyph[lscript] = c 0 /glyph[lscript] γ . The bound (72) can be immediately obtained from (71) using the fact that e -x ≤ 1 -x/ 2 for x ∈ [0; 1] .

Corollary 5. Under conditions of Corollary 4 it holds with P - probability at least 1 -1 /n 2 that

Proof. It is sufficient to choose q = 2log n and use Markov's inequality together with the union bound.

Proposition 7. Assume A1, A2, A3 with γ = 1 / 2 , and A4. Then on the set Ω 5 defined in (47) , it holds for any n ≤ m ≤ k ≤ 2 n , that

Proof. Our proof relies on the auxiliary result of Lemma 13 below together with the blocking technique. Indeed, let us represent

where r < h and h = h ( n ) is a block size defined in (18). Then we obtain, using the independence of bootstrap weights w m +1 , . . . , w k , that

In the last inequality we applied Lemma 13 to each of the blocks of length h in the first bound. It remains to upper bound the residual terms. Since the remainder block has length less then h , we have due to (81) (which holds according to A4), that

where the last inequality is due to Proposition 1. Next,

Since

we get from previous bound

where in the last line we additionally used (19).

Lemma 13. Assume A1, A2, A3 with γ = 1 / 2 , and A4. On the set Ω 5 defined in (47) , it holds for h = h ( n ) defined in (18) and any m ∈ [ n ; 2 n -h ] , that

Proof. Recall that we use the notation E b [ · ] = E [ ·|Z 2 n ] , where Z 2 n = ( Z 1 , . . . , Z 2 n ) are the random variables used in the construction of the iterates { θ k } 1 ≤ k ≤ n in (1).

Let h ∈ N be a block length, which value will be determined later, and consider a product

.

Expanding the product of matrices (77), we obtain

where S = ∑ m + h glyph[lscript] = m +1 α glyph[lscript] ( w glyph[lscript] -1) A glyph[lscript] is a linear statistics in { w glyph[lscript] } m + h glyph[lscript] = m +1 , and the remainder R collects the higher-order terms in the products

with I glyph[lscript] r = { ( i 1 , . . . , i r ) ∈ { m +1 , . . . , m + h } r : i 1 < · · · < i r } . We first consider the contracting part in matrix Q -norm. Indeed, applying (8), we obtain that

provided that h is set in such a manner that ∑ m + h glyph[lscript] = m +1 α glyph[lscript] ≤ α ∞ , where α ∞ is defined in (7). Hence, we get from the above inequality that for any u ∈ R d , it holds that

Now we need to estimate the remainders in the representation (78). On the set Ω 5 , it holds that

Moreover, it is straightforward to check that

In order to bound the remainder term R , we note that

To complete the proof it remains to set the parameter h in such a way that we can guarantee

keeping at the same time the constraint

Recall that α glyph[lscript] = c 0 / √ glyph[lscript] . Thus, using the bounds (75) and (76), we obtain that

and

Hence, taking into account (81) and (82), and 1 m +1 ≤ 1 m , the inequality (79) would follow from the bound

Since log (1 + x ) ≤ x for x ≥ 0 and c 0 C 2 A κ Q e ≤ 1 , the latter inequality is satisfied if

Now we use one more lower bound

which follows from an elementary inequality √ 1 + x ≥ 1 + ( √ 2 -1) x , valid for 0 ≤ x ≤ 1 . Hence, (83) would from the inequality

Setting h is such a manner that inequality (84) would follow from

The latter inequality is satisfied, if the block size h satisfies

Thus, setting h ( n ) as in (18), all previous inequalities will be fulfilled, provided that

Here last inequality follows from (80) and the following simple bounds, where we use that m ≥ n and √ 1 + x ≤ 1 + x/ 2 :

Now (78) implies that

and the statement follows from an elementary inequality 1 + x ≤ e x .

E Applications to the TD learning

Recall that the temporal difference learning algorithm in the LSA's setting can be written as

Recall that our aim is to estimate the agent's value function

where a k ∼ π ( ·| s k ) , and s k +1 ∼ P( ·| s k , a k ) , for any k ∈ N . We define the transition kernel under policy π

which corresponds to the 1 -step transition probability from state s to a set B ∈ B ( S ) . We denote by µ the invariant distribution over the state space S induced by the transition kernel P π ( ·| s ) in (87). In this case the TD learning updates (85) correspond to the approximate solution of the deterministic system ¯ A θ glyph[star] = ¯ b , where we have set, respectively,

E.1 Proof of Proposition 2

We first need to check that the matrix ¯ A + ¯ A glyph[latticetop] , where ¯ A is defined in (88), is positive-definite. In order to show this fact we closely follow the exposition of [61, Lemma 18] and [51, Lemma 5]. Define a random matrix A as an independent copy of A k from (86), that is,

where s ∼ µ , and s ′ ∼ P π ( ·| s ) . With the definition of A , we get that

where we used an elementary inequality uv glyph[latticetop] + vu glyph[latticetop] glyph[precedesequal] ( uu glyph[latticetop] + vv glyph[latticetop] ) valid for any u, v ∈ R d . Hence, with the definition of Σ ϕ in (22), we get

Hence, ¯ A + ¯ A glyph[latticetop] is positive-definite, and we can set P = ¯ A + ¯ A glyph[latticetop] in the right-hand side of the Lyapunov equation (69). Obviously, Q = I is a solution to the corresponding Lyapunov equation

Moreover, applying [61, Lemma 18], we obtain

Hence, we get for α ≤ (1 -γ ) / (1 + γ ) 2 , and applying (89) and (90), that

Hence, the bound (8) holds with a = (1 -γ ) λ min (Σ ϕ ) and α ∞ = (1 -γ ) / (1 + γ ) 2 .

where A k and b k are given by

F Experimental details for the TD learning

Here we provide some details on numerical experiments. Code to run experiments is provided in https://github.com/svsamsonov/BootstrapLSA . For the considered Garnet problem we choose the policy π in the following way. For any a ∈ A , we set

where the U ( s ) i are independent random variables following uniform distribution U [0 , 1] . Here we assume that each action a ∈ A can be selected at any state s ∈ { 1 , . . . , N s } . We generate an instance of Garnet problem with mentioned parameters, and find analytically the true parameter θ glyph[star] . In order to estimate the supremum

η ∼ N (0 , I N s ) , and show that this supremum scales as n -1 / 4 when γ = 1 / 2 and admits slower decay for other powers of γ . We first approximate true probability P ( ‖ Σ 1 / 2 ∞ η ‖ ≤ x ) by the corresponding empirical probabilities based on sample of size M glyph[greatermuch] n . We fix M = 5 · 10 7 . We choose trajectory lengths n ∈ { 1600 , 3200 , 6400 , 12800 , 25600 , 51200 , 102400 , 204800 , 409600 , 819200 , 1638400 } ,

fix the length of burn-in period n 0 = 102400 , and generate N = 6553600 independent trajectories starting in the fixed point θ 0 ∈ R N s . Weset the learning rate schedule as α k = c 0 /k γ and try different values γ ∈ { 0 . 5 , 0 . 65 , 0 . 7 } , and c 0 = 4 . 0 . Unfortunately, even the chosen order of trajectory length n seems to be insufficient in order to significantly distinguish, for example, between γ = 0 . 5 and γ = 0 . 65 . However, learning rate schedule with faster decay performs worse in terms of ∆ n . Note that the current experiment is already rather computationally intense for artificial problem and takes about 12 hours of compute on a Core i9 - 10920x processor with 12 cores with 3.7 GHz.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Main results are, respectively, the ones of Theorem 2 and Theorem 3, their statements are complete and supported by the proofs in the Appendix section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our setting related to the LSA problem, and not more general non-linear stochastic optimisation problems. We highlight the potential generalizations of Theorem 2 to the non-linear setting and discuss why generalizing Theorem 3 might be more challenging. We also discuss the limitation related to i.i.d. observations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All our theoretical results are provided with references to assumptions, that are stated in Section 3 and Section 4. All results are given with proofs, that are correctly referenced for each theorem and corollary.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Numerical results are stated with a complete description of the environments that are used, as well as the precise sets of hyperparameters that we used. The code (in Python) is provided as supplementary with the paper, making it easy for one to reproduce our numerical experiments. At the same time, tracing the second-order terms in the normal approximation is computationally involved and can take sufficiently large amount of time.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All code is open source, link to a github repository is included.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The algorithm used in the numerical experiments are exactly the algorithms described in the paper. The Garnet environements are given with the parameters used for generation, and with reference to the original problem.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Unfortunately, error bars for computing the second-order terms in normal approximation are quite computationally intense, moreover, tracing the terms of order n 1 / 4 requires quick increase of trajectory length n .

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All necessary information to reproduce experiments is provided in Appendix F. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper is of purely theoretical nature, and the proposed methods do not deal with sensitive attributes that could induce unfairness or privacy issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is of purely theoretical nature. We do not foresee any societal harm from the proof of non-asymptotic bootstrap validity and normal approximation bounds in Kolmogorov distance.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Not applicable: no existing assets are used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Not applicable: paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable: paper does not involve crowdsourcing nor research on human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable: paper does not involve crowdsourcing nor research on human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents advancements in the theoretical understanding of the linear stochastic approximation (LSA) algorithm. \n\nIt establishes the Berry–Esseen bound for the normal approximation of Polyak-Ruppert averaged iterates, achieving an optimal rate with an aggressive step size of $\\alpha_k \\approx k^{-1/2}$. Additionally, it demonstrates the non-asymptotic validity of confidence intervals using a novel multiplier bootstrap procedure, marking a first in this domain. \n\nThe practical utility of these theoretical results is showcased through applications in temporal difference (TD) learning for reinforcement learning.\n\nSTRENGTHS:\n1. **Theoretical Advancements**: The paper makes theoretical contributions by establishing the Berry–Esseen bound for the normal approximation of Polyak-Ruppert averaged iterates. Though some previous works have done this in special cases before, this work differs from them in providing a tighter bound.\n\n2. **Good Clarity:** The paper is easy to follow and well-written. The proof seems correct (not checked very carefully).\n\nWEAKNESSES:\n1. **Strong Assumptions**: The requirement that $\\epsilon(z)$ is uniformly bounded is a strong assumption that might limit the applicability of the results. Relaxing this condition to weaker ones, such as finite moments, could enhance the paper's generality.\n\n2. **Missing References**: There are some missing references, see the limitations.\n\n3. **Discussion on Lower Bounds**: The paper focuses on deriving upper bounds but lacks a discussion on the tightness and potential lower bounds. Addressing this aspect could provide a more comprehensive understanding of the bounds' efficacy and limitations.\n\nQUESTIONS:\nSee the Limitations.\n\nLIMITATIONS:\n1. **Missing References on Statistical Inference**: Statistical inference for nonlinear stochastic approximation has been considered by [1*] and [2*], which is highly related to this manuscript but hasn’t been cited. Note that [1*] provides a Berry-Esseen-like bound for the whole trajectory rather than the averaged iterates. These references could be cited after [36] in line 116.\n     - [1*] Li, Xiang, Jiadong Liang, and Zhihua Zhang. \"Online statistical inference for nonlinear stochastic approximation with Markovian data.\" arXiv preprint arXiv:2302.07690 (2023).\n     - [2*] Li, Xiang, et al. \"A statistical analysis of Polyak-Ruppert averaged Q-learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.\n\n2. **Assumption on Bounded $\\epsilon(z)$**: Theorem 1 requires that $\\epsilon(z)$ is uniformly bounded, which is a strong condition. Is it possible to relax this condition to a weaker one, such as $\\epsilon(z)$ only having a finite order of moments (such as the fourth order moment or smaller)?\n\n3. **Tightness of Derived Upper Bounds**: The paper provides upper bounds, but it would be insightful to discuss the tightness of these bounds. Are there any thoughts or conjectures regarding potential lower bounds?\n\n#### Minor Corrections:\n\n1. **Figure 1**: The last subfigure should be labeled as (c).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present paper studies linear stochastic approximation with martingale difference noise and diminishing step-sizes. The authors obtain Berry-Esseen bounds for the parameter sequence with Polyak-Ruppert averaging as well as a generalization of finite-time bounds for estimation confidence intervals for parameters in LSA. The obtained Berry-Esseen bounds are illustrated by a TD learning numerical example.\n\nSTRENGTHS:\nTo the best of the reviewer's knowledge, both of the contributions are novel. In particular, I believe that this is the first Berry-Esseen bound type bound to be obtained for general linear SA, which is is exciting to see.\n\nThe assumptions, contributions and approach to analysis are objectively identified. The authors also did a great job in providing discussions/remarks/intuition for their results.\n\nThe paper is well-written but some proofreading is recommended.\n\nWEAKNESSES:\nAlthough the authors did a good job in outlining the scope and contributions of the paper, the analysis and main text are hard to follow given the number of symbols and equations. It is easy for a reader to get lost/distracted midway and it is very hard to keep track of the definitions of each of the terms \n\nI understand that this is an issue with theory papers like this, but would encourage the authors to move unnecessary terms or inequalities to the appendix (e.g. (9) in A3. The exact lower bound adds very little in the main text in my opinion. Its definition could have been postponed to the Appendix.)\n\nThe numerical experiments are also weak since many of the plots are not related to the contributions of the paper itself. Plot c seems to be the only one directly related to the theorems of the paper, but it still does not illustrate the theory that well. Maybe including a plot of C k^{-1/4}  to plot (b) where C is a constant for comparison could help in inferring convergence rates.\n\nAlso, there is a mistake in the label of Figure 1. Subfigure (b) is mentioned twice.\n\nQUESTIONS:\n- Could the authors clarify if using Polyak Ruppert averaging is necessary to obtain such a Berry-Esseen bound? It would be exciting to see bounds for unaveraged estimates as well.\n\n-Could the authors  run  the experiment  supporting figure (c) for longer to see if the curve with \\gamma = 1/2 willnot continue to go upwards eventually?\n\nLIMITATIONS:\nThe authors clearly identified the limitations of their results through a clear list of assumptions", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Overview\n\nLet $Z, Z_1, \\dots, Z_n$ be i.i.d. random elements with a common distribution $\\pi$ over $\\mathbf{Z}$. Given $A : \\mathbf{Z} \\to \\mathbb{R}^{d\\times d}$ and $b : \\mathbf{Z} \\to \\mathbb{R}^d$, the goal of the LSA procedure is to find the unique solution $\\theta^\\star$ of\n$$\\mathbb{E}\\left(A(Z)\\theta^\\star - b(Z)\\right) = 0. $$\n\nGiven a decreasing sequence of step sizes $\\alpha_k$ and a starting point $\\theta_0$, the standard LSA is given by\n$$\\theta_k = \\theta_{k-1} - \\alpha_k(A(Z_k)\\theta_{k-1} - b(Z_k)) $$\nand the Polyak-Ruppert averaged LSA is given by\n$$\\overline{\\theta_n} = \\frac{1}{n} \\sum_{k=n}^{2n-1} \\theta_k. $$\n\nThe authors provide Berry-Essen-type bounds for the Gaussian approximation of $\\sqrt{n} \\left( \\overline{\\theta_n} - \\theta^\\star \\right)$ and for the corresponding multiplicative bootstrap process. Namely, for the Gaussian approximation result they upper bound the quantity\n$$\\rho_n = \\sup_\\text{B convex} \\left| \\mathbb{P}\\left( \\sqrt{n} \\left( \\overline{\\theta_n} - \\theta^\\star \\right) \\in B \\right) - \\mathbb{P}\\left( \\Sigma_\\infty^\\frac{1}{2} \\eta \\in B \\right)\\right| $$\nwhere $\\eta \\sim \\mathcal{N}(0,I_d)$. And for the bootstrap approximation result they upper bound\n$$\\rho_n^b = \\sup_\\text{B convex} \\left| \\mathbb{P}\\left( \\left.\\sqrt{n} \\left( \\overline{\\theta_n^b} - \\overline{\\theta_n} \\right) \\in B \\right| Z_1, \\dots, Z_{2n} \\right) - \\mathbb{P}\\left( \\sqrt{n} \\left( \\overline{\\theta_n} - \\theta^\\star \\right) \\in B \\right)\\right| $$\nwhere $\\overline{\\theta_n^b}$ are obtained from a multiplier bootstrap process. It is interesting to notice that this multiplier process can be evaluated online without keeping a history in memory.\n\n## Overall proof arguments\n\nTo provide the Gaussian approximation result the authors write\n$$\\theta_n - \\theta^\\star = (I - \\alpha_n A(Z_n))(\\theta_{n-1} - \\theta^\\star) - \\alpha_n \\varepsilon(Z_n) $$\nwhere\n$$\\varepsilon(z) = A(z)\\theta^\\star - b(z). $$\n\nThe authors observe that $\\varepsilon(Z_n)$ can be viewed as a noise, which is assumed to be bounded. Meanwhile, the operator $ I - \\alpha_n A(Z_n) $ is a random perturbation around $ I - \\alpha_n \\mathbb{E}A(Z) $, which is shown to act as a contraction in an appropriate norm, provided that $-\\mathbb{E}A(Z)$ is Hurwitz. Thus, when one take the PR average the noise is expected to behave as the sum of i.i.d. noises and the contraction term must shrink. This is formally done writing the PR average in the form of Theorem 2.1 [reference 60 of the paper] and bounding the terms given by the latter theorem.\n\nThe proof the bootstrap approximation result follows the standard practice in this literature:\n- First, conditionally on the sample, a Gaussian approximation is obtained relating the bootstrap to a Gaussian with its covariance.\n- Second, a Gaussian comparison theorem relates the latter Gaussian with the desired Gaussian, with some concentration results being used to bound their difference in high probability.\n\n## Main claims and observations\n\nThe authors show that taking a step size of order $\\frac{1}{\\sqrt{k}}$ yields the best possible convergence rate in their bounds. They provide empirical evidence that this rate is optimal.\nThey also claim to be the first ones to fully provide a non-asymptotic bootstrap approximation result.\n\nSTRENGTHS:\nThe paper seems to be the first to provide non-asymptotic Gaussian and bootstrap approximation bounds for LSA. Their assumptions are quite mild and are in line with the ones made in similar papers in other domains. For instance, assumption A.2 is similar in nature to the boundness assumptions and the strong-covariance assumption made in [A]. The paper is mathematically sound and poses interesting research directions. I'm particularly curious about the convergence rate of $n^{-\\frac{1}{4}}$ suggested by their theoretical results and supported by their experiment. The proposed application to policy evaluation in RL is also interesting. Finally, I point out that the code on the supplementary material was easy to reproduce.\n\n[A] Chernozhukov, Victor, Denis Chetverikov, and Yuta Koike. \"Nearly optimal central limit theorem and bootstrap approximations in high dimensions.\" The Annals of Applied Probability 33.3 (2023): 2374-2425.\n\nWEAKNESSES:\nThe main claim of the paper is that their bounds suggest an optimal convergence rate of $n^{-\\frac{1}{4}}$ when taking $\\alpha_k = \\frac{c}{\\sqrt{k}}$. The bottleneck of this convergence rate comes from an application of Cauchy-Schwarz inequality, the boundness of $\\varepsilon$ and the MSE bound on $D$ given by Theorem 1. Meanwhile, the authors also provide experimental evidence that this convergence rate is indeed optimal (although not considering gamma values below 0.5 in Figures 1 or 3). A counter-example or a deeper discussion on this convergence rate would be beneficial to the paper: is it an artifact of the proof or is there reason to believe it cannot be improved?\n\nQUESTIONS:\nSome observations:\n- Line 54 has a typo in \"Berry-Essee\".\n- In Equation (15) the $\\ell$ can be removed from $\\theta_{k}^{b,\\ell}$ to enhance clarity. It is only used to explain that to evaluate the probability in practice one must run several samples of the bootstrap process.\n- Line 263 has a typo in \"date\".\n- The Equation after line 303 is using $\\phi$ instead of $\\varphi$.\n- Line 307 asks for a sequence of \"TD(0) updates\", what does the \"(0)\" stands for?\n- Figure 1 lacks x and y labels. The legend can also be improved for clarity.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and the potential impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-25"}