Title: Computing the Bias of Constant-step Stochastic Approximation with Markovian Noise
Venue: NeurIPS 2024
Accepted: yes
Keywords: Stochastic approximation; Polyak-Ruppert averaging; Stein's method
Abstract: We study stochastic approximation algorithms with Markovian noise and constant step-size $\alpha$. We develop a method based on infinitesimal generator comparisons to study the bias of the algorithm, which is the expected difference between $\theta_n$ ---the value at iteration $n$--- and $\theta^*$ ---the unique equilibrium of the corresponding ODE. We show that, under some smoothness conditions, this bias is of order $O(\alpha)$. Furthermore, we show that the time-averaged bias is equal to $\alpha V + O(\alpha^2)$, where $V$ is a constant characterized by a Lyapunov equation, showing that $E[\bar{\theta}_n] \approx \theta^*+V\alpha + O(\alpha^2)$, where $\bar{\theta}_n$ is the Polyak-Ruppert average.  We also show that $\bar{\theta}_n$ converges with high probability around $\theta^*+\alpha V$. We illustrate how to combine this with Richardson-Romberg extrapolation to derive an iterative scheme with a bias of order $O(\alpha^2)$.

Computing the Bias of Constant-step Stochastic Approximation with Markovian Noise

Abstract

We study stochastic approximation algorithms with Markovian noise and constant step-size α . We develop a method based on infinitesimal generator comparisons to study the bias of the algorithm, which is the expected difference between θ n -the value at iteration n -and θ ˚ -the unique equilibrium of the corresponding ODE. We show that, under some smoothness conditions, this bias is of order O p α q . Furthermore, we show that the time-averaged bias is equal to αV ` O p α 2 q , where V is a constant characterized by a Lyapunov equation, showing that E ' ¯ θ n ‰ « θ ˚ ` V α ` O p α 2 q , where ¯ θ n ' p 1 { n q ř n k ' 1 θ k is the Polyak-Ruppert average. We also show that ¯ θ n converges with high probability around θ ˚ ` αV . We illustrate how to combine this with Richardson-Romberg extrapolation to derive an iterative scheme with a bias of order O p α 2 q .

1 Introduction

Stochastic approximation (SA) is a widely used algorithmic paradigm to solve fixed-point problems under noisy observations. While SA was introduced in the 1950s [30, 6], it is still widely used today in many applications to solve optimization problems, or to implement machine learning or reinforcement learning algorithms [23, 5]. A typical SA is a stochastic recurrence of the form

where θ n P R d is a vector of parameters, X n and M n ` 1 are sources of randomness, and α n is the step-size. The step-size might be decreasing to 0 as n grow or can be fixed to a small constant (in which case we simply call it α ).

The goal of running a SA algorithm is to obtain a sequence θ n that gets close to the root θ ˚ of the function ¯ f p θ q ' E r f p θ, X q ` M | θ s , the expectation of the iterate (1), as n goes to infinity. The SA procedure described in (1) models many algorithms that are used in machine learning, like first-order optimization and stochastic gradient descent [24, 16], and Q -learning or policy gradient algorithms [5, 35].

The theory of stochastic approximation has been studied extensively and is covered by a lot of textbooks, e.g. , [4, 8, 23]. This theory shows that the limit points of θ n as n grows are similar to the limit points of the ODE 9 x ' ¯ f p x q [3, 9]. In particular, one can show that if all solutions of the ODE converge to θ ˚ , then θ n gets close to θ ˚ as n grows large (under technical conditions on f , the step size α n , and the noise X n or M n ` 1 ). The asymptotic behavior of θ n depends on the nature of the step-size: When the step-size α n depends on n and converges to 0 as n goes to infinity, then θ n generally converges almost surely to θ ˚ . When α n is kept constant and does not depend on n , then θ n will in general not converge to θ ˚ but keeps oscillating around θ ˚ , with fluctuation amplitudes of order O p ? α q . In this work, we focus on the latter case and propose a method to quantify these fluctuations.

We consider a SA algorithm with Markovian noise. That is, θ n evolves as in (1) where the random variable X n evolves as a Markov chain that depends on θ n , and the random variable M n ` 1 is a Martingale difference sequence. One thing that sets this paper apart from others is that we allow the Markovian noise to be dependent on θ n . This is particularly valuable in reinforcement learning application such as Q -learning where the exploration policy might depend on θ n (this would for instance be the case for ε -greedy). It is known that when the step size α is constant, then the expectation of the iterates, E r θ n s , does not converge to θ ˚ as n goes to infinity but has a bias [41, 20]. The goal of our paper is to provide a framework to characterize and compute this bias.

Our main contribution is to provide a framework based on semi-groups, to quantify the convergence rate of θ n to θ ˚ . This framework is similar to Stein's method, that has been recently popularized to obtain accuracy results and refinement terms for fluid limits [10, 17, 18, 40]. It allows us to quantify the distance between the stochastic recurrence (1) and its deterministic counterpart (8) as a function of the distance between the infinitesimal generators of the stochastic and deterministic systems. By using this framework, we obtain two main results. Our first result (Theorem 1) states that, under smoothness conditions on f , there exists a constant C ą 0 such that for all small enough α :

This guarantees that that the bias and variance of θ n are of order at most α .

A classical way to reduce the variance of SA is to use Polyak-Ruppert tail averaging [28, 31], i.e. , to look at the convergence of ¯ θ n ' p 1 { n q ř n k ' 1 θ n instead of θ n . In this paper, we show that, for the constant step-size case, the use of Polyak-Ruppert averaging removes the variance ( i.e. lim n Ñ8 var ' ¯ θ n ‰ ' 0 ), but not the bias: Theorem 2 shows that there exists a vector V and a constant C 1 ą 0 such that:

This shows that the bias of the averaging is exactly of order α . We also show in Theorem 3 that ¯ θ n converges in probability to a point in θ ˚ ` V α ` O p α 2 q as n goes to infinity.

We also provide numerical simulation on a synthetic example that illustrates how to combine the Polyak-Ruppert averaging with a Richardson-Romberg extrapolation [19] to construct a stochastic approximation algorithm whose bias is of order O p α 2 q . This leads to an algorithm that enjoys both a small bias and a fast converge rate, compared to using a time-step of order α 2 .

Roadmap . The rest of the paper is organized as follows. We describe related work in Section 2. We introduce the model and give assumptions and notations in Section 3. We state the main results and provide numerical illustrations in Section 4. We give the main ingredients of the proofs in Section 5. The appendices contain technical lemmas and additional numerical examples.

2 Related work

Stochastic approximation algorithms were first introduced with decreasing step-size [30, 6]. Since these seminal papers, there has been a considerable amount of work aiming at characterizing the asymptotic properties of SA, by relating the asymptotic behavior of θ n with the one of the ODE 9 θ ' ¯ f p θ q [3, 9]. These topics are today well covered by textbooks such as [4, 8, 23].

In this original theory, the goal was to show that θ n is close to the behavior of an ODE but not necessarily to obtain tight rates of convergence. This lead to the development of a new line of research focussing on non-asymptotic properties of SA [27]. This paper derives a unified framework to study the rate of convergence of stochastic gradient descent by relating it to SA with decreasing step-size, and martingale noise ( i.e. , where the function f does not depend on X k in (1)). This framework has been extended by a series of papers for applications in reinforcement learning, see for instance [11, 15, 29, 26], still in the decreasing step-size case, with the goal of understanding the fluctuations of the algorithm around the equilibrium.

For the decreasing step size, the fluctuations of θ n around θ ˚ vanish as n goes to infinity. This is not the case for constant step-size SA, for which understanding the magnitude of the fluctuations is therefore important. To bound the fluctuations, a part of the literature seeks to obtain bounds on the

mean squared error (MSE), E ' p θ n ´ θ ˚ q 2 ‰ as a function of the problem's parameter and the stepsize α . These papers show that the MSE is generally of order O p α q , see for instance [33] for the case of linear model plus Markovian noise, or [13, 15, 14] for contractive stochastic approximations (with Markovian or martingale noise).

The case of constant-step size is also interesting because the θ n has a bias that does not vanish as n goes to infinity. In particular, the Polyak-Ruppert averaging does not converge to θ ˚ : generally lim n Ñ8 ¯ θ n ‰ θ ˚ . It is shown in particular in [16, 20, 25, 32, 41, 42] that a constant step-size SA has an asymptotic bias of order α , which shows that lim n Ñ8 ¯ θ n ' θ ˚ ` V α ` O p α 2 q . Some of these papers, and in particular [16, 20], show that this bias characterization can be coupled with a Richardson-Romberg extrapolation to obtain a more accurate algorithm. In our paper, we also study the case of constant-step size SA with Markovian noise and use the same Polyak-Ruppert and Richardson-Romberg analysis. One of the distinguishable property of our model is that we allow the evolution of the Markovian noise X n to depend on θ n , whereas most of the cited paper consider X n has an external source of noise. This dependence is particularly interesting when studying asynchronous Q -learning algorithms [37, 36], for which the navigation policy is often derived from the current values of the parameter (a popular example is to use an ϵ -greedy policy).

A second distinguishable feature of our paper is the methodology that we use. Our main tool is to relate the distance between the expectation of θ n and θ ˚ as a distance between the infinitesimal generators of the stochastic recurrence (1) and of the deterministic recurrence (8). The method that we develop is tightly connected to Stein's method [34] and in particular with the line of work that use this method to obtain accuracy bounds for mean-field approximation [17, 22, 39, 40]. In particular, the start of our proof, which is to study a hybrid system composed of a stochastic and a deterministic recurrence in equation (9), can be seen as a discrete-time version of what is termed as a 'classical trick' in [17, 22]. Apart from this difference between discrete and continuous time, one of the major features of our model is to have a Markovian noise. Our model can be viewed as a discrete-time version of the recent paper [1]. Some of our results (like our Theorem 1) are analogous to the results of [1] but others (like Theorem 2) are different and require time-averaging, mostly because of the possible periodicity of the discrete-time Markov chains. Also, the model of the current paper is slightly more general than the one of [1]. The fact that our methodology directly studies the expectation of θ n makes it different from convergence results that use concentration equalities [12].

3 Model and preliminaries

3.1 Model and first assumptions

Throughout the paper p θ n , X n q n ě 0 is a discrete-time stochastic process adapted to a filtration p F n q n ě 0 . This stochastic process has two components of different nature. The first component θ , that we call the parameter, is continuous and lives in a compact subset Θ Ă R d : for each n : θ n P Θ Ă R n . The second component X lives in a finite set X , i.e., X n P X .

The evolution of θ and X are coupled in the following way. The parameter θ evolves according to the recurrence equation given by (1). At every time-step n , the process X makes a Markovian transition according to a Markovian kernel K p θ n q . More precisely, for all x, x 1 P X and θ P Θ :

To obtain our results, we will make the following assumptions:

(A1) The process M is a martingale difference sequence (that is: for all n : E r M n ` 1 | F n s ' 0 ) and its conditional covariance is E ' M n ` 1 M T n ` 1 | F n ‰ : ' Q p θ n , X n q . Moreover, we assume that E ' M n ` 1 M T n ` 1 | F n ^ X n ` 1 ‰ ' R p θ n , X n , X n ` 1 q .

(A2) The functions f , K , Q and R are four times differentiable in θ .

(A3) For any given θ P Θ , the matrix K p θ q is unichain 1 .

In addition to these three assumptions, we later add an assumption (A4) about the stability of the ODE around its fixed point. We do not state this assumption here as it needs extra definitions.

3.2 Averaged values, average ODE and stability assumption

Assumption (A3) implies that a Markov chain with kernel K p θ q has a unique stationary measure π p θ q , and we denote by π x p θ q the stationary probability of x P X for such a Markov chain. For any function g defined on Θ ˆ X , we call ¯ g its averaged version, i.e., the function that associates to θ P Θ the value ¯ g p θ q , defined as:

It is shown in [1, Lemma 4], that under Assumption (A3), for each θ P Θ , the kernel K p θ q has a unique stationary distribution π p θ q . The value ¯ g p θ q is equal to E r g p θ, X qs where X is distributed according to the stationary distribution associated to K p θ q .

Following our notation introduced in (3), we denote by ¯ f the averaged version of f . By [1, Lemma 4], the function θ ÞÑ π p θ q is twice differentiable under assumptions (A2) and (A3), which implies that the function ¯ f is also twice differentiable. This implies, for an initial ϑ p 0 q ' θ , the ODE 9 ϑ ' ¯ f p ϑ q has a unique local solution, and we denote the value of this solution at time t by ϕ t p θ q . In order to prove this result, we will need that this ODE has a unique fixed point to which all trajectories converge and that this fixed point is an exponentially stable attractor. This is summarized in the following asusmption:

(A4) There exists a θ ˚ such that for any θ , the solution of the ODE 9 ϑ ' ¯ f p ϑ q is defined for all t ą 0 and converges to θ ˚ : lim t Ñ8 ϕ t p θ q ' θ ˚ for all θ P Θ . Moreover, the derivative of ¯ f at θ ˚ is Hurwitz (i.e., the real parts of all of its eigenvalues are negative).

By classical results on the stability of ODES, this assumption implies that the convergence to θ ˚ occurs exponentially fast, that is, there exists a, b ą 0 such that for all θ P Θ : } ϕ t p θ q ´ θ ˚ } ď ae ´ bt } θ ´ θ ˚ } . To prove that, one can use [21, Theorem 4.13], that shows that θ ˚ is an exponentially stable point of the ODE, i.e. , } ϕ t p θ q ´ θ ˚ } ď a 1 e ´ bt } θ ´ θ ˚ } in a neighborhood N of θ ˚ , because D ¯ f p θ ˚ 0 q is Hurwitz. Then, as Θ is compact, there exists T such that ϕ T p θ q P N for all θ P Θ . The result then follows by choosing a ' a 1 e Tb sup θ P Θ } θ ´ θ ˚ } { sup θ P N } θ ´ θ ˚ } .

3.3 Discussion on the assumptions and limits

Most of the assumptions used in the paper are classical when studying stochastic approximation algorithms with Markovian noise. In this section, we discuss the limits of each assumption (from the most classical one to the most original one). Assumption (A3) is necessary to define the notion of average dynamics ¯ f p θ q and is therefore present in virtually all papers about Markovian noise stochastic approximation. The originality of our assumption is that we allow the transition kernel to depend on θ . For assumption (A1), we add to the classical result the existence of a co-variance matrix Q , which is needed as it appears in the expression of V . Assumption (A4) imposes that the stochastic approximation has a unique exponentially stable attractor. Our results could probably be adapted to a model with multiple attractors (by using large deviation techniques similar to the one of [2, 38] for a two time-scale setting as ours) but this would be another paper.

One important assumption we make is that θ lives in a compact set Θ . The bounded condition on Θ simplifies greatly the proofs because it allows us to use uniform bounds that do not depend on θ (for instance, the identity function h p θ q ' θ is bounded thanks to this assumption. Similarly, the time taken by the solution of the ODE ϕ t p θ q to reach θ ˚ is also uniformly bounded independently on θ ). This assumption could be relaxed by imposing high probability bounds (for instance, a large deviation result like [2] that would guarantee that θ n stays close to θ ˚ ), or a bound on the higherorder moment or exponential moments of θ n . We left this result for future work as it would greatly impact the readability of the proofs.

The most questionable of our assumption is Assumption (A2) that imposes that all parameters of the problem are four times differentiable in θ . While this assumption might seem as technical, the fact that the parameters are twice differentiable is crucial in our analysis. In particular, the constant V does depend on the first two derivatives of the function ¯ f . In general, if the parameters of the systems are not differentiable, then the bias will not be of order O p α q but of order O p ? α q . Treating a non-differentiable ¯ f would need a completely different methodology. Our assumption of having

four times differentiable functions and not just twice has two reasons: it guarantees that the error › › ¯ θ n ´p θ ˚ ` αV q › › is O p α 2 q , and it simplifies the proof by allowing us to reuse Theorem 1 to obtain the bound O p α 2 q in Theorem 2. We believe that if we only impose twice differentiability, this error would be o p α q (and in fact probably α 3 { 2 as long as all derivatives are Lipschitz-continuous).

3.4 Notations

Recall that Θ is a compact subset of R d . We suppose that it is equipped with a norm }¨} . For a function h : E Ñ R , where E Ă R d 1 , we denote by } h } ' sup e P E } h p e q} the supremum of this function. If h is i times differentiable, we denote by D i h its i th derivative. Its value evaluated in e P E is denoted by D i h p e q . It is a multi-linear map and we denote by D i h p e q a b i its value applied to p a, . . . , a q for a given a P R d 1 . We denote by › › D i h › › the operator norm of its i th derivative, defined as › › D i h › › ' sup e P E ,a P R d 1 › › D i h p e q a b j › › { } a } j . By Taylor remainder theorem, for all e P E and a P R d 1 , we have: › › › h p e ` a q ´ h p e q ´ ř i ´ 1 j ' 1 D j h p e q a b j › › › ď › › D i h › › } a } i .

Wedefine by › › D ď i h › › ' max p} h } , max j ď i › › D j h › › q the maximum of the norm of the first i th derivatives of h . We denote by C i p E , R q the set of functions whose first i derivatives are bounded and by C i ď 1 p E , R q the set of functions whose first i derivatives are bounded by 1 : › › D ď i h › › ď 1 . As X is a discrete set, the above notions extend to functions h : E ˆ X Ñ R . For such a function, by abuse of notation, we call D i h the i th derivative with respect to the continuous variable only.

In the paper, the value at time t of the solution of the ODE 9 ϑ ' ¯ f p ϑ q starting in θ is denoted by ϕ t p θ q . Wewill introduce later a quantity φ n p θ q that corresponds to a Euler discretization of the ODE with constant step-size α . To lighten notations, we will omit the dependence on α in the notation φ but it should be clear that φ depends on α . To help distinguishing between the solutions of the the ODE ϕ t and the discrete-time recurrence φ n , we will reserve the index t for a continuous time variable and the indices n , k or N for discrete-time indices.

To ease notations in some part of the proofs, we will sometimes use big-O notations, like O p 1 q or O p α q . When using this, we allow the hidden constants to depends on all parameters of the problems defined in Assumptions (A1) to (A4) but they cannot depend on varying quantities likes an index k , n or α or norm of functions, like } g } or } h } that are introduced in the proofs. Note that in all of our lemmas, we always consider functions whose norm is bounded by 1 . This can, of course, be readily extended to functions not bounded by 1 by adding an extra factor like › › D ď i h › › . We avoid doing this to lighten the notations.

4 Main results and illustrations

This section presents the main theorems and illustrate their consequence. The proof of the theorems are postponed to the next Section 5.

4.1 Theoretical results

Our first result is Theorem 1 that shows that the expected value of θ n is at distance at most O p α q of the solution of the ODE. This shows that the bias of stochastic approximation is of order α with respect to the ODE.

Theorem 1. Assume (A1)-(A4). Then, there exists a constant C ą 0 and α 0 such that for all n , α ď α and all h P C 3 p Θ , R q :

Note that the bound of Theorem 1 is valid independently of n . As we will see in the proof, this is a consequence of Assumption (A4). Without the latter assumption, one would naturally obtain a constant C that grows exponentially with n . As C does not depend on time, a direct consequence of Theorem 1 is to show that the asymptotic bias of θ n is of order O p α q as n goes to infinity, that is:

Our second result is Theorem 2, that shows that the inequality of (4) is essentially an equality. This theorem provides an asymptotic expansion of the bias term in α .

Theorem 2. Assume (A1)-(A4). Then, there exists a constant C 1 ą 0 and α 0 such that for all α ď α 0 and h P C 5 ď 1 p Θ , R q :

An important difference between Theorem 1 and 2 is that the former studies the convergence of the iterates θ n whereas the latter provides a refinement term for the average of the iterates, i.e., ¯ θ n : ' 1 n ř n k ' 1 E r θ k s , by showing that ¯ θ n is essentially equal to h p ϕ 8 q plus a bias term V α . One may wonder if Theorem 2 would be true for the (non-averaged) iterates θ n . The answer is no and a counter-example is provided in Appendix A. This example illustrates that when the Markovian component X n can be periodic, the O p α q term of E r θ n s does not necessarily stabilize to a constant V but can be periodic as well.

Theorem 2 concerns the convergence of the expectation of ¯ θ n but not the value of ¯ θ n itself. As we will see in Section 5, this is mostly due to our proof techniques that works with generators and therefore is most suitable to obtain precise convergence results for the expectation. In fact, we will prove in Section B.2 that we can obtain a high-probability convergence result as an almost-direct consequence of Theorem 2, as expressed by the following Theorem.

Theorem 3. Assume (A1)-(A4). Then, there exists a constant C 2 ą 0 and α 0 such that for all α ď α 0 :

This result is illustrated on a synthetic example whose parameters are given in Appendix A.2, and for which θ ˚ ' 1 . We run the stochastic approximation algorithm for various values of α and report the results in Figure 1. In all cases, we use the same source of randomness (the values of the Markov chain X n and of the the Martingale noise M n ` 1 are the same for all trajectories). We plot a sample trajectory of θ n , ¯ θ n , and ¯ θ n { 2: n defined 2 as:

for α P t 0 . 001 { 4 , 0 . 001 { 8 , 0 . 001 { 16 u (more values of α are displayed in Appendix A). We observe that if θ n is quite noisy, the Polyak-Ruppert averaging ¯ θ n or ¯ θ n { 2: n are much closer to θ ˚ ' 1 , which is a clear advantage for ¯ θ n { 2: n in terms of rate of convergence, especially for small values of α .

0

20000 40000 60000 80000100000

n

0.9

1.0

1.1

1.2

n

or

n

n

n

n

/2 :

n

0

20000 40000 60000 80000100000

n

0.9

1.0

1.1

1.2

n

or

n

n

n

n

/2 :

n

0

20000 40000 60000 80000100000

n

0.9

1.0

1.1

1.2

n

or

n

n

n

n

/2 :

n

(a)

α

'

0

.

0025

(=

0

.

01

{

4

)

(b)

α

'

0

.

00125

(=

0

.

01

{

8

)

(c) α ' 0 . 000625 (= 0 . 01 { 16 )

4.2 The value of extrapolation: Illustration of Theorem 2 and 3

As we will see in the proof, the constant V of Theorem 2 can be expressed as a function of the problem's parameters, which allows one to construct a quantity θ ˚ ` V that is a tight approximation

0

20000

40000

60000

80000

100000

n

10

2

10

1

10 0

Error (logscale)

theoretical error as

n

=0.02

=0.01

=0.005

=0.0025

=0.00125

=0.000625

0

20000

40000

60000

80000

100000

n

10

2

10

1

10 0

Error

2

(0.005)

n

/2 :

n

(0.01)

n

/2 :

n

2

(0.0025)

n

/2 :

n

(0.005)

n

/2 :

n

2

(0.000625)

n

/2 :

n

(0.00125)

n

/2 :

n

(a) Error

ˇ

ˇ

¯

θ

n

{

2:

n

´

θ

˚

ˇ

ˇ

.

(b) Error of the extrapolation (6).

of ¯ θ n . Yet, stochastic approximation algorithms are most used when one does not have access to the problem's parameter. Here, we illustrate how we can run two algorithms with two different step-sizes in order to obtain an algorithm that has both a fast convergence rate and a high precision.

Let ¯ θ p 2 α q n { 2: n and ¯ θ p α q n { 2: n be two trajectories of the stochastic recurrence (1) each with respective stepsize 2 α and α . By Theorem 2, for large n we have:

We see that we can suppress the term in 2 V α and V α by using a linear combination of ¯ θ p 2 α q n { 2: n and ¯ θ p α q n { 2: n , which leads to an equation that has a distance to θ ˚ that is of order O p α 2 q :

To explore the benefit of using the extrapolation (6), we plot in Figure 2 the error of the averaged iterate 3 ˇ ˇ ¯ θ n { 2: n ´ θ ˚ ˇ ˇ , and the error of the extrapolation (6) for various values of α ' 0 . 01 ˚ 2 ´ k , with k ' t´ 1 . . . 4 u . The parameters are the same as the one of Figure 1 and the various α s use the same source of randomness. As we expect by the statement of the theorems, the error ¯ θ n { 2: n ´ θ ˚ is approximately equal to V α (the stars on the right are the theoretical values of lim n Ñ8 ˇ ˇ ¯ θ n { 2: n ´ θ ˚ ˇ ˇ ). On the right panel, we observe that 2 ¯ θ p α q n ´ ¯ θ p 2 α q n is closer to θ ˚ than the corresponding ¯ θ p α q n (the scale of the y -axis is the same for both panels). Note that even for n ' 10 5 , the valus of α are still noisy.

5 Proof overview and generator method

5.1 Proof overview

The first idea of our proof is to construct new random variables φ k p θ n ´ k q that corresponds to a system where we apply the stochastic recurrence (1) for k steps and then apply the deterministic recurrence (8) up to time n . By introducing these new variables and comparing their expectation for k and k ` 1 , we reduce our problem to a comparison of the generators of the stochastic and of the deterministic system. This step is done in Section 5.2. In an independent lemma, we use Assumption (A4) to show that the derivatives of the function φ n converge exponentially fast to 0 as n goes to infinity. This leads to Lemma 6 whose proof is technical and postponed to Section C.

Once these two basic points are obtained, we will use them to prove Proposition 4 which shows that the expectation of θ n is close to the solution of the deterministic ODE. More precisely, this

proposition shows that there exists a bounded sequence of constants V p α q n and a constant C ą 0 such that for all n P N : ˇ ˇ ˇ E r h p θ n q ´ h p ϕ nα p θ 0 qqs ´ αV p α q n ˇ ˇ ˇ ď Cα 2 . A main difficulty is to show that V p α q n is bounded. For that we will use Lemmas 7 and 8 that use properties of the Poisson Equation (21) to show that if the functions g k do not vary too much with k , then they can be replaced by their averaged versions:

Theorem 1 is a direct consequence of Proposition 4. by using that V p α q n are bounded.

The next step is to show that the time average versions of V p α q n , V p α q n ' n ´ 1 ř n k ' 1 V p α q n , converges (as n goes to infinity) to a term V p α q . Here, the main technical difficulty is to obtain a refinement of the averaging property of (7), which is done in Lemmas 9 and 10. We then use Theorem 1 and Lemma 11 to show that V p α q « V ` O p α q where V is given in Proposition 5. This gives Theorem 2. As a side-product, Lemma 11 also shows that the constant V can be computed by solving a linear system. The high-probability bound (Theorem 3) is a consequence of Theorem 2.

The proof structure is illustrated in Figure 3 that shows the dependencies between the lemmas.

Generator method

(Section 5.2)

Exponential decay of

›

›

›

B

i

pB

θ

q

i

φ

n

›

›

›

as

n

grows (Lemma 6)

Telescopic

sums

(Lemma 7)

Refined averaging

(Lemmas 9 and 10)

Derivation of

V

p

α

q

n

(Proposition 4)

lim

n

Ñ8

V

p

α

q

n

«

V

(Proposition 5)

O

p

α

q

(Theorem 1)

V α

`

O

p

α

2

q

(Theorem 2)

High probability

bound (Theorem 3)

V

is the solution of a

linear system (Lemma 11)

Poisson Equation

and averaging

(Lemma 8)

5.2 Deterministic recurrence and comparison of generators

To compare the stochastic variable θ n and the solution of the ODE 9 ϑ ' ¯ f p ϑ q , we introduce a deterministic recurrence equation, that is a first-order discretization of the ODE. This recurrence equation is obtained by replacing the sources of randomness of (1) by their expectation. The stochastic approximation (1) contains two sources of randomness: M n ` 1 and X n . In our analysis, we will use a deterministic counterpart of (1) that corresponds to setting the noise M n ` 1 to 0 and to using ¯ f p θ q instead of f p θ n , X n q . More precisely, for an initial value θ P Θ and k P Z ` , we define φ n p θ q as:

with the convention that φ 0 p θ q ' θ .

Let h : Θ Ñ R be an arbitrary function. By using the definition of the deterministic recurrence, for any k P t 0 . . . n u , we introduce the variable

The quantity z n,k is the expected value of a recurrence at time n if one starts by applying the stochastic recurrence for the first k steps and then the deterministic recurrence for the remaining n ´ k

steps. Our proof method consists in obtaining precise bounds on the difference E r θ n s ´ E r φ n p θ 0 qs . By using the notation z n,k , this quantity is equal to z n,n ´ z n, 0 . To obtain a bound on this quantity, we will use a trick, that essentially consists in comparing z n,k ` 1 ´ z n,k . This quantity is simpler to analyze because the only modification between the two is to replace one stochastic transition by one deterministic transition. We can then recover the original bound by using that:

This method is illustrated in Figure 4 on a synthetic example whose parameters are given in Appendix A.3. In the right panel, plot two functions (for two different values of k ) that are equal to θ n for k ď n and to φ n ´ k p θ k q for k ą n .

0

25

50

75

100

125

150

175

200

n

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

n

(

0) or

n

n

(

0)

n

0

25

50

75

100

125

150

175

200

n

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

n

k

(

k

) if

n

>

k

or

k

for

n

<

k

n

k

(

k

) for

k

=75

n

k

(

k

) for

k

=100

(a) Stochastic vs deterministic recurrences

(b) Impact of changing k in φ n ´ k p θ k q

5.3 Derivation of V p α q n and of V by comparing the generators

Using the above notations, we are now ready to prove the first proposition.

Proposition 4. Assume (A1)-(A4). Then, there exists a constant C ą 0 and α 0 such that for all n , α ď α 0 and all h P C 3 ď 1 p Θ , R q :

where the quantity V p α q n is given by:

Moreover, V p α q n is bounded independently of α and n .

Proof. Following the definition of the stochastic and deterministic recurrences, the difference z n,k ´ z n,k ` 1 that appears in (10) is equal to

If g : R d Ñ R is a thrice differentiable function whose third derivative is bounded by › › D 3 g › › , by using a Taylor expansion, we have that g p x ` a q´ g p x ` b q ' Dg p x qp a ´ b q` D 2 g p x qp a ´ b q b 2 ` R , where R is a remainder term that is smaller than c 6 } a ´ b } 3 . In our proof, we use this Taylor

expansion with the function g ' h ˝ φ n ´p k ` 1 q , x ' θ k , a ' α p f p θ k , X k q` M k ` 1 q and b ' α ¯ f p θ k q . Applying this to (11) shows that

where A n,k and B n,k are equal to

and R n,k is a remainder term.

By assumption (A1), the conditional expectation and variance of the martingale term are equal to E r M k ` 1 | F k s ' 0 and E ' M b 2 k ` 1 | F k ‰ ' Q p θ k , X k q . By using the total law of expectation, this shows that A n,k and B n,k can be simplified as:

By definition, V p α q n ' ř n ´ 1 k ' 0 p A n,k ` αB n,k { 2 q . Hence, combining this with (10) shows that

To complete the proof, it remains to be shown that there exists a constant such that ř n ´ 1 k ' 0 R n,k ď Cα 2 and that V p α q n is bounded independently of α and n . There are three terms, corresponding to R , B , and ř n k ' 1 A n,k :

Case of R . In Lemma 6, we show that the norm of the derivatives of h ˝ φ n ´ k are smaller than c 1 e ´ c 2 α p n ´ k q . As the derivatives of h are bounded by 1 , this implies there exists c independent of α and n such that › › D 3 p h ˝ φ n ´ k q › › ď ce ´ c 2 α p n ´ k q . Hence, there exists c 1 ą 0 such that:

Case of B . The proof that ř n k ' 1 αB n,k is bounded independently of n and α is similar since the factor α cancels out with the factor 1 { α that comes out of ř 8 k ' 0 e ´ c 2 αk .

Case of A Showing that ř n k ' 1 A n,k is bounded independently of n and α is more subtle and uses the fact that ¯ f is the 'averaged' version of f with respect to the transition kernel K . The proof of this is a consequence of the first point of Lemma 8 that is stated in Section C.

The proof of the main results is then a consequence of the next proposition, whose proof is given in Appendix B. This proposition shows that the Cesaro limit of V p α q n is approximately equal to a term V that does not depend on α plus a term of order O p α q .

Proposition 5. Assume (A1)-(A4). Then, there exist a constant C ą 0 and α 0 ą 0 such that for all h P C 5 ď 1 p Θ , R q , there exists a vector V such that for all n , α ď α 0 :

where V p α q n : ' n ´ 1 ř n k ' 1 V p α q k with V p α q k as in Proposition 4.

6 Conclusion / discussion

In this paper, we presented an analysis of the bias of constant-step size algorithms with Markov noise. We developed a novel technic, based on generator comparison, that allows to obtain a very fine comparison between the expectation of the stochastic trajectory and the value of its deterministic counterpart. This methodology is quite generic and we believe that it could easily be adapted to obtain more general results. We are in particularly targeting: relaxing the bounded of Θ , and obtaining non-asymptotic results.

References

Sebastian Allmeier and Nicolas Gast. Bias and refinement of multiscale mean field models. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 7(1):1-29, 2023.

Wa¨ ıss Azizian, Franck Iutzeler, J´ erˆ ome Malick, and Panayotis Mertikopoulos. What is the long-run distribution of stochastic gradient descent? a large deviations analysis. arXiv preprint arXiv:2406.09241 , 2024.

Michel Bena¨ ım. Dynamics of stochastic approximation algorithms. In Seminaire de probabilites XXXIII , pages 1-68. Springer, 2006.

Albert Benveniste, Michel M´ etivier, and Pierre Priouret. Adaptive algorithms and stochastic approximations , volume 22. Springer Science & Business Media, 2012.

Dimitri Bertsekas. Reinforcement learning and optimal control , volume 1. Athena Scientific, 2019.

Julius R Blum. Approximation methods which converge with probability one. The Annals of Mathematical Statistics , pages 382-386, 1954.

Nicoletta Bof, Ruggero Carli, and Luca Schenato. Lyapunov theory for discrete time systems. arXiv preprint arXiv:1809.05289 , 2018.

Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint , volume 48. Springer, 2009.

Vivek S Borkar and Sean P Meyn. The ode method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization , 38(2):447-469, 2000.

Anton Braverman, JG Dai, and Xiao Fang. High-order steady-state diffusion approximations. Operations Research , 72(2):604-616, 2024.

Siddharth Chandak, Vivek S Borkar, and Parth Dodhia. Concentration of contractive stochastic approximation and reinforcement learning. Stochastic Systems , 12(4):411-430, 2022.

Hao Chen, Abhishek Gupta, Yin Sun, and Ness Shroff. Hoeffding's inequality for markov chains under generalized concentrability condition. arXiv preprint arXiv:2310.02941 , 2023.

Zaiwei Chen, Siva T Maguluri, Sanjay Shakkottai, and Karthikeyan Shanmugam. A lyapunov theory for finite-sample guarantees of markovian stochastic approximation. Operations Research , 2023.

Zaiwei Chen, Siva Theja Maguluri, and Martin Zubeldia. Concentration of contractive stochastic approximation: Additive and multiplicative noise. arXiv preprint arXiv:2303.15740 , 2023.

Zaiwei Chen, Sheng Zhang, Thinh T Doan, John-Paul Clarke, and Siva Theja Maguluri. Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning. Automatica , 146:110623, 2022.

Aymeric Dieuleveut, Alain Durmus, and Francis Bach. Bridging the gap between constant step size stochastic gradient descent and markov chains. The Annals of Statistics , 48(3):pp. 1348-1382, 2020.

Nicolas Gast. Expected values estimated via mean-field approximation are 1/n-accurate. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 1(1):1-26, 2017.

Nicolas Gast and Benny Van Houdt. A refined mean field approximation. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 1(2):1-28, 2017.

Francis Begnaud Hildebrand. Introduction to numerical analysis . Courier Corporation, 1987.

Dongyan Huo, Yudong Chen, and Qiaomin Xie. Bias and extrapolation in markovian linear stochastic approximation with constant stepsizes. In Abstract Proceedings of the 2023 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems , pages 81-82, 2023.

Hassan K Khalil. Nonlinear systems . Prentice Hall, Upper Saddle River, N.J., 2002.

Vassili N. Kolokoltsov, Jiajie Li, and Wei Yang. Mean Field Games and Nonlinear Markov Processes. arXiv:1112.3744 , April 2012.

H. Kushner and G.G. Yin. Stochastic Approximation and Recursive Algorithms and Applications . Stochastic Modelling and Applied Probability. Springer New York, 2003.

Guanghui Lan. First-order and stochastic optimization methods for machine learning , volume 1. Springer, 2020.

Caio Kalil Lauand and Sean Meyn. The curse of memory in stochastic approximation. In 2023 62nd IEEE Conference on Decision and Control (CDC) , pages 7803-7809. IEEE, 2023.

Wenlong Mou, Ashwin Pananjady, Martin J Wainwright, and Peter L Bartlett. Optimal and instance-dependent guarantees for markovian linear stochastic approximation. arXiv preprint arXiv:2112.12770 , 2021.

Eric Moulines and Francis Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in neural information processing systems , 24, 2011.

Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization , 30(4):838-855, 1992.

Guannan Qu and Adam Wierman. Finite-time analysis of asynchronous stochastic approximation and q -learning. In Conference on Learning Theory , pages 3185-3205. PMLR, 2020.

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400-407, 1951.

David Ruppert. Efficient estimations from a slowly convergent robbins-monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.

Marina Sheshukova, Denis Belomestny, Alain Durmus, Eric Moulines, Alexey Naumov, and Sergey Samsonov. Nonasymptotic analysis of stochastic gradient descent with the richardsonromberg extrapolation. arXiv preprint arXiv:2410.05106 , 2024.

Rayadurgam Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation andtd learning. In Conference on Learning Theory , pages 2803-2830. PMLR, 2019.

Charles Stein. Approximate computation of expectations. Lecture Notes-Monograph Series , 7:i-164, 1986.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

John N Tsitsiklis. Asynchronous stochastic approximation and q-learning. Machine learning , 16:185-202, 1994.

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8:279-292, 1992.

Sarath Yasodharan and Rajesh Sundaresan. Large deviations of mean-field interacting particle systems in a fast varying environment. The Annals of Applied Probability , 32(3):1666-1704, 2022.

Lei Ying. On the approximation error of mean-field models. In Proceedings of the 2016 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Science , pages 285-297. ACM, 2016.

Lei Ying. Stein's method for mean field approximations in light and heavy traffic regimes. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 1(1):12, 2017.

Lu Yu, Krishnakumar Balasubramanian, Stanislav Volgushev, and Murat A Erdogdu. An analysis of constant step size sgd in the non-convex regime: Asymptotic normality and bias. Advances in Neural Information Processing Systems , 34:4234-4248, 2021.

Yixuan Zhang and Qiaomin Xie. Constant stepsize q-learning: Distributional convergence, bias and extrapolation. arXiv preprint arXiv:2401.13884 , 2024.

Appendix

A Additional numerical results

A.1 Reproducibility

All parameters given in this section should suffice to reproduce all figures. The total computation time to obtain all figures of the paper does not exceeds a few minutes (on a 2018 laptop, all implementation being done in Python/Numpy/Matplotlib). To ensure reproducibility, the code necessary to reproduce the paper (including latex, code to run the simulations and code to plot the figures) is available at https://github.com/ngast/paper_bias_stochastic_approximation2024 .

A.2 Parameters for the example of Figure 1 and Figure 2

For Figure 1 and 2, the state space of the Markovian part is X ' t 0 , 1 u , and the transition matrix is

The drift is f p θ, x q ' θ ´ 2 x . The martingale noise M n ` 1 is a sequence of i.i.d. random variables with P p M n ` 1 ' 1 | F n q ' P p M n ` 1 ' ´ 1 | F n q ' 0 . 5 . One can show that θ ˚ ' 1 . The initial value of the algorithm is set to θ 0 ' 3 , X 0 ' 0 .

In Figure 5, we complete the results of Figure 1 by showing more values of α . We observe that in all cases, the averaged values ¯ θ n and ¯ θ n { 2: n are much close to θ ˚ ' 1 than the original θ n that is very noisy. This explain why using a Polyak-Ruppert averaging is important. Also, if for large values of α P t 0 . 02 , 0 . 01 , 0 . 005 u , the two quantities ¯ θ n and ¯ θ n { 2: n have similar behavior, the situation is quite different when α gets smaller. In the latter case, the convergence of ¯ θ n { 2: n is much faster than the one of ¯ θ n because of the transient behavior of θ n at the beginning. This is why we use ¯ θ n { 2: n in our simulations.

0

20000 40000 60000 80000 100000

n

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

n

or

n

=0.02

n

n

n

/2 :

n

0

20000 40000 60000 80000 100000

n

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

n

or

n

=0.01

n

n

n

/2 :

n

0

20000 40000 60000 80000 100000

n

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

n

or

n

=0.005

n

n

n

/2 :

n

0

20000 40000 60000 80000 100000

n

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

n

or

n

=0.0025

n

n

n

/2 :

n

0

20000 40000 60000 80000 100000

n

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

n

or

n

=0.00125

n

n

n

/2 :

n

0

20000 40000 60000 80000 100000

n

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

n

or

n

=0.000625

n

n

n

/2 :

n

A.3 Parameters for Figure 4

For Figure 4, the state space of the Markovian part is X ' t 0 , 1 u and the transition matrix is

The drift is f p θ, x q ' 1 ´ p 1 ` θ q x . The initial state is θ 0 ' X 0 ' 0 and there is no martingale noise 4 . The step-size is set to α ' 0 . 05 .

A.4 Why is Theorem 1 for θ n and Theorem 2 and 3 for ¯ θ n ?

Note that compared to (4) of Theorem 1, the result of (5) in Theorem 2 is stated for the averaged iterates 1 N ř N n ' 1 E r θ n s and not directly for θ n . In fact, there are many cases for which (5) is also valid for θ n . In particular, if E r h p θ n qs converges as n goes to infinity, then one necessarily has:

This is for instance the case if p θ n , X n q is an ergodic Markov chain. Yet, there are cases for which the result of (5) only holds for the averaged iterates. This occurs typically when X n has a periodic behavior, in which case the averaged iterates eliminate eventual oscillations caused by periodic noise sequences. To illustrate this, we consider a one-dimensional model with f p θ, x q ' 2 x ´ θ , X n ` 1 ' 1 ´ X n and M n ` 1 ' 0 . This model satisfies all assumptions of the problems and θ ˚ ' 1 . The initial conditions are set to θ 0 ' 0 . 85 , and X 0 ' 0 . The system is therefore deterministic. In Figure 6 we plot the trajectories of θ n and of ¯ θ p α q N : ' 1 N ř N n ' 1 θ n for two stepsize values: α ' 0 . 05 and α ' 0 . 1 . We observe that, as stated by Theorem 1, the smaller is α , the closer is lim n Ñ8 θ n from θ ˚ . One can also observe that p θ n ´ 1 q{ α oscillates and does not converge as n goes to infinity, contrary to p ¯ θ n ´ 1 q{ α that does converge (to 0 here).

0

25

50

75

100

125

150

n

0.85

0.90

0.95

1.00

1.05

n

,

n

or

*

n

,

=0.1

n

,

=0.1

Equilibirum

*

0

25

50

75

100

125

150

n

0.85

0.90

0.95

1.00

1.05

n

,

n

or

*

n

,

=0.05

n

,

=0.05

Equilibirum

*

(a)

α

'

0

.

1

(b)

α

'

0

.

05

B Proof of the main results

B.1 Proof of Proposition 5

In this section, we prove Proposition 5 that states, that under(A1)-(A4), there exist a constant C ą 0 and α 0 ą 0 such that for all h P C 5 ď 1 p Θ , R q , there exists a vector V such that for all n , α ď α 0 :

This proposition shows that the Cesaro limit of the constants V p α q n defined in Proposition 4 converges to a limit that is equal to V ` O p α q . In particular, this result implies that

Proof of Proposition 5. Recall from the proof of Proposition 4 that V p α q n ' ř n ´ 1 k ' 0 E r A n,k ` αB n,k { 2 s . We first treat the term ř n ´ 1 k ' 0 B n,k that is the easiest (essentially because it is multiplied by a factor α ). Applying Lemma 8 to the function g p θ, X q : ' p f p θ, X q ´ ¯ f p X qq b 2 ` Q p θ, X q shows that:

By applying Theorem 1, this shows that

The treatment of A n,k requires more work, and in particular one can show that ř n ´ 1 k ' 0 A n,k does not necessarily converge as n goes to infinity 5 . This is why we study the convergence of ¯ A N ' ř N n ' 1 ř n ´ 1 k ' 0 A n,k and not the one of A n,k . By applying Lemma 10 with g ' f and then applying Theorem 1, it holds that lim N Ñ8 ˇ ˇ ¯ A N ` αm p θ ˚ q ˇ ˇ ď Cα , with m p θ ˚ q equal to

Note that to apply Theorem 1, we need m to be three times differentiable which is why we need h to be four times differentiable.

Let V p α q ' α p ř 8 k ' 0 D 2 p h ˝ φ k qp θ ˚ q ¯ g p θ ˚ q ` m p θ ˚ qq . By using Lemma 11, there exists a vector V such that › › V p α q ´ V › › ď Cα .

B.2 Proof of Theorem 3 (High-probability bound)

Fix some i P t 1 . . . d u and let Y n : ' p θ n ´ θ ˚ ´ αV q i , be the i th component of the vector where the constant V is the one defined in Proposition 5 when applied to the function h p θ q ' θ ´ θ ˚ . By Proposition 4 and Proposition 5, there exists C ą 0 such that for n large enough:

It is easy to modify the proofs of those propositions (by adding a conditioning on F mK and starting the summation at mK ` 1 ) to show that for all K and for m large enough:

Let Z m ' 1 K ř p m ` 1 q K k ' mK ` 1 Y m . The above result shows that ř M m ' 1 Z m ´ CM p α 2 ` 1 αK q is a supermartingale. Moreover, as Θ is bounded, there exists C 1 ą 0 (independent of C and α ) such that | Z n ´ C p α 2 ` 1 αK q| ď C 1 (almost surely). Hence, by using Azuma-Hoeffding inequality, for all ε :

By setting K ' 1 { α 3 , ε ' Cα 2 , and taking the limit as M goes to infinity, this implies that:

By using that 1 n ř n k ' 1 Y k ' 1 M ř M m ' 1 Z m ` O p 1 { M q , with M ' t M { K u , it follows that for all P t 1 . . . d u :

A symmetric proof shows that lim n Ñ8 P ` p ¯ θ n ´p θ ˚ ` αV qq i ď ´ 3 Cα 2 ˘ ' 0 . The theorem then follows by using a union bound on all i .

This shows that the constant of Theorem 3 can be chosen as C 2 ' 3 C where C is the constant of Theorem 1. By choosing more carefully the values of K and ε , this constant can be reduced to C 2 ' p 1 ` δ q C for all δ ą 0 .

C Technical lemmas

This section contains the technical lemmas that are used in the proof of the main theorems. This section is divided in two parts. We first show in Lemma 6 that the derivatives of φ n are exponentially small as n goes to infinity. Using this exponential convergence, we then prove Lemma 7 that provides bound for an almost telescopic sum that appears in the proof of Lemma 8. The latter shows how to bound the difference between f p θ n , X n q and ¯ f p θ n q by using a Poisson approach.

C.1 Exponential decay of the derivatives of φ n

The first lemma is at the core of our analysis. It shows that if the ODE has a unique attractor (as imposed by (A4)), then for α small enough, the derivatives of the discrete-time recurrence φ n are exponentially small as n goes to infinity.

Lemma 6. Assume (A2)-(A4). There exist α 0 ą 0 and constants c 1 , c 2 , c 3 , c 4 ą 0 such that for all α ă α 0 , i P t 1 , 2 , 3 u and n ,

(ii) For all h P C i ď 1 p Θ , R q , the i th derivative of h ˝ φ n satisfies:

Proof. The proof of (ii) is just a comparison between a sum and an integral. Indeed, as x ÞÑ e ´ x is decreasing, one have:

This shows that:

This concludes the proof by setting c 3 ' p α 0 ` 1 { c 2 q .

To prove (ii) , let us first prove the result for the first derivative ( i ' 1 ). By Assumption (A4), all trajectories of the ODE 9 ϑ ' ¯ ϑ converge to the fixed point θ ˚ and the Jacobian H : ' D ¯ f p θ ˚ q is Hurwitz. By classical results [7] on the discretization of ODE 6 , this implies that, for α small enough,

the discretization φ n p θ q converges exponentially fast to θ ˚ , that is: there exists α 0 ą 0 and a, b such that for all α ă α 0 and θ P Θ : } φ n p θ q ´ θ ˚ } ď ae ´ c 2 αn . Using this and the fact that ¯ f is twice continuously differentiable in θ ˚ implies the existence of a 1 such that:

Recall that the definition of φ n in Equation (8) implies that φ n ` 1 p θ q ´ φ n p θ q ' α ¯ f p φ n p θ qq with φ 0 p θ q ' θ . Hence, as ¯ f is differentiable with respect to its initial condition, by using the chain rule, the derivative of φ n ` 1 exists and satisfies:

To simplify notation, let use denote by X n ` 1 : ' Dφ n ` 1 p θ q (the above equation is X n ` 1 ´ X n ' αX n D ¯ f p φ n p θ qq ), and let us compute 7 X n ` 1 p I ` αH q ´p n ` 1 q . By adding a substracting the term X n p I ` αH q ´ n , it holds that:

where we used (15) for the last line.

By using that the last term X n p I ` αH q ´ n corresponds to the X n ` 1 p I ` αH q ´p n ` 1 q where we replace n ` 1 by n , a direct induction shows that

n

ÿ

k

'

1

Multiplying this equation by p I ` αH q n ` 1 implies that:

By using (15), and denoting x n ` 1 : ' } X n ` 1 } , it holds that

To conclude, let y n ` 1 : ' α p a 1 q 2 ř n k ' 1 y k e ´ c 2 αk with y 0 : ' x 0 . It can be shown by induction on k that x n ď e c 2 αk y k . Moreover, we have:

By using the concavity of the log, we have that:

By using the point (i), the last quantity is bounded by some quantity d . This shows that x n ď de ´ c 2 αn .

The proof of the higher derivatives (case i ě 2 ) are very similar to the proof for the first derivative i ' 1 and we only give an overview. For the second derivative, differentiating (15) with respect to θ gives:

X

n

1

`

p

I

`

αH

q

´p

n

`

1

q

'

α

X

k

p

D

¯

f

p

φ

k

p

θ

qq ´

H

qp

I

`

αH

q

´p

k

`

1

q

.

This equation is very close to (15) with an additional term Z n : ' α p Dφ n p θ q , Dφ n p θ qq ¨ D 2 ¯ f p φ n p θ qq . Indeed, denoting by X 1 n ' D 2 φ n p θ q , one can reapply the steps used to obtain (16) and get instead:

One can then use our result for the first derivative to show that › › p X 1 k p D ¯ f p φ k p θ qq ´ H q ` Z k q › › ď a 2 e ´ c 2 αk and obtain an equation similar to (17) but for x 1 n ` 1 : ' › › D 2 φ p θ q › › . The proof can be modified mutatis-mutandis for higher derivatives.

C.2 Bound on the almost-telescopic sum

In this section, we prove a lemma that we will later use in the proof of Lemma 8. This lemma concerns a summation that is almost-telescopic : the first (18) is of the form ř n ´ 1 k ' 0 u k p v k ` 1 ´ v k q . If it would hold that u k ' u k ´ 1 , then the sum would be telescopic and there would be no need for a lemma. Here, we show that the difference } u k ´ u k ´ 1 } is small and use this to prove our result. In the second lemma, we make a second round of summation.

Lemma 7. Assume (A1)-(A4). Then, there exists a constant C ą 0 and α 0 such that for all n , α ď α 0 and all i P t 1 , 2 , 3 u , h P C i ď 1 p Θ , R q and g P C 0 ď 1 p Θ , R q :

Proof. Let us denote u k : ' D p h ˝ φ n ´p k ` 1 q qp θ k q and v k : ' g p θ k , X k q . The quantity (18) is equal to E ' ř n ´ 1 k ' 0 u k p v k ` 1 ´ v k q ı . By shifting the indices of the sum (which corresponds to a discrete-time integration by part), we get:

As the norm of g and h are bounded by 1 , } u n ´ 1 v n ´ u 0 v 0 } and } v k } are bounded independently of α . Hence, the result follows if we can show that ř n ´ 1 k ' 1 } u k ´ 1 ´ u k } is bounded regardless of α . To show this, by adding and subtracting the term D p h ˝ φ n ´ k qp θ k q , we have:

By Lemma 6, the function D p h ˝ φ n ´ k q is Lipschitz continuous with a constant ce ´p n ´ k q αc 2 . As the difference between θ k and θ k ´ 1 is of order α , this shows that the first line is bounded by c 1 αe ´p n ´ k q αc 2 . For the second line, the definition of the function φ n ´ k in (8) implies that for any θ :

This shows that difference between D p h ˝ φ n ´ k q and D p h ˝ φ n ´p k ` 1 q q is of order c 2 αe ´p n ´ k q αc 2 . The two facts combined imply that:

C.3 Poisson equation and treatment of the averaging term

For a given g : Θ ˆ X Ñ R , we say that the function G g : Θ ˆ X Ñ R is a solution of the Poisson equation if it satisfies

with ¯ g p θ q ' ř x π x p θ q g p θ, x q as before.

It is shown in [1, Lemma 4] that under assumption (A3), for each g , there exists such a function G g . Moreover, under the regularity assumption (A2), this same lemma shows that the function G g can be chosen to be four times differentiable and that is derivatives satisfy › › D ď i G g › › ' O p › › D ď i g › › q .

Lemma 8. Assume (A1)-(A4). Then, there exists a constant C ą 0 and α 0 such that for all n , α ď α 0 and all i P t 1 , 2 , 3 u , h P C i ď 1 p Θ , R q and g P C 0 ď 1 p Θ , R q :

Proof. To lighten the notations, we consider the case i ' 1 , the proof holds for i ą 1 by replacing D p h ˝ φ q by D i p h ˝ φ q . By adding and subtracting some terms, the quantity (22) is equal to the expectation of

¯

We examine the three lines separately:

By using the fact that X k ` 1 makes a Markovian transition (2), and by using the definition of G h p x, y q , we get:

from which follows that the expectation of (23) is equal to zero.

To bound Equation (24), we use the definition (1) of θ k ` 1 to implies that there exists a constant C ą 0 such that | θ k ` 1 ´ θ k | ď Cα . By using that G g is Lipschitz-continuous (see [1, Lemma 4]) and that the norm of the derivative of h ˝ φ n ´ k is bounded by c 1 e ´ c 2 p n ´ k q α (Lemma 6), this shows that there exists c 1 ă 8 such that:

The fact that the expectation of (25) is bounded is a direct consequence of Lemma 7 applied with g ' G g and the fact that } DG g } ' O p} Dg }q .

This conclude the proof.

C.4 Refined treatment of the averaging term

The next two lemmas prove properties of the average sums that refine the analysis of Lemma 8. They are used to study the term A n in the proof of Proposition 5.

Lemma 9. Assume (A1)-(A4). Then, there exists a constant C ą 0 and α 0 such that for all n , α ď α 0 :

where ¯ m ' ř x,x 1 P X DG ℓ p θ, x 1 qp f p θ, x q ` R p θ, x, x 1 qq K x,x 1 p θ q π x p θ q .

Proof. By applying the same trick as for (23)-(25) in the one of Lemma 8 but replacing D i p h ˝ φ n ´p k ` 1 q qp θ k q by 1 { n , the left-hand-side of (26) is equal to the expectation of

where there is no equivalent of (23) because we already use that the expectation of this term is 0 .

In (28), each element of the first sum is O p α q by using a Taylor expansion of G ℓ and using that } DDG ℓ } ' O p} Dℓ }q . Hence the first term is of order O p α q . The second sum is telescopic. As it is multiplied by 1 { n , this leads to the term O p 1 { n q of (26). Combining both leads to (26).

To obtain (27), we refine the analysis of the first term of (28). By using a Taylor expansion on the function θ ÞÑ G ℓ p θ, X q , the quantity G ℓ p θ k , X k ` 1 q ´ G ℓ p θ k ` 1 , X k ` 1 q is equal to

By definition, } θ k ` 1 ´ θ k } 2 is of order α 2 .

Let m p θ, x q ' ř x 1 P X DG ℓ p θ, x 1 qp f p θ, x q ` R p θ, x, x 1 qq K x,x 1 p θ q . By using the law of total expectation,

By using (26) with the function m instead of ℓ , we obtain that 1 n ř n ´ 1 k ' 0 E r m p θ k , X k q ´ ¯ m p θ k qs ď C p α ` 1 { n q . This shows that the first term of (28) is equal to α ř n ´ 1 k ' 0 ¯ m p θ k q plus a term of order C p α 2 ` α { n q . As the term O p 1 { n q from the telescopic sum of the second term of (28) dominates the term α { n , this implies (27).

Lemma 10. Assume (A1)-(A4). Then, there exists a constant C ą 0 and α 0 such that for all n , α ď α 0 , h P C 4 ď 1 p Θ , R q and g P C 3 ď 1 p Θ , R q :

where the function ¯ m is equal to

Proof. Let us denote by u n ´p k ` 1 q p θ k q : ' D p h ˝ φ n ´p k ` 1 q qp θ k q and let us study the sum S : ' 1 N ř N n ' 1 ř n ´ 1 k ' 0 D p h ˝ φ n ´p k ` 1 q qp θ k q . By reordering the sum, this is equal to:

By Lemma 6, there exists c 1 , c 2 ą 0 such that } u n } ď c 1 e ´ c 2 αn . This shows that the second term of the previous sum (30) is bounded (in absolute value) by

for a suitable constant c 1 . This term goes to 0 when N goes to infinity.

To treat the first term of (30), we apply Lemma 9 with the function ℓ p θ, X q : ' ř 8 n ' 0 D i p h ˝ φ n qp θ q g p θ, X q ' ř 8 n ' 0 u n p θ q g p θ, X q .

C.5 Characterization of V

The next Lemma show how an approximate computable expression of the bias term V can be obtained. Recall that by definition, V is dependent on φ with step-size α . The subsequently defined and computable expression ˜ V is resolves this dependence on α and is further justified as it admits an accurate of order α with respect to V .

Lemma 11 (Approximate Computable Expression of V ) . Assume (A1)-(A4). Define

with H p 1 q i,j : ' B ¯ f i B θ i p θ ˚ q is the Jacobien matrix of ¯ f at θ ˚ , H p 2 q i,jk ' B ¯ f i B θ i B θ j p θ ˚ q is the second derivative of ¯ f in θ ˚ . In the above notation, H p 2 q ¨ W : ' p ř j,k H p 2 q i,jk W j,k q i ' 1 ...d , where W is the unique solution to the Sylvester equation H p 1 q W ` W ` H p 1 q ˘ T ` O ' 0 and where O and S are defined by

with ¯ g p θ ˚ q : ' ř x π x p θ ˚ q ´ p f p θ ˚ , x q ´ ¯ f p θ ˚ qq b 2 ` Q p θ ˚ , x q ¯ is defined as in the proof of Proposition 5.

There exists a constant C ą 0 and α 0 such that for all α ď α 0 :

where V p α q ' α p ř 8 k ' 0 D 2 p h ˝ φ k qp θ ˚ q ¯ g p θ ˚ q ` m p θ ˚ qq is defined as in the proof of Proposition 5.

Note that the constant V does not depend on n or α .

Proof. Using the definitions (33) and (34) for O and S respectively, and the definition of V , ¯ g and m in the proof of Proposition 5, we can rewrite V p α q as:

Using that φ k p θ ˚ q ' θ ˚ yields the identity

where the first line corresponds to the first term of (35) and the second line to the second term of (35). The notations correspond to Dh p θ ˚ q D 2 φ k p θ ˚ q ' p ř i B h B θ i B φ i B θ m B θ n q m,n and ¨ denotes the sum over the element wise product between the matrices.

From hereon out, we suppress the dependence of the therms on θ ˚ in order to ease the notation. As said before, it is shown in [1, Lemma 4], under assumptions (A2) and (A3), that G f is computable and four times differentiable, which implies that the terms S and O are computable. Therefore, in the rest of the proof we are concerned with obtaining computable expressions for the terms

We recall that by definition Dφ 1 ' I ` αH p 1 q . Using the chain rule, we have Dφ k ' D p φ 1 ˝ φ k ´ 1 q ' p I ` αH p 1 q q Dφ k ´ 1 ' p I ` αH p 1 q q k . By assumption of exponential stability, H p 1 q is Hurwitz and thus, for small enough α , all its eigenvalues have negative real parts. This implies that for such α , ř 8 k ' 0 p I ` αH p 1 q q k ' p αH p 1 q q ´ 1 . This gives the first term Dh p θ ˚ q ` H p 1 q ˘ ´ 1 S of (32).

Define W p α q as the second sum of (38) which is equal to

As H p 1 q is Hurwitz, W p α q is well-defined for α small enough and is a solution to the discrete-time Sylvester equation p I ` αH p 1 q q W p α q p I ` αH p 1 q q T ´ W p α q ` αO ' 0 . To obtain a computable expression independent of α , we consider the corresponding continuous-time Sylvester equation given by H p 1 q W ´ W p H p 1 q q T ` O ' 0 for which the Hurwitz property of H p 1 q ensures the existence of a unique solution W . By definition of the two Sylvester equations, we then have that W p α q ' W ` αH p 1 q W p H p 1 q q T , i.e., the difference between the two solutions is of order α . This gives the second term Dh p θ ˚ q ` H p 1 q ˘ ´ 1 H p 2 q ¨ W of (32).

For the last sum of (38), apply the chain rule and substitute Dφ 1 ' I ` αH p 1 q and D 2 φ 1 ' αH p 2 q to obtain the identity

Therefore,

with H p 2 q ¨p I ` αH p 1 q q j O p I ` α p H p 1 q q T q j ' ř m,n H p 2 q i,mn ` p I ` αH p 1 q q j O p I ` α p H p 1 q q T q j ˘ m,n . Consequently, for ř 8 k ' 0 D 2 φ k ¨ O the above yields

with C 1 ' ` p H p 1 q q ´ 1 H p 2 q ¨ ` H p 1 q W p α q ` W p α q p H p 1 q q T ` αH p 1 q W p α q p H p 1 q q T ˘˘ which is bounded by assumption. Using the previously discussed difference between W and W p α q , we conclude that p H p 1 q q ´ 1 H p 2 q ¨ W is an approximate solution to ř 8 k ' 0 D 2 φ k ¨ O with error of order α .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The purpose of the paper is to obtain bounds on the accuracy of constant step-size stochastic approximation. This is directly reflected by the title and abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limits of the assumptions are discussed in a separate subsection 3.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proofs of the main results are given in the main body of the paper while some more technical lemmas are give in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper does not contain important experimental results but only numerical illustrations of the theorems (with small synthetic examples). All parameters of the examples are provided in the appendix and the code of all simulation is available at https://github.com/ngast/paper_bias_stochastic_approximation2024 .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper is fully reproducible (code to produce simulation, figures and latex) at https://github.com/ngast/paper_bias_stochastic_approximation2024 .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: There is no dataset used.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not contain numerical experiments but only small numerical illustrations of theorems.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The numerical illustration are very limited (less than a few minutes of computation time in total). This is mentioned in the text.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper is a theoretical paper and its direct ethical implications seem limited.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is a theoretical paper and there is no direct broader impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: There are no models or data released.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: There is no significant code, data or model used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There are no new assets released.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies stochastic approximation with constant stepsize and Markovian noise. The authors provide a characterization of the bias (i.e., the difference between the expectation of the iterate and the desired limit), show that Polyak-Ruppert averaging can help reduce the variance but not the bias, and numerically demonstrate that Richardson-Romberg extrapolation helps reduce the bias.

STRENGTHS:
The writing is mostly clear and the proof is written with high quality. The result is an extension of [17,36] to the case where the Markov chain can have its transition probability matrix as a function of the stochastic iterate, which enables the authors to apply the results to RL algorithms with time-varying behavior policies (though the results are not formally presented in this paper). Overall, this is a strong theoretical work and the techniques developed in this work are quite novel.

WEAKNESSES:
(1) Assumption (A2) is relatively strong and difficult to verify in practice.

(2) The authors claim that Assumption (A4) implies the global exponential stability of the ODE, but did not provide a reference for the claim. Also, the claim in lines 466-471 needs formal justification.

QUESTIONS:
Both theorems require $h$ to be uniformly bounded by a constant. This is not true even for $h(\theta)=\theta$, which corresponds to bounding the expectation of $\theta_n$. Am I missing something?

LIMITATIONS:
The results are mostly asymptotic. A characterization of the stochastic approximation algorithm behavior for a finite $n$ would be preferred for practical purposes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the asymptotic bias in non-linear stochastic approximation algorithms with Markovian noise and fixed step-size. Upon applying the averaging technique of Polyak and Ruppert, the authors identify that, in general, the bias is of the same order of the step-size. The main source of bias is characterized, and an extrapolation technique is employed so that bias is attenuated. Finally, a few numerical studies are presented for illustration of the theoretical contributions.\n\nSTRENGTHS:\nI find the overall contribution of the paper to be very interesting. As far as the reviewer is aware, the characterization of bias for nonlinear SA with parameter dependent Markovian noise is novel and original. \n\nThe analysis seems sound. The reviewer did not have time to look over every single proof carefully, but haven't found egregious errors in the proofs revised.\n\nThe problem setup, assumptions and approach to analysis are clearly stated. The paper is well-written, but some polishing is needed.\n\nWEAKNESSES:\nIn particular, there have been recent papers, achieving similar bias characterizations and higher order error bounds for stochastic approximation with Markovian noise (see for example [R1] and [R2] which both deal with linear recursions). The approach to analysis in [R2] is similar to the present paper in the sense that the bias characterization is also given in terms of solutions to Poisson's equations. Moreover, this is not the first time that Richardson-Romberg extrapolation is used for bias attenuation in stochastic approximation: it was previously proposed in [R1] to kill the dominant term of order O(\\alpha) just like the present paper.\n\nThe authors do cite [R1] in the present work, but the reviewer feels that a deeper discussion is needed on how the present paper improves upon [R1] and [R2]. I encourage the authors to include a citation on previous uses of this technique as well.\n\n\n[R1] Huo, Dongyan, Yudong Chen, and Qiaomin Xie. \"Bias and extrapolation in Markovian linear stochastic approximation with constant stepsizes.\" Abstract Proceedings of the 2023 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems. 2023.\n\n[R2] Lauand, Caio Kalil, and Sean Meyn. \"The curse of memory in stochastic approximation.\" 2023 62nd IEEE Conference on Decision and Control (CDC). IEEE, 2023.\n\nQUESTIONS:\n- Is it possible to relax the assumption that the parameters remain within the compact set \\Theta with probability one?   It seems likely that the authors can obtain their results subject to a moment bound, which I believe is more realistic based upon standard SA stability theory.\n\n- One thing that sets this paper apart from others is that they allow Markovian noise that is parameter dependent.   This is extremely valuable in RL applications such as Q-learning (for example, epsilon-greedy policies induce such a model), and actor-critic methods.  Could the authors provide more discussion regarding applications in which the Markovian noise is parameter dependent?\n\n-Could the authors clarify whether the experiments display in Figure 1 (a) and (b) pertain to the same run? It is hard to identify the improvement between RR exploration and the regular algorithm for the choice of \\alpha =0.01. I encourage the authors to include sample paths for \\alpha =0.01 and \\alpha =0.005 instead of the \\alpha =0.0025 for visualization.\n\nLIMITATIONS:\nThe authors discuss the limitations of their work and assumptions in section 3.3.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper studies non-linear stochastic approximation scheme ($\theta_n$) driven by a uniformly geometrically ergodic MC $(X_n)$. Moreover, it is allowed the evolution of the Markovian noise $X_n$ to depend on $\theta_n$.  The authors study the asymptotic behaviour of the last iterate, Polyak-Ruppert and Richardson-Romberg procedures for differentiable test functions. The result generalizes recent result of Aymeric Dieuleveut, Alain Durmus, and Francis Bach. Bridging the gap between constant step size stochastic gradient descent and markov chains. The Annals of Statistics, 48(3):pp. 1348–1382, 2020.

STRENGTHS:
Novel technique based on infinitesimal generator comparison

WEAKNESSES:
- It would be good to provide non-asymptotic results rather than asymptotic
- I think that the result of theorem 3 is very weak since it is obtained using Chebyshev’s inequality. Could you please comment on $\alpha^{5/4}$ term?

QUESTIONS:
- I think it is better to give a definition of a unichain.
- Is it possible to relax conditions of the theorems? For example, what will be if we reduce the number of derivatives in Th 2? Is it true that in this case one will obtain \alpha^{3/2} in the remainder term? Is it possible to achieve this using the suggested technique?

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-17