Title: The Collusion of Memory and Nonlinearity in Stochastic Approximation With Constant Stepsize
Venue: NeurIPS 2024
Accepted: yes
Keywords: stochastic approximation

nonlinearity

Markov chain

weak convergence

Wasserstein metric

asymptotic bias

Richardson-Romberg extrapolation

Polyak-Ruppert averaging
Abstract: In this work, we investigate stochastic approximation (SA) with Markovian data and nonlinear updates under constant stepsize $\alpha>0$. Existing work has primarily focused on either i.i.d. data or linear update rules. We take a new perspective and carefully examine the simultaneous presence of Markovian dependency of data and nonlinear update rules, delineating how the interplay between these two structures leads to complications that are not captured by prior techniques. By leveraging the smoothness and recurrence properties of the SA updates, we develop a fine-grained analysis of the correlation between the SA iterates $\theta_k$ and Markovian data $x_k$. This enables us to overcome the obstacles in existing analysis and establish for the first time the weak convergence of the joint process $(x_k, \theta_k)$. Furthermore, we present a precise characterization of the asymptotic bias of the SA iterates, given by $\mathbb{E}[\theta_\infty]-\theta^\ast=\alpha(b_\textup{m}+b_\textup{n}+b_\textup{c})+\mathcal{O}(\alpha^{3/2})$. Here, $b_\textup{m}$ is associated with the Markovian noise, $b_\textup{n}$ is tied to the nonlinearity of the SA operator, and notably, $b_\textup{c}$ represents a multiplicative interaction between the Markovian noise and the nonlinearity of the operator, which is absent in previous works. As a by-product of our analysis, we derive finite-time bounds on higher moment $\mathbb{E}[||\theta_k-\theta^\ast||^{2p}]$ and present non-asymptotic geometric convergence rates for the iterates, along with a Central Limit Theorem.

The Collusion of Memory and Nonlinearity in Stochastic Approximation With Constant Stepsize

Abstract

In this work, we investigate stochastic approximation (SA) with Markovian data and nonlinear updates under constant stepsize α > 0 . Existing work has primarily focused on either i.i.d. data or linear update rules. We take a new perspective and carefully examine the simultaneous presence of Markovian dependency of data and nonlinear update rules, delineating how the interplay between these two structures leads to complications that are not captured by prior techniques. By leveraging the smoothness and recurrence properties of the SA updates, we develop a fine-grained analysis of the correlation between the SA iterates θ k and Markovian data x k . This enables us to overcome the obstacles in existing analysis and establish for the first time the weak convergence of the joint process ( x k , θ k ) k ≥ 0 . Furthermore, we present a precise characterization of the asymptotic bias of the SA iterates, given by E [ θ ∞ ] -θ ∗ = α ( b m + b n + b c ) + O ( α 3 / 2 ) . Here, b m is associated with the Markovian noise, b n is tied to the nonlinearity of the SA operator, and notably, b c represents a multiplicative interaction between the Markovian noise and the nonlinearity of the operator, which is absent in previous works. As a by-product of our analysis, we derive finite-time bounds on higher moment E [ ∥ θ k -θ ∗ ∥ 2 p ] and present non-asymptotic geometric convergence rates for the iterates, along with a Central Limit Theorem.

1 Introduction

Stochastic Approximation (SA) is an iterative scheme for solving fixed-point equations using noisy observations. Its application spans various domains including stochastic control [9, 40], reinforcement learning (RL) [4, 62] and stochastic optimization [43]. A typical SA algorithm takes the form θ k +1 = θ k + αg ( θ k , x k ) , where ( x k ) k ≥ 0 represents the underlying noisy data sequence and α > 0 is a constant stepsize. The goal of SA is to approximate the target solution θ ∗ that solves E x ∼ π [ g ( θ ∗ , x )] = 0 , with π being the stationary distribution of the stochastic process ( x k ) k ≥ 0 .

SA subsumes many important algorithms. A prime example is stochastic gradient descent (SGD) for minimizing a function J ( θ ) given noisy estimates g ( θ, x ) of its gradient. Linear SA schemes include SGD for quadratic objective functions, as well as various RL algorithms such as linear TD-Learning (in which g is not the gradient of any function and standard SGD results do not apply).

Of particular interest to us are SA updates given by a nonlinear function g ( θ, x ) of θ . One motivating example is learning a Generalized Linear Model (GLM) y ≈ σ ( z ⊤ θ ) with a nonlinear mean function σ : R → R . Apower approach, developed in [19, 36, 37, 64], uses a surrogate loss function, where the corresponding SGD update takes the form θ k +1 = θ k + α ( σ ( w ⊤ k θ k ) -y k ) w k , where x k = ( w k , y k ) is the observed covariate-response pair. Common choices of σ include the identity map for linear regression, the sigmoid function for logistic regression, as well as Rectified Linear Unit (ReLU) and its various smoothed versions (e.g., ELU and SoftPlus) for ReLU regression [7, 18, 19, 30, 36].

Furthermore, we are interested in the setting where the data sequence ( x k ) k ≥ 0 forms a Markov chain , going beyond the common i.i.d. data setting. The Markovian model captures a wide range of SA problems in machine learning where stochastic data exhibit serial dependence [5, 33, 38, 53].

Classical work on SA focuses on diminishing stepsizes [10, 58]. Constant stepsize schemes have recently gained popularity due to easy parameter tuning, fast initial convergence, and robust empirical performance. Non-asymptotic error bounds have been obtained for constant stepsize SA [17, 60]. Recent work further provides fine-grained characterization of the distributional and steady-state behaviors of the iterates [20, 33, 45, 67, 69]. Two recurring themes in these results are weak convergence of the distribution of θ t and the presence of an asymptotic bias E [ θ ∞ ] -θ ∗ ∝ α , both having important implications for iterate averaging, bias reduction and statistical inference [34].

Note that most previous work studied the nonlinear update setting and Markovian data setting separately -e.g., in [20, 67] for nonlinear SGD with i.i.d. data, and in [33, 34] for Markovian linear SA. The linearity or i.i.d. assumptions imposed in these prior works are restrictive, especially in the face of modern machine/reinforcement learning paradigms where nonlinear models are the norm and dependent data is common. Moreover, the absence of prior work dealing with Markovian nonlinear SA is not merely an overlook-as argued below, this setting is significantly more challenging.

Our Contributions In this work, we study constant-stepsize SA with both Markovian data and nonlinear update. In Section 3, we elucidate the new challenges that arise from the simultaneous presence of these two structures, which break key steps in previous analyses of the i.i.d. or linear setting. Due to the interaction between these two structures, establishing weak convergence is far from obvious, and the asymptotic bias exhibits new behaviors. Consequently, analyzing the nonlinear Markovian setting requires more than simply combining previous techniques.

To address the above confounding complication, we exploit the smoothness and recurrence structures of the SA update, thereby developing a fine-grained analysis of the correlation of the parameter θ k and data x k . This allows us to establish for the first time the weak convergence of the joint process ( x k , θ k ) k ≥ 0 to a unique invariant distribution, represented by the limiting random variable ( x ∞ , θ ∞ ) . As a by-product of our analysis, we derive finite-time bounds on E [ ∥ θ k -θ ∗ ∥ 2 p ] , the 2 p -th moments of the errors, generalizing the results in [17, 20, 60] to higher moments and to the nonlinear Markovian setting. In addition, we prove a Central Limit Theorem (CLT) for averaged iterates.

Moreover, we show that nonlinearity and Markovian structure contribute in a multiplicative way to the asymptotic bias of the SA iterates. We obtain the following bias characterization: E [ θ ( α ) ∞ ] -θ ∗ = α ( b m + b n + b c ) + O (( ατ α ) 3 / 2 ) . We provide explicit expressions for the vectors b m , b n , b c, which are independent of α . b m represents the bias component due to Markovian data (quantified by the mixing property of x k ), and b n the bias due to the nonlinearity of g (quantified by the second derivative g ′′ ). Importantly, we identify the compound term b c, which is absent in both nonlinear SA with i.i.d. data and linear SA with Markovian data. We explore the algorithmic implications of the above results on Polyak-Ruppert (PR) averaging [35, 56, 59] and Richardson-Romberg (RR) extrapolation [31]. We show that PR averaging reduces the variance but not the bias, whereas RR extrapolation eliminates the leading bias term α ( b m + b n + b c ) , reducing the asymptotic bias to a higher order of α .

Related Work Postponing a detailed literature review to Section 6, here we remark on the recent works most relevant to ours [1, 44, 45, 46], all studying Markovian nonlinear SA. The authors of [1] present an upper bound for the PR-averaged iterates and, similar to our results, demonstrate the effectiveness of RR-extrapolation in reducing bias, but lack weak convergence result for last iterates. In [44], the authors suggest adopting the ordinary differential equation framework to prove weak convergence of iterates and derive an upper bound for the asymptotic bias, which contrasts with our equality characterization with a closed-form solution for the leading-order bias. In [45], the authors prove weak convergence of ( x t , θ t ) using coupling, but only in the linear setting, not for nonlinear SA. In the latter setting, their weak convergence analysis is thwarted by challenges similar to what we elucidate in Section 3, the interplay between nonlinearity and Markovian data leading to 'double recursions.' The coupling technique differs as well: we couple two processes by sharing data x t = x ′ t , while in [45], they initialize two processes with different x 0 and x ′ 0 , and analyze the stopping time τ when θ τ = θ ′ τ . Moreover, they only present an upper bound for asymptotic bias, while ours presents a fine-grained characterization in Theorem 4.6 necessary for justifying RR-extrapolation. Lastly, the paper [46] discusses stepsize selection and its impact on the asymptotic statistics of PR-averaged SA with both constant and diminishing stepsizes.

̸

Notations The Euclidean norm is denoted by ∥ · ∥ . The notation ' u ⊗ v ' represents the tensor product between vectors u and v , and ' u ⊗ k ' denotes the k -th tensor power of vector u . The ball with radius β is B ( β ) := { θ ∈ R d : ∥ θ ∥ ≤ β } . L ( z ) denotes the distribution of a random vector z and Var( z ) its covariance matrix. Let P 2 ( R d ) be the space of square-integrable distributions on R d and P 2 ( X × R d ) be the space of distributions ¯ ν on X × R d with square-integrable second marginal on R d . The Wasserstein-2 between two probability measures µ and ν in P 2 ( R d ) is defined as W 2 ( µ, ν ) = inf ψ ∈ Π( µ,ν ) {( E [ ∥ θ -θ ′ ∥ 2 ] ) 1 2 : L ( θ ) = µ, L ( θ ′ ) = ν } , where Π( µ, ν ) is the set of all couplings between µ and ν . Extending to X × R d , we define the metric ¯ d ( ( x, θ ) , ( x ′ , θ ′ ) ) := √ { x = x ′ } + ∥ θ -θ ′ ∥ 2 , and denote by ¯ W 2 the extended Wasserstein-2 distance w.r.t. ¯ d.

✶ The lowercase letter c and its derivatives c ′ , c 0 , etc. denote universal numerical constants, whose value may change from line to line. We use s ≡ s ( θ 0 , θ ∗ , µ, L, R ) and its derivatives to denote quantities (scalars, vectors, or matrices) that are independent of the stepsize α and the iteration index k , but may depend on the initialization θ 0 , SA primitives θ ∗ , µ and L , and the coefficient R for the geometric mixing rate of ( x k ) in Assumption 1. As we are primarily interested in dependence on α and k , we adopt the following big-O notation: ∥ f ∥ = O ( h ( α, k )) if it holds that ∥ f ∥ ≤ s · ∥ h ( α, k ) ∥ .

2 Problem Setup and Preliminaries

Let ( x k ) k ≥ 0 be a Markov chain on a general state space X . Consider the following projected stochastic approximation (SA) iteration:

where g : R d ×X → R d is a deterministic function, { ξ k } k ≥ 1 are i.i.d. zero-mean random fields, α > 0 is a constant stepsize, and Π B ( β ) ( θ ) := arg min z : ∥ z ∥≤ β ∥ z -θ ∥ is the projection operator. We shall omit the superscript ( α ) in θ k when the dependence on α is clear from the context. In this work, we also consider the projection-free variant of the iteration (2.1) with β = ∞ .

We denote by π the stationary distribution of the Markov chain ( x k ) k ≥ 1 and define the shorthand ¯ g ( θ ) := E π [ g ( θ, x )] , where E π [ · ] denotes the expectation with respect to x ∼ π . The algorithm (2.1) computes an estimation of the target vector θ ∗ that solves the steady-state equation E π [ g ( θ, x )] = 0 . Our general goal is to characterize the relationship between the iterate θ k and the target solution θ ∗ .

In the following, we state the assumptions needed for our main results. For a more detailed discussion of the assumptions, we refer readers to Appendix B.

Assumption 1 (Uniform Ergodicity) . ( x k ) k ≥ 0 is a uniformly ergodic Markov chain on a countable state space ( X , B ( X )) with transition kernel P and a unique stationary distribution π . That is, there exist constants r ∈ [0 , 1) and R > 0 such that ∥ P k ( x, · ) -π ∥ TV ≤ Rr k , ∀ x ∈ X .

̸

The countable state space ensures separability under the ✶ { x = x ′ } metric, necessary for constructing a valid coupling in the invariance proof and establishing a well-defined P ∗ for bias characterization. Wekeep the notation general to allow future extensions to general state space and broader applicability of our results. All irreducible, aperiodic, and finite state space Markov chains are uniformly ergodic. The uniform ergodicity assumption is common in prior work on SA with Markovian noise [6, 21, 24, 33, 48]. Relaxing this uniform ergodicity assumption, in the style of [45, 52, 60] is possible but orthogonal to our focus, and thus we do not pursue this direction in this work.

We allow the chain ( x k ) k ≥ 0 to be arbitrarily initialized rather than from the stationary distribution π . An important quantity is the mixing time of the Markov chain, defined as follows.

Definition 2.1. For ϵ ∈ (0 , 1) , the ϵ -mixing time of ( x k ) k ≥ 0 , denoted by τ ϵ ≥ 1 , is defined as τ ϵ := min { k ≥ 1 : sup x ∈ X ∥ P k ( x, · ) -π ∥ TV ≤ ϵ } .

Under Assumption 1, the ϵ -mixing time satisfies τ ϵ ≤ K log 1 ϵ for all ϵ ∈ (0 , 1) , where K ≥ 1 is independent of ϵ . In the sequel, unless otherwise specified, we always choose ϵ = α and let τ ≡ τ α .

The following assumptions on the nonlinear function g in (2.1) is standard in the literature [17, 20, 32, 45, 48]. A wide family of g functions satisfies these assumptions, with the L 2 -regularized logistic regression of GLM being a standard example.

Assumption 2 (Differentiability and Linear Growth) . For each x ∈ X , the function g ( θ, x ) is three times continuously differentiable in θ with uniformly bounded first to third derivatives, i.e., sup θ ∈ R d ∥ g ( i ) ( θ, x ) ∥ < + ∞ for i = 1 , 2 , 3 , x ∈ X . Moreover, there exists a constant L 1 > 0 such that (1) ∥ g ( i ) ( θ, x ) -g ( i ) ( θ ′ , x ) ∥ ≤ L 1 , for all θ, θ ′ ∈ R d , i = 0 , 1 , 2 and x ∈ X , and (2) ∥ g (0 , x ) ∥ ≤ L 1 for all x ∈ X .

The linear growth condition in Assumption 2 implies that g ( θ, x ) is L 1 -Lipschitz w.r.t. θ uniformly in x . When g is a linear function, i.e., g ( θ, x ) = A ( x ) θ + b ( x ) , this assumption is satisfied with sup x ∈X ∥ A ( x ) ∥ < ∞ and sup x ∈X ∥ b ( x ) ∥ < ∞ , which are commonly assumed for linear SA. The above assumption immediately implies that the growth rate of ∥ g ∥ and ∥ ¯ g ∥ will be at most linear in θ , i.e., ∥ g ( θ, x ) ∥ ≤ L 1 ( ∥ θ -θ ∗ ∥ +1) and ∥ ¯ g ( θ ) ∥ ≤ L 1 ( ∥ θ -θ ∗ ∥ +1) .

Assumption 3 (Strong Monotonicity) . There exists µ > 0 such that ⟨ θ -θ ′ , ¯ g ( θ ) -¯ g ( θ ′ ) ⟩ ≤ -µ ∥ θ -θ ′ ∥ 2 , ∀ θ, θ ′ ∈ R d . Consequently, the target equation ¯ g ( θ ) = 0 has a unique solution θ ∗ .

When g is a gradient field, Assumption 3 is equivalent to strong convexity. For notational simplicity, we assume the strong monotonicity parameter satisfies µ ≤ 1 -r , where r is the convergence factor in Assumption 1. For general µ , our results remain valid with µ replaced by min { µ, 1 -r } .

We next consider the noise. Denote by F k the filtration generated by { x t , θ t , ξ t +1 } t =0 ∪ { x k , θ k } . Assumption 4 (Noise Sequence) . Let p ∈ Z + be given. The noise sequence ( ξ k ) k ≥ 1 is a collection of i.i.d. random fields satisfying the following conditions with L 2 ,p > 0 :

Define C ( θ ) = E [ ξ 1 ( θ ) ⊗ 2 ] and assume that C ( θ ) is at least twice differentiable. There also exist M ϵ , k ϵ ≥ 0 such that for θ ∈ R d , we have max i =1 , 2 ∥ ∥ C ( i ) ( θ ) ∥ ∥ ≤ M ϵ { 1 + ∥ θ -θ ∗ ∥ k ϵ } .

In the sequel, we set L := L 1 + L 2 , and without loss of generality, we assume L ≥ 1 .

When p = 1 , the second inequality in (2.2) only requires linear growth in expectation , which relaxes the almost sure linear growth condition in [17]. The constraint on the covariance matrix C ( θ ) is lenient and satisfied in most regular enough settings, as shown in [20].

3 Analytical Challenges and Techniques

In this section, we elaborate on the challenges and techniques in proving the above results.

Previous work has established weak convergence of ( x k , θ k ) separately for nonlinear SA with i.i.d. data, and for Markovian linear SA. The high-level approaches used in two representative prior works can be summarized as follows. The work [20] on nonlinear SGD leverages local linearization of g through Taylor expansion. The work [33] on Markovian linear SA exploits the mixing property of the Markovian noise to regain approximate independence, particularly between x k and θ k -τ for sufficiently large τ. It is tempting to expect that nonlinear SA can be analyzed by combining these two approaches. Perhaps surprisingly, such a simple combination would not work due to the interplay between nonlinearity and Markovian structures.

To demonstrate this challenge, let us seek to establish weak convergence in the Wasserstein distance W 2 via forward coupling [29], an approach employed by both [20, 33] as well as others [25]. Specifically, we consider two SA iterate sequences ( θ [1] k ) k ≥ 0 and ( θ [2] k ) k ≥ 0 from different initializations θ [1] 0 and θ [2] 0 coupled by sharing the data sequence ( x k ) k ≥ 0 : θ [1] k +1 = θ [1] k + αg ( θ [1] k , x k ) and θ [2] k +1 = θ [2] k + αg ( θ [2] k , x k ) . To establish convergence in W 2 , we consider the difference sequence

and it suffices to prove w k converges to 0 in mean square: E [ ∥ w k +1 ∥ 2 ] ≲ ρ k E [ ∥ w 0 ∥ 2 ] for ρ < 1 .

With this goal in mind and following the idea from [20], one may first linearize the right-hand side of the difference dynamic (3.1) and obtain the approximation

Next, to analyze the drift of the Lyapunov function E [ ∥ w k ∥ 2 ] and handle the Markovian noise ( x k ) , we use the conditioning technique from [33]. We condition on the information of τ steps before, denoted by F k -τ := σ ( ( θ [1] t , θ [2] t , x t ) : t ≤ k -τ ) . Ignoring higher-order terms and assuming a one-dimensional problem for simplicity, we obtain that

where we use w k ≈ w k -τ for small α (this argument, which is made precise in [33, 60], essentially exploits the fact that x k evolves faster than θ k ).

To prove dynamic (3.3) converges, it boils down to showing the 'gain matrix' E [ g ′ ( θ [2] k , x k ) | F k -τ ] is negative/Hurwitz. To further simplify, we assume k is large so that the chain ( x k ) is distributed per its stationary distribution π , in which case the gain matrix simplifies to E x ∞ ∼ π [ g ′ ( θ [2] ∞ , x ∞ )] .

Analyzing this gain matrix is where our analysis diverges from previous work. If the SA update were linear , i.e., g ( θ, x ) = A ( x ) θ, then the gain E [ g ′ ( θ [2] ∞ , x ∞ )] = E π [ A ( x ∞ )] would be independent of θ [2] ∞ , and its Hurwitz property is a standard and necessary condition for proving convergence of linear SA. If the data sequence ( x k ) were i.i.d. , then θ k would be independent of x k and hence the gain becomes E [ g ′ ( θ [2] ∞ , x ∞ )] = E [ E [ g ′ ( θ [2] ∞ , x ∞ ) | θ [2] ∞ ]] = E [¯ g ′ ( θ [2] ∞ )] with ¯ g ( · ) := E x ∼ π [ g ( · , x )] , where the Hurwitz property again follows from standard assumptions on ¯ g.

However, both arguments fail for the Markovian nonlinear setting. Common assumptions for nonlinear SA only ensure Hurwitz E x ∼ π [ g ′ ( θ, x ) | θ ] given θ . This does not imply the desired Hurwitz E [ g ′ ( θ [2] ∞ , x ∞ )] , precisely owing to the simultaneous presence of (i) the dependence of g ′ on both θ ∞ and x ∞ (due to nonlinearity) and (ii) the correlation between θ ∞ and x ∞ (due to Markovian).

Our Approach We overcome this challenge by carefully analyzing the properties of the above dependence and correlation. Therefore, for sufficiently large τ , we further decompose (3.3) as

where we approximate w k ≈ w k -τ , w t g ′ ( θ [2] t , x k ) ≈ ( g ( θ [1] t , x k ) -g ( θ [2] k , x k )) for t = k, k -τ and obtain the second term in the last inequality. Next, we propose employing two different Taylor expansions to prove that ♠ is of higher orders of α . Wefirst apply the Taylor expansion to g ( θ [1] k , x k ) -g ( θ [2] k , x k ) and g ( θ [1] k -τ , x k ) -g ( θ [2] k -τ , x k ) . However, this only achieves ♠ ≲ ∥ w k ∥ 2 ( ∥ w k ∥ + ατT 1 ) , where T 1 = min( ∥ θ [1] k ∥ , ∥ θ [2] k ∥ , ∥ θ [1] k -τ ∥ , ∥ θ [2] k -τ ∥ ) + 1 . When θ [1] k and θ [2] k are not close to each order, i.e., when ∥ w k ∥ is large, ♠ is not necessarily of higher order. Therefore, we consider a second type of Taylor expansion on g ( θ [1] k , x k ) -g ( θ [1] k -τ , x k ) and g ( θ [2] k , x k ) -g ( θ [2] k -τ , x k ) . The intuition for the second type of Taylor expansion is to analyze and bound ♠ by the small distance between θ [ j ] k and θ [ j ] k -τ for j ∈ { 1 , 2 } , even when ∥ w k ∥ is large. This achieves ♠ ≲ ∥ w k ∥ ατT 1 ( ∥ w k ∥ + ατT 1 ) . Simultaneously applying the two Taylor expansions will yield ♠ ≲ ατ ∥ w k ∥ 2 T 1 . Finally, we overcome this challenge by carefully analyzing the boundness of T 1 ; see Theorem 4.1 and its proof.

In parallel to the above coupling approach, we also explore an alternative approach by verifying the joint Markov chain ( x k , θ k ) satisfies certain irreducibility and Lyapunov drift conditions, which in turn imply the chain is ergodic. To apply this approach, we exploit additional properties of the SA noise, namely miniorization, which is satisfied in many applications where additional randomness is injected to the SA update. While the high level strategy of this approach is well developed [23, 50],

carrying out the analysis of each step is technically involved. In particular, we need to translate the minorization property of the noise to the irreducibility of the joint chain ( x k , θ k ) , which is nontrivial in the presence of Markovian noise and nonlinearity.

4 Main Results

4.1 Weak Convergence of Projected SA

Our first main result proves the ergodicity of the joint process ( x k , θ k ) k ≥ 0 of the projected SA (2.1).

Theorem 4.1 (Ergodicity of Projected SA) . Suppose that Assumption 1-4 ( p = 1) hold. The projected SA (2.1) is applied with radius parameter 2 ∥ θ ∗ ∥ ≤ β < ∞ . For stepsize α > 0 that satisfies the constraint ατ α ≤ µ (940+96 β ) L 2 , the Markov chain ( x k , θ k ) k ≥ 0 converges to a unique stationary distribution ¯ ν α ∈ P 2 ( X × R d ) . Let ν α := L ( θ ∞ ) be the second marginal of ¯ ν α . For k ≥ 2 τ α , it holds that

Theorem 4.1 generalizes prior weak convergence results for constant stepsize SA/SGD either under i.i.d. noise [20, 67] or linear update [33, 45]. Our stepsize condition ατ α ≲ µ/L 2 coincides with [33, 60] on linear SA, a special case of our setting.

The proof of Theorem 4.1 highlights the stabilizing effect of the projection operation in (2.1). This effect, together with the smoothness of update function g , controls how the Markovian correlation propagates through the nonlinear update, allowing us to overcome the challenges discussed in Section 3. It is unclear whether our proof, which is based on Markov chain coupling, can be fully generalized to SA without projection. Nevertheless, we show that such a generalization is possible for a sub-family of nonlinear SA where g possesses the additional structure termed 'asymptotic linearity', which is satisfied by, e.g., SGD applied to certain settings of logistic regression. For a formal statement of this result and proof, we refer the readers to Appendix E.

As a by-product of our analysis, we establish the following non-asymptotic 2 p -th moment bound on the error θ k -θ ∗ . Let θ t +1 / 2 := θ t + α ( g ( θ t , x t ) + ξ t +1 ( θ t )) denote the pre-projection iterate.

Proposition 4.2. Consider ( θ k ) k ≥ 0 of iteration (2.1) with β ∈ [2 ∥ θ ∗ ∥ , ∞ ] . Let Assumption 1-4 (2 p ) hold. If stepsize α satisfies ατ α L 2 ≤ c p µ, with c p ≤ 1 , the following holds for all k ≥ τ α ,

Proposition 4.2 implies that E [ ∥ θ k -θ ∗ ∥ 2 p ] ≲ ( ατ ) p for sufficiently large k , generalizing the results of [17, 20, 60] to higher moments and the nonlinear Markovian setting. Notably, this result holds even without the projection operation in the SA update (2.1), i.e., β = ∞ . Furthermore, Proposition 4.2 can be used to derive high-probability tail bounds using the Markov inequality.

4.2 Weak Convergence without Projection

Parallel to the coupling approach, we consider an alternative approach for establishing weak convergence via verifying irreducibility, positive Harris recurrence, and V -uniform ergodicity [50] of the Markov chain ( x k , θ k ) . This approach applies to nonlinear SA even without projection. To verify irreducibility, we exploit the following additional noise structure.

Assumption 5 (Noise Minorization) . For each θ ∈ R d , the distribution of the random variable ξ 1 ( θ ) , denoted by ζ θ , can be decomposed as ζ θ = ζ 1 ,θ + ζ 2 ,θ , where the measure ζ 1 ,θ has a density, denoted by p θ , which satisfies inf θ ∈ C p θ ( t ) > 0 for any bounded set C and any t ∈ R d .

Asimilar assumption is considered in [5, 67]. This assumption is mild and satisfied by any continuous random field supported on R d . Introducing such (small) continuous noise is often part of the algorithm design for inducing privacy [2, 22] or exploration [28, 55]. Without Assumption 5, the chain may fail to be irreducible even when the other assumptions are satisfied; see [33] for a counterexample.

Under Assumption 5, we obtain the following ergodicity result paralleling Theorem 4.1.

Theorem 4.3 (Ergodicity of SA - Minorization) . Suppose that Assumption 1-3, Assumption 4 ( p = 1) , and Assumption 5 hold. For stepsize α > 0 that satisfies the constraint ατ α L 2 < c 2 µ , the Markov chain ( x k , θ k ) k ≥ 0 of (2.1) with β = ∞ is V -uniformly ergodic with Lyapunov function V ( x, θ ) = ∥ θ -θ ∗ ∥ 2 +1 and a unique stationary distribution ¯ ν α ∈ P 2 ( X × R d ) . Moreover, defining the V -norm ∥ ν ∥ V := ∫ | ν ( d x ) | V ( x ) , we have

where the constants ρ ∈ (0 , 1) and κ ∈ (0 , ∞ ) may depend on α .

4.3 Non-Asymptotic Convergence Rate and Central Limit Theorem

In the sequel, let ( x ∞ , θ ( α ) ∞ ) denote the random vector whose law is the stationary distribution ¯ ν α given in Theorem 4.1. As a corollary, we have geometric convergence for the first 2 moments of θ k .

Corollary 4.4 (Non-Asymptotic Convergence Rate) . Under the setting of Theorem 4.1, for any initialization of θ 0 ∈ R d , we have

Moreover, the convergence rate established in Theorem 4.1 is fast enough that we can use it to prove a Central Limit Theorem for the average iterates.

Corollary 4.5 (Central Limit Theorem) . Under the setting of Theorem 4.1, as k → ∞ we have 1 √ k ∑ k -1 t =0 ( θ t -E [ θ ∞ ] ) ⇒N (0 , Σ ( a ) ) , where Σ ( α ) := lim k →∞ 1 k E [( ∑ k -1 t =0 ( θ t -E [ θ ( α ) ∞ ] )) ⊗ 2 ] .

Establishing the CLT sets the stage for using the SA iterates for statistical inference tasks such as confidence interval estimation. We discuss this in greater detail in Section 4.4 after characterizing the asymptotic bias, another important ingredient for using SA for inference.

4.4 Bias Characterization

In this subsection, we characterize the asymptotic bias E [ θ ( α ) ∞ ] -θ ∗ . Understanding the bias structure has important algorithmic implications for bias reduction, which we explore in Section 4.5, as well as for more efficient statistical inference and confidence interval estimation [34].

Theorem 4.6 (Bias Characterization) . Suppose Assumptions 1-4 ( p = 3) hold. For each stepsize α > 0 satisfying ατ α L 2 < c 3 µ , the following holds for some vector b independent of α :

More specifically, the leading bias can be decomposed as b = b m + b n + b c , where

with A = (¯ g ′ ( θ ∗ ) ⊗ I + I ⊗ ¯ g ′ ( θ ∗ )) -1 and h ( θ ∗ , x ) = ∫ X ( I -P ∗ +Π) -1 ( P ∗ -Π)( x, d x ′ ) g ( θ ∗ , x ′ ) , with the kernel P ∗ being a regular conditional probability on X that satisfies ∫ B π ( d x ) P ( x, C ) = ∫ C π ( d y ) P ∗ ( y, B ) , for all B,C ∈ B ( X ) .

̸

We defer the detailed proof to Appendix I. A few remarks are in order. First, we emphasize that (4.2) is essentially an equality, indicating a non-zero bias of order α whenever b = 0 (up to higher order terms). Notably, the Polyak-Ruppert averaging of the iterates cannot eliminate this bias. Note that the bias expansion in (4.2) applies to both weakly converged projected and non-projected SA. Our analysis shows that compared with the non-projected SA, the projection operator induces an extra bias term of the order O ( α 2 τ 3 α ) , which is negligible relative to the main terms in in (4.2).

More importantly, Theorem 4.6 provides an explicit expression of the leading bias, which decomposes into three components: the Markovian part, the nonlinearity contribution, and a compound term,

which is unique in nonlinear Markovian SA. Specifically, b m in (4.3) is associated with the Markovian multiplicative noise, where the matrix P ∗ -Π in the h function determines the mixing time of the data sequence ( x k ) k ≥ . The term b n in (4.4) is linked to nonlinearity, as reflected by the Hessian term ¯ g ′′ ( θ ∗ ) = E [ g ′′ ( θ ∗ , x ∞ )] , which quantifies the nonlinearity of g and is equal to zero in the case of a linear g . Lastly, b c in (4.5) is the compound term, due to its dependence on both the Markov noise ( h function) and the nonlinearity measure ¯ g ′′ . In particular, we note the following two special cases: (1) When g is a linear function, ¯ g ′′ ( θ ∗ ) = 0 . Hence, b n = b c = 0 , and b m recovers the result in [33]; (2) When ( x k ) k ≥ 0 is i.i.d. sampled from the stationary distribution π , we have h ( θ ∗ , x ) ≡ 0 ∀ x ∈ X , for P = P ∗ = Π . As such, b m = b c = 0 , recovering the result in [20]. The presence of the compound term b c suggests that as the SA structure becomes more nonlinear and the underlying Markov chain mixes more slowly, the impact on the bias is multiplicative rather than simply additive, a surprising phenomenon not unveiled in previous studies. It is possible to improve the residual order from O ( α 3 / 2 ) to O ( α 2 ) with a more refined characterization of the asymptotic second moment E [( θ ∞ -θ ∗ ) ⊗ 2 ] by following a similar strategy as our current approach. We leave this refinement out of the scope of the current paper.

4.5 Algorithmic Implications

We examine the practical implications of our weak convergence and bias characterization results, particularly for Polyak-Ruppert (PR) tail averaging and Richardson-Romberg (RR) extrapolation. In this subsection, we focus on the dependence on the stepsize α and iteration index k , and make use of the big-O notation from Section 1. Recall that b is the bias vector defined in Theorem 4.6.

PR averaging [56, 59] is a classical approach for reducing the variance and accelerating the convergence of SA. Here we consider the tail-averaging variant of PR averaging, defined as ¯ θ k 0 ,k := 1 k -k 0 ∑ k -1 t = k 0 θ t , for k ≥ k 0 , with a user-specified burn-in period k 0 ≥ 0 (a common choice is k 0 = k/ 2 ). The following corollary, proved in Appendix J, provides a non-asymptotic bound on the mean squared error (MSE) for the averaged iterates ¯ θ k 0 ,k .

Corollary 4.7 (Tail Averaging) . Under the setting of Theorem 4.6, the tail-averaged iterates satisfy the following bounds for all k > k 0 +2 τ α and k 0 ≥ τ α + 1 αµ log ( 1 ατ α ) ,

Corollary 4.7 shows that the MSE can be decomposed into three terms and elucidates how these terms depend on α, k , and other problem parameters. In particular, the term T 1 corresponds to the asymptotic squared bias ∥ E [ θ ( α ) ∞ -θ ∗ ] ∥ 2 , which is not affected by averaging. The term T 2 is associated with the variance Var( ¯ θ k 0 ,k ) , which decays at rate 1 /k due to averaging. Lastly, the term T 3 represents the optimization error ∥ E ¯ θ k 0 ,k -θ ∞ ∥ 2 , which decays geometrically in k 0 thanks to the use of a constant stepsize α and the tail-averaging procedure.

Note that averaging does not affect the bias of order α . With the precise bias characterization in Theorem 4.6, we can order-wise reduce the bias to O ( α 3 / 2 ) by employing the RR extrapolation technique [61]. Let ¯ θ ( α ) k 0 ,k and ¯ θ (2 α ) k 0 ,k denote the tail-averaged iterates using two stepsizes α and 2 α with the same data ( x k ) k ≥ 0 . The RR extrapolated iterates are defined as ˜ θ ( α ) k 0 ,k = 2 ¯ θ ( α ) k 0 ,k -¯ θ (2 α ) k 0 ,k .

Corollary 4.8 (RR-Extrapolation) . Under the setting of Theorem 4.6, the RR-extrapolated iterates satisfy the following bounds for all k > k 0 +2 τ α and k 0 ≥ τ α + 1 αµ log ( 1 ατ α ) ,

Backed by the CLT in Corollary 4.5, the iterates of constant-stepsize SA can be used to construct confidence intervals of θ ∗ . For i.i.d. data or linear SA, this approach has been explored in [34, 47, 66, 67] along with an appropriate variance estimator [26, 66]. In our Markovian nonlinear setting, where the iterates are biased, it is crucial to use RR extrapolation for bias reduction. Once the bias is accounted for, the power of using constant stepsizes reveals itself as it leads to rapid mixing and

low correlation of the iterates. Together, they lead to efficient confidence interval estimation schemes using nonlinear Markovian SA; see the empirical results in [34] showing its efficacy. In contrast, the classical diminishing stepsize paradigm often suffers from high correlation [13] and in turn inaccurate variance estimation, resulting in unsatisfactory coverage probability with finite data [34].

4.6 Implications for Learning GLM

Generalized linear models (GLM) extend linear regression to the model E [ Y | W ] = σ ( W ⊤ θ ∗ ) , where W is the covariate, Y the response variable, and σ is called the mean function . For any monotone (and potentially nonlinear) σ , the powerful framework developed in [19, 36, 37, 64] allows one to formulate the estimation of θ ∗ as minimizing an appropriate convex (surrogate) loss function. Applying SGD to this loss leads to a nonlinear SA update, to which our results are applicable. Below we discuss their applications in two concrete examples of GLMs.

Logistic Regression Logistic regression uses a sigmoid mean function σ ( x ) = 1 1+exp( -x ) . Suppose the covariate w k is sequentially sampled from a uniformly ergodic Markov chain with a bounded state space W ⊂ R d , and conditioned on w k the response y k is Bernoulli distributed with parameter (1 + exp( -w ⊤ k θ ∗ )) -1 . SGD applied to the L 2 -regularized negative log-likelihood function takes the form of the SA update θ k +1 = θ k + αg ( θ k , x k ) , where x k = ( w k , y k ) ∈ W × { 0 , 1 } and g ( θ k , x k ) = -w k ( σ ( -w ⊤ k θ k ) -y k ) -λθ k . For simplicity, we do not consider ξ -perturbation, i.e., ξ k +1 ( θ k ) ≡ 0 . It is easy to verify that this g is strongly monotone and sufficiently smooth with at most linear growth in | θ | , hence satisfying Assumption 1-3. Therefore, all the results in Sections 4.1-4.5 apply to logistic regression with constant stepsizes and Markovian data.

Smooth ReLU Regression The mean function σ can be interpreted as playing a similar role as the activation function in neural networks. Widely adopted is ReLU activation σ ( x ) = max(0 , x ) as well as its various smooth approximations [7, 30]. The problem of learning θ ∗ in this setting, sometimes called ReLU Regression, has been studied in the last decade and recently regained attention [19, 36, 37, 64]. Unlike linear or logistic regression, the least squares and maximum likelihood formulation associated with such nonlinear mean functions σ is non-convex. Nevertheless, the convex surrogate loss framework in [19, 64] still applies. As an example, we focus on the SoftPlus activation σ ( x ) = log(1 + exp( ιx )) /ι with a temperature parameter ι > 0 [30]. With L 2 -regularization the resulting SGD iteration is θ k +1 = θ k -α ( w k ( 1 ι log(1 + exp( ιw ⊤ k θ k )) -y k ) + λθ k ) , where the covariate-response pair ( θ k , x k ) is as before. This problem can again be cast as nonlinear SA with a strongly monotone and smooth g , satisfying Assumptions 1-3. All results in Sections 4.1-4.5 apply.

5 Numerical Experiments

In this section, we provide numerical experiment results to verify our theoretical results. We run SGD on L 2 -regularized logistic regression with Markovian data and constant stepsizes, where the covariate x t ∈ R is sequentially sampled from an autoregressive (AR) model of order 1; specifically, x t +1 = 0 . 9 x t + ζ t +1 with ζ t i.i.d. following N (0 , 1) , and the stationary distribution is x ∞ ∼ N (0 , 1 / (1 -0 . 9 2 )) . The binary dependent variable y t is sampled from Bernoulli (1 / (1+exp( -w ∗ x t )) with w ∗ = 1 . The regularized parameter is set to λ = 0 . 0001 . 1

To examine the asymptotic bias, we run the experiment for an episode length of 10 7 , with Markovian data as well as i.i.d. data sampled from N (0 , 1 / (1 -0 . 9 2 )) . We plot the errors (distance to θ ∗ ) of the PR averaged iterates and the RR extrapolated iterates, for different stepsizes α . Figure 1(a) verifies the presence of an asymptotic bias approximately proportional to the stepsize α , and illustrates the effectiveness of RR extrapolation in reducing this bias. In Figure 1(b), we compare the bias under Markovian data ( x t +1 ∼ P ( ·| x t ) ) and i.i.d. data ( x t ∼ x ∞ ). Interestingly, Figure 1(b) reveals that Markovian data does not necessarily lead to a larger bias than i.i.d. data. This is consistent with our theory, as the three bias terms b m , b n , b c may have opposite signs leading to cancellation. This result suggests that in the presence of nonlinearity, one should not avoid Markovian data simply for the sake of reducing bias. Rather, RR extrapolation may be more effective for bias reduction.

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

10

3

10

2

10

1

10 0

10 1

=0.8 Mkv PR

=0.2 Mkv PR

=0.8 Mkv RR

(a) This plot shows the errors of PR-averaged iterates and RR-extrapolated iterates, all generated using Markovian data.

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

10

3

10

2

10

1

10 0

10 1

=0.8 iid PR

=0.8 Mkv PR

=0.4 iid PR

Mkv RR

(b) This plot compares the errors under Markovian data ( x t ∼ P ( ·| x t -1 )) and iid data ( x t ∼ π ) , where all other settings are the same.

To verify the CLT in Corollary 4.5, we repeat the experiment 1000 times with an episode length of 10 6 and stepsize α = 0 . 8 . We compute the PR averaged iterates and plot the histogram and the quantile-quantile (QQ) plot in Figure 2 in Appendix C. The close alignment between the histogram and the normal curve in Figure 2(a) and the linearity of the points along the 45-degree reference line in the QQ plot in Figure 2(b) confirm that the empirical distribution follows a normal distribution.

6 Related Work

General SA and SGD. SA and SGD can be traced back to the seminal work of [58]. Classical work assumes a diminishing stepsize sequence, and has shown almost sure asymptotic convergence to θ ∗ [8, 58]. Subsequent works propose the iterate averaging technique, now known as PolyakRuppert (PR) averaging, to reduce variance and accelerate convergence [56, 59], and also establish a Central Limit Theorem for the asymptotic normality of the averaged iterates [57]. The asymptotic convergence theory of SA and SGD is well developed and extensively addressed in many exemplary textbooks, see [3, 40, 65]. There are also recent works studying the non-asymptotic convergence with diminishing stepsizes [12, 14]. The recent work [15] establishes the high probability bound on the estimation error of contractive SA with diminishing stepsize.

SA and SGD with Constant Stepsizes. There has been an increasing interest in studying SA with constant stepsize. Many works in this line provide non-asymptotic upper bounds on mean squared error (MSE) E [ ∥ θ t -θ ∗ ∥ 2 ] . Works in [25, 42, 51] study linear SA (LSA) under i.i.d. data. Recent works extend the analysis of the MSE to LSA with Markovian data, such as [24, 52, 60]. There are also works providing upper bounds of MSE for general contractive SA with Markovian noise [14, 17].

In addition to non-asymptotic guarantees, some works focus on the asymptotic behavior of SA iterates. Recent works have shown that when using constant stepsize, one loses the almost sure convergence guarantee in the diminishing stepsize sequence regime, and at best can achieve distributional convergence, as demonstrated in [16, 20, 25, 33, 66, 67, 69]. The presence of asymptotic bias is also a recurring theme in recent literature, with precise characterization given in [20] for strongly-convex SGD with i.i.d. data and in [33] for LSA with Markovian data. Works in [34, 51, 66, 67, 69] also establish Central Limit Theorems for averaged SA iterates with constant stepsizes.

7 Conclusion

We provide the first weak convergence and steady-state analysis for constant-stepsize SA with both nonlinear update and Markovian data. Our analysis elucidates the compound effect of nonlinearity and memory, which leads to new analytical challenges and behaviors. A limitation of our results is the use of a projection step or the noise minorization assumption. Whether they can be removed is worth investigating. Other future directions include refining the dimension dependence in our results, as well as a theoretical investigation of statistical inference.

Acknowledgement Y. Chen is partially supported by National Science Foundation (NSF) grants CCF-1704828 and NSF CCF-2233152. Y. Zhang and Q. Xie are supported in part by NSF grants CNS-1955997, EPCN-2339794 and EPCN-2432546. Y. Zhang is also supported in part by NSF Award DMS-2023239.

References

Sebastian Allmeier and Nicolas Gast. Computing the bias of constant-step stochastic approximation with Markovian noise, 2024. 2, 16

Borja Balle and Yu-Xiang Wang. Improving the Gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 394-403. PMLR, 10-15 Jul 2018. 6

Albert Benveniste, Michel Metivier, and Pierre Priouret. Adaptive Algorithms and Stochastic Approximations . Springer Berlin Heidelberg, 1st edition, 1990. 10

Dimitri P. Bertsekas. Reinforcement learning and Optimal Control . Athena Scientific, Belmont, Massachusetts, USA, 2019. 1

Aleksandr Beznosikov, Sergey Samsonov, Marina Sheshukova, Alexander Gasnikov, Alexey Naumov, and Eric Moulines. First order methods with Markovian noise: from acceleration to variational inequalities, 2023. 2, 6

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. Operations Research , 69(3):950-973, May 2021. 3, 16

Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey. Smooth maximum unit: Smooth activation function for deep networks using smoothing maximum technique. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 784-793, 2022. 1, 9

Julius R. Blum. Approximation methods which converge with probability one. The Annals of Mathematical Statistics , 25(2):382 - 386, 1954. 10

Vivek S. Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint . Hindustan Book Agency Gurgaon, 2008. 1

Vivek S. Borkar and Sean P. Meyn. The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization , 38(2):447-469, Jan 2000. 2

Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning , 8(3-4):231-357, November 2015. 16

Siddharth Chandak, Vivek S. Borkar, and Parth Dodhia. Concentration of contractive stochastic approximation and reinforcement learning. Stochastic Systems , Jul 2022. 10

Xi Chen, Jason D Lee, Xin T Tong, and Yichen Zhang. Statistical inference for model parameters in stochastic gradient descent. Annals of Statistics , 48(1):251-273, 2020. 9

Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, and Karthikeyan Shanmugam. A Lyapunov theory for finite-sample guarantees of Markovian stochastic approximation. Operations Research , 2023. 10

Zaiwei Chen, Siva Theja Maguluri, and Martin Zubeldia. Concentration of contractive stochastic approximation: Additive and multiplicative noise. arXiv preprint arXiv:2303.15740 , 2023. 10

Zaiwei Chen, Shancong Mou, and Siva Theja Maguluri. Stationary behavior of constant stepsize sgd type algorithms: An asymptotic characterization. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 6(1), 02 2022. 10

Zaiwei Chen, Sheng Zhang, Thinh T. Doan, John-Paul Clarke, and Siva Theja Maguluri. Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning. Automatica , 146:110623, 2022. 2, 3, 4, 6, 10, 18, 34

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs), 2016. 1

Ilias Diakonikolas, Surbhi Goel, Sushrut Karmalkar, Adam R. Klivans, and Mahdi Soltanolkotabi. Approximation schemes for ReLU regression. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 1452-1485. PMLR, 09-12 Jul 2020. 1, 9

Aymeric Dieuleveut, Alain Durmus, and Francis Bach. Bridging the gap between constant step size stochastic gradient descent and Markov chains. The Annals of Statistics , 48(3):1348 - 1382, 2020. 2, 3, 4, 6, 8, 10, 16

Jing Dong and Xin T. Tong. Stochastic gradient descent with dependent data for offline reinforcement learning, 2022. 3

Jinshuo Dong, Aaron Roth, and Weijie J. Su. Gaussian Differential Privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(1):3-37, 02 2022. 6

Randal Douc, Eric Moulines, Pierre Priouret, and Philippe Soulier. Markov Chains . Springer Cham, 1st edition, 2018. 5

Alain Durmus, Eric Moulines, Alexey Naumov, and Sergey Samsonov. Finite-time highprobability bounds for Polyak-Ruppert averaged iterates of linear stochastic approximation, 2022. 3, 10

Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov, Kevin Scaman, and Hoi-To Wai. Tight high probability bounds for linear stochastic approximation with fixed stepsize. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 30063-30074. Curran Associates, Inc., 2021. 4, 10

James M. Flegal and Galin L. Jones. Batch means and spectral variance estimators in Markov chain Monte Carlo. The Annals of Statistics , 38(2):1034 - 1070, 2010. 8

Gerald B. Folland. Real analysis: modern techniques and their applications . Wiley, New York, 2nd ed. edition, 1999. 16

Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration, 2019. 6

Sergey G. Foss and Richard L. Tweedie. Perfect simulation and backward coupling. Communications in Statistics. Stochastic Models , 14(1-2):187-203, 1998. 4

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pages 315-323, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. 1, 9

F B Hildebrand. Introduction to numerical analysis . Dover Books on Mathematics. Dover Publications, Mineola, NY, 2 edition, June 1987. 2

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. 3

Dongyan (Lucy) Huo, Yudong Chen, and Qiaomin Xie. Bias and extrapolation in Markovian linear stochastic approximation with constant stepsizes. In Abstract Proceedings of the 2023 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems , pages 81-82, 2023. 2, 3, 4, 5, 6, 8, 10, 45, 55, 56

Dongyan (Lucy) Huo, Yudong Chen, and Qiaomin Xie. Effectiveness of constant stepsize in Markovian LSA and statistical inference, 2023. 2, 7, 8, 9, 10

Prateek Jain, Sham M. Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic gradient descent for least squares regression: Mini-batching, averaging, and model misspecification. Journal of Machine Learning Research , 18(223):1-42, 2018. 2

Sham M Kakade, Varun Kanade, Ohad Shamir, and Adam Kalai. Efficient learning of generalized linear and single index models with isotonic regression. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011. 1, 9

Adam Tauman Kalai and Ravi Sastry. The isotron algorithm: High-dimensional isotonic regression. In COLT , 2009. 1, 9

Kyurae Kim, Jisu Oh, Jacob Gardner, Adji Bousso Dieng, and Hongseok Kim. Markov chain score ascent: A unifying framework of variational inference with markovian gradients. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 34802-34816. Curran Associates, Inc., 2022. 2

Harold Kushner. Stochastic approximation: a survey. WIREs Computational Statistics , 2(1):8796, 2010. 16

Harold J. Kushner and G. George Yin. Stochastic Approximation and Recursive Algorithms and Applications . Stochastic Modelling and Applied Probability. Springer, New York, NY, USA, 2nd edition, 2003. 1, 10

Simon Lacoste-Julien, Mark Schmidt, and Francis Bach. A simpler approach to obtaining an o(1/t) convergence rate for the projected stochastic subgradient method, 2012. 16

Chandrashekar Lakshminarayanan and Csaba Szepesvári. Linear stochastic approximation: How far does constant step-size and iterate averaging go? In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1347-1355. PMLR, 09-11 Apr 2018. 10

Guanghui Lan. First-order and stochastic optimization methods for machine learning , volume 1. Springer, 2020. 1

Caio Kalil Lauand and Sean Meyn. Bias in stochastic approximation cannot be eliminated with averaging. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1-4, 2022. 2

Caio Kalil Lauand and Sean Meyn. The curse of memory in stochastic approximation. In 2023 62nd IEEE Conference on Decision and Control (CDC) , pages 7803-7809, 2023. 2, 3, 6, 16

Caio Kalil Lauand and Sean Meyn. Revisiting step-size assumptions in stochastic approximation, 2024. 2

Tianyang Li, Liu Liu, Anastasios Kyrillidis, and Constantine Caramanis. Statistical inference using SGD. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), Apr. 2018. 8

Xiang Li, Jiadong Liang, and Zhihua Zhang. Online statistical inference for nonlinear stochastic approximation with Markovian data, 2023. 3

Michael Maxwell and Michael Woodroofe. Central limit theorems for additive functionals of Markov chains. The Annals of Probability , 28(2):713 - 724, 2000. 39

Sean P. Meyn and Richard L. Tweedie. Markov Chains and Stochastic Stability . Cambridge Mathematical Library. Cambridge University Press, Cambridge, 2nd edition, 2009. 5, 6, 40, 44, 45

Wenlong Mou, Chris Junchi Li, Martin J. Wainwright, Peter L. Bartlett, and Michael I. Jordan. On linear stochastic approximation: Fine-grained Polyak-Ruppert and non-asymptotic concentration. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 2947-2997. PMLR, 09-12 Jul 2020. 10

Wenlong Mou, Ashwin Pananjady, Martin J. Wainwright, and Peter L. Bartlett. Optimal and instance-dependent guarantees for Markovian linear stochastic approximation. Mathematical Statistics and Learning , 7(1):41-153, March 2024. 3, 10

Dheeraj Nagaraj, Xian Wu, Guy Bresler, Prateek Jain, and Praneeth Netrapalli. Least squares regression with Markovian data: Fundamental limits and algorithms. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 16666-16676. Curran Associates, Inc., 2020. 2

A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization , 19(4):1574-1609, 2009. 16

Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations , 2018. 6

Boris T. Polyak. New stochastic approximation type procedures. Automation and Remote Control , 51(7):98-107, Jul 1990. 2, 8, 10

Boris T. Polyak and Anatoli B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838-855, Jul 1992. 10

Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics , 22(3):400 - 407, 1951. 2, 10

David Ruppert. Efficient estimations from a slowly convergent Robbins-Monro process. Technical report, Cornell University, February 1988. 2, 8, 10

Rayadurgam Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation and TD learning. In Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 2803-2830. PMLR, 25-28 Jun 2019. 2, 3, 5, 6, 10, 18

Josef Stoer and Roland Bulirsch. Introduction to Numerical Analysis . Springer, New York, NY, USA, 3rd edition, 2002. 8

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . A Bradford Book, Cambridge, MA, USA, 2018. 1

Cédric Villani. Optimal Transport: Old and New . Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2009. 30, 39, 40

Puqian Wang, Nikos Zarifis, Ilias Diakonikolas, and Jelena Diakonikolas. Robustly learning a single neuron via sharpness. arXiv preprint arXiv:2306.07892 , 2023. 1, 9

Stephen J. Wright and Benjamin Recht. Optimization for Data Analysis . Cambridge University Press, 2022. 10

Chuhan Xie and Zhihua Zhang. A statistical online inference approach in averaged stochastic approximation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 8998-9009. Curran Associates, Inc., 2022. 8, 10

Lu Yu, Krishnakumar Balasubramanian, Stanislav Volgushev, and Murat A. Erdogdu. An analysis of constant step size SGD in the non-convex regime: Asymptotic normality and bias. In Advances in Neural Information Processing Systems , volume 34, pages 4234-4248. Curran Associates, Inc., 2021. 2, 6, 8, 10

Yixuan Zhang, Dongyan (Lucy) Huo, Yudong Chen, and Qiaomin Xie. Prelimit coupling and steady-state convergence of constant-stepsize nonsmooth contractive SA. In Abstracts of the 2024 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems , SIGMETRICS/PERFORMANCE '24, page 35-36, New York, NY, USA, 2024. Association for Computing Machinery. 16

Yixuan Zhang and Qiaomin Xie. Constant stepsize Q-learning: Distributional convergence, bias and extrapolation. Reinforcement Learning Journal , 3:1168-1210, 2024. 2, 10

A Additional Notations

General Probability We write z 1 ⊥ ⊥ z 2 | z 3 if random variables z 1 and z 2 are conditionally independent given z 3 .

̸

Recall that we define the metric ¯ d ( ( x, θ ) , ( x ′ , θ ′ ) ) := √ ✶ { x = x ′ } + ∥ θ -θ ′ ∥ 2 for the space X× R d . Thus, for ¯ µ and ¯ ν in P 2 ( X × R d ) , the Wasserstein-2 distance w.r.t. ¯ d is computed as

̸

General State Space Markov Chains Throughout the paper, we assume that X is a Borel space. Let P denote the transition kernel. We call π the stationary distribution of P if it satisfies ∫ X π ( d x ) P ( x, B ) = π ( B ) , for B ∈ B ( X ) . Define the π -weighted inner product ⟨ f, g ⟩ L 2 ( π ) = ∫ X π ( d x ) f ⊤ ( x ) g ( x ) and the induced norm ∥ f ∥ L 2 ( π ) = ( ⟨ f, f ⟩ L 2 ( π ) ) 1 / 2 . Let L 2 ( π ) = { f : ∥ f ∥ L 2 ( π ) < ∞} denote the corresponding Hilbert space of R d -valued, squareintegrable and measurable functions on X . For an operator T : L 2 ( π ) → L 2 ( π ) , its operator norm is defined as ∥ T ∥ L 2 ( π ) = sup ∥ f ∥ L 2 ( π ) =1 ∥ Tf ∥ L 2 ( π ) . The transition kernel is a bounded linear operator on L 2 ( π ) , in particular with norm ∥ P ∥ L 2 ( π )=1 . Also, we define the kernel/operator Π = 1 ⊗ π by Π( x, · ) = π .

Throughout the paper, we assume that X is a Borel space. Let P denote the transition kernel. We call π the stationary distribution of P if it satisfies ∫ X π ( d x ) P ( x, B ) = π ( B ) , for B ∈ B ( X ) . There exists a kernel P ∗ as a regular conditional probability that satisfies ∫ A π ( d s ) P ( x, B ) = ∫ B π ( d y ) P ∗ ( y, A ) , for A,B ∈ B ( X ) [27, Chapter 21.4, Theorem 19], and P ∗ defines the probability law for the time-reversed chain of ( x k ) k ≥ 0 .

B Additional Discussion on Assumptions

In this section, we provide a more detailed discussion of the assumptions taken in this work.

Projection and Minorization Projection steps have a longstanding presence in SA literature for tractability in convergence theory, as seen in many analyses of SGD [6, 11, 39, 41, 54]. Although not an algorithmic proposal, this additional projection step does not incur much computational cost in practice, as it only involves rescaling the iterates, and the projection radius can be estimated a priori. Before our work, no studies had proven weak convergence for non-linear SA with Markovian data and constant stepsize, with or without the projection. Thus, our result is the first to prove detailed weak convergence in this setting. Nonetheless, we provide an alternative proof of weak convergence in Theorem 4.3 using the Drift and Minorization technique, which does not require a projection.

Differentiability The differentiability condition of the SA update operator g in Assumption 2 ensures controlled evolution of the iterates θ . This differentiability assumption supports a third-order Taylor expansion of g with a bounded remainder, which is crucial for both analyzing the ♠ term in the convergence proof as discussed in Section 3 and for bias characterization. Moreover, some form of differentiability assumption is standard in SA literature, such as [1, 20, 45], particularly when one seeks a fine-grained characterization of the iterates' distributional property. Such an assumption is satisfied by many GLMs, such as logistic regression and Poisson regression. When g is not differentiable, the bias of SA iterates behaves drastically different [68], which is beyond the scope of this paper.

Strong Monotonicity The strong monotonicity assumption is common in SA literature. Together with smoothness, it allows us to establish geometric distributional convergence. While some GLMs by themselves do not satisfy this condition, it is a common practice to apply L 2 -regularization (equivalently, weight decay) to ensure strong convexity and improve statistical performance. It is a standard calculation that one can appropriately choose the regularization parameter to derive tight results for non-strongly-convex functions.

C Additional Plots

In this section, we present the plots from numerical experiments in Section 5 that verify the Central Limit Theorem (CLT) in Corollary 4.5. In these plots, we plot the centered and scaled PR-averaged iterates, i.e, √ T ( ¯ θ ( k ) T -∑ 1000 l =1 ¯ θ ( l ) T ) , where ¯ θ ( k ) = ( ∑ T t =1 θ t ) /T with T = 10 6 is the PR-averaged iterates for the k -th repeat.

20

10

0

10

20

0

20

40

60

80

3

2

1

0

-1

-2

4

-2

1

0

1

2

3

D Proof of Pilot Results (Proposition 4.2)

In this section, we prove the pilot result, namely Proposition 4.2. We prove the desired moments for β = ∞ , i.e., without any projection. It is easy to see that when the projection radius β ∈ [2 ∥ θ ∗ ∥ , ∞ ] ,

where θ t +1 / 2 denotes the iterate before projection. The term on the right hand side can be further bounded by the moment bounds for iteration without projection. Therefore, it suffices for us to prove the respective moment bounds without any projection.

Given Assumption 4 hold for 2 p -th moment, with p ≥ 1 , we prove the moment bound in Proposition 4.2 for n with 1 ≤ n ≤ p by induction.

D.1 Base Case

In this section, we prove the base case of Proposition 4.2, i.e., with n = 1 . The base case gives the desired mean squared error (MSE) convergence bound, which will subsequently be used in the proof of weak convergence.

We start by noting the following decomposition,

It is easy to see that under Assumption 3, we have

Additionally, under Assumption 2 and 4, we have the following upper bound

Therefore, the key to analyze the remaining inner product ⟨ θ k -θ ∗ , g ( θ k , x k ) -¯ g ( θ k ) ⟩ . Consider the following decomposition

Hence, we need some upper bound on ∥ θ k -θ k -τ ∥ .

We next note the following technical Lemma, which is adapted from [17, 60] for the updated unbounded i.i.d. noise assumption in Assumption 4. The proof of the technical Lemma is delayed to Appendix D.1.1.

Lemma D.1. For 16 ατ ≤ µ/ (4 L 2 ) , we have

Given (D.10), we additionally note that

We next use the above four technical inequalities to analyze the four terms in (D.3)-(D.4). To bound (D.3), we first note that

where (i) holds for the Cauchy-Schwarz inequality and (ii) holds for (D.10). To bound (D.4), we next note that

where (iii) holds due to the mixing property of Markov chain ( x k ) k ≥ 0 and (iv) holds for (D.11).

To bound (D.5), we have

where (v) holds for (D.7) and (vi) holds for (D.11).

Lastly, to bound (D.6), we apply the similar technique used in bounding the third term in (D.5) and obtain a similar result

Combining all analyses above, we have

Hence, making use of (D.1), (D.2), and (D.12), we obtain the following

Therefore, when we have α satisfying the constraint, i.e., ατL 2 < c 2 , 1 µ, we obtain

Recursively, we get

Lastly, we note that

where (i) holds by the Bernoulli inequality, that (1 + x ) r ≥ 1 + rx for x ≥ -1 and r ≥ 1 ; (ii) holds for µ ≤ L ; (iii) holds for ατL < µ/ (114 L ) < 1 2 .

Hence, for k ≥ τ , we have

for c 2 , 1 and c 2 , 2 some universal constants. As such, we have completed the proof of base case for Proposition 4.2.

D.1.1 Proof of Lemma D.1

In this section, we provide the proofs of the four technical inequalities in Lemma D.1.

Proof of (D.7) .

Proof. Note that

so we start with analyzing ∥ θ t +1 -θ t ∥ .

Recall that we assume

then we have for k -τ ≤ t ≤ k ,

Hence, for 0 ≤ n ≤ τ ,

We next note that

Hence, at this stage, if we require ατL < µ/ (4 L ) < 1 / 4 , we have the following upper bound (1 + αL ) n ≤ (1 + αL ) τ ≤ 1 + 2 ατL ≤ 2 .

Therefore, for 0 ≤ n ≤ τ ,

As such, we have

and prove the desired inequality.

Proof of (D.8) .

Proof. We prove this inequality based on the claim that we have just shown,

We simply note that

Hence,

Therefore, we obtain

Proof of (D.9) .

Proof. To analyze E [ ∥ θ k -θ k -τ ∥ 2 |F k -τ ] , we consider the following attempt.

Next, we study E [ ∥ θ t -θ ∗ ∥ 2 |F k -τ ] . We start with the following, for k -τ ≤ t < k , E [ ∥ θ t +1 -θ ∗ ∥ 2 |F k -τ ] = E [ ∥ θ t -θ ∗ ∥ 2 |F k -τ ] + 2 α E [ ⟨ θ t -θ ∗ , g ( θ t , x t ) ⟩|F k -τ ] + α 2 E [ ∥ g ( θ t , x t ) + ξ t +1 ( θ t ) ∥ 2 |F k -τ ] ≤ E [ ∥ θ t -θ ∗ ∥ 2 |F k -τ ] + 2 α 2 ( E [ ∥ g ( θ t , x t ) ∥ 2 |F k -τ ] + E [ ∥ ξ t +1 ( θ t ) ∥ 2 |F k -τ ] ) +2 α E [ ∥ θ t -θ ∗ ∥∥ g ( θ t , x t ) ∥|F k -τ ] .

We note that

Substituting the above inequality back, we obtain

We further recall that

and hence we obtain the following upper bound

Then, recursively, for 0 ≤ n ≤ τ , we have

As such, under the assumption that 4 ατL < µ/ (4 L ) < 1 / 4 , then for k -τ ≤ t ≤ k , we have

E

[

∥

θ

t

-

θ

∗

∥

2

|F

k

-

τ

]

(1 + 10

2

∥

θ

k

-

τ

Combining all the analyses above, we have

Proof of (D.10) .

Proof. This inequality simply extends the result from (D.9),i.e.,

We first note that

Hence,

Again, under the assumption that 16 ατL 2 < µ/ 4 , we can conclude that

ατL

)

-

θ

∗

∥

∥

θ

2

k

-

τ

-

+10

θ

∗

∥

ατL.

2

+10

ατL

D.2 Induction Step

In this step, assume that the moment bound in Proposition 4.2 has been proven for k ≤ n -1 , we now proceed to show that the desired moment convergence holds for n with 2 ≤ n ≤ p .

We start with the following decomposition of ∥ θ k +1 -θ ∗ ∥ 2 n

We note the following cases.

i = n , j = l = 0 . In this case, the summand is simply ∥ θ k -θ ∗ ∥ 2 i .

When i = n -1 , j = 1 and l = 0 . In this case, the summand is of order α , i.e., α 2 n ⟨ θ k -θ ∗ , g ( θ k , x k ) + ξ k +1 ( θ k ) ⟩ j ∥ θ k -θ ∗ ∥ 2( n -1) . We can further compose it as

Note that, when ( x k ) is i.i.d. or from a martingale noise sequence, we have

However, when ( x k ) is Markovian, the above equality then does not hold and T 1 requires a careful analysis.

Nonetheless, under the strong monotonicity assumption, we have

For the remaining terms, we see that they are of higher orders of α . Therefore, when α is selected sufficiently small, these terms do not raise concern.

Therefore, to prove the desired moment bound, we spend the remaining section analyzing T 1 . Immediately, we note that

Subsequently, we focus on analyzing T ′ 1 .

We start with the following decomposition of T ′ 1 .

We note the following technical lemma, which will offer significant help in the analysis of T ′ 1 . We postpone the proof of the lemma to the end of this subsection.

LemmaD.2. For ˜ c n ατ ≤ µ/ (4 L 2 ) , where ˜ c n denotes some constant dependent of the higher-moment 2 n , we have

Following the lemma, we observe that a natural consequence is for any m ≤ 2 n , we have

where we use the inequality a + b > ( a + b ) for a, b > 0 , p ∈ (0 , 1)

p p p to obtain the final inequality.

Now, we are ready to analyze (D.14)-(D.18). Firstly, for (D.14), we make use of the mixing assumption of τ , and have that

where we make use of the inequality 2 | x | 3 ≤ x 2 + x 4 to obtain the final step.

Next, we proceed to analyze (D.15). It is easy to see that

The term in (D.16) can be analyzed in a similar fashion as the (D.15).

For (D.17), we first derive the following

We next analyze the two terms T a and T b respectively. Starting with T a , we have

For T b , we have

Combining the analyses of the two terms, we get the following upper bound to (D.17)

Lastly, we analyze (D.18). We first make use of the mean-value theorem, with a ∈ [0 , 1] , we have

Substituting the above upper bound back into (D.18), we obtain

Combining the analyses above, we have the following bound for T 1 ,

where c n, 1 , c n, 2 and c n, 3 are some constants that depend on n .

Additionally, we note that

Therefore, for sufficiently small ατL < µ/ ( c ′ n L ) , we have

As such, for sufficiently small α , we have

E [ | T 1 ||F k -τ ]

Hence, up til this point, we have obtained

Following the induction hypothesis, when k is sufficiently large, we have

Substituting the above upper bound back into our analysis of the 2 n -th moment bound, we obtain

Subsequently, if we set α sufficiently small, such that

we obtain

where s ( θ 0 , L, µ ) is some constant that may depend on the initialization θ 0 and the problem primitives µ and L but is independent of α .

Recursively, we get

Lastly, we recall that

Substituting back, we obtain for sufficiently large k ,

As such, we have proven the desired n -th moment bound.

D.2.1 Proof of Lemma D.2

We now come back to Lemma D.2 and provide the complete proof.

Proof. The proof follows a similar strategy as (D.9) and (D.10) in Section D.1.1.

We start with the following relaxation and obtain that

Next, in order to obtain a bound on ∥ θ t -θ ∗ ∥ 2 n , we study the following term.

Note that

Substituting back, we obtain

Consolidating the terms, we have

Recursively, for 0 ≤ l ≤ τ , we have

Then, for we have for k -τ ≤ t ≤ k ,

Finally, we have

As such, we have completed the proof.

E Proof of Theorem 4.1

In this section, we prove the weak convergence result in Theorem 4.1. In fact, the proof of the projected SA weak convergence result can be seen as a special case of unprojected SA with the asymptotic linearity condition, which we have briefly discussed in Section 4. Therefore, the proof proceeds in the following two subsections. First, we formally define the asymptotic linearity condition and present our weak convergence result for unprojected SA under this additional assumption. Next, we relate this result for unprojected SA to projected SA and specialize the proof to obtain Theorem 4.1.

E.1 Asymptotic Linearity

In this subsection, we formally introduce the asymptotic linearity condition, which is crucial for establishing weak convergence in the context of unprojected SA ( β = ∞ ). Additionally, we explore the implications of this condition.

Assumption 6 (Asymptotic Linearity) . The noise sequence ( ξ k ) k ≥ 1 is a collection of i.i.d. random fields satisfying the following conditions: (1) E [ ξ k +1 ( θ ) |F k ] = 0 , (2) there exists a constant L 3 > 0 such that ξ 1 is L 3 -Lipschitz, i.e., ∥ ξ 1 ( θ ) -ξ 1 ( θ ′ ) ∥ ≤ L 3 ∥ θ -θ ′ ∥ , for all θ, θ ′ ∈ R d , and (3) ∥ ξ 1 (0) ∥ ≤ L 3 .

Moreover, there exists a function G ( · ) : X → R d × d such that given ϵ > 0 , define

and we have lim ϵ → 0 ϵδ ( ϵ ) = 0 .

The first part of Assumption 6 states that the random field grows at most linearly in θ. The second part of Assumption 6 implies that g ′ ( θ, x ) converges to a limit G ( x ) when ∥ θ ∥ → ∞ for all x ∈ X , which shows the asymptotic linearity of g ( θ, x ) . Furthermore, Assumption 6 also requires how fast

g ′ ( θ, x ) converges to G ( x ) . A sufficient condition under which the second part of Assumption 6 holds is that there exists ω > 0 such that ∥ θ ∥ 1+ ω ∥ g ′ ( θ, x ) -G ( x ) ∥ < ∞ for ∀ θ ∈ R d and x ∈ X . We can verify that to ensure ∥ g ′ ( θ, x ) -G ( x ) ∥ < ϵ , we can set ∥ θ ∥ ∈ Θ( ϵ -1 1+ ω ) , which can ensure ϵδ ( ϵ ) ∈ O ( ϵ ω 1+ ω ) → 0 as ϵ → 0 . This sufficient condition implies that g ′ ( θ, x ) uniformly converge to G ( x ) with convergence rate of O ( ∥ θ ∥ -(1+ w ) ) . By definition, we conclude that the structure of linear SA is also asymptotic linear. Besides that, the 1-dimensional logistic regression also satisfies Assumption 6. For 1-dimensional logistic regression, we have g ( θ, x, y ) = x ( 1 1+ e -θx -y ) + λθ , where ( x, y ) presents the data. Therefore, we have g ′ ( θ, x, y ) = x 2 e -θx (1+ e -θx ) 2 + λ and g ′ ( θ, x, y ) uniformly converges to λ with geometric convergence rate, thereby satisfies the Assumption 6.

E.2 Proof Under Assumption 6

With the asymptotic linearity condition now formally defined, we proceed to prove the weak convergence for unprojected SA. For convenient reference, we state the theorem below.

Theorem E.1 (Ergodicity of SA-Asymptotic Linearity) . Suppose that Assumption 1-Assumption 4 hold. Additionally, assume 6. For stepsize α > 0 that satisfies the constraint ατ α L 2 < min( c 2 µ, κ µ ) , with c 2 formalized in Proposition 4.2 and κ µ defined in (E.1) , the Markov chain ( x k , θ k ) k ≥ 0 converges to a unique stationary distribution ¯ ν α ∈ P 2 ( X × R d ) .

Moreover, there exist κ µ > 0 and some universal constant c ′ such that

We let ν α := L ( θ ∞ ) be the second marginal of ¯ ν α . For k ≥ 2 τ α , it holds that

The proof of Theorem E.1 consists of two major steps. Firstly, we assume that x 0 ∼ π , and show that ( x k , θ k ) k ≥ 0 converges to a unique limiting invariant distribution. Next, we relax the assumption of x 0 ∼ π , and prove that for arbitrary initialization ( x 0 , θ 0 ) ∈ X × R d , the Markov chain will converge to the same limit.

Step 1: Initialization with x 0 ∼ π . To prove the convergence of the Markov chain, we consider the following coupling construction. We have a pair of Markov chains ( x k , θ [1] k ) k ≥ 0 and ( x k , θ [2] k ) k ≥ 0 sharing the same underlying process and noise, i.e., ( x k , ξ k +1 ) k ≥ 0 , i.e.,

We assume that the initial iterates θ [1] 0 and θ [2] 0 may depend on each other and on x 0 , but are independent of subsequent ( x k ) k ≥ 1 given x 0 . For the iterates difference θ [1] k -θ [2] k , we have the following Proposition E.2, whose proof is given at the end of this subsection.

Proposition E.2. ∀ k ≥ τ and ατ ≤ min( µ 908 L 2 , κ µ L 2 ) , we have

where κ µ > 0 and ϵδ ( ϵ ) ≤ µ 768 , ∀ ϵ ≤ κ µ .

By Proposition E.2 and the definition of W 2 and ¯ W 2 , we have

where (i) and (ii) hold by the definition of W 2 and ¯ W 2 and (iii) holds by applying Proposition E.2.

Note that equation (E.4) always holds for any joint distribution of initial iterates ( x 0 , θ [1] 0 , θ [2] 0 ). Recall that P ∗ represents the transition kernel for the time-reversed Markov chain of { x k } k ≥ 0 , and the initial distribution of x 0 is assumed to be mixed already. Given a specific x 0 , we sample x -1 from P ∗ ( · | x 0 ) . Additionally, we use θ [2] -1 to denote the random varible that satisfies θ [2] -1 d = θ [1] 0 and is independent of { x k } k ≥ 0 . Finally, we set θ [2] 0 as

By the property of time-reversed Markov chain, we have { x k } k ≥-1 d = { x k } k ≥ 0 . Given that θ [2] -1 d = θ [1] 0 and θ [2] -1 is independent with { x k } k ≥-1 , we can prove ( x k , θ [2] k ) d = ( x k +1 , θ [1] k +1 ) for k ≥ 0 . We thus have for all k ≥ τ :

where (i) holds by inequality (E.4). Then, we have

Consequently, {L ( x k , θ [1] k ) } k ≥ 0 forms a Cauchy sequence w.r.t. the metric ¯ W 2 . Since the space P 2 ( X × R d ) endowed with ¯ W 2 is a Polish space, every Cauchy sequence converges [63, Theorem 6.18]. Furthermore, convergence in Wasserstein 2-distance also implies weak convergence [63, Theorem 6.9]. Therefore, we conclude that the sequence ( L ( x k , θ [1] k )) k ≥ 0 converges weakly to a limit distribution ¯ µ ∈ P 2 ( X × R d ) .

Now that we have established the existence of a limiting distribution, we next proceed to show the uniqueness. We prove this by contradiction. Note that we currently assume that x 0 ∼ π , hence to show that the limit ( x ∞ , θ ∞ ) is unique, we only need to show that the limit is independent of the initial distribution of θ 0 , which can be correlated to x 0 .

Consider two Markov chains ( x k , θ k ) k ≥ 0 and ( x k , θ ′ k ) k ≥ 0 , sharing ( x k , ξ k +1 ) k ≥ 0 but with arbitrary initialization of θ 0 and θ ′ 0 . For the sake of contradiction, we assume that ( x 0 , θ 0 ) ⇒ ( x ∞ , θ ∞ ) and ( x 0 , θ ′ 0 ) ⇒ ( x ∞ , θ ′ ∞ ) respectively. Then, by the triangle inequality, we have that

As such, we have shown that the limit ¯ ν is unique.

Lastly, we prove that ¯ ν is invariant. Suppose that we initialize the joint process at its limit, i.e., ( x 0 , θ 0 ) ∼ ¯ ν . We first apply the triangle inequality, and we obtain

Clearly, as k →∞ , ¯ W 2 ( ( x k +1 , θ k +1 ) , ( x 0 , θ 0 ) ) → 0 . To bound ¯ W 2 ( ( x 1 , θ 1 ) , ( x k +1 , θ k +1 ) ) , we need the following lemma.

Lemma E.3. Consider two copies of the SA trajectory, where L ( x 0 , θ 0 ) = ¯ ν and L ( x ′ 0 , θ ′ 0 ) is allowed to be arbitrary.

where

are independent of L ( x , θ )

′ 0 ′ 0 .

Proof. Consider the following coupling between the two processes ( x k , θ 0 ) k ≥ 0 and ( x ′ k , θ ′ k ) k ≥ 0

Then, it is clear that

̸

Recall the metric d 0 ( x, x ′ ) = { x = x ′ } and hence, we have

Therefore, it is easy to see that

whence

As such, we see that

Next, we make use of Cauchy-Schwarz inequality and obtain

Because Assumption 6 implies Assumption 4 ( p = 2) , by Proposition 4.2 and Fatou's lemma, we have

which implies E [ ∥ θ ∞ ∥ 4 ] < ∞ . Hence, the desired inequality follows through.

By Lemma E.3, we can set L ( x ′ 0 , θ ′ 0 ) = L ( x k , θ k ) , then

Therefore, ¯ W 2 2 ( L ( x 1 , θ 1 ) , L ( x k +1 , θ k +1 )) → 0 as k → 0 , which implies ¯ W 2 ( ( x 1 , θ 1 ) , ( x 0 , θ 0 ) ) = 0 . As such, we have proved the joint sequence ( x k , θ k ) k ≥ 0 converges weakly to the unique invariant distribution ¯ ν ∈ P 2 ( X × R d ) . As a result, { θ k } k ≥ 0 converges weakly to µ ∈ P 2 ( R d ) , where µ is the second marginal of ¯ µ over R d .

Lastly, before proceeding to the next step, in which we remove the assumption x 0 ∼ π , we first derive the convergence rate of { θ k } k ≥ 0 under x 0 ∼ π as presented in the following lemma. This lemma will help us to establish the convergence rate without x 0 ∼ π .

Lemma E.4. Under x 0 ∼ π , Assumption 1-4 and 6 and the same setting as Proposition E.2,

Proof. Let us consider the coupled processes defined as equation (E.3). Suppose that the initial iterate ( x 0 , θ [2] 0 ) follows the stationary distribution ¯ ν , thus L ( x k , θ [2] k ) = ¯ ν and L ( θ [2] k ) = ν for all k ≥ 0 . By equation (E.4), we have for all k ≥ τ :

where we make use of the derivation in (D.13) to obtain the last inequality.

We note that

the last inequality holds for ατ ≤ min( µ 908 L 2 , κ µ L 2 ) . Therefore, we prove the desired inequality

Step 2: Arbitrary Initialization for ( x 0 , θ 0 ) . In this step, we remove the assumption of x 0 ∼ π needed in the previous step. We need the following lemma to prove our result.

Lemma E.5. Consider two trajectories ( x k , θ k ) k ≥ 0 and ( x ′ k , θ ′ k ) k ≥ 0 . Suppose that θ 0 = θ ′ 0 , x ′ 0 ∼ π and x 0 is initialized from some arbitrary distribution that satisfies ∥L ( x 0 ) -π ∥ TV = ϵ. Then for k ≥ τ, we have

Proof. We consider the following coupling between two joint processes ( x k , θ k ) k ≥ 0 and ( x ′ k , θ ′ k ) k ≥ 0 . We first apply the maximal coupling on x 0 and x ′ 0 such that

̸

For the case x 0 = x ′ 0 , we can couple the two Markov chains { x k } k ≥ 0 and { x ′ k } k ≥ 0 such that

Under this coupling, we have θ k ≡ θ ′ k , ∀ k ≥ 0 .

̸

For the case x 0 = x ′ 0 , we let the two processes ( x k , θ k ) k ≥ 1 and ( x ′ k , θ ′ k ) k ≥ 1 evolve independently. Given the above coupling, we first observe that

̸

̸

̸

̸

The second equality holds since E [ ¯ W 2 ( L ( x k , θ k ) , L ( x k , θ ′ k )) | x 0 = x ′ 0 ] = 0 and P ( x 0 = x ′ 0 ) = ϵ. Next, we note the following upper bound of the Wasserstein distance,

̸

Making use of Proposition 4.2, we have

where the second inequality holds for θ 0 = θ ′ 0 by assumption.

Note that the above upper bound to the Wasserstein distance is independent of the choice of ( x 0 , x ′ 0 ) . Hence, we can conclude that

̸

We complete the proof of the lemma.

By Lemma E.5, we see that when x 0 is close to its stationary distribution π , θ k would not deviate too much from θ ′ k , as if it were initialized from the stationary distribution.

̸

Now we consider a joint process ( x k , θ k ) k ≥ 0 with arbitrary initialization. By the property of uniform ergodicity of ( x k ) k ≥ 0 , we know that ∥L ( x k ) -π ∥ TV ≤ Rr k . Choose time t 0 ≥ 0 . We construct a second Markov chain ( x ′ k , θ ′ k ) k ≥ t 0 with the following properties: (1) x ′ t 0 ∼ π and is maximally coupled to x k 0 , i.e., ∥L ( x t 0 ) -L ( x ′ t 0 ) ∥ TV = P ( x t 0 = x ′ t 0 ) and (2) θ ′ t 0 = θ k 0 . Under this construction, for k ≥ t 0 + τ , we have

where the last inequality follows from Lemma E.4 and Lemma E.5.

For each t with t 0 ≥ τ , set t 0 = t/ 2 . From the above inequality, we obtain that

¯

W

2

(

L

(

x

t

, θ

t

)

,

¯

ν

)

where s ( θ 0 , L, µ ) denote some constant that depends on the initialization of θ 0 , and problem primitives L, µ , but independent of stepsize α and iteration index t . Last inequality holds because µ ≤ 1 -r and α ≤ µ 908 L 2 ≤ 1 .

Therefore, as t →∞ , we obtain that ¯ W 2 ( L ( x k , θ k ) , ¯ ν ) → 0 , which implies that the Markov chain ( x k , θ k ) k ≥ 0 with arbitrary initailization converges to the same ¯ ν . As such, we have proved the desired weak convergence result without the assumption on x 0 ∼ π initialization.

Additionally, we obtain the following convergence rate. For any initialization ( x 0 , θ 0 ) ∈ X × R d , we have

E.2.1 Proof of Proposition E.2

First, we present the following lemma that is similar to [17, Lemma 2.3]. Lemma E.6. For any k < k satisfying α ( k -k ) ≤ 1 , the following six inequalities hold:

1 2 2 1 8 L

Proof. Consider the coupling given by equation (E.3), by Assumption 2 and 6, we have

Given k 1 < k 2 , for ∀ t ∈ [ k 1 , k 2 ] , since 1 + x ≤ e x for ∀ x ∈ R , we have

where (i) holds for e x ≤ 1 + 2 x ∀ x ∈ [0 , 1 2 ] and α ( k 2 -k 1 ) ≤ 1 8 L . Then, we have

Therefore, following α ( k 2 -k 1 ) ≤ 1 8 L , we have

Then, by rearranging the terms, we have

thereby we have proved the first two inequalities of Lemma E.6.

By Assumption 2 and 6 and [17, Lemma 2.3], we can prove the last four inequalities and we omit the details here.

Now, we are ready to prove Proposition E.2. We start with the following decomposition. By equation (E.3), we first have

For T 2 , by Assumption 2 and 6, we have

We denote ε ( · , x k ) := g ( · , x k ) -¯ g ( · ) to be the noise function. Then, by Assumption 2, we conclude that ε ( · , x k ) is 2 L -Lipschitz continuous. Therefore, T 1 can be rewritten as:

For E [ ⟨ θ [1] k -θ [2] k , ¯ g ( θ [1] k ) -¯ g ( θ [2] k ) ⟩ ] , by Assumption 3, we have

We assume ατ ≤ 1 8 L . For T 3 , by definition of mixing time τ , we obtain

where the last inequality holds by Lemma E.6. For T 4 , we obtain

Below, we bound term ♠ by two different Taylor expansions. One the one hand, there exist λ 1 , λ 2 ∈ [0 , 1] such that h k = λ 1 θ [1] k +(1 -λ 1 ) θ [2] k , h k -τ = λ 2 θ [1] k -τ +(1 -λ 2 ) θ [2] k -τ and

where we note that choosing the second iterates for triangle inequality in equation (E.7) and choosing θ [2] k for bounding θ [2] k -θ [2] k -τ in equation (E.8) are both symmetric, which implies that we can replace the θ [2] k in term A with arbitrary one in { θ [1] k -τ , θ [1] k , θ [2] k -τ , θ [2] k } .

On the other hand, there exist ¯ λ 1 , ¯ λ 2 ∈ [0 , 1] such that p k = ¯ λ 1 θ [1] k +(1 -¯ λ 1 ) θ [1] k -τ , q k = ¯ λ 2 θ [2] k + (1 -¯ λ 2 ) θ [2] k -τ and

where adding and subtracting the second iterates in equation (E.7) and choosing θ [2] k for bounding θ [2] k -θ [2] k -τ in equation (E.8) are both symmetric. We have

Therefore, we obtain

and we can replace the θ [2] k in term B and C with arbitrary one in { θ [1] k -τ , θ [1] k , θ [2] k -τ , θ [2] k } . By (Bound1) and (Bound2) , we have

Below, we discuss the upper bound for |♠| by three cases.

Case 1: If one of { θ [1] k -τ , θ [1] k , θ [2] k -τ , θ [2] k } has norm that is less than 4 δ ατL 2 , where we define δ ατL 2 := δ ( ατL 2 ) , without loss of generality, we assume ∥ θ [2] k ∥ ≤ 4 δ ατL 2 . Multiply ✶ ( ∥ θ [2] k ∥ ≤ 4 δ ατL 2 ) to both sides of the above inequality, and we have

where we can replace the θ [2] k in the left hand of the above inequality with an arbitrary one in { θ [1] k -τ , θ [1] k , θ [2] k -τ , θ [2] k } by the similar argument under (Bound1) and (Bound2).

Therefore, by definition of ♠ and Lemma E.6, we obtain

Case 3: If all four variables in { θ [1] k -τ , θ [1] k , θ [2] k -τ , θ [2] k } have a norm larger than 4 δ ατL 2 , and ∥ θ [1] k ∥ ≥ 2 ∥ θ [1] k -θ [2] k ∥ , we obtain

where (i) holds by choosing ατ ≤ 1 16 L . Therefore, we have ∥ h k ∥ ≥ δ ατL 2 and ∥ h k -τ ∥ ≥ δ ατL 2 . Therefore, by equation (E.6) and Assumption 6, we obtain

Therefore, we obtain

By Assumption 6, there exists κ µ > 0 such that ϵδ ( ϵ ) ≤ µ 3072 for ∀ ϵ ≤ κ µ . By the above bounds, when ατ ≤ min( cµ L 2 , κ µ L 2 ) , where we specify the constant c < 1 later, we obtain

where the last inequality holds by choosing c = 1 4696 .

Therefore, for k ≥ τ and ατ ≤ min( µ 906 L 2 , κ µ L 2 ) we have

E.3 Proof of Projected SA

Now, we specialize the proof of Theorem E.1 to the projected SA iterates.

We consider the same coupling:

We first need to verify that Proposition E.2 holds for projected SA. By the non-expansion property of Π B ( β ) with respect to ∥ · ∥ , we obtain the following inequality:

which implies Lemma E.6 still holds for projected SA. For Proposition E.2, we can notice that the iterates of projected SA will always satisfy Case 1 with a finite bound β and we do not need to discuss Cases 2 and 3.

Therefore, when ατ ≤ min( cµ L 2 , 1 8 L ) , where we specify the constant c < 1 later, we obtain

where we set c = 1 940+96 β .

Therefore, ∀ k ≥ τ and ατ ≤ µ (940+96 β ) L 2 , we have

Then, still by the non-expansion of Π B ( β ) with respect to ∥ · ∥ , the rest of the proof simply follows the same proof for non-projected SA. As such, we have proven Theorem 4.1.

F Proof of Corollary 4.4

In this section, we present the proof of Corollary 4.4.

Recall that by Theorem 4.3, we obtain for k ≥ 2 τ ,

By [63, Theorem 4.1], there exists a coupling between θ k and θ ∞ such that

Applying Jensen's inequality twice, we obtain that

We thus have for all k ≥ 2 τ ,

For the second moment, we first note that

where we apply Cauchy-Schwarz to obtain the last inequality.

Meanwhile, we have

Substituting the above bounds into the right-hand side of inequality (F.1) yields

G Proof of Corollary 4.5

In this section, we prove the CLT result.

Proof. Consider the following centered test function ¯ h : X × R d → R d defined as

To prove that the CLT for function ¯ h , we need to verify the Maxwell-Woodroofe condition [49], i.e.,

where Q denotes the transition kernel of the joint Markov chain. If we can show the following

with r ∈ [0 , 1 / 2) , then the Maxwell-Woodroofe condition is verified, as

We now proceed to prove the desired order in (G.1). For sufficiently large n ≥ 2 τ α , we observe

We now show that both terms T 1 and T 2 are of order O (1) with respect to the parameter n .

For T 1 , since Q is a transition kernel, so its ∥ Q ∥ L 2 ( µ ) operator norm equals to 1 . Hence, T 1 can be upper bounded as

where the last inequality follows from Tr(Var( θ ∞ )) ≤ ατ established in (H.4) and ατ 2 α → 0 by Definition 2.1.

Before proceeding to analyze the summation in T 2 , we first recall (E.2), that for t ≥ 2 τ α ,

which holds for any ( x, θ ) ∈ X × R d . Hence, by the property of Wasserstein distance [63], there always exists a coupling that attains the optimality, i.e.,

̸

Making use of this relationship, we can therefore bound T 2 ,

where the last O ( · ) is asymptotic in n .

Combining the analysis of T 1 and T 2 , we have shown the desired order in (G.1). Therefore, the Maxwell-Woodroofe condition has been verified and we establish the CLT for averaged nonlinear iterates with constant stepsize and Markovian data.

H Proofs under Minorization Condition

When assuming the perturbed continuous noise condition in Assumption 5, one takes the alternative route to prove weak convergence. This is achieved by establishing the satisfaction of both a minorization condition and a drift condition. In this section, we prove the weak convergence result in Theorem 4.3 by following this alternative approach. The subsequent corollaries of weak convergence, namely the non-asymptotic convergence rate in Corollary 4.4 and the Central Limit Theorem (CLT) in Corollary 4.5, also hold, and we will provide the proofs for these results as well.

H.1 Proof of Theorem 4.3

In this section, we prove the weak convergence under Assumption 5(a). The proof consists of two major steps. Firstly, built upon the MSE convergence established in Proposition 4.2, we derive a multi-state drift condition. Subsequently, we show that under the minorization condition, the Markov chain is ( x k , θ k ) k ≥ 0 is φ -irreducible. Then, follow [50, Theorem 19.1.3], we can conclude that the Markov chain ( x k , θ k ) k ≥ 0 is geometrically ergodic.

For completeness, we include the Theorem 19.1.3 from [50] below.

Theorem H.1. Suppose that Φ is a φ -irreducible chain on X , and let n ( x ) be a measurable function from X → Z + . The chain is geometrically ergodic if it is aperiodic and there exists some petite set C , a nonnegative function V ≥ 1 and bounded on C , and positive constants λ < 1 and b satisfying

We note that the function n ( x ) can be interpreted as the number of steps we must wait, starting from any x , for the drift to become negative.

Step 1: Deriving the Drift Condition Given the iteration step

we have already shown the following convergence rate on the MSE in Proposition 4.2, that

From (H.3), we derive that

Q k V ( x 0 , θ 0 ) -V ( x 0 , θ 0 ) ≤ -βV ( x 0 , θ 0 ) + m ✶ ¯ C ( x 0 , θ 0 ) , where ¯ C = X × C Θ . Rewriting the above Lyapunov drift condition, with b = m/ (1 -β ) , we obtain

Setting λ such that λ k = 1 -β , we have

which gives the desired multi-step drift condition.

Step 2: Proving the Minorization Condition Now that we have established the desired multi-step drift condition, it remains for us to show that ¯ C is accessible, small, and aperiodic.

Under this setup of ξ t ( θ ) in Assumption 5(a), it is straightforward to verify the accessibility of ¯ C . For any ( x, θ ) ∈ X × R d , we have

As such, we have shown that ¯ C is accessible.

Assuming ¯ C is small, we can directly conclude aperiodicity following the definition of period of an accessible small set, d ( C ) = g.c.d. { n ∈ N ∗ : inf x ∈ C P n ( x, C ) > 0 } .

Therefore, what remains to show is that ¯ C is small. For ( x, θ ) ∈ ¯ C and ¯ A ∈ B ( X ) ×B ( R d ) , we define the following projection sets,

Therefore,

]

Inspired by the MSE convergence bound, we define the Lyapunov function V : X × R d → [1 , ∞

Therefore, the major goal in this step is to obtain the desired drift condition as shown in (H.1).

Therefore, from the above MSE convergence rate, we first obtain that

Consider k = min { t ≥ 0 : c 2 , 1 (1 -αµ ) t < 1 } . Then, set η = 8(1 -αµ ) k , β = 1 -η 2 , and m = c 2 , 2 L 2 µ ατ +1 , and we consider the following bounded sublevel set,

Next, for ( x, θ ) ∈ ¯ C , we observe that

We next recall the linear growth assumption in Assumption 2 that ∥ g ( x, θ ) ∥ ≤ L ( ∥ θ ∥ +1) . Hence, given θ ∈ C Θ , ∀ x ∈ X , we have

for some bounded value B .

We now define the measure ς † on R d as ς † ( A ) = 1 α d inf ∥ z ∥≤ B ∫ θ ′ ∈ A ∩ C Θ inf ˆ θ ∈ C Θ p ˆ θ ( θ ′ -z α ) d θ ′ . Hence, it is easy to see that ς † ( C c Θ ) = 0 . Moreover, we note the following property of measure ς † . Claim 1. For A ⊆ C Θ and λ ( A ) > 0 , ς † ( A ) > 0 , where λ denotes the Lebesgue measure.

We delay the proof to the end. Taking the claim as true, we derive that

Therefore, by combining all the analyses, we obtain

where ζ = ς † ( C Θ ) m -1 and ϕ × ς † being the unique induced product measure on X × R d .

As such, we have proven that ¯ C is ( m,ϕ × ς † ) -small, and hence ( δ m , ϕ × ς † ) -petite, and subsequently shown that ( x k , θ k ) k ≥ 0 is geometrically ergodic.

By [50, Theorem 16.0.1 (iv)], we can further conclude that the geometrically ergodic ( θ t , x t ) t ≥ 0 is also V -uniformly ergodic with the same V as defined in (H.2). Therefore, we have the following convergence rate in the V -norm ∥ ∥ L ( x k , θ k ) , -¯ ν α ∥ ∥ V ≤ κρ k , where κ and ρ implicitly depend on the stepsize α .

Lastly, we provide the proof of Claim 1.

Proof. We first recall that for any set B ⊆ R d such that λ ( B ) > 0 , where λ refers to the Lebesgue measure, then for θ ∈ C Θ , we know that ∫ t ∈ B p θ ( t ) d t ≥ ∫ t ∈ B inf θ ∈ C Θ p θ ( t ) d t > 0 . Moreover, for a given (translation) z ∈ R d and θ ∈ C Θ , we have

Following the properties stated above, we define h ( z ) = ∫ t ∈ B inf θ ∈ C p θ ( t -z ) d t, and it is easy to verify that h ( z ) is a continuous function. Hence, for a bounded set D , we have inf z ∈ D h ( z ) > 0 . Subsequently, we define the measure ς † induced by h and we verify that

H.2 Proof of Corollary 4.4

As shown in the previous section, the joint process ( x k , θ k ) k ≥ 0 is V -uniformly ergodic and hence also exhibits a geometric non-asymptotic convergence rate under the V -weighted norm. Subsequently, this corresponds to a version of Corollary 4.4 resulting in a different set of convergence rate coefficients. Thus, we restate Corollary 4.4 in the context of the minorization setting and provide the proof below.

Corollary H.2 (Non-Asymptotic Convergence Rate) . For any initialization of θ 0 ∈ R d , under the setting of Theorem 4.3, we have

∥ ∥ E [ θ k ] -E [ θ ( α ) ∞ ] ∥ ∥ ≤ κ · ρ k · s ′ ( θ 0 , L, µ ) , and ∥ ∥ E [ θ k θ ⊤ k ] -E [ θ ( α ) ∞ ( θ ( α ) ∞ ) ⊤ ] ∥ ∥ ≤ κ · ρ k · s ′′ ( θ 0 , L, µ ) , where κ and ρ are defined in (4.1) and implicitly depend on α .

Proof. For V -uniformly ergodic Markov chain ( x k , θ k ) k ≥ 0 , when functions f : X × R d → R d is dominated by the Lyapunov function, i.e., ∥ f ∥ ≤ V , it enjoys the following convergence property,

Consider test function f ( θ, x ) = θ -θ ∗ . It is easy to see that ∥ f ∥ ≤ V . Hence, we obtain

Next, consider f ′ ( x ) = ( θ -θ ∗ )( θ -θ ∗ ) ⊤ . Clearly, ∥ f ′ ∥ ≤ V . Therefore,

For the LHS, we have

Subsequently,

The above results imply the convergence of the first two moments. Moreover, we conclude that

Additionally,

H.3 Proof of Corollary 4.5

After establishing the V -uniform ergodicity of the joint process ( x k , θ k ) k ≥ 0 , the central limit theorem for averaged iterates follows as a straightforward consequence.

Proof. For any test function h : X × R d → R d that satisfies ∥ h ∥ 2 ≤ V , by Theorem 17.0.1 in [50], it has the following CLT results,

Therefore, consider h ( x, θ ) = θ -θ ∗ , it is easy to see that ∥ h ∥ 2 ≤ V , and hence we naturally obtain the desired CLT result, 1 √ k [ ∑ k -1 t =0 ( θ t -E [ θ ∞ ] ) ] ⇒N (0 , Σ ( a ) ) as k →∞ .

I Proof of Theorem 4.6

We now provide the proof of Theorem 4.6 on characterizing the asymptotic bias of nonlinear SA.

I.1 BAR and Preliminaries

The proof utilizes the basic adjoint relationship (BAR) approach to study the stationary distribution

via carefully designed test functions h . We refer readers to [33] for the derivation of the following properties of Markovian SA at stationarity,

Following the Borel state space assumption in 1, E [ ✶ { θ ∞ ∈ ·}| x ∞ = x ] induce a regular conditional probability measure, which we denote as ˜ ν ( · , x ∞ = x ) , and hence (I.1) can be reformulated as ˜ ν ( θ ∞ ∈ S, x ∞ = x ) = ˜ ν ( θ ∞ +1 ∈ S, x ∞ +1 = x ) .

Before proceeding to the proof, we introduce the following shorthands and notations. For x ∈ X ,

Following the differentiability assumption of g in Assumption 3, and we can apply Taylor expansion to g and we note the following notation on residuals.

By Assumption 3 and results from Proposition 4.2 and 4.2, we note that the residual R n ( θ, x ) satisfies

Hence, we have

Lastly, we denote

We are now ready to present our proof. The proof consists of two major steps. For the complexity of this problem, we first set aside the projection constraint and focus on the BAR analysis, and we shall present the asymptotic bias characterization without the projection step. The analysis in this step thus shall work for the minorization proof technique as well. Then, in the second step, we elaborate on the impact brought along by the projection analysis and conclude our proof.

I.2 Step 1: Bias Characterization without Projection

I.2.1 Step 1: First Moment Analysis

Consider test function h 1 ( x, θ ) = θ -θ ∗ . Therefore, we first have

which immediately implies that

Substituting the Taylor expansion (I.4) back into (I.6), we have

where we make use of E [ g ( θ ∗ , x )] = ¯ g ( θ ∗ ) = 0 by definition and the order or E [ R 3 ( θ ∞ , x ∞ )] = O (( ατ ) 3 / 2 ) to obtain the second equality.

Next, we proceed to analyze the two terms in (I.7). For the first term, we have

We now move on to analyze the second term, and obtain

Hence, substituting (I.8) and (I.9) back into (I.7), we reorganize the terms and arrive at

To obtain a refined characterization of the asymptotic bias, we carefully analyze the remaining three terms in (I.10)-(I.11). We focus on each term in the next three sections respectively.

I.2.2 Step 2: Second Moment Analysis

We start with analyzing E [( θ ∞ -θ ∗ ) ⊗ 2 ] in this section.

Following the BAR approach, we consider the test function h 2 ( x, θ ) = ( θ -θ ∗ ) ⊗ 2 and obtain

Simplifying the above expression, we have

We adopt a similar approach in analyzing the above relationship that contains g ( θ ∞ , x ∞ ) as in the previous step. We make use of the Taylor expansion of g at θ ∗ but at a lower order. We substitute the Taylor expansion (I.5) into (I.12) and obtain

Therefore, we proceed to analyze the terms in (I.13)-(I.16). Starting with the terms in (I.13), we have

Similarly,

Next, for the terms in (I.14), we have

Similarly,

Moving on to the second term in (I.15)

Last, for the terms in (I.16)

Similarly,

Lastly, by a second order Taylor expansion around θ ∗ of C ( θ ∞ ) = E [( ξ ∞ +1 ( θ ∞ )) ⊗ 2 ] , we have

where R ′ 2 ( θ ) satisfies sup x ∈ R d { ∥ R ′ 2 ( θ ) ∥ / ( ∥ θ -θ ∗ ∥ 2 + ∥ θ -θ ∗ ∥ k ϵ +2 ) } < ∞ .

Substituting the above analyses of the terms and consolidating the terms, we obtain

By far, we observe that the remaining terms all contain δ 1 and δ 2 . Therefore, we conclude our analysis of E [( θ ∞ -θ ∗ ) ⊗ 2 ] at this step and leave the analysis of δ 1 and δ 2 to the next section.

I.2.3 Step 3: Analysis of the δ -System

In this section, we analyze δ 1 and δ 2 .

Analysis of δ 1 . Starting with δ 1 , we first consider the following recursive relationship induced by (I.2).

where in (i) we make use of the update rule in (2.1), E [ ξ ∞ +1 ( θ ∞ )] = 0 and conditional independence x ∞ +1 ⊥ ⊥ θ ∞ | x ∞ . Next, we substitute the Taylor expansion (I.5) to obtain (ii).

Writing with notation shorthands z and δ , we have

If we apply π to both sides of (I.18), we obtain

Analyzing the three terms closely, we observe that

and hence we first obtain

We now subtract πz 1 on both sides of (I.18), and obtain

Consolidating the terms, we have

We next note the following properties,

Therefore, we can first conclude that ∥ δ ∥ L 2 ( π ) = O ( α ) .

Subsequently, from (I.19), we can derive that

Hence, together with the relationship between δ 1 and z 1 , we have

Lastly, we substitute (I.22) back into (I.20) and obtain

where

Therefore, for the terms that involve δ 1 , we can conclude that

where M and v are independent of α .

Note that (I.23) together with (I.21) implies that

Analysis of δ 2 . For z 2 , we first note that

and hence this implies that

Next, following (I.3), we obtain the following recursive relationship.

where in (i) we make use of the update rule in (2.1) and conditional independence x ∞ +1 ⊥ ⊥ θ ∞ | x ∞ . Next, we substitute the Taylor expansion (I.5) to obtain (ii).

Writing with z 2 shorthand, we have

Making use of the relationship ( P ∗ -Π) z 2 = ( P ∗ -Π) δ 2 , we have

Hence,

To analyze the above system of δ 2 , we make use of the Taylor expansion of g ( θ ∞ , s ) and ξ ∞ +1 ( θ ∞ ) . Starting with the first term, we have

Similarly, we have the following relationship for the second term,

Next, we proceed to analyze the third term.

Lastly, for the noise term, we derive that

Leveraging on the respective orders, we can conclude that

Therefore, for second-moment cross-terms, we have the following orders

I.2.4 Step 4: Bias Characterization

Finally, we are ready to consolidate the above analyses and conclude the characterization of the asymptotic bias.

We recall that we have already shown the following expansion of the asymptotic bias

By our analyses above, we have shown that

Hence, we derive that

For the second term, we simply use the order shown in (I.24).

Lastly, we substitute our analyses of δ 1 and δ 2 into the expansion of MSE (I.17) and derive that

Therefore, combining all the analyses, we have shown the bias characterization in Theorem 4.6,

where

Therefore, we see that assuming weak convergence without projection, the bias admits a leading term of order α . We emphasize that the expansion holds as equality, rather than an upper bound.

I.3 Step 2: Impact of Projection on Bias

Now, we proceed to analyze the impact of having the additional projection step on the asymptotic bias characterization. In the following, we use the shorthand θ t +1 / 2 to denote the iterate we obtain before the projection step, i.e.,

Therefore, we see that our analysis from Step 1 can be understood as the analysis for θ ∞ +1 / 2 .

Starting with the first moment analysis with test function h 1 ( x, θ ) = θ -θ ∗ , we have

which implies that

Therefore, we turn our focus to analyzing E [ θ ∞ +1 -θ ∞ +1 / 2 ] .

✶ where we note that when ∥ θ t +1 / 2 -θ ∗ ∥ ≤ β implies that ∥ θ t +1 / 2 ∥ ≤ β + ∥ θ ∗ ∥ ≤ 2 β and hence θ ∞ +1 = θ t +1 / 2 in this case. To analyze the remaining term, we use Hölder's inequality and obtain

Setting p = 6 and q = 6 / 5 , and making use of the property that E [ ∥ θ ∞ +1 / 2 -θ ∗ ∥ 6 ] = O (( ατ ) 3 ) from Proposition 4.2, we have

Hence, we can conclude that

Substituting this order information back into (I.25), we can see that E [ θ ∞ +1 -θ ∞ +1 / 2 ] /α = O ( α 2 τ 3 ) . Recall that τ = O (log(1 /α )) , and hence we can assimilate this residual order from projection into the existing O (( ατ ) 3 / 2 ) residual term.

For the remaining terms, we follow the existing analysis in Section I.2.1 and again obtain

Now, we proceed to analyze E [( θ ∞ -θ ∗ ) ⊗ 2 ] and examine the impact of projection. Consider test function h 2 ( x, θ ) = ( θ -θ ∗ ) ⊗ 2 and follow a similar strategy as the first moment analysis, we have

Hence, by reorganizing the terms, we have

Therefore, we turn our focus to analyzing E [( θ ∞ +1 -θ ∗ ) ⊗ 2 -( θ ∞ +1 / 2 -θ ∗ ) ⊗ 2 ] . Similar as the first moment analysis, we have

To analyze the term on the right-hand side, we again make use of Hölder's inequality and obtain

Setting p = 3 and q = 3 / 2 , and making use of the property that E [ ∥ θ ∞ +1 / 2 -θ ∗ ∥ 6 ] = O (( ατ ) 3 ) from Proposition 4.2, we have

Hence, we can conclude that

From the analyses above, we can also conclude that

Therefore, combining the analyses above, we see that the projection only introduces error terms of order O ( α 2 τ 3 ) . Hence, combining the analysis from Section I.2.1, we can conclude the same desired order that

J Additional Insights on TA and RR

In this section, we present more detailed results that characterize the first and second moment of Polayk-Ruppert (PR) tail-averaged iterates and Richardson-Romberg (RR) extrapolated iterates.

The following corollary provides non-asymptotic characterization for the first two moments of PR tail-averaged iterates ¯ θ k 0 ,k .

Corollary J.1 (Tail Averaging) . Under the setting of Theorem 4.6, the tail-averaged iterates satisfy the following bounds for all k > k 0 +2 τ and k 0 ≥ τ + 1 αµ log ( 1 ατ α ) :

With this result, taking the trace on both sides of (J.2) recovers Corollary 4.7.

Proof. First, we have

Hence, it follows that

Together with Theorem 4.6, we have

thereby finishing the proof of the first moment.

To bound the second moment of the tail-averaged iterate, we follow the proof technique in [33, Section A.6.2] . We notice that

For T 2 , we have

The term T 3 is similar to T 2 and obeys the same bound. For T 4 , we have

For T 1 , we have

By Corollary 4.4 and Proposition 4.2 we have

where we bound Var ( θ ∞ ) with O ( ατ α ) by Proposition 4.2 and Fatou's lemma.

Then, for (J.3), we have

We restate the following claim, whose proof closely resembles Claim 4 in [33].

Then, by [33, Claim 4], we have term (J.4) = O ( τ α k -k 0 ) . Similarly, we have term (J.5) = O ( τ α k -k 0 ) . Therefore, we have

By adding T 1 -T 4 together, we obtain

Next, we present the following corollary formalizes the non-asymptotic characterization for the first two moments of the RR-extrapolated iterate ˜ θ ( α ) k 0 ,k .

Corollary J.2 (Richardson-Romberg Extrapolation) . Under the setting of Theorem 4.6, the RR extrapolated iterates with stepsizes α and 2 α satisfy the following bounds for all k > k 0 +2 τ α and k 0 ≥ τ α + 1 αµ log ( 1 ατ α ) :

Proof. By equation (J.1), we obtain

With these notations, ˜ θ k 0 ,k -θ ∗ = 2 u 1 -u 2 + v . We then have the following bound

By equation (J.6), we have

Similarly, we have

By Theorem 4.6, we have ∥ v ∥ 2 2 = O ( ( ατ α ) 3 ) . Combining these bounds, we have

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The main results are elaborately discussed in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions for this paper are stated in Section 2 and Section 4, with a more detailed discussion provided in Appendix B. Complete proofs detailed in Appendices D-J.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiment details are fully described in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is publicly available at https://github.com/lucyhuodongyan/ nonlinear-sa-bias .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment details are fully described in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Given the experiment set-up, error bars are omitted for the convergence plots in Figure 1, because the focus is on the asymptotic bias, to which Polyak-Ruppert averaged iterates converge almost surely. As the bias approaches its limit, variability decreases, making error bars unnecessary.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computation resources are disclosed in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper is theoretical and has no potential negative social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies constant step-size stochastic approximation algorithms with Markovian noise. It is known that in this case, the error of a stochastic does not vanish asymptotically, even when consider averaging. This is \\theta_n has a bias.  Previous work have show how to study the bias of such algorithm for linear stochastic approximation. In this paper, the authors tackle the more challenging case of non-linear stochastic approximation. Most of the paper is devoted to proving that the bias is of order O(\\alpha). Some discussion on the applicability of results is presented.\n\nSTRENGTHS:\nThe paper proposes a characterization of the bias under the challenging setting of Markovian noise plus non-linear drifts.\n\nThe analysis is asymptotically tight as the step size \\alpha converges to 0, i.e., the expression for the bias is not a bound but an equality plus smaller order terms.\n\nThe implications of the results are discussed.\n\nWEAKNESSES:\nMost of the assumptions needed to obtain the results are relatively mild but two conditions are quite strong:\n1. strong monotonicity (A3)\n2. Smoothness (A2)\nI think that the paper should discuss in more details these assumption and highlight that they are really limiting the applicability of the result.\n\nThe assumption developed in part 4.2 to avoid assuming that the iterates are bounded seems quite strong, not verified in many practical cases (for instance any instance of stochastic gradient descent where the noise has a finite support would not satisfy this assumption), and very artificial (is the only purpose of this assumption to\n\nThe paper is extremely long (54 pages including proofs and references). To me, this says that there is either too much content or that the results are too diluted. As a result, the paper is very technical and hard to read. There should be more effort to make the paper more readable. Some suggestions:\n- consider a slightly less general setting\n- avoid considering sub-cases (like section 4.2) that are a bit orthogoal to the paper\n\nThe practical applications of the results are unclear. Some potential applications are presented in Section 4.5 / 4.6 but there are no experiments or simulations to confirm that this actually work (these sections are interesting, though).\n\nThere is a lot of papers on the subject. This can be seen as a good sign (this is an active area of research), but at the same time, it is hard for me to really assess the novelty of the results. Note that there are some (recent) papers that seem related to these work and that might be cited. This last point is not a weakness since some of them are extremely recent (available online after the submission deadline):\n- Computing the Bias of Constant-step Stochastic Approximation with Markovian Noise. S Allmeier, N Gast (2024)\n- Bias in Stochastic Approximation Cannot Be Eliminated With Averaging, Caio Kalil Lauand, Sean P. Meyn (2022)\n- Revisiting Step-Size Assumptions in Stochastic Approximation. Caio Kalil Lauand, Sean Meyn (2024)\nA comparison with these papers might be useful (even of this is not mandatory for the two 2024 papers).\n\nQUESTIONS:\nThere are some questions / comments in the limit that would deserve some comments from the authors.\n\nThe Markovian noise (x_k) is assumed exogeneous (it does not depend on \\theta). For some application (like Q-learning with a navigating policy derived from \\theta), this would not be satisfied. Could this assumption be lifted?\n\nThe next order term is a O(\\alpha^{3/2}): is it the sharpest bound or could O(\\alpha^2) be obtained?\n\nLIMITATIONS:\nThe limitation of the assumptions (see \"weaknesses\") should be better discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present paper obtains a representation for the asymptotic bias of constant step-size nonlinear stochastic approximation with Markovian noise. In particular, this characterization makes the hindering effect of memory and nonlinearity explicit in terms of the algorithm's performance. Moreover, the authors establish ergodicity of the parameter-noise joint process (with and without projection of estimates), obtain finite-time bounds on L_p moments of the estimation error and establish a central limit theorem for the constant gain algorithm. Finally, a bias attenuation technique based upon the Richardson-Romberg extrapolation is proposed for the nonlinear algorithm.\n\nSTRENGTHS:\nThe contributions and assumptions are clearly identified. To the best of the reviewer's knowledge, the results are novel and exciting: it is great to see the hindering effect caused by the interplay between memory and nonlinearity in SA.\n\nThis paper is well written, but could use some polishing. \n\nI did not have enough time to review all proofs in detail, but the analysis seems correct.\n\nWEAKNESSES:\nOne of the weaknesses of this paper is the fact that no numerical experiments are provided to illustrate any of the main results. Although a discussion on how the theory fits within Generalized Linear Models is given before the conclusions, I encourage the authors to include a simple toy example to illustrate some of their main results such as the CLT or the bias attenuation technique.\n\nQUESTIONS:\n-\tCould the authors clarify if assuming strong monotonicity and uniform boundedness of g and its Lipschitz constant over x are indeed needed for the result in Thm. 4.6? I can see their importance for finite-time bounds, but if there are needed for the asymptotic bias bound, I believe that the authors should mention the employment of stronger assumptions when comparing their work with previous research on asymptotic results.\n\n-\tI am not sure I understand what the authors mean by ``fine-grained’’  when talking about the result in Thm. 4.6. Upon further inspection, it seems that equation (4.2) is an extension of the result in [40] for nonlinear recursions: it consists a representation for the dominant bias term plus an upper bound as in this previous work. Is the fine-grained part related to the upper bound for \\alpha?\n\n-\tI might have missed this in the text, but could the authors clarify if the results in Sections 4.3 and 4.5 require projection?\n\nLIMITATIONS:\nThe authors addressed the limitations in their work through a clear list of assumptions and discussions after presenting the main results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper considers a nonlinear stochastic approximation (SA) problem with Markov noise (MC). It is assumed that the MC is uniformly geometrically ergodic. Instead of the standard iterative procedure, a projection onto a bounded set is additionally introduced (the latter can be relaxed under the additional assumption of the existence of a positive density). Under these assumptions the authors manage to write a decomposition that characterizes the bias. At the same time, they manage to identify three factors influencing the bias: the factor of MC, the factor of nonlinearity of the procedure, and the factor of interaction between MC and nonlinearity. In addition, bounds for the Polyak-Ruppert averaging and the Richardson Romberg procedure are given.

STRENGTHS:
- Decomposition that characterizes the bias with explicit dependence on MC, non-linearity and interactions between MC and non-linearity.

WEAKNESSES:
It would be good to obtain:
- high probability bounds instead of the MSE
- remove additional projection step and assumption on the density (it could be useful to consider convergence in the weighted W distance instead of V norm)
-  explicit dependence on the asymptotic variance of MC in the $O(\tau/(k - k_0))$.

QUESTIONS:
Could you please comment on the fact that there is no dependence between stepsize and number of iterates in the Polyak Ruppert avaraging?

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-17

comments: SUMMARY:
This paper investigates stochastic approximation (SA) with Markovian data and nonlinear updates under constant stepsize, and establishes the weak convergence of the joint process $(x_t, \theta_t)$. It also presents a precise characterization of the asymptotic bias of the SA iterates.

STRENGTHS:
I find this paper well-written. The analysis appears to be correct (although details are not checked). Also, both the literature review and motivation are very clear. It seems to me that this paper has solved a challenging problem, caused by Markovian data and nonlinear updates.

WEAKNESSES:
I find the presentation of this paper a bit technical for people that are not very familiar with this area. In addition, perhaps some numerical experiments should be performed to better illustrate the theory.

QUESTIONS:
1. About Assump. 3: for many GLMs, we actually do not have strong convexity. I feel this assumption is a bit strong. 
2. Page 4, line 147: is the notation superscript "\cross 2" defined anywhere? I assume this denotes the outer product of a vector. 
3. Similar to Assump. 3, Assump. 4 should be justified further as well.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-18