Title: Learning from A Single Markovian Trajectory: Optimality and Variance Reduction
Venue: NeurIPS 2025
Accepted: yes
Keywords: stochastic optimization

Markov sampling

sample complexity
Abstract: In this paper, we consider the general stochastic non-convex optimization problem when the sampling process follows a Markov chain. This problem exhibits its significance in capturing many real-world applications, ranging from asynchronous distributed learning to reinforcement learning. In particular, we consider the worst case where one has no prior knowledge and control of the Markov chain, meaning multiple trajectories cannot be simulated but only a single trajectory is available for algorithm design. We first provide algorithm-independent lower bounds with $\Omega(\epsilon^{-3})$ (and $\Omega(\epsilon^{-4})$) samples, when objectives are (mean-squared) smooth, for any first-order methods accessing bounded variance gradient oracles to achieve $\epsilon$-approximate critical solutions of original problems. Then, we propose \textbf{Ma}rkov-\textbf{C}hain \textbf{SPIDER} (MaC-SPIDER), which leverages variance-reduced techniques, to achieve a $\mathcal{O}(\epsilon^{-3})$ upper bound for mean-squared smooth objective functions. To the best of our knowledge, MaC-SPIDER is the first to achieve $\mathcal{O}(\epsilon^{-3})$ complexity when sampling from a single Markovian trajectory. And our proposed lower bound concludes its (near) optimality.

Learning from A Single Markovian Trajectory: Optimality and Variance Reduction

Abstract

In this paper, we consider the general stochastic non-convex optimization problem when the sampling process follows a Markov chain. This problem exhibits its significance in capturing many real-world applications, ranging from asynchronous distributed learning to reinforcement learning. In particular, we consider the worst case where one has no prior knowledge and control of the Markov chain, meaning multiple trajectories cannot be simulated but only a single trajectory is available for algorithm design. We first provide algorithm-independent lower bounds with Ω( ϵ -3 ) (and Ω( ϵ -4 ) ) samples, when objectives are (mean-squared) smooth, for any first-order methods accessing bounded variance gradient oracles to achieve ϵ -approximate critical solutions of original problems. Then, we propose Ma rkovC hain SPIDER (MaC-SPIDER), which leverages variance-reduced techniques, to achieve a O ( ϵ -3 ) upper bound for mean-squared smooth objective functions. To the best of our knowledge, MaC-SPIDER is the first to achieve O ( ϵ -3 ) complexity when sampling from a single Markovian trajectory. And our proposed lower bound concludes its (near) optimality.

1 Introduction

Many modern learning tasks can be modeled as stochastic optimization problems. Specific applications range from (un)supervised learning, reinforcement learning to large-scale generative models. In particular, the surge in deep learning and large models has made first-order methods-those that rely solely on gradient information-attractive due to their ease of implementation and computational efficiency [1, 40].

Yet, the commonly assumed i.i.d. data setting rarely reflects real-world scenarios. In many practical applications, data arrive sequentially and exhibit temporal dependencies, often shaped by underlying dynamical systems that can be modeled as Markov processes. For instance, in reinforcement learning, data are collected through interactions with an environment governed by a Markov decision process (MDP), producing inherently correlated samples [35]. In recommendation systems, user feedback unfolds over time, where each action depends on previous interactions in a Markovian fashion [2]. Time-series data-from sensors to financial markets-are also naturally Markovian [8, 13]. MCMCmethods, widely used in Bayesian inference, rely on Markov chains to sample from complex posteriors [9, 28]. Even in language modeling, transformer architectures produce token sequences with local dependencies that can be interpreted through a Markovian lens [1, 27].

These examples point to a clear fact: i.i.d. assumptions are sometimes insufficient, and optimizing in the presence of Markovian data requires more careful analysis. As a result, a growing body of work has emerged to study stochastic optimization under Markov sampling, spanning areas such as reinforcement learning [7, 39], distributed optimization [33, 15], and federated learning [34]. These

developments underscore the need for a more unified and general theoretical foundation that can account for the challenges introduced by Markovian dynamics.

Compared to the extensively studied i.i.d. setting, analyzing first-order optimization methods under Markov sampling introduces new and significant challenges due to data correlation. In the i.i.d. case, each data point is sampled independently from a fixed distribution, yielding unbiased estimates of the stochastic gradient. This independence enables the direct application of standard tools such as variance bounds and concentration inequalities [29, 22, 25]. However, when data are generated by a Markov process, consecutive samples are inherently correlated, complicating the statistical properties of gradient estimates. Specifically, the Markovian dependence introduces bias into the gradient estimates, which disrupts the foundational assumptions that are crucial for i.i.d.-based analysis. As a result, many classical techniques no longer apply directly, and alternative tools are required to handle the joint effects of bias and temporal correlation [7, 14, 31, 33, 21].

There are fruitful results about stochastic optimization with Markovian samples in literature, where a large amount focus on designing new algorithms and deriving corresponding sample complexity upper bounds. A common assumption placed by these works is that the algorithm is implemented in the simulated environment, meaning multiple trajectories can be incorporated to facilitate the performance of the algorithm. For example, in [36, 23] multiple trajectories are sampled from the underlying Markov chain for an update of policy in the RL problem. And variance-reduced techniques are incorporated by utilizing multiple trajectories to obtain better convergence rates and sample complexities [30, 26, 37, 38]. However, in practice one may have no access to multiple trajectories due to no prior knowledge and uncontrollability of the Markov chain. In such case, once the sampling begins, the chain keeps evolving and cannot be restarted, i.e., only a single trajectory is available for algorithm design. Based on this restriction, [14, 11] establish SGD-based approaches with batched samples drawn from a single trajectory and [6] further proposes the accelerating versions and generalizes them to variational inequalities. However, these methods suffer from slow convergence rates or high sample complexity, which is mainly because no variance reduction technique is considered as the i.i.d. counterpart. Moreover, when considering variance reduction for the Markov setting, biasness and time dependence make the analysis developed for the i.i.d. case fail to hold, which bring challenges in the derivation of new mathematical tools. Therefore, in this paper we are interested in understanding whether variance reduction can be achieved in the Markov sampling case when only one trajectory is available and aim to derive a new variance-reduced method to reduce sample complexity. Besides, we would also like to figure out what is the fundamental limit of our method and all existing methods. To achieve this, we further develop lower bound of sample complexity under both smooth and mean-squared smooth (which is commonly assumed for variance reduction applications) settings for any first-order algorithms, which nearly match the upper bound of our method. This concludes the near min-max optimality of our algorithm. Our main contributions are summarized as follows:

We provide the algorithm-independent sample complexity lower bound for any first-order methods of stochastic non-convex optimization problems, given data samples are generated by a Markov chain for smooth functions. Our lower bound shows a complexity with the order of ϵ -4 , which nearly matches the upper bound of algorithm MAG provided in [11, 6].

We also consider the case of mean-squared smooth functions, which shows broad applications in variance reduction for the i.i.d. case. We provide a lower bound of order ϵ -3 when samples are Markovian.

We then propose a new algorithm, called MaC-SPIDER. The convergence and sample complexity analysis is provided for MaC-SPIDER, which indicates nearly the same order of ϵ -3 , hence combining with our lower bound demonstrates its (near) min-max optimality.

Variance reduction for non-convex stochastic optimization. Variance reduction has drawn much attention very recently. It effectively controls the variance to a low level and hence improves sample efficiency of solving stochastic optimization problems. Mature variance-reduced techniques have been successfully applied to the i.i.d. case to reduce sample complexity. Popular methods include but not limited to SAG [32], SVRG [20, 4], SPIDER [16]. In particular, it is shown that SPIDER can achieve the optimal order of ϵ -3 sample complexity. However, in practice data might not be i.i.d. but Markovian. For the RL problem, variance-reduced methods have been developed by assuming multiple trajectories can be used for every iteration of the algorithm. Under this assumption,

incorporating with SVRG gives a sample complexity of O ( ϵ -10 / 3 ) [30, 37] and O ( ϵ -3 ) is achievable when combining with SPIDER [38].

Lower bounds for stochastic optimization. The sample complexity of first-order stochastic optimization has been a central topic of study over the past two decades. Among these developments, lower bounds play a fundamental role, as they characterize the inherent difficulty of optimization and set performance limits for all algorithms within a given class. In the full-batch setting, where the algorithm has access to all data, a lower bound of Ω( ϵ -2 ) has been established for smooth non-convex objectives [10], and this rate is achieved by standard Gradient Descent [18]. Under the i.i.d. sampling assumption, lower bounds of Ω( ϵ -2 ) for convex objectives [3] and Ω( ϵ -4 ) for general non-convex objectives [5] have been proven. The latter improves to Ω( ϵ -3 ) under a mean-square smoothness assumption [5]. Correspondingly, stochastic gradient descent (SGD) [17, 18] and SPIDER [16] achieves matching upper bounds in both settings, respectively. In the Markovian sampling regime, where data are temporally correlated, complexity results explicitly depend on the chain's mixing or hitting time, denoted by τ . For strongly convex functions, [6] establishes a lower bound of Ω( τ log( ϵ -1 )) , while for general convex objectives, [12] proves a lower bound of Ω( τϵ -2 ) , which matches the known upper bound for SGD in that setting. For non-convex objectives, [14] recently derived a loose lower bound of Ω( τϵ -1 ) , whereas the best-known upper bound remains ˜ O ( τϵ -4 ) .

2 Problem Formulation

Consider the general stochastic optimization problem,

where s ∈ S for S being the support, and π denotes some unknown underlying distribution. In this paper we focus on the Markovian case, i.e., we assume that the samples { s t } ∞ t =0 form a sequence generated by some underlying Markov chain with its stationary distribution being π . Moreover we focus on countable-state Markov chains, meaning the state spaces are countable but may not be finite. Note that the Markovian setting reduces to the i.i.d. setting by decoupling the dependence across time.

Since exactly solving (1) is NP-hard [19], by restricting to first-order methods, we search for an ϵ -approximate critical solution, which is widely adopted by literature [10, 5, 6] of F ( x ) defined in (1). In particular, given differentiable function F : R d → R , our goal is to find some x such that

for any ϵ > 0 .

2.1 Function class

Particularly, we consider all smooth functions in the following class:

where ∆ ≥ 0 and L > 0 are fixed parameters. The condition F (0) -inf x F ( x ) ≤ ∆ on F (0) can be generalized to any initial value F ( x 0 ) . However, for zero-respecting algorithms (to be defined in Section 2.2), we have x 0 = 0 . In particular, we consider the case where the objective F is smooth and has bounded initial gap to the optimum.

2.2 Algorithm class

Our algorithm class is inspired by [5]. We consider the following first-order algorithms such that:

the algorithm can access an unknown F ∈ F (∆ , L ) by a stochastic first-order oracle O ;

the oracle O returns a sequence of samples z := { s i } B i =1 ( B can be time-dependent) generated by a Markov chain and a mapping

where g ( x ; s ) := ∇ f ( x ; s ) is the stochastic gradient.

with

at iteration t , the algorithm queries a batch of M points 1

for each batch query x t , O responses with

where z t,i is the sequence of sample drawn for x t,i and z t := ⋃ M i =1 z t,i .

Then algorithm A consists of a sequence of measurable mappings {A t } ∞ t =0 to generate a sequence of iterates { x t } ∞ t =0 satisfying the following conditions:

the ( t +1) -th iterate is the output of A t when taking all previous oracle responses as input, i.e.,

Algorithm A is zero-respecting , i.e., for any O and samples z 0 , z 1 , . . . with any M , it satisfies for any t ≥ -1 and any m ∈ [ M ]

where g k,m ′ is the stochastic gradient for x A [ O F ] k,m ′ and support( x ) represents non-zero coordinates of x .

We denote A zr ( M ) the class of all zero-respecting algorithms. It is worth noting that for any A ∈ A zr ( M ) , x A [ O F ] 0 , 1 = 0 by definition.

We note that the above-mentioned algorithm class is general, which captures many existing first-order algorithms. For example, the vanilla MC-SGD [14]

corresponds to M = 1 , B = 1 , ∀ t ≥ 0 . For Randomized ExtraGradient [6], which maintains the update by

where by generating J t ∼ Geom (1 / 2)

one can clearly see that it fits in the case of M = 2 and B = 2 J t .

2.3 Markov-Chain Classes

In this section, we are interested in any sampling schemes characterized by finite-state Markov chains, i.e., |S| < ∞ . Before formally defining the classes of Markov chains, we present the definitions of hitting and mixing times which are two critical quantities characterizing the structure of a chain.

Definition 2.1 (Hitting time) . For any state w ∈ S , define

as the time Markov chain firstly reaches state w . The hitting time τ hit is defined by

Intuitively, the hitting time measures the maximal number of steps for which any pair of states take to transit between each other.

Definition 2.2 (Mixing time) . Consider Markov chain P . For any ϵ > 0 , define

where d TV ( P, Q ) represents the total variation distance between probability measures P and Q . We call τ mix the mixing time of P , where τ mix := t mix (1 / 4) .

The mixing time measures the convergence speed of a Markov chain to its stationary distribution. Then we consider the class of Markov chains for which the stationary distribution π exists and the hitting time τ hit is upper bounded by parameter τ ≥ 1 (scaled by some numerical constant). We denote the chain by P . Specifically,

where c 0 > 0 is some numerical constant; τ hit is defined in Definition 2.1; µP t represents the distribution of the chain after t -step transitions starting from the initial distribution µ .

2.4 Oracle Classes

Recalling that the oracle O returns a sequence of stochastic gradient evaluated at each query, we place the following assumption.

Assumption 2.3. There exists some 0 < σ 2 < ∞ , such that E s ∼ π ∥ g ( x ; s ) -∇ F ( x ) ∥ 2 ≤ σ 2 and E s ∼ π [ g ( x ; s )] = ∇ F ( x ) for any x .

Basically, Assumption 2.3 only limits the asymptotic behaviors of the stochastic gradient: 1) gradient estimate, i.e., E s ∼ π [ g ( x ; s )] = ∇ F ( x ) ; and 2) bounded variance, i.e., E s ∼ π ∥ g ( x ; s ) -∇ F ( x ) ∥ 2 ≤ σ 2 . Note that this assumption does not place any restriction when the chain has not yet converged to its stationary distribution. Moreover, it becomes aligned with the bounded variance assumption of stochastic first-order methods under i.i.d. sampling by further forcing independence across samples [18, 4].

Then, we are interested in two oracle classes. The first natural class, denoted by O ( σ 2 , τ ) , is that the stochastic gradient g is sampled from a chain contained in M ( τ ) by (4) and such that Assumption 2.3 is satisfied. This oracle class is considered in SGD-based analysis in literature [6].

The second class we consider is inspired by applications of variance reduction in the i.i.d. setting [16, 5], where besides bounded variance of stochastic gradients, an stronger requirement is placed. In particular, we assume that the stochastic gradient g satisfies the mean-squared smoothness:

Assumption 2.4. There exists ¯ L > 0 such that E s ∼ π ∥ g ( x ; s ) -g ( y ; s ) ∥ 2 ≤ ¯ L 2 ∥ x -y ∥ 2 , ∀ x, y ∈ R d .

It is straightforward to observe that mean-squared smoothness implies smoothness of F by observing ∥ E s ∼ π [ g ( x ; s ) -g ( y ; s )] ∥ 2 ≤ E s ∼ π ∥ g ( x ; s ) -g ( y ; s ) ∥ 2 ≤ ¯ L 2 ∥ x -y ∥ 2 . Then, in the second oracle class, denoted by O ( σ 2 , τ, ¯ L 2 ) , we force the stochastic gradient g is sampled from a chain in M ( τ ) and such that both Assumptions 2.3 and 2.4 are satisfied.

2.5 Sample Complexity Measures

Our results of lower bounds are established in terms of the sample complexity for finding an ϵ -approximate critical solution of F . Define P as the collection of all well-defined probability measures supported on S . Let S t ( A ) = ⋃ s ≤ t z t be the collection of all samples utilized til time t by algorithm A .

Concretely, the sample complexity measure for the smooth setting is defined by

Similarly, for the mean-squared smooth setting, the sample complexity measure is given by

When N ϵ ( M, ∆ , L, σ 2 , τ ) is lower bounded by N T , i.e., N ϵ ( M, ∆ , L, σ 2 , τ ) ≥ N T with N T denoting all collected samples up to time T , it indicates that there exists some stationary Markov sampling process P with bounded hitting time and an oracle O ∈ O ( σ 2 , τ ) such that for any A ∈ A zr ( M ) there exists F ∈ F (∆ , L ) for which E ∥∇ F ( x A [ O F ] T, 1 ) ∥ > ϵ , where the expectation is taken over randomness in A and O . In other words, at least N T number of samples must be required to (possibly) achieve an ϵ -approximate critical solution for any first-order algorithm (and similarly for the mean-squared smooth setting).

3 Algorithm-independent Lower Bounds

In this section, we show our main results on the lower bounds of sample complexity for stochastic non-convex optimization under Markov sampling. The result is algorithm-independent, implying that all first-order methods that are zero-respecting take at least such samples to reach an ϵ -approximate critical point of the non-convex objective function.

We show the following sample complexity lower bounds for any finite Markov sampling processes under both smooth and mean-squared smooth settings. Please refer to Appendices B and C for detailed proof.

Theorem 3.1. For the smooth setting, there exist numerical constants c 1 , c 2 > 0 such that for any M,L, ∆ , σ, τ > 0 ,

For the mean-squared smooth setting, we have

Remark 3.2 . Note that the extreme case τ = 1 corresponds to the i.i.d. sampling case. To see this, recalling the definition of hitting time τ = 1 indicates exactly one step is taken transiting from one state to any other, which then implies the samples are drawn exactly from the stationary distribution π and there is no time dependence across samples drawn at different time steps, hence reducing to i.i.d. case. Thus, when σ 2 ⪰ L ∆ our lower bound results are aligned with those provided in [5]. Moreover, noting that for the smooth setting the lower bound reduces to Ω ( τL ∆ ϵ 2 + τσ 2 L ∆ ϵ 4 ) if σ 2 ≿ L ∆ , it nearly matches the best-known upper bound in literature [6], which is ˜ O ( τ mix L ∆ ϵ 2 + τ mix σ 2 L ∆ ϵ 4 ) 2

4 Min-max Optimality for Mean-squared Smooth Functions

In this section, we propose Ma rkovC hain SPIDER (MaC-SPIDER) ,which is a variant of SPIDER [16] under Markov sampling, manifesting the (near) min-max optimality of sample complexity. In particular, we show that the sample complexity of MaC-SPIDER is O ( τ mix π -1 / 2 min ϵ -3 ) (with π min = min s π ( s ) ), which is comparable to our proposed lower bound Ω( τϵ -3 ) up to some constant gap (independent of ϵ ) between π -1 / 2 min τ mix and the hitting time τ . Proof for this section is deferred to Appendix D.

We present MaC-SPIDER in Algorithm 1, which exhibits effective variance reduction ability, hence improving sample complexity. Algorithmically, it is similar to SPIDER [16], while now the sampled

Algorithm 1 Ma rkovC hain SPIDER (MaC-SPIDER)

1: Input: initial point x 0 , N 0 = 0 , batch size M 1 and M 2 , integers T and r , stepsizes { η t } T -1 t =0

2: for t = 0 , 1 , . . . , T -1 do

3: if t mod r = 0 then

4: Draw M 1 samples and compute v t = 1 M 1 ∑ M 1 i =1 g ( x t ; s N t + i ) .

5: N t +1 = N t + M 1 .

6: else

7: Draw M 2 samples and compute v t = v t -1 + 1 M 2 ∑ M 2 i =1 ( g ( x t ; s N t + i -1 ) -g ( x t -1 ; s N t + i )) .

9: end if

10: Set the learning rate η t = min { 1 8 L , ϵ 4 L ∥ v t ∥ } .

11: x t +1 = x t -η t v t .

12: end for

13: Output: ˜ x T sampled uniformly from { x t } T -1 t =0 .

gradients are drawn from some unknown Markov chain rather than identically and independently. One crucial characteristic of MaC-SPIDER is that only one single trajectory generated by the chain is used for implementation. In other words, one needs not to care about when or where to restart the chain once the sampling process begins, but just keeps in mind how many samples should be drawn every iteration. This also enables its broader application in practice. For example, in literature [30, 26] multiple trajectories must be simulated before variance-reduced techniques to be applied, which costs extra resources and waiting time since every trajectory requires restart and then wait until enough collected samples. More problematically, multiple trajectories may even become unavailable in practical scenarios, when the chain is uncontrollable to simulate. MaC-SPIDER only uses a single trajectory with no further limitation and knowledge on the chain, which in this sense distinguishes its simplicity and applicability for practical implementation.

In particular, v t maintained in Algorithm 1 serves as an estimate of ∇ F ( x t ) that incorporates the history of sampled gradients for a better estimation with fewer samples to control the variance. This is intuitive and understandable when samples are i.i.d., as each sample is an independent and unbiased estimate of the true gradient. However, such unbiasedness no longer holds in our case, since Markovian samples are essentially biased and time-dependent, which makes it unknown whether or not variance can still be reduced effectively. Fortunately, by carefully analyzing the coupled correlation across different samples, we provide a positive answer to the controllability of the variance of v t , formally stated by Proposition 4.3.

We first present the following useful lemma, which generally provide the basic guidance on the variance control of Markov sampling.

Lemma 4.1. For a Markov chain { s t } ∞ t =0 with state space S and stationary distribution π , consider any function h : S → R d and define h π := E s ∼ π [ h ( s )] . Denoting E t ( · ) as the expectation conditioning on filtration F t and τ mix as the mixing time, the following holds:

(2). If E s ∼ π ∥ h ( s ) ∥ 2 ≤ B 2 , then for any s t

Remark 4.2 . Note that similar variance bounds as Part (1) are provided in [6, 11]. However, either of their analysis requires bounded gradient assumption, i.e., ∥∇ F ( x ) ∥ ≤ G < ∞ , ∀ x or bounded

noise assumption, i.e., ∥ g ( x ; s ) -∇ F ( x ) ∥ 2 ≤ σ 2 < ∞ , ∀ x, s , while ours do not depend on such restrictive assumptions.

Then leveraging Lemma 4.1 gives the following result, which indicates that the variance of v t can be well controlled to an arbitrary level.

Proposition 4.3. Considering Algorithm 1 and supposing Assumptions 2.3, 2.4 hold, by setting

we have for any ϵ ≤ 1

It is worth noting that Proposition 4.3 implies that to control the variance to a level of O ( ϵ 2 ) , the average number of samples per iteration that Algorithm 1 uses is O ( ϵ -1 ) , while simply using batched samples requires O ( ϵ -2 ) samples [11, 6]. Thus MaC-SPIDER successfully reduces the variance with fewer samples. This is the main reason why the sample complexity can be improved to O ( ϵ -3 ) . It is formally summarized by the following theorem.

Theorem 4.4. For Algorithm 1 with any ϵ ≤ 1 assuming Assumptions 2.3, 2.4 hold, setting

where ∆ = F ( x 0 ) -min x F ( x ) , we guarantee

Moreover, the total number of samples is O ( τ mix π -1 / 2 min ϵ -3 ) .

Remark 4.5 . Combining with the lower bound in Theorem 3.1, we note that MaC-SPIDER achieves almost the same order of ϵ -3 up to some gap (independent of ϵ ) between τ and π -1 / 2 min τ mix . This shows its near min-max optimality.

5 Proof Idea of the Lower Bounds

In this section we present the proof idea of how we obtain the sample complexity lower bounds for Markov sampling. We first clarify the proof sketch by focusing on the case where B = 1 , i.e., only one sample is drawn from the underlying Markov chain by the algorithm. Then, we generalize it to the case of B ≥ 1 which can also be time-dependent. Full proofs are presented in Appendices B and C.

The core technique inspired by [5] is to construct a "hard" function F with f ( · ; s ) supported on each state of a Markov chain lying in the required class such that the gradient norm, ∥∇ F ( x ) ∥ , is small only if each coordinate of x has a large enough absolute value. We use the progress function to mathematically evaluate the largest coordinate whose absolute value is larger than some nonnegative scalar α , i.e.,

where [ x ] k represents the k -th coordinate of x . We set prog α ( x ) = 0 if | [ x ] k | ≤ α, ∀ k ∈ [ d ] . Then the task of finding an ϵ -approximate critical solution is equivalently transformed to finding a solution x whose coordinate progress is high. Formally it is stated by the following lemma.

Lemma 5.1. There exists some F ∗ ∈ F ( O (∆ ϵ 2 d ) , L ) such that ∥∇ F ∗ ( x ) ∥ > ϵ, ∀ ϵ > 0 if prog 0 ( x ) < d .

Indicated by Lemma 5.1 ensuring ∥∇ F ∗ ( x ) ∥ ≤ ϵ requires all coordinates of x to be nonzero. Then for the smooth setting, we construct a chain with its hitting time upper bounded by τ and at least Ω( τ/q ) (with q ∈ (0 , 1) ) iterations are needed to make one increase in prog 0 ( x ) . It is stated by the following lemma.

Lemma 5.2. For any q ∈ (0 , 1) and any zero-respecting algorithm A ∈ A zr , there exist a Markov chain contained in M ( τ ) and some F ∗ ∈ F ( O (∆ dϵ 2 ) , L ) with g ∗ ( x ; s ) satisfying ∇ F ∗ ( x ) = E s ∼ π [ g ∗ ( x ; s )] and E s ∼ π ∥ g ∗ ( x ; s ) -∇ F ∗ ( x ) ∥ 2 ≤ O ( σ 2 ϵ 2 /q ) such that for any 0 < δ < 1 , with probability at least 1 -δ

In fact the constructive function F ∗ in Lemma 5.2 coincides with the one in Lemma 5.1, which then implies that at least Ω( τd/q ) iterations are needed to guarantee an ϵ -approximate critical solution output by any algorithm and hence Ω( τd/q ) samples (due to B = 1 ). Finally setting d = Ω( ϵ -2 ) and q = O ( ϵ 2 ) concludes the lower bound Ω( τϵ -4 ) shown by the smooth setting of Theorem 3.1.

For the mean-squared smooth setting, we have the following result.

Lemma 5.3. For any q ∈ (0 , 1) and any zero-respecting algorithm A ∈ A zr , there exist a Markov chain contained in M ( τ ) and some ¯ F ∗ ∈ F ( O (∆ dϵ ) , L ) with ¯ g ∗ ( x ; s ) satisfying ∇ ¯ F ∗ ( x ) = E s ∼ π [¯ g ∗ ( x ; s )] and E s ∼ π ∥ ¯ g ∗ ( x ; s ) -∇ ¯ F ∗ ( x ) ∥ 2 ≤ O ( σ 2 ϵ 2 /q ) such that for any 0 < δ < 1 , with probability at least 1 -δ

Thus, taking d = O ( ϵ -1 ) and q = O ( ϵ 2 ) recovers the bound for the mean-squared smooth setting in Theorem 3.1.

Note that the above proof derivations are established on the precondition when B = 1 by which we are able to directly obtain the sample complexity bounds through the iteration complexity analysis, since the iteration complexity is the same as the sample complexity. To generalize our results to B ≥ 1 , we present the following result.

Lemma 5.4. There exist a Markov chain in (4) and some functions F ∗ , g ∗ ( ¯ F ∗ , ¯ g ∗ ) satisfying corresponding conditions in Lemma 5.2 (or Lemma 5.3) such that for any zero-respecting algorithm A zr with B ≥ 1 , there is a zero-respecting algorithm A ∗ zr with B = 1 for which the following holds: for any t ≥ 0 if max m ∈ [ M ] max s ≤ t prog 0 ( x A ∗ [ O F ] s,m ) ≤ k , then max m ∈ [ M ] max s ≤ t prog 0 ( x A [ O F ] s,m ) ≤ k , ∀ 0 ≤ k ≤ d .

The above lemma indicates that we can always find an algorithm that only draws one sample per iteration to achieve no worse progress in its update than other algorithms that access multiple samples per iteration. In other words, combining with Lemmas 5.1 and 5.2 yields that accessing multiple samples every iteration has no benefit on improving the sample complexity for the algorithm, which hence implies the lower bounds that holds for B = 1 also holds for B ≥ 1 .

6 Limitation

The paper only considers the case of finite-state Markov chains, while in practice it is highly possible that the underlying Markovian sampling process admits infinitely many states. Besides, we only consider stationary Markov chains in the paper, which limits the generalization to the non-stationary setting. Therefore, extension to non-stationary Markov chains is an interesting direction and may need extra efforts.

7 Conclusion

In this paper, we study the first-order non-convex stochastic optimization problems. Unlike the conventional i.i.d. sampling, we focus on the case where data samples and stochastic gradient estimates are generated by an unknown Markov chain, which introduces additional data correlation and hence non-trivial analysis difficulties. Due to the lack of sample complexity lower bound results and the gap to the best-known upper bound under the Markovian setting, we provide an improved complexity lower bound with the order of ϵ -4 for general smooth functions, which nearly matches the best-known upper bound. We also consider the mean-squared smooth setting, which exhibits broad applications in variance reduction literature. We prove that for the mean-squared smooth setting, the sample complexity lower bound is the order of ϵ -3 for Markovian samples. Finally, we propose a new algorithm MaC-SPIDER, which to the best of our knowledge is the first variance-reduced method under a single Markovian trajectory, such that its sample complexity upper bound nearly matches our proposed lower bound, implying its near min-max optimality and the tightness of the lower bound.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

MMehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys , 55(7):1-38, 2022.

Alekh Agarwal, Martin J Wainwright, Peter Bartlett, and Pradeep Ravikumar. Informationtheoretic lower bounds on the oracle complexity of convex optimization. Advances in Neural Information Processing Systems , 22, 2009.

Zeyuan Allen-Zhu and Elad Hazan. Variance reduction for faster non-convex optimization. In International conference on machine learning , pages 699-707. PMLR, 2016.

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199(1):165-214, 2023.

Aleksandr Beznosikov, Sergey Samsonov, Marina Sheshukova, Alexander Gasnikov, Alexey Naumov, and Eric Moulines. First order methods with markovian noise: from acceleration to variational inequalities. Advances in Neural Information Processing Systems , 36, 2024.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory , pages 16911692. PMLR, 2018.

George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control . John Wiley & Sons, 2015.

Stephen Brooks. Markov chain monte carlo method and its application. Journal of the royal statistical society: series D (the Statistician) , 47(1):69-100, 1998.

Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming , 184(1):71-120, 2020.

Ron Dorfman and Kfir Yehuda Levy. Adapting to mixing time in stochastic optimization with markovian data. In International Conference on Machine Learning , pages 5429-5446. PMLR, 2022.

John C Duchi, Alekh Agarwal, Mikael Johansson, and Michael I Jordan. Ergodic mirror descent. SIAM Journal on Optimization , 22(4):1549-1578, 2012.

Philippe Esling and Carlos Agon. Time-series data mining. ACM Computing Surveys (CSUR) , 45(1):1-34, 2012.

Mathieu Even. Stochastic gradient descent under markovian sampling schemes. In International Conference on Machine Learning , pages 9412-9439. PMLR, 2023.

Mathieu Even, Anastasia Koloskova, and Laurent Massoulié. Asynchronous sgd on graphs: a unified framework for asynchronous decentralized and federated optimization. In International Conference on Artificial Intelligence and Statistics , pages 64-72. PMLR, 2024.

Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal nonconvex optimization via stochastic path-integrated differential estimator. Advances in neural information processing systems , 31, 2018.

Dylan J Foster, Ayush Sekhari, Ohad Shamir, Nathan Srebro, Karthik Sridharan, and Blake Woodworth. The complexity of making the gradient small in stochastic convex optimization. In Conference on Learning Theory , pages 1319-1345. PMLR, 2019.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization , 23(4):2341-2368, 2013.

Christopher J Hillar and Lek-Heng Lim. Most tensor problems are np-hard. Journal of the ACM (JACM) , 60(6):1-39, 2013.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems , 26, 2013.

Kyurae Kim, Jisu Oh, Jacob Gardner, Adji Bousso Dieng, and Hongseok Kim. Markov chain score ascent: A unifying framework of variational inference with markovian gradients. Advances in Neural Information Processing Systems , 35:34802-34816, 2022.

Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming , 133(1):365-397, 2012.

Yuheng Lei, Yao Lyu, Guojian Zhan, Tao Zhang, Jiangtao Li, Jianyu Chen, Shengbo Eben Li, and Sifa Zheng. Zeroth-order actor-critic: An evolutionary framework for sequential decision problems. IEEE Transactions on Evolutionary Computation , 2025.

David A Levin and Yuval Peres. Markov chains and mixing times , volume 107. American Mathematical Soc., 2017.

Shaojie Li and Yong Liu. High probability guarantees for nonconvex stochastic gradient descent with heavy tails. In International Conference on Machine Learning , pages 12931-12963. PMLR, 2022.

Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variancereduced) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems , 33:7624-7636, 2020.

Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Martin Jaggi, Hyeji Kim, and Michael Gastpar. Attention with markov: A framework for principled analysis of transformers via markov chains. arXiv preprint arXiv:2402.04161 , 2024.

Christopher Nemeth and Paul Fearnhead. Stochastic gradient markov chain monte carlo. Journal of the American Statistical Association , 116(533):433-450, 2021.

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4):15741609, 2009.

Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. In International conference on machine learning , pages 4026-4035. PMLR, 2018.

Abhishek Roy, Krishnakumar Balasubramanian, and Saeed Ghadimi. Constrained stochastic nonconvex optimization with state-dependent markov data. Advances in neural information processing systems , 35:23256-23270, 2022.

Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162:83-112, 2017.

Tao Sun, Dongsheng Li, and Bao Wang. On the decentralized stochastic gradient descent with markov chain sampling. IEEE Transactions on Signal Processing , 2023.

Zhenyu Sun, Ziyang Zhang, Zheng Xu, Gauri Joshi, Pranay Sharma, and Ermin Wei. Debiasing federated learning with correlated client participation. arXiv preprint arXiv:2410.01209 , 2024.

Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.

Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International conference on machine learning , pages 2139-2148. PMLR, 2016.

Pan Xu, Felicia Gao, and Quanquan Gu. Sample efficient policy gradient methods with recursive variance reduction. arXiv preprint arXiv:1909.08610 , 2019.

Pan Xu, Felicia Gao, and Quanquan Gu. An improved convergence analysis of stochastic variance-reduced policy gradient. In Uncertainty in Artificial Intelligence , pages 541-551. PMLR, 2020.

Sheng Zhang, Zhe Zhang, and Siva Theja Maguluri. Finite sample analysis of average-reward td learning and q -learning. Advances in Neural Information Processing Systems , 34:1230-1242, 2021.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Both the abstract and introduction discuss the sample complexity lower bound and the new proposed algorithm MaC-SPIDER with near optimality, which reflects the contribution and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation of the paper is clearly discussed.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides complete proof for each theoretical result and clearly states the corresponding assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper is purely theoretical and has no empirical result.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper is purely theoretical and has no experiment.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper is purely theoretical and does not have experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper is purely theoretical and has no experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper is purely theoretical and has no experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors reviewed the NeurIPS Code of Ethics and affirm the paper conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper studies the theory of stochastic optimization and has no societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper studies the theory of stochastic optimization and has no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper is purely theoretical and does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper is purely theoretical and does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper studies the theory of stochastic optimization and does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper is purely theoretical and does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The paper provides the sample complexity theory of stochastic optimization and does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Constructions of the Markov Chain and "Hard" Function F

A.1 Construction of Markov chains

In this section, we construct a Markov chain that is used for the lower bound proofs. The idea is to construct a chain such that 1) there exist two states between which at least τ steps must be take to transit; 2) the hitting time of the constructed chain is upper bounded by τ . Without loss of generality we assume τ is even.

In particular, we consider a directed cyclic-like chain with self-loops. Denote s = i be the i -th state of the Markov chain for i ∈ { 0 , 1 , . . . , 2 τ ′ +1 } with τ ′ = τ/ 2 . Then for any q ∈ (0 , 1 / 2) the transition of the chain is defined as follows:

Then letting v ∗ 1 = 0 , v ∗ 2 = 1 , w ∗ 1 = τ ′ + 1 , w ∗ 2 = τ ′ + 2 , it is straightforward that the above constructed Markov chain guarantees that transitioning between v ∗ 1 and w ∗ 1 takes at least τ ′ = τ/ 2 steps. Moreover, the hitting time of the chain is O ( τ ) by noting the hitting time of directed cyclic chain with self-loops and n states is O ( n ) . We denote this chain by P ∗ , and hence P ∗ ∈ M ( τ ) .

A.2 Construction of function F

Now we construct a "hard" function that is difficult for any first-order algorithm to search for the critical point. Specifically we consider the following two functions

where and

with u ∈ R .

We denote π s as the corresponding probability of state s of the stationary distribution π . Then, given the Markov chain P ∗ constructed in Appendix A.1, we know that at least 1 2 τ steps are required to take transiting from v ∗ 1 to w ∗ 1 and vice versa. Then, we construct function F such that F ( x ) = π v ∗ h 1 ( x ) + π w ∗ h 2 ( x ) , where we denote v ∗ = { v ∗ 1 , v ∗ 2 } , w ∗ = { w ∗ 1 , w ∗ 2 } and π v ∗ = π v ∗ 1 + π v ∗ 2 , π w ∗ = π w ∗ 1 + π w ∗ 2 . For any x and i ≥ 0 define x ≤ i := ([ x ] 1 , . . . , [ x ] i , 0 , . . . , 0) as the truncated version by only keeping the first i coordinates. We also set x ≤ 0 = x . Then we have the following properties of F .

Lemma A.1. Let F ( x ) = π v ∗ h 1 ( x ) + π w ∗ h 2 ( x ) for h 1 , h 2 defined by (10) , (11) . Then we have the following:

(3). F ( x ) is l 1 -smooth for some constant l 1 > 0 .

(4). If prog 1 ( x ) < d , ∥∇ F ( x ) ∥ ≥ 1 .

(6). If prog 0 ( x ) is odd, prog 0 ( ∇ h 1 ( x )) ≤ prog 1 2 ( x ) , prog 0 ( ∇ h 2 ( x )) ≤ prog 1 2 ( x ) + 1 . If prog 0 ( x ) is even, prog 0 ( ∇ h 1 ( x )) ≤ prog 1 2 ( x ) + 1 , prog 0 ( ∇ h 2 ( x )) ≤ prog 1 2 ( x ) .

Proof. For Part (1), observing that F (0) < 0 and noting that 0 ≤ ψ ( u ) ≤ e , 0 ≤ ϕ ( u ) ≤ √ 2 πe ,

with ∆ 0 = e √ 2 πe , which completes its proof.

For Part (2), noting that 0 ≤ ψ ′ ( u ) ≤ √ 54 e -1 and 0 ≤ ϕ ′ ( u ) ≤ √ e , combining with the fact that for each i = 1 , 2

yields implying ∥∇ h i ( x ) ∥ ≤ 23 and ∥∇ h i ( x ) ∥ ≤ 23 √ d, ∀ i = 1 , 2 .

Parts (3) and (4) follow directly from [10]. Parts (5)-(7) follow from the observation that

B Lower Bound for the Smooth Setting

In this section, we show the lower bound of the smooth setting in Theorem 3.1. Based on the contructive F in the last section, we consider the following gradient oracle g : for each i -th coordinate of g

Recalling the definition of P ∗ , we note P ( s = v ∗ 1 | s ∈ { v ∗ 1 , v ∗ 2 } ) = P ( s = w ∗ 1 | s ∈ { w ∗ 1 , w ∗ 2 } ) = q ∈ (0 , 1 / 2) . Then, we have the following lemma.

Lemma B.1. Considering stochastic gradient g ( x ; s ) constructed as (12) , the following statements hold:

(1). For s ∈ { v ∗ 1 , v ∗ 2 , w ∗ 1 , w ∗ 2 } , with probability at least 1 -q , prog 0 ( g ( x ; s )) ≤ prog 1 2 ( x ) and g ( x ; s ) = g ( x ≤ prog 1 2 ( x ) ; s ) for all x .

(2). For s / ∈ { v ∗ 1 , v ∗ 2 , w ∗ 1 , w ∗ 2 } , with probability 1 , prog 0 ( g ( x ; s t )) ≤ prog 1 2 ( x ) and g ( x ; s t ) = g ( x ≤ prog 1 2 ( x ) ; s t ) for all x .

(3). For any s , with probability 1, prog 0 ( g ( x ; s )) ≤ 1 + prog 1 2 ( x ) and g ( x ; s ) = g ( x ≤ 1+ prog 1 2 ( x ) ; s ) for all x .

(4). E s ∼ π [ g ( x ; s )] = ∇ F ( x ) .

Proof. We firstly show Part (3). Note that by (12) and Part (7) of Lemma A.1, for any x, s , [ g ( x ; s t )] i = 0 , ∀ i > 1+ prog 1 2 ( x ) in the sense that [ ∇ h 1 ( x )] i = [ ∇ h 2 ( x )] i = 0 , ∀ i > 1+ prog 1 2 ( x ) , which implies prog 0 ( g ( x ; s )) ≤ 1 + prog 1 2 ( x ) . Moreover, by Part (7) of Lemma A.1, defining x ′ := x ≤ 1+ prog 1 2 ( x ) gives ∇ h 1 ( x ) = ∇ h 1 ( x ′ ) and ∇ h 2 ( x ) = ∇ h 2 ( x ′ ) . Thus, we obtain g ( x ; s ) = g ( x ′ ; s ) for any x, s , implying Part (3).

For Part (1), we note that when i ≥ 1 + prog 1 2 ( x ) and s ∈ { v ∗ 2 , w ∗ 2 } , g ( x ; s ) = [ ∇ h j ( x )] ≤ prog 1 2 ( x ) for j = 1 , 2 , which implies prog 0 ( g ( x ; s )) ≤ prog 1 2 ( x ) , ∀ s ∈ { v ∗ 2 , w ∗ 2 } . Further, according to (5) of Lemma A.1, we have g ( x ; s ) = g ( x ≤ prog 1 2 ( x ) ; s ) for s ∈ { v ∗ 2 , w ∗ 2 } and all x . Since P ( z = 0) = 1 -q , hence Part (1) is proved.

Part (2) holds trivially in the sense that g ( x ; s t ) = 0 when s / ∈ { v ∗ 1 , v ∗ 2 , w ∗ 1 , w ∗ 2 } . Finally, Part (4) holds since E [ ✶ s /q | s ∈ { v ∗ 1 , v ∗ 2 } ] = E [ ✶ s /q | s ∈ { w ∗ 1 , w ∗ 2 } ] = 1 . Also, we show in the following lemma that g has bounded variance.

Lemma B.2. For F ( x ) = π v ∗ h 1 ( x ) + π w ∗ h 2 ( x ) and g defined as (12) , then for any Markov chain with stationary distribution π , given any x ∈ R d ,

for some constant a 1 , a 2 > 0 .

Proof. By Part (4) of Lemma B.1, we know E s ∼ π [ g ( x ; s )] = ∇ F ( x ) .

Denote i ∗ = 1 + prog 1 2 ( x ) . For any s ∈ { v ∗ 1 , v ∗ 2 , w ∗ 1 , w ∗ 2 } , we have

✶ When i ∗ -1 is odd, from Part (6) of Lemma A.1 we know that [ ∇ h 1 ( x )] i ∗ = 0 . Therefore,

and

where we use (2) of Lemma A.1. Combining the above three inequalities, it yields that when i ∗ -1 is odd, for any Markov chain, any x , t ≥ 0 and any initial distribution of the chain,

where a 1 = 6 · 23 2 , a 2 = 3 · 23 2 and we use that E [( ✶ s /q -1) 2 | s ∈ { w ∗ 1 , w ∗ 2 } ] = (1 -q ) /q . The case when i ∗ -1 is even can be derived similarly.

Then, we are ready to show Lemmas 5.1 and 5.2. We first focus on the case B = 1 and then generalize it to B ≥ 1 .

Proof of Lemmas 5.1 and 5.2: Given any ϵ > 0 , we consider the following F ∗

And we consider the following gradient g ∗

with g ( x ; s ) defined as (12). Since ∇ F ( x ) = E s ∼ π [ g ( x ; s )] , ∇ F ∗ ( x ) = E s ∼ π [ g ∗ ( x ; s )] . We note that which implies that F ∗ is L -smooth by Part (3) of Lemma A.1. Moreover, by Part (1) of Lemma A.1 we obtain that

All the above concludes Lemma 5.1.

To see Lemma 5.2, note by Lemma B.2, we have

Then, define

✶ Note that under the construction of the Markov chain P ∗ and F ∗ and g ∗ , for any zero-respecting algorithm A

That is to say within every 1 2 τ iterations B t can be 1 at most once. And Part (1) of Lemma B.1 indicates that the probability of B t being 1 is no greater than q . Let k ( t ) := max m ∈ [ M ] max l ≤ t prog 0 ( x A [ O ] F ∗ l,m ) . Then, the above implies that

Also recalling the definition of P ∗ guarantees for any t in the ideal case the number of possible non-zero B l can be at most 2 t/τ with each being 1 with probability at most q , it implies

where z i denotes the Bernoulli random variable with succeeding probability being at most q . Note that z i s are independent in the sense that conditioning on the chain hits v ∗ = { v ∗ 1 , v ∗ 2 } and will hit w ∗ = { w ∗ 1 , w ∗ 2 } at exactly τ/ 2 steps later, whether w ∗ 1 or w ∗ 2 will be visited is independent of which of v ∗ 1 or v ∗ 2 has been visited. Thus

Therefore, we conclude that for any δ ∈ (0 , 1) and q ∈ (0 , 1 / 2) with probability at least 1 -δ ,

which completes the proof of Lemma 5.2.

Proof of Lemma 5.4: To see the part of the smooth setting of Lemma 5.4, note that for any algorithm A with B ≥ 1 , we simply observe that by the construction of F ∗ and g ∗ and the Markov chain P ∗ , for any t ≥ 0 and m ∈ [ M ] , there exists an algorithm ˜ A with B = 1 for which prog 0 ( x A [ O F ∗ ] t,m ) ≤ prog 0 ( x ˜ A [ O F ∗ ] t,m ) , since multiple samples do not contribute to additional progress of x , which then proves the part of smooth setting (Similar claims can be achieved for the mean-squared setting of Lemma 5.4).

Proof of the smooth setting of Theorem 3.1: Now to show the lower bound for the smooth setting of Theorem 3.1, setting

and yields that F ∗ ∈ F (∆ , L ) and Assumption 2.3 is satisfied. By Part (4) of Lemma A.1 and Lemma 5.2, choosing δ = 1 / 2 renders that for any m ∈ [ M ] with probability at least 1 / 2 ,

which implies that

Therefore, we conclude that

by the selections of d, q as (14),(15) for some constants c 1 , c 2 > 0 .

C Lower Bound for the Mean-squared Smooth Setting

In this section, we show the lower bound of the mean-squared smooth setting in Theorem 3.1. The idea is similar to the proof for the smooth setting, except that we replace the indicator function in (12) by its smoothed surrogate:

where Γ is defined by with

Then, we consider the following stochastic gradient ¯ g :

It is straightforward to see E s ∼ π [¯ g ( x ; s )] = ∇ F ( x ) . According to Observation 1 of [5], we know Θ i ( x ) = 0 , ∀ i ≤ prog 1 2 ( x ) and hence [¯ g ( x ; s )] i = [ ∇ h j ( x )] i , j = 1 , 2 , ∀ i ≤ prog 1 2 ( x ) when s ∈ { v ∗ 1 , v ∗ 2 , w ∗ 1 , w ∗ 2 } . Moreover, according to Part (6) of Lemma A.1, we have for any s ∈ S

Defining δ ( x ; s ) := ¯ g ( x ; s ) -∇ F ( x ) , it yields that there is eactly one non-zero coordinate of δ ( x ; s ) , which is the 1 + prog 1 2 ( x ) -th coordinate. Moreover, we have the following results for ¯ g .

Lemma C.1. Consider ¯ g defined as (17) . Then,

for some constants a 3 , a 4 > 0 . And for some constant ¯ l 1 > 0

Proof. Similar to the proof of Lemma B.2, we can easily obtain (18). To show (19), note that

where i ∗ x = 1 + prog 1 2 ( x ) , i ∗ y = 1 + prog 1 2 ( y ) . Since

and by Observation 1.3 of [5] | Θ i ( x ) -Θ i ( y ) | ≤ 36 ∥ x -y ∥ and noting | Θ i ( x ) | ≤ 1 , ∥∇ F ( x ) ∥ ∞ ≤ 23 , we obtain

Finally leveraging Part (3) of Lemma A.1 gives

Then we show the lower bound corresponding to the mean-squared smooth setting of Theorem 3.1.

Proof of the mean-squared smooth setting of Theorem 3.1 and Lemma 5.3: Noting that ¯ L -meansquared smoothness implies ¯ L smoothness, we thus consider the case L ≤ ¯ L with L to be determined later. Consider the same F ∗ as (13) and let ¯ g ∗ ( x ; s ) = ( Lλ/l 1 )¯ g ( x/λ ; s ) . Similarly, we have F ∗ is L -smooth. Also

and

Then taking which completes the proof.

D Convergence Analysis of MaC-SPIDER

In this section, we provide the proof for Section 4. We first present the following technical lemma. Lemma D.1. We have the following claims:

Proof. The first two claims are directly from [24].

To see the third claim, we note that

which completes the proof with d TV ( µ, π ) + 2 ≤ 3 .

D.1 Proof of Lemma 4.1 and Proposition 4.3

Proof of Lemma 4.1: Let ˜ h i t = h ( s t + i ) -h π and ˜ h ( s ) = h ( s ) -h π . We have

we guarantee that F ∗ ∈ F ( L, ∆) and Assumptions 2.3, 2.4 are satisfied. Similar to the proof of the smooth setting, we can easily obtain Lemma 5.3 and then conclude that

First, we show the following useful bound: for any s ∈ S , given t ≥ 0 and 1 ≤ i < j ,

Then, noting that for any 1 ≤ i < j ≤ M ,

where we use (20) and note ∑ s ′ π ( s ′ ) ⟨ ˜ h ( s ) , ˜ h ( s ′ ) ⟩ = ⟨ ˜ h ( s ) , ∑ s ′ π ( s ′ ) ˜ h ( s ′ ) ⟩ = ⟨ ˜ h ( s ) , E s ′ ∼ π [ ˜ h ( s ′ )] ⟩ = 0 . Further using the fact ∑ T t =1 max s d TV ( P t ( · | s ) , π ) ≤ 3 τ mix by Lemma D.1, we obtain

Similarly we note that

which then implies by Lemma D.1

Combining all above gives (7). To show (9), we note that

where we use ( a + b ) 2 ≤ 2 a 2 +2 b 2 and E [ X -E ( X )] 2 ≤ E [ X ] 2 We note that by replacing ˜ h ( s ) in (20) by h ( s ) ,

.

where we use E s ∼ π ∥ h ( s ) ∥ ≤ ( E s ∼ π ∥ h ( s ) ∥ 2 ) 1 / 2 ≤ B and this concludes (8). Moreover, similar to the analysis of E t ⟨ ˜ h i t , ˜ h j t ⟩ , we have

and hence

which concludes (9).

Proof of Proposition 4.3: We denote E t ( · ) as the expectation conditioning on filtration F t .

Note that for t mod r = 0 ,

by (7) of Lemma 4.1 and noting M 1 = 112 τ mix π -1 / 2 min ϵ -2 max { σ, σ 2 } .

̸

For t mod r = 0 , conditioning on F t +1 and letting ˜ g M 2 t +1 = 1 M 2 ∑ M 2 i =1 g ( x t +1 ; s N t +1 + i ) -g ( x t ; s N t +1 + i ) yields

Noting that E t +1 ,s ∼ π ∥ g ( x t +1 ; s ) -g ( x t ; s ) ∥ ≤ L ∥ x t +1 -x t ∥ , ∀ s ∈ S and E t +1 ,s ∼ π [ g ( x t +1 ; s ) -g ( x t ; s )] = ∇ F ( x t +1 ) -∇ F ( x t ) , combining with (8) and (9) of Lemma 4.1 gives

where B := L max t ∥ x t +1 -x t ∥ . Further noting

we have

where we use ϵ 2 4 ∥ v t - ∇ F ( x t ) ∥ ≤ ϵ 2 ∥ v t - ∇ F ( x t ) ∥ 2 + 1 2 ϵ ( ϵ 2 4 ) 2 in the second inequality. Then noting that for rt 0 ≤ t < r ( t 0 +1) given any t 0 ≥ 0 , we have for r = 1 /ϵ

where we use the fact (1 + ϵ/ 2) 1 /ϵ ≤ √ e ≤ 2 and E ∥ v rt 0 -∇ F ( x rt 0 ) ∥ 2 ≤ ϵ 2 / 8 .

D.2 Proof of Theorem 4.4

Noting ¯ L -mean-squared smoothness implies ¯ L -smoothness of F , we have

Noting that η t ≤ 1 2 ¯ L , then using the fact that min {| x | , x 2 / 2 } ≥ | x | -2 , ∀ x ∈ R ,

which hence induces

Taking expectation on both sides and using Lemma 4.3 gives

By T = 16 ¯ L ∆ 0 ϵ -2 we conclude

It is straightforward to see that the total number of samples is upper bounded by

which indicates

Peer reviews:
comments: SUMMARY:
This paper explores a general stochastic non-convex optimization problem, which lies outside the scope of my current research expertise and academic focus. As such, I do not feel adequately qualified to provide a fair and informed evaluation of the technical contributions and significance of the work. I respectfully request that my review be disregarded in the decision-making process, and that greater consideration be given to the assessments provided by reviewers with relevant expertise in this area.

STRENGTHS AND WEAKNESSES:
This paper explores a general stochastic non-convex optimization problem, which lies outside the scope of my current research expertise and academic focus. As such, I do not feel adequately qualified to provide a fair and informed evaluation of the technical contributions and significance of the work. I respectfully request that my review be disregarded in the decision-making process, and that greater consideration be given to the assessments provided by reviewers with relevant expertise in this area.

QUESTIONS:
This paper explores a general stochastic non-convex optimization problem, which lies outside the scope of my current research expertise and academic focus. As such, I do not feel adequately qualified to provide a fair and informed evaluation of the technical contributions and significance of the work. I respectfully request that my review be disregarded in the decision-making process, and that greater consideration be given to the assessments provided by reviewers with relevant expertise in this area.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-27

comments: SUMMARY:
This paper studies stochastic non-convex optimization with samples from a single uncontrolled Markov chain. It establishes an $\Omega(\epsilon^{-3})$ lower bound under the mean-squared smoothness assumption and an $\Omega(\epsilon^{-4})$ bound without it. The proposed MaC-SPIDER algorithm matches these bounds, achieving near-optimal sample complexity.

STRENGTHS AND WEAKNESSES:
Strengths:

The paper is well presented. It proposes a new algorithm based on the variance reduction (SPIDER) algorithm in optimization for the setting of learning from a single Markov chain. Lower bound analysis shows that the result is nearly minimax optimal.

Weaknesses:

(1) The abstract and the introduction are inconsistent: the lower bound in the abstract is $\epsilon^{-3}$, while that in the intro is $\epsilon^{-4}$.

(2) There are some typos in the sample complexity definition (equation 6, the left part should depend on both $\bar L$ and $L$). Moreover, it’s hard for me to understand the meaning of this metric. For example, it is not clear to me what the final infimum is taken over. Could you clarify more about its definition?

(3) The algorithm requires prior knowledge of some parameters like $\pi_{min}$ and $\tau_{\min}$, which is hard to know in advance.

QUESTIONS:
(1) In Remark 4.5, we see that there is still a gap between $\tau$ and $\pi_{min}^{1/2}\tau_{mix}$. Could you provide a concrete example to show this gap?

(2) Could you provide any empirical validation?

(3) What result does MaC-SPIDER have without the mean-squared smooth assumption? Can it also match the lower bound?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
This paper looks at optimizing non-convex functions when data comes from a single Markov chain trajectory—no resets, no i.i.d. samples. That makes things harder due to data correlation. The authors prove a lower bound showing you need at least Ω(ϵ⁻³) samples in the best case. Then they propose MaC-SPIDER, a smart variance-reduced method that hits this bound (up to constants), making it nearly optimal. It’s the first method to do this using just one trajectory. The work is purely theoretical and highlights that it doesn't cover infinite or non-stationary chains.

STRENGTHS AND WEAKNESSES:
Weakness:
1. The novelty is limited. The paper extends known SPIDER-style variance reduction to the Markov setting, but the core idea is a relatively straightforward adaptation. The analysis borrows heavily from prior i.i.d. literature and lower bound techniques.

2.The core technical difficulty (dealing with Markovian bias and variance) is acknowledged. The novelty is incremental.

3. The method is motivated by practical problems like RL, but the paper does not test on any real or synthetic Markovian datasets to show relevance.

4.The algorithm relies on mixing time and stationary distribution parameters, but how to estimate them or deal with their unknown nature?

5. The technique can reduce variance, but how to handle the bias introduced by Markovian sampling?

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of general stochastic non-convex optimization for $F(x)$ under the setting that the sampling process (randomness) follows a finite-state stationary Markov chain (MC), which has many applications in machine learning. It considers the case where only one trajectory is sampled and no restarts are allowed. The interaction protocol is through a stochastic first-order oracle, which returns unbiased and variance-bounded samples of the gradient ($\\mathbb{E} [g(x, s)] = \\nabla F(x)$), and the random seed $s$ is from the MC. This work then focuses on zero-respecting algorithms, where at each step $t$, the evaluation point $x_t$ can only be taken from previous points with non-zero gradients. This is a broad algorithm class which stochastic gradient descent and randomized extragradient fall in. First, a sample complexity lower bound of $\\Omega (\\epsilon^{-3})$ is proven, which is the first in the literature. Then, an algorithm, MaC-SPIDER, built upon a previous algorithm SPIDER, is proposed, and is shown that its upper bound is also $O (\\epsilon^{-3})$ up-to other factors such as the mixing time and the minimal stationary probability.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Important Problem:** This work advances the theoretical understanding of stochastic optimization, which is crucial for modern ML applications where i.i.d. assumptions often fail. The results could influence algorithm design in machine learning, both theoretically and practically.\n\n- **Fundamental Contribution:** First to establish matching (the gap being only problem-dependent constants) lower and upper bounds for single-trajectory Markovian optimization, filling an important gap in the literature.\n\n- **Technical Innovation:** Successfully adapts variance reduction to handle bias and temporal dependence in Markovian samples, which is non-trivial.\n\n\n### Weaknesses\n\n- **Lack of Intuitive Explanation:** Section 2.2 introduces many concepts, which may confuse readers not specialized in this problem. The authors did not provide enough explanations for these settings.\n\n- **Limited Scope:** Restricts to finite-state Markov chains, excluding many practical scenarios with continuous or countably infinite state spaces. Restricts to stationary MCs, limiting applicability to non-stationary settings.\n\n- **Lack of Numerical Simulation:** An experiment showing MaC-SPIDER is indeed converging faster than previous algorithms is missing.\n\nQUESTIONS:\nDo you think the problem-dependent quantities of $\\tau_{mix}$ and $\\pi_{min}$ are necessary in the lower-bound (in the minimax sense)? Why is the specific value of $1/4$ chosen for the mixing time?\n\nLIMITATIONS:\nAs the authors admitted, the setting of finite-state stationary MCs is restrictive. Though the dependence of $\\epsilon$ is tight, the necessity of other problem-dependent quantities is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}