Title: First Order Methods with Markovian Noise: from Acceleration to Variational Inequalities
Venue: NeurIPS 2023
Accepted: yes
Keywords: convex optimization

stochastic optimization

Markovian noise

acceleration

variational inequalities

lower bounds
Abstract: This paper delves into stochastic optimization problems that involve Markovian noise. We present a unified approach for the theoretical analysis of first-order gradient methods for stochastic optimization and variational inequalities. Our approach covers scenarios for both non-convex and strongly convex minimization problems. To achieve an optimal (linear) dependence on the mixing time of the underlying noise sequence, we use the randomized batching scheme, which is based on the multilevel Monte Carlo method. Moreover, our technique allows us to eliminate the limiting assumptions of previous research on Markov noise, such as the need for a bounded domain and uniformly bounded stochastic gradients. Our extension to variational inequalities under Markovian noise is original. Additionally, we provide lower bounds that match the oracle complexity of our method in the case of strongly convex optimization problems.

First Order Methods with Markovian Noise: from Acceleration to Variational Inequalities

Abstract

This paper delves into stochastic optimization problems that involve Markovian noise. We present a unified approach for the theoretical analysis of first-order gradient methods for stochastic optimization and variational inequalities. Our approach covers scenarios for both non-convex and strongly convex minimization problems. To achieve an optimal (linear) dependence on the mixing time of the underlying noise sequence, we use the randomized batching scheme, which is based on the multilevel Monte Carlo method. Moreover, our technique allows us to eliminate the limiting assumptions of previous research on Markov noise, such as the need for a bounded domain and uniformly bounded stochastic gradients. Our extension to variational inequalities under Markovian noise is original. Additionally, we provide lower bounds that match the oracle complexity of our method in the case of strongly convex optimization problems.

1 Introduction

Stochastic gradient methods are an essential ingredient for solving various optimization problems, with a wide range of applications in various fields such as machine learning [36, 37], empirical risk minimization problems [96], and reinforcement learning [93, 85, 69]. Various stochastic gradient descent methods (SGD) and their accelerated versions [75, 31] have been extensively studied under different statistical frameworks [17, 97]. The standard assumption for stochastic optimization algorithms is to consider independent and identically distributed noise variables. However, the growing usage of stochastic optimization methods in reinforcement learning [10, 87, 25] and distributed optimization [63, 18, 65] has led to increased interest in problems with Markovian noise. Despite this, existing theoretical works that consider Markov noise have significant limitations, and their analysis often results in suboptimal finite-time error bounds.

Our research aims to fill the gap in the existing literature on the first-order Markovian setting. By focusing on uniformly geometrically ergodic Markov chains, we obtain finite-time complexity bounds for achieving ε -accurate solutions that scale linearly with the mixing time of the underlying Markov chain. Our approach is based on careful applications of randomized batch size schemes and provides a unified view on both non-convex and strongly convex minimization problems, as well as variational inequalities.

Our contributions. Our main contributions are the following:

/diamondmath Accelerated SGD. We provide the first analysis of SGD, including the Nesterov accelerated SGD method, with Markov noise without the assumption of bounded domain and uniformly bounded stochastic gradient estimates. Our results are summarised in Table 1 and Section 2.1 and cover both strongly convex and non-convex scenarios. Our findings for non-convex minimization problems complement the results obtained in [21].

/diamondmath Lower bounds. In Section 2.2 we give the lower bounds showing that the presence of mixing time in the upper complexity bounds is not an artefact of the proof. This is consistent with the results reported in [71].

/diamondmath Extensions. In Section 2.4 we provide, as far as we know, the first analysis for variational inequalities with general stochastic Markov oracle, arbitrary optimization set, and arbitrary composite term. Our finite-time performance analysis provides complexity bounds in terms of oracle calls that scale linearly with the mixing time of the underlying chain, which is an improvement over the bounds obtained in [99] for the Markov setting.

Related works. Next, we briefly summarize the related works.

/diamondmath Stochastic gradient methods. Numerous research papers have reported significant improvements achieved by accelerated methods for stochastic optimization with stochastic gradient oracles involving independent and identically distributed (i.i.d.) noise. These methods have been extensively studied in theory [44, 14, 16, 58, 61, 26, 30, 97, 94, 3, 39, 102] and have shown practical success [55, 91]. The finite-time analysis of first-order methods in i.i.d. noise settings has been extensively studied by many authors, as discussed in [59] and references therein. In Table 1 we include only some important results because i.i.d. setting is not in the interest of this paper.

While the literature on i.i.d. noise is extensive, existing research on the first-order Markovian setting is relatively sparse. In this study, we focus on Markov chains that are uniformly geometrically ergodic, and we refer the reader to Section 2 for detailed definitions. We note that the complexity bounds which scale linearly with the mixing time of the underlying general Markov chain are currently available only for general convex and non-convex minimization problems. Namely, [23] has investigated a version of the ergodic mirror descent algorithm that yields optimal convergence rates for Lipschitz, general convex and non-convex problems. Recently, [21] proposed a random batch size algorithm that adapts to the mixing time of the underlying chain for non-convex optimization with a compact domain. In particular, [21, Theorem 4.3] yields optimal complexity rates in terms of the number of oracle calls required for non-convex problems, which is consistent with the results obtained in [23]. Unlike previous studies, this method is insensitive to the mixing time of the noise sequence.

For the general case of Markovian noise the finite-time analysis of non-accelerated SGD-type algorithms was carried out in [90] and [19]. However, [90] heavily relies on the bounded domain assumption and uniformly bounded stochastic gradient oracles, while its bound in [90, Theorem 5] has a suboptimal dependence on the mixing time of the underlying chain, see Table 1. Additionally, [90] does not cover the strongly convex setting. On the other hand, [19] covers both non-convex and strongly convex settings, but the bounds of [19, Theorem 1] has terms that are exponential in the mixing time, and a careful examination reveals suboptimal dependence on the initial condition for strongly convex problems when SGD is applied.

In the study of Nesterov-accelerated SGD with Markovian noise, the authors of [20] considered the use of a batch size of 1 and achieved a rate of forgetting the initial condition that matches that of the i.i.d. noise setting. However, their result is suboptimal in terms of the variance terms in both non-convex and strongly convex settings, as detailed in Table 1. We emphasize that the case of unbounded gradient oracles with Markov noise is not treated in contrast to the i.i.d. setup [97, 62].

The above papers deal with general Markovian noise optimization. But there are also results that deal with Markovian stochasticity with a finite state space. Here we can highlight the work [28], where the author gives quite extensive results and achieves linear scaling by mixing time in the non-convex as well as strongly convex cases. Recently, numerous papers have appeared dealing with the special scenario of distributed optimization [89]. [99] investigates the generalization and stability of Markov SGD with special attention to the excess variance guarantees. We note that first, these algorithms only need to deal with a very special case of Markov gradients, and second, the corresponding dependence on the mixing time of the Markov chain is again quadratic. At the same time, there exist particular results, e.g. [71], which provide a lower bound for the particular finite sum problems in the Markovian setting.

/diamondmath Variational inequalities. Variational inequalities [29] have been an active area of research in applied mathematics for more than half a century [78, 41, 86]. VI cover important special cases, e.g., minimization over a convex domain, saddle point or min-max and fixed point problems. computational game theory [29], robust [7] and nonsmooth [73, 72] optimization, supervised [51, 4] and unsupervised [103, 5] learning, image denoising [27, 11]. In the last 5 years, variational inequalities and their special cases have attracted much interest in the machine learning community

due to new connections to reinforcement learning [79, 50], adversarial training [64], and GANs [15, 33, 66, 12, 60, 82].

Variational inequalities (VI) and saddle point problems have their own well-established theory and methods. Unlike minimization problems, solving variational inequalities doesn't rely on (accelerated) gradient descent. Instead, the extragradient method [57], various modified versions [72, 42], or similar techniques [95] are recommended as the basic and theoretically optimal methods. While deterministic methods have long been used for solving variational inequalities, stochastic methods have gained importance only in the last 15 years, following pioneering works by [49, 52]. We summarise the results on methods for stochastic variational inequalities with the Lipschitz operator and smooth stochastic saddle point problems in Table 2. The number of papers dealing with stochastic VIs and saddle point problems is small compared to those dealing with stochastic optimization, we include in Table 2 papers with the i.i.d. noise (which we do not do for stochastic optimization). The only competing work dealing with Markovian noise in saddle point problems consider the finite sum problem and thus the finite Markov chain [99], therefore we do not include it in Table 2. Moreover, the results from [99] has much worse oracle complexity guarantees O ( τ 2 / ε 2 ) in terms of τ . There are more papers dealing with stochastic finite-sum variational inequalities or saddle point problems, but in the i.i.d. setting [12, 80, 104, 2, 8]. We also do not consider in Table 2 because of the difference in the stochastic oracle structure. It is important to note that, unlike most previous works, we consider the most general formulation of VI itself for an arbitrary optimization set and composite term.

Notations and definitions. Let ( Z , d Z ) be a complete separable metric space endowed with its Borel σ -field Z . Let ( Z N , Z ⊗ N ) be the corresponding canonical process. Consider the Markov kernel Q defined on Z × Z , and denote by P ξ and E ξ the corresponding probability distribution and the expected value with initial distribution ξ . Without loss of generality, we assume that ( Z k ) k ∈ N is the corresponding canonical process. By construction, for any A ∈ Z , it holds that P ξ ( Z k ∈ A | Z k -1 ) = Q( Z k -1 , A ) , P ξ -a.s. If ξ = δ z , z ∈ Z , we write P z and E z instead of P δ z and E δ z , respectively. For x 1 , . . . , x k being the iterates of any stochastic first-order method, we denote F k = σ ( x j , j ≤ k ) and write E k as an alias for E [ ·|F k ] . We also write N ∗ := N \ { 0 } . For the sequences ( a n ) n ∈ N and ( b n ) n ∈ N we write a n /lessorsimilar b n if there exists a constant c such that that a n ≤ cb n for all n ∈ N .

Unbounded

Method

Domain

Gradient

noise

General

MC

Acceleration

Oracle complexity

(Smooth and non-convex)

Oracle complexity

(Smooth and strongly convex)

i.i.d.

SGD

[84, 70, 32]

✓

✗

N/A

✗

˜

O

(

L

(

f

(

x

0

)

-

f

(

x

∗

))

[

1

ε

2

+

σ

2

ε

4

])

˜

O

(

L

µ

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

µ

2

ε

)

ASGD

[97, 13]

(1)

✓

✓

N/A

✓

˜

O

(

L

(

f

(

x

0

)

-

f

(

x

∗

))

[

1+

δ

2

ε

2

+

σ

2

ε

4

])

˜

O

( (

1 +

δ

2

L

µ

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

µ

2

ε

)

EMD

[23]

(2)

✗

✗

✓

✗

˜

O

(

τ

G

2

D

2

ε

4

)

✗

MC

SGD

[90]

(3)

✓

✗

✗

✗

˜

O

(

h

(

G,L

)

(

τ

ε

2

)

1

/

(1

-

q

)

)

✗

MC

SGD

[19]

(4)

✓

✓

✓

✗

˜

O

(

τ

L

2 (1+

‖

x

∗ ‖

2 +

‖

x

0

-

x

∗ ‖

2 )

ε

4

)

˜

O

(

e

τ

(

L/µ

) 2

[

h

(

L

µ

) log

‖

x

0

-

x

∗ ‖

2

ε

+

τ

2

L

2 (1+

‖

x

∗ ‖

2 )

µ

2

ε

])

ASGD

[20]

(5)

✗

✗

✗

✓

˜

O

(

1

ε

4

[

B

2

+

G

6

(

L

2

τ

2

+1)

] )

˜

O

( √

L

µ

‖

x

0

-

x

∗ ‖

2

ε

1

/

2

+

τ

2 (

G

2 +

µGD

+

µLD

2 )

µ

2

ε

)

MAG

[21]

(6)

✓

✗

✓

✗

˜

O

(

τ

(

G

+

L

+

B

) 2

G

2

ε

4

)

✗

MC

SGD

[28] (Sec. 5.1)

(7)

✓

✗

✗

✗

O

(

τ

(

L

(

f

(

x

0 )

-

f

(

x

∗

))+

σ

2 )

ε

2

+

τ

(

L

(

f

(

x

0 )

-

f

(

x

∗

))+

σ

2 )

σ

2

ε

4

)

O

(

τ

L

µ

log

(

f

(

x

0 )

-

f

(

x

∗

))

/µ

+

σ

2

/

(

µL

)

ε

+

τσ

2

µ

2

ε

)

MC

SGD

[28] (Sec. 5.2)

(8)

✓

✗

✗

✗

✗

O

(

L

µ

log

‖

x

0

-

x

∗ ‖

2

ε

+

L

τσ

2

∗

µ

3

ε

)

RASGD

(ours)

✓

✓

✓

✓

˜

O

(

τ

L

(

f

(

x

0

)

-

f

(

x

∗

))

[

1+

δ

2

ε

2

+

σ

2

ε

4

])

˜

O

(

τ

[

(1 +

δ

2

)

√

L

µ

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

µ

2

ε

])

notation: µ and L are as in A 1 and A 2, G = sup x,z ‖∇ F ( x, z ) ‖ . Note that G ≥ L and G 2 ≥ σ 2 under A 4. We also set B = sup x | f ( x ) | ; x 0 - starting point, x ∗ - solution, D -optimisation domain; D = sup x ∈ D ‖ x -x ∗ ‖ , σ and δ - stochasticity parameters (see A 4); σ ∗ - stochasticity parameter in x ∗ ; τ - mixing time of the chain (see A 3), ε - accuracy of the solution, measured as E [ ‖∇ f ( x ) ‖ 2 ] /lessorsimilar ε 2 for non-convex problems and E [ ‖ x -x ∗ ‖ 2 ] /lessorsimilar ε for the strongly convex ones. Functions h ( L/µ ) and h ( G,L ) stands for an implicit dependence of the respective parameters.

(1) gives results with stepsize as a parameter, we choose it the close way as in our Corollary 1. (2) covers more general noise setting, then just Markovian. (3) for general state-space Markov noise the analysis of [90] requires reversibility. Parameter q ∈ (1 / 2; 1) refers to the step size ∼ 1 /k q . (4) The fluctuation terms in [19, Theorem 1,3] contain hidden dependence on the initial error and ‖ x ∗ ‖ in the fluctuation terms, making the result comparison complicated. They also contain hidden factors, which are exponential in C = τ / log 4 in the notations of our paper. Moreover, the analysis of [19] requires that F ( x, z ) is Lipschitz w.r.t. x for any z ∈ Z . (5) considers Markovian noise with finite state space and a specifically decreasing step size. Moreover, in the proof of [20, Theorem 3] (equations (64) -(66) ) the authors lost the factor C 2 , with C = τ / log 4 . The result in the table accounts for this lost factor. (6) considers the adaptive tuning of batch size, which is oblivious to τ . (7) considers Markov noise with finite state space and additionally assumes that all stochastic realization F ( · , Z ) are L -smooth. (8) considers Markovian noise with finite state space, σ ∗ bounds noise only in x ∗ , but additionally assumes that all stochastic realization F ( · , Z ) are L -smooth and µ -strongly convex.

2 Main results

Assumptions. In this paper we study the minimization problem

where the access to the function f and its gradient is available only through the (unbiased) noisy oracle F ( x, Z ) and ∇ F ( x, Z ) , respectively. In the following presentation we impose at least one of the following regularity constraint on the underlying function f itself:

Statement

Stochasticity

Method

VI?

Any set?

Composite?

Unbounded?

Oracle complexity

Strongly monotone

SPEG

[33, 42]

✓

✓

✗

✗

✗

˜

O

(

L

µ

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

µ

2

ε

)

(1)

SEG

[53]

✓

✓

✗

✗

✗

˜

O

(

‖

x

0

-

x

∗ ‖

2

ε

+

B

2 +

σ

2 +(

B

+

σ

)(1+

LD

)

σ

2

ε

)

SS-SEG

[68, 38]

✓

✓

✓

✓

✗

˜

O

(

L

µ

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

∗

µ

2

ε

)

SEG

[9]

✗

✓

✗

✗

✗

˜

O

(

L

µ

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

µ

2

ε

)

DSEG

[43]

✓

✗

✗

✓

✗

O

( [

L

2

σ

2

µ

4

ε

]

3

)

(2)

UEG

[38]

✓

✗

✗

✓

✗

O

( (

L

+

∆

µ

+

∆

2

µ

2

)

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

µ

2

ε

)

SGDA

[8]

✓

✓

✓

✗

✗

O

(

L

2

µ

2

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

µ

2

ε

)

(3)

REG

(ours)

✓

✓

✓

✓

✓

˜

O

(

τ

·

[ (

L

+

∆

µ

+

∆

2

µ

2

)

log

‖

x

0

-

x

∗ ‖

2

ε

+

σ

2

µ

2

ε

])

Monotone

SMP

[52]

✓

✓

✗

✗

✗

O

(

LD

2

ε

+

σ

2

∆

2

ε

2

)

VR-SEG

[45]

✓

✓

✗

✓

✗

O

(

(

σ

+

∆

) 8

D

4

ε

2

+

D

4

ε

2

)

IPM

[46]

✓

✓

✓

✓

✗

O

(

˜

O

(

L

4

D

4

ε

2

+

σ

2

∗

D

4

ε

2

))

SS-SEG

[68]

✓

✓

✓

✓

✗

˜

O

(

L

2

D

4

ε

+

σ

4

∗

L

2

ε

2

)

SEG

[9]

✗

✓

✗

✗

✗

O

(

LD

2

ε

+

σ

2

∆

2

ε

2

)

REG

(ours)

✓

✓

✓

✓

✓

˜

O

τ

·

LD

2

ε

+

σ

2

D

2

ε

2

+

∆

2

D

4

ε

2

notation: µ = constant of strong monotonicity of operator F , L = Lipschitz constant of F , B = uniform bound of F , D = uniform bound of iterations x k , x 0 = starting point, x ∗ = solution, ∆ and σ = stochasticity parameters (see A 7, [52, 33, 42, 53, 9, 8] take ∆ = 0 ), σ ∗ = stochasticity parameter in x ∗ (see [68]), τ = mixing time of the chain (see A 3), ε = accuracy of the solution. (1) give results with stepsize as a parameter, we choose it according to Section 3 from [88]. (2) consider A 7, but do not provide explicit rates if ∆ = 0 (see also [38, Table 1]). (3) consider the cocoercive case, for which in general /lscript = L 2 /µ .

/negationslash

A 1. The function f is L -smooth on R d with L > 0 , i.e., it is differentiable and there is a constant L > 0 such that the following inequality holds for all x, y ∈ R d :

A 2. The function f is µ -strongly convex on R d , i.e., it is continuously differentiable and there is a constant µ > 0 such that the following inequality holds for all x, y ∈ R d :

Next we specify our assumptions on the sequence of noise variables { Z i } ∞ i =0 . We consider here the general setting of { Z i } ∞ i =0 being a time-homogeneous Markov chain. Such problems naturally arise in stochastic optimization. In the empirical risk minimization problems it naturally appears in the context of non-random minibatch choice. Indeed, a random choice of a batch number may lose to a non-random one, see [67, 56]. A wide range of problems dealing with Markovian noise is spawned by the reinforcement learning methods. The usual MDP setting falls naturally inside this paradigm, moreover, the analysis of non-tabular RL problems requires to deal with the general state-space Markov noise. Here the potential range of applications include the policy evaluation methods, such as the temporal difference methods [92], and policy optimization algorithms, such as policy gradient family, e.g. the celebrated REINFORCE algorithm [100].

We denote by Q the Markov kernel corresponding to the sequence { Z i } ∞ i =0 and impose the following assumption on the mixing properties of Q :

A 3. { Z i } ∞ i =0 is a stationary Markov chain on ( Z , Z ) with Markov kernel Q and unique invariant distribution π . Moreover, Q is uniformly geometrically ergodic with mixing time τ ∈ N , i.e., for every k ∈ N ,

The assumption A 3 is classical in the literature on optimization methods with Markovian noise and has been considered in particular in recent works [90, 21, 20]. In particular, this assumption covers finite state-space Markov chains with irreducible and aperiodic transition matrix considered in

(

[

])

[28]. Yet our definition of the mixing time τ is more classical in the probability literature [81], and is slightly different from the one considered e.g. in [28, 65]. Next we specify our assumptions on stochastic gradient:

A 4. For all x ∈ R d it holds that E π [ ∇ F ( x, Z )] = ∇ f ( x ) . Moreover, for all z ∈ Z and x ∈ R d it holds that

The assumption A 4 resembles the strong growth condition [97], which is classical for the overparametrized learning setup [97, 98]. The main difference is that A 4 concerns the almost sure bound in (4), which is unavoidable when dealing with uniformly geometrically ergodic Markovian noise A 3. Note that it is possible that the quantity δ 2 in (4) is not instance-independent and scales with the ratio L/µ from A 1-A 2 in the particular problems. With the assumptions A 3 and A 4 we can prove the result on the mean squred error of the stochastic gradient estimate computed over batch size n under arbitrary initial distribution. This result is summarized below in Lemma 1:

Lemma 1. Assume A 3 and A 4. Then, for any n ≥ 1 and x ∈ R d , it holds that

Moreover, for any initial distribution ξ on ( Z , Z ) , that

where C 1 = 16(1 + 1 ln 2 4 ) .

Proof. We first prove (5). Note that due to [81, Proposition 3 . 4 ] the Markov kernel Q under A 3 admits a positive pseudospectral gap γ ps > 0 such that 1 / γ ps ≤ 2 τ . Thus, applying the statement of [81, Theorem 3 . 2 ], we get under A 4 that

To prove the second part we use the maximal exact coupling construction and follow, e.g., [24, Theorem 1]. The complete proof is given in Appendix B.1.

The proof of Lemma 1 simplifies the arguments in [21, Lemma 4] and allows us to obtain tighter values for the constants when dealing with the randomized batch size. Note that it is especially important to have the result for MSE under arbitrary initial distribution ξ , since in the proofs of our main results we will inevitably deal with the conditional expectations w.r.t. the previous iterate. We provide more details on the bias and variance of the Markov SGD gradients in the next section.

2.1 Accelerated method

We begin with a version of Nesterov accelerated SGD with randomized batch size, described in Algorithm 1. Due to the unboundedness of the stochastic gradient variance (see A 4), using of the classical Nesterov accelerated method [76, Section 2.2.] does not give the desired result, it is necessary to introduce an additional momentum [74, 97]. We use our own version, but partially similar to [74, 97]. The main feature of Algorithm 1 is that the number of samples used during the k -th gradient computation scales as 2 J k , where J k comes from a truncated geometric distribution. The truncation parameter needs to be adopted (see Theorem 1) in order to control the computational complexity of the algorithm.

Randomized batch size allows for efficient bias reduction in the stochastic gradient estimates and can be seen as a particular case of the so called multilevel MCMC [35, 34]. In the optimization context this approach was successfully used by [21] for the non-convex problems. Indeed, this bias naturally appears under the Markovian stochastic gradients oracles. It is easy to see that, with the counter T k defined in Line 9, we have

/negationslash

Below we show how the bias of the gradient estimate scales with the truncation parameter M . The statement of Lemma 2 yields that the gradient estimates g k introduced above have the bias, which decreases quadratically with M .

Lemma 2. Assume A 3 and A 4. Then for the gradient estimates g k from Algorithm 1 it holds that E k [ g k ] = E k [ g k /floorleft log 2 M /floorright ] . Moreover,

Algorithm 1 Randomized Accelerated GD

1: Parameters: stepsize γ > 0 , momentums θ , η , β , p , number of iterations N , batchsize limit M

2: Initialization: choose x 0 = x 0 f

3: for k = 0 , 1 , 2 , . . . , N -1 do

4: x k g = θ x k f +(1 -θ ) x k

5: Sample J k ∼ Geom (1 / 2)

7: x k +1 f = x k g -p γ g k

9: T k +1 = T k +2 J k B

10: end for

The proof and the statement with explicit constants are given in Appendix B.2. Note that the Lemma 2 is a natural counterpart of the deterministic bound Lemma 1. Moreover, it gives the idea of the trade-off between the parameters B and M . Namely, the expected number of oracle calls to compute g k is O ( B log 2 ( M )) with the bias scaling as M -2 . Thus the increase of M drastically reduced the bias with only a logarithmic payment in variance. At the same time, gradient variance scales as ( τ /B ) 2 , but the increase of B is much more expensive for the computational cost of the whole procedure. Taking into account the considerations above, we can prove the following result:

Theorem 1. Assume A 1 - A 4. Let problem (1) be solved by Algorithm 1. Then for any b ∈ N ∗ , γ ∈ (0; 3 4 L ] , and β , θ , η , p, M, B satisfying it holds that

The proof is provided in Appendix B.3. The result of Theorem 1 can be rewritten as an upper complexity bound under an appropriate choice of the remaining free parameter b :

Corollary 1. Under the conditions of Theorem 1, choosing b = τ and γ as

in order to achieve ε -approximate solution (in terms of E [ ‖ x -x ∗ ‖ 2 ] /lessorsimilar ε ) it takes

The results of Corollary 1 are obtained with fixed parameters of the method. In Corollary 1 these parameters are selected a bit artificially, e.g., the stepsize γ depends on the iteration horizon N . In Appendix B.4 we show how one can similar results, but with a decreasing stepsize.

Comparison. Running the procedure above requires to know the mixing time τ . Estimating the mixing time from a single trajectory of the running Markov chain is known to be computationally hard problem, see e.g. [101] and references therein. At the same time, methods, which share the same (optimal) linear scaling of the sample complexity w.r.t. the mixing time also share the same drawback as our method. In particular, it holds true for the EMD algorithm [23], SGD-DD algorithm [71], and usual SGD with Markovian data [28]. At the same time, in the non-convex scanario the paper [21] is truly oblivious to mixing time, allowing to obtain sample complexity rates for non-convex problems, which are homogeneous w.r.t. τ with AdaGrad-type learning rate. An interesting direction for the future work to suggest a procedure that would allow to generalize the results of [21] to accelerated SGD setting.

It is possible that the sample complexity bound (8) is worse than the respective bounds for nonaccelerated SGD with Markov data, provided that δ 2 grows quickly with L/µ . At the same time, this drawback is shared by the classical results on learning under the strong growth condition, see e.g. [97]. As it is shown in [62], the respective rates can be worse than the ones obtained by usual SGD even under the i.i.d. noise setting, see Appendix F.3 in [62]. Making the analysis of accelerated SGD 'backward compatible' w.r.t. the rates of usual SGD requires to perform analysis in terms of additional problem-specific quantities, see [47, 62].

The closest equivalent of the result Corollary 1 is given by [20, Theorem 3]. However, the corresponding bound of [20, Theorem 3] is incomplete, since the factor τ 2 is lost in the proof (see equations (64 -66) ). With this completion, the bound of [20, Theorem 3] yields a variance term of order ˜ O ( σ 2 τ 2 µ 2 ε ) , which is suboptimal with respect to τ . Moreover, the corresponding analysis relies heavily on the assumption of a bounded domain. In [28], the author considers Markovian noise with a finite number of states and manages to obtain a rather interesting result of the form O ( L µ log 1 ε + L τσ 2 ∗ µ 3 ε ) . Here the first term does not depend on τ , and the second consists only σ ∗ (stochasticity in x ∗ ), but the price for this is an additional factor L/µ in the second term and more strict assumption that all realizations F ( · , z ) are smooth and strongly convex. In the context of overparameterized learning, our results are almost consistent with the bound of [97, Theorem 1] under i.i.d. sampling. The difference is that the term δ 2 in A 4 can be more pessimistic than the expectation bound in [97].

2.2 Lower bounds

We start with a lower bound for the complexity of Markovian stochastic optimization under the assumptions A 1 -A 4. Below we provide a result that highlights that the bound of Theorem 1 is tight provided that δ does not scale with the instance-dependent quantities, e.g., condition number L/µ .

Theorem 2. There exists an instance of the optimization problem satisfying assumptions A 1 -A 4 with δ = 1 and arbitrary σ ≥ 0 , L, µ > 0 , τ ∈ N ∗ , such that for any first-order gradient method it takes at least oracle calls in order to achieve E [ ‖ x N -x ∗ ‖ 2 ] ≤ ε .

The proof is provided in Appendix B.5. The idea of the constructed lower for deterministic part bound Ω ( τ √ L µ log 1 ε ) goes back to [76, Theorem 2.1.13]. The stochastic part lower bound goes back to the classical statistical reasoning, and is well explained for i.i.d. noise in [59, Chapter 4.1]. Our adaptation for Markovian setting is based on Le Cam's theory, see [1, Theorem 8], and also [105]. For the case of Markov noise this lower bound is, to the best of our knowledge, original. The closest result to ours is the stochastic term lower bound in [28, Proposition 1], but it is valid only for the vanilla stochastic gradient methods. Below we provide another lower bound showing that the dependence of the sample complexity Corollary 1 on δ is not an artefact of the proof.

Proposition 1. There exists an instance of the optimization problem satisfying assumptions A 1 -A 4 with arbitrary L, µ > 0 , τ ∈ N ∗ , δ = L µ , and σ = 0 , such that for any first-order gradient method it takes at least gradient calls in order to achieve E [ ‖ x N -x ∗ ‖ 2 ] ≤ ε .

This lower bound is adapted from the information-theoretic lower bound [71]. The detailed proof can be found in Appendix B.5. Recent studies [54, 71, 13] have revealed the impossibility of accelerating stochastic gradient descent (SGD) for online linear regression problems with specific noise structures. To address this issue, researchers have proposed various solutions, such as the MaSS algorithm [62] and the approach presented in [48]. However, these methods rely heavily on the particular structure of the online regression setup. Another question that naturally arises is whether one can get rid of the dependence on τ in the deterministic part of (8) if δ = 0 . The following counterexample shows that this is not the case in general.

Proposition 2. There exists an instance of the optimisation problem satisfying assumptions A 1 -A 4 with with arbitrary L, µ > 0 , τ ∈ N ∗ , σ = 1 , δ = 0 , such that for any first-order gradient method it takes at least

oracle calls in order to achieve E [ ‖ x N -x ∗ ‖ 2 ] ≤ ε .

2.3 Non-convex problems

Now we proceed with a randomized batch size version of the simple SGD algorithm. It is summarized in Algorithm 2 and can be shown to achieve optimal rates of convergence for smooth non-convex problems. For the case of non-convex problems with Markov noise similar analysis appeared in [21, Theorem 4].

By balancing the values of B and M with Lemma 2, we establish the following result:

Theorem 3. Assume A 1, A 3, A 4. Let problem (1) be solved by Algorithm 2. Let f ∗ be a global (maybe not unique) minimum of f . Then for any b ∈ N ∗ , and γ , M satisfying

The proof is provided in Appendix B.6. The next corollary immediately follows from the theorem.

Corollary 2. Under the conditions of Theorem 3, if we choose b = τ and γ given by then to achieve ε -solution (in terms of E [ ‖∇ f ( x ) ‖ 2 ] /lessorsimilar ε 2 ) we need

Comparison. The respective bound for the non-convex setting provided in [20, Theorem 1] yields the sample complexity of order ˜ O ( τ 2 L ( f ( x 0 ) -f ( x ∗ )) σ 2 ε 4 ) . Also we can note the results of [28, Theorem

To achieve linear convergence rates in the non-convex setting we can use the Polyak-Lojasiewicz (PL) condition [83]. The respective result is provided in Appendix B.7.

2.4 Variational inequalities

In this section, we are interested in the following problem:

Here F : R d → R d an operator, X a convex set, and r : R d → R is a regularization term (a suitable lower semicontinuous convex function) which is assumed to have a simple structure. As mentioned earlier, this problem is quite general and covers a wide range of possible problem formulations. For example, if the operator F is the gradient of a convex function f , then the problem (9) is equivalent to the composite minimization problem [6], i.e., minimization of f ( x ) + r ( x ) . In the meantime, (9) is also a reformulation of the min-max problem

with convex-concave continuously differentiable g , convex sets X 1 , X 2 and convex functions r 1 , r 2 . Using the first-order optimality conditions, it is easy to verify that (10) is equivalent to (9) with x = ( x T 1 , x T 2 ) T , F ( x ) = ( ∇ x 1 f ( x 1 , x 2 ) T , -∇ x 2 f ( x 1 , x 2 ) T ) T , and r ( x ) = r 1 ( x 1 ) + r 2 ( x 2 ) .

A 5. The operator F is L -Lipschitz continuous on X with L > 0 , i.e., the following inequality holds for all x, y ∈ X :

A 6. The operator F is µ F -strongly monotone on X , i.e., the following inequality holds for all x, y ∈ X :

The function r is µ r -strongly convex on X , i.e. for all x, y ∈ X and any r ′ ( x ) ∈ ∂ r ( x ) we have

These two assumptions are more than standard for the study of variational inequalities and are found in all the papers from Table 2. We consider two cases: strongly monotone/convex with µ F + µ r > 0 and monotone/convex with µ F + µ r = 0 .

A 7. For all x ∈ R d it holds that E π [ F ( x, Z )] = F ( x ) . Moreover, for all z ∈ Z and x ∈ X it holds that

where x ∗ is some point from the solution set.

A 7 is found in the literature on variational inequalities [43, 45, 38] and is considered to be analog to A 4 on overparametrized learning.

Just as the Nesterov accelerated method is optimal for smooth convex minimization problems, the ExtraGradient method [57, 72, 52] is optimal for monotone variational inequalities. Therefore, we take it as a base. On the extrapolation step (Line 4) of Algorithm 3, we simply collect a batch of size B , but on the main step (Line 8) we use the randomization as in Algorithm 1. The next theorem gives the convergence of our method.

Theorem 4. Assume A 5, A 6 with µ F + µ r > 0 , A 3, A 7. Let problem (9) be solved by Algorithm 3. Then for any b ∈ N ∗ , and γ , M satisfying it holds that

The proof is postponed to Appendix B.8. One can get an estimate on oracle complexity.

Algorithm 3 Randomized ExtraGradient

1: Parameters: stepsize γ > 0 , number of iterations N

2: Initialization: choose x 0 ∈ X

3:

,

,

, . . . , N

4:

5:

6:

7:

8:

9:

10:

= 0

k

x

+1

1

2

k

/

T

+1

2

k

/

Sample

2

=

=

J

g

g

=

x

k

+1

T

k

+1

end for

Corollary 3. Under the conditions of Theorem 4, if we choose b = τ and γ as follows

then to achieve ε -solution (in terms of E [ ‖ x -x ∗ ‖ 2 ] /lessorsimilar ε ) we need

k

k

0

prox

γ

r

T

-

x

+

k

k

+

=

=

{

B

∼

2

0

prox

,

γ

r

T

k

+1

/

J

k

J

(

(

g

k

k

k

1

do

-

(

x

k

B

γ

1

2

-

-

+2

)

γ

J

g

k

J

g

k

k

k

B

-

)

for

∑

,

)

+

i

)

with

2

(

-

1

1

)

B

i

=1

F

(

x

, Z

if

2

M

k

J

otherwise

k

T

k

g

k

j

= 2

-

j

B

-

1

∑

2

B

j

i

=1

·

F

(

x

k

+1

/

2

, Z

T

k

+1

/

2

+

i

)

Note that one provide an (almost) matching lower complexity bounds for variational inequalities via lower bounds for saddle point problems, which are a special case of variational inequalities. The method for obtaining lower bounds for saddle point problems is reduced to obtaining estimates for the strongly convex minimization problem (see [106, 40] for respective deterministic lower bounds), which we provide in Section 2.2. Similarly, the question of constructing a lower bound which is tight w.r.t. ∆ remains open.

For the monotone case, we use the gap function as a convergence criterion:

Such a criterion is standard and classical for monotone variational inequalities [72, 52]. An important assumption for the gap function is the boundedness of the set X .

A 8. The set X is bounded and has a diameter D , i.e., for all x, y ∈ X : ‖ x -y ‖ 2 ≤ D 2 .

A8can be slightly relaxed. We need to use a simple trick from [77]. In particular, we need to consider C - a compact subset of X and change X to C in (14). But such a technique is rather technical and does not change the essence. Finally, the following result holds.

Theorem 5. Assume A 5, A 6 with µ F + µ r = 0 , A 8, A 3, A 7. Let problem (9) be solved by Algorithm 3. Then for any B ∈ N ∗ , and γ , M satisfying γ /lessorsimilar L -1 , M = √ N , it holds that

The proof is postponed to Appendix B.9. The following corollary holds.

Corollary 4. Under the conditions of Theorem 5, if we choose B = τ and γ as follows

then to achieve ε -solution (in terms of E [ Gap ( x )] /lessorsimilar ε ) we need

Comparison. These results is the first for variational inequalities with Markovian stochasticity, either in the strongly monotone or monotone cases. The only close work is [99]. The authors work with convex-concave saddle point problems and provide the following estimate on the oracle complexity O ( τ 2 · G 4 ε 2 + D 2 ε 2 ) (with G - the uniform bound of the operator), which is worse than ours at least in terms of τ . Moreover, the authors consider the case of a finite Markov chain, which is a special case of our setup.

3 Conclusion

In this paper, we present a unified random batch size framework that achieves optimal finite-time performance for non-convex and strongly convex optimization problems with Markov noise, as well as for variational inequalities. Unlike existing methods, our framework relaxes the assumptions typically imposed on the domain and stochastic gradient oracle. We also provide a variety of lower bounds, which are to the best of our knowledge original in the Markov setting.

Acknowledgments

This research of A. Beznosikov has been supported by The Analytical Center for the Government of the Russian Federation (Agreement No. 70-2021-00143 dd. 01.11.2021, IGK 000000D730321P5Q0002). E. Moulines received support from the grant ANR-19-CHIA-002 SCAI and parts of his work has been done under the auspices of Lagrange Center for maths and computing.

References

Eddie Aamari and Clément Levrard. Nonasymptotic rates for manifold, tangent space and curvature estimation. The Annals of Statistics , 47(1):177 - 204, 2019.

Ahmet Alacaoglu and Yura Malitsky. Stochastic variance reduction for variational inequality methods. In Conference on Learning Theory , pages 778-816. PMLR, 2022.

Necdet Serhat Aybat, Alireza Fallah, Mert Gurbuzbalaban, and Asuman Ozdaglar. A universally optimal multistage accelerated stochastic gradient method. Advances in neural information processing systems , 32, 2019.

Francis Bach, Rodolphe Jenatton, Julien Mairal, and Guillaume Obozinski. Optimization with sparsity-inducing penalties. arXiv preprint arXiv:1108.0775 , 2011.

Francis Bach, Julien Mairal, and Jean Ponce. Convex sparse matrix factorizations. arXiv preprint arXiv:0812.1869 , 2008.

Amir Beck. First-order methods in optimization . Society for Industrial and Applied Mathematics (SIAM), 2017.

Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust Optimization . Princeton University Press, 2009.

Aleksandr Beznosikov, Eduard Gorbunov, Hugo Berard, and Nicolas Loizou. Stochastic gradient descent-ascent: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics , pages 172-235. PMLR, 2023.

Aleksandr Beznosikov, Valentin Samokhin, and Alexander Gasnikov. Distributed saddle-point problems: Lower bounds, optimal and robust algorithms. arXiv preprint arXiv:2010.13112 , 2020.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory , pages 16911692. PMLR, 2018.

Antonin Chambolle and Thomas Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of mathematical imaging and vision , 40(1):120-145, 2011.

Tatjana Chavdarova, Gauthier Gidel, François Fleuret, and Simon Lacoste-Julien. Reducing noise in gan training with variance reduced extragradient. Advances in Neural Information Processing Systems , 32, 2019.

You-Lin Chen, Sen Na, and Mladen Kolar. Convergence analysis of accelerated stochastic gradient descent under the growth condition. arXiv preprint arXiv:2006.06782 , 2020.

Andrew Cotter, Ohad Shamir, Nati Srebro, and Karthik Sridharan. Better mini-batch algorithms via accelerated gradient methods. Advances in neural information processing systems , 24, 2011.

Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. arXiv preprint arXiv:1711.00141 , 2017.

Olivier Devolder et al. Stochastic first order methods in smooth convex optimization. Technical report, CORE, 2011.

Aymeric Dieuleveut, Nicolas Flammarion, and Francis Bach. Harder, better, faster, stronger convergence rates for least-squares regression. The Journal of Machine Learning Research , 18(1):3520-3570, 2017.

Alexandros G. Dimakis, Soummya Kar, José M. F. Moura, Michael G. Rabbat, and Anna Scaglione. Gossip algorithms for distributed signal processing. Proceedings of the IEEE , 98(11):1847-1864, 2010.

Thinh T. Doan. Finite-time analysis of markov gradient descent. IEEE Transactions on Automatic Control , 68(4):2140-2153, 2023.

Thinh T Doan, Lam M Nguyen, Nhan H Pham, and Justin Romberg. Convergence rates of accelerated markov gradient descent with applications in reinforcement learning. arXiv preprint arXiv:2002.02873 , 2020.

Ron Dorfman and Kfir Yehuda Levy. Adapting to mixing time in stochastic optimization with markovian data. In International Conference on Machine Learning , pages 5429-5446. PMLR, 2022.

R. Douc, E. Moulines, P. Priouret, and P. Soulier. Markov chains . Springer Series in Operations Research and Financial Engineering. Springer, 2018.

John C Duchi, Alekh Agarwal, Mikael Johansson, and Michael I Jordan. Ergodic mirror descent. SIAM Journal on Optimization , 22(4):1549-1578, 2012.

Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov, and Marina Sheshukova. Rosenthal-type inequalities for linear statistics of markov chains. arXiv preprint arXiv:2303.05838 , 2023.

Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov, and Hoi-To Wai. On the stability of random matrix product with markovian noise: Application to linear stochastic approximation and td learning. In Conference on Learning Theory , pages 1711-1752. PMLR, 2021.

Pavel Dvurechensky and Alexander Gasnikov. Stochastic intermediate gradient method for convex problems with stochastic inexact oracle. Journal of Optimization Theory and Applications , 171:121-145, 2016.

Ernie Esser, Xiaoqun Zhang, and Tony F Chan. A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM Journal on Imaging Sciences , 3(4):1015-1046, 2010.

Mathieu Even. Stochastic gradient descent under Markovian sampling schemes. arXiv preprint arXiv:2302.14428 , 2023.

F. Facchinei and J.S. Pang. Finite-Dimensional Variational Inequalities and Complementarity Problems . Springer Series in Operations Research and Financial Engineering. Springer New York, 2007.

Alexander Vladimirovich Gasnikov and Yu E Nesterov. Universal method for stochastic composite optimization problems. Computational Mathematics and Mathematical Physics , 58:48-64, 2018.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming , 155(1-2):267-305, 2016.

Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. A variational inequality perspective on generative adversarial networks. arXiv preprint arXiv:1802.10551 , 2018.

Michael B. Giles. Multilevel monte carlo path simulation. Operations Research , 56(3):607617, 2008.

Peter W Glynn and Chang-han Rhee. Exact estimation for markov chain equilibrium expectations. Journal of Applied Probability , 51(A):377-389, 2014.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS) , 2014.

Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, Cambridge, MA, USA, 2016. http://www.deeplearningbook.org .

Eduard Gorbunov, Hugo Berard, Gauthier Gidel, and Nicolas Loizou. Stochastic extragradient: General analysis and improved rates. In International Conference on Artificial Intelligence and Statistics , pages 7865-7901. PMLR, 2022.

Eduard Gorbunov, Marina Danilova, Innokentiy Shibaev, Pavel Dvurechensky, and Alexander Gasnikov. Near-optimal high probability complexity bounds for non-smooth stochastic optimization with heavy-tailed noise. arXiv preprint arXiv:2106.05958 , 2021.

Yuze Han, Guangzeng Xie, and Zhihua Zhang. Lower complexity bounds of finite-sum optimization problems: The results and construction. arXiv preprint arXiv:2103.08280 , 2021.

P. T. Harker and J.-S. Pang. Finite-dimensional variational inequality and nonlinear complementarity problems: a survey of theory, algorithms and applications. Mathematical programming , 1990.

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. Advances in Neural Information Processing Systems , 32, 2019.

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. Explore aggressively, update conservatively: Stochastic extragradient methods with variable stepsize scaling. Advances in Neural Information Processing Systems , 33:16223-16234, 2020.

Chonghai Hu, Weike Pan, and James Kwok. Accelerated gradient methods for stochastic optimization and online learning. Advances in Neural Information Processing Systems , 22, 2009.

A. N. Iusem, A. Jofré, R. I. Oliveira, and P. Thompson. Extragradient method with variance reduction for stochastic variational inequalities. SIAM Journal on Optimization , 27(2):686-724, 2017.

Alfredo N Iusem, Alejandro Jofré, and Philip Thompson. Incremental constraint projection methods for monotone stochastic variational inequalities. Mathematics of Operations Research , 44(1):236-263, 2019.

Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Accelerating stochastic gradient descent for least squares regression. In Conference On Learning Theory , pages 545-604. PMLR, 2018.

Prateek Jain, Sham M. Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic gradient descent for least squares regression: Mini-batching, averaging, and model misspecification. Journal of Machine Learning Research , 18(223):1-42, 2018.

Houyuan Jiang and Huifu Xu. Stochastic approximation approaches to the stochastic variational inequality problem. IEEE Transactions on Automatic Control , 53(6):1462-1475, 2008.

Yujia Jin and Aaron Sidford. Efficiently solving MDPs with stochastic mirror descent. In Proceedings of the 37th International Conference on Machine Learning (ICML) , volume 119, pages 4890-4900. PMLR, 2020.

Thorsten Joachims. A support vector method for multivariate performance measures. pages 377-384, 01 2005.

Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems , 1(1):17-58, 2011.

Aswin Kannan and Uday V Shanbhag. Optimal stochastic extragradient schemes for pseudomonotone stochastic variational inequality problems and their variants. Computational Optimization and Applications , 74(3):779-820, 2019.

Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, and Sham Kakade. On the insufficiency of existing momentum schemes for stochastic optimization. In 2018 Information Theory and Applications Workshop (ITA) , pages 1-9. IEEE, 2018.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Anastasia Koloskova, Nikita Doikov, Sebastian U Stich, and Martin Jaggi. Shuffle SGD is always better than SGD: Improved analysis of SGD with arbitrary data orders. arXiv preprint arXiv:2305.19259 , 2023.

G. M. Korpelevich. The extragradient method for finding saddle points and other problems. Matecon , 12:35-49, 1977.

Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming , 133(1-2):365-397, 2012.

Guanghui Lan. First-order and Stochastic Optimization Methods for Machine Learning . 01 2020.

Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , volume 89 of Proceedings of Machine Learning Research , pages 907-915. PMLR, 16-18 Apr 2019.

Qihang Lin, Xi Chen, and Javier Pena. A smoothing stochastic gradient method for composite optimization. Optimization Methods and Software , 29(6):1281-1301, 2014.

Chaoyue Liu and Mikhail Belkin. Accelerating sgd with momentum for over-parameterized learning. arXiv preprint arXiv:1810.13395 , 2018.

Cassio G. Lopes and Ali H. Sayed. Incremental adaptive strategies over distributed networks. IEEE Transactions on Signal Processing , 55(8):4064-4077, 2007.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR) , 2018.

Xianghui Mao, Kun Yuan, Yubin Hu, Yuantao Gu, Ali H. Sayed, and Wotao Yin. Walkman: A communication-efficient random-walk algorithm for decentralized optimization. IEEE Transactions on Signal Processing , 68:2513-2528, 2020.

Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. arXiv preprint arXiv:1807.02629 , 2018.

Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Random reshuffling: Simple analysis with vast improvements. Advances in Neural Information Processing Systems , 33:17309-17320, 2020.

Konstantin Mishchenko, Dmitry Kovalev, Egor Shulgin, Peter Richtárik, and Yura Malitsky. Revisiting stochastic extragradient. In International Conference on Artificial Intelligence and Statistics , pages 4573-4582. PMLR, 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

Eric Moulines and Francis Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011.

Dheeraj Nagaraj, Xian Wu, Guy Bresler, Prateek Jain, and Praneeth Netrapalli. Least squares regression with markovian data: Fundamental limits and algorithms. Advances in neural information processing systems , 33:16666-16676, 2020.

Arkadi Nemirovski. Prox-method with rate of convergence O(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Yu Nesterov. Smooth minimization of non-smooth functions. Mathematical programming , 103(1):127-152, 2005.

Yu. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization , 22(2):341-362, 2012.

Yu. E. Nesterov. A method for solving the convex programming problem with convergence rate O (1 /k 2 ) . Dokl. Akad. Nauk SSSR , 269(3):543-547, 1983.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2003.

Yurii Nesterov. Dual extrapolation and its applications to solving variational inequalities and related problems. Mathematical Programming , 109(2):319-344, 2007.

J. Von Neumann and O. Morgenstern. Theory of games and economic behavior . Princeton University Press, 1944.

Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Proceedings of the 34th International Conference on Machine Learning (ICML) , volume 70, pages 2681-2690. PMLR, 2017.

Balamurugan Palaniappan and Francis Bach. Stochastic variance reduction methods for saddle-point problems. Advances in Neural Information Processing Systems , 29, 2016.

Daniel Paulin. Concentration inequalities for Markov chains by Marton couplings and spectral methods. Electronic Journal of Probability , 20(none):1 - 32, 2015.

Wei Peng, Yu-Hong Dai, Hui Zhang, and Lizhi Cheng. Training gans with centripetal acceleration. Optimization Methods and Software , 35(5):955-973, 2020.

B.T. Polyak. Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics , 3(4):864-878, 1963.

Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics , 22(3):400 - 407, 1951.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

Gesualdo Scutari, Daniel Palomar, Francisco Facchinei, and Jong-shi Pang. Convex optimization, game theory, and variational inequality theory. Signal Processing Magazine, IEEE , 27:35 - 49, 06 2010.

Rayadurgam Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation andtd learning. In Conference on Learning Theory , pages 2803-2830. PMLR, 2019.

Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232 , 2019.

Tao Sun, Dongsheng Li, and Bao Wang. Adaptive Random Walk Gradient Descent for Decentralized Optimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 20790-20809. PMLR, 17-23 Jul 2022.

Tao Sun, Yuejiao Sun, and Wotao Yin. On Markov chain gradient descent. Advances in neural information processing systems , 31, 2018.

Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning , volume 28 of Proceedings of Machine Learning Research , pages 1139-1147, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3:9-44, 1988.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018.

Adrien Taylor and Francis Bach. Stochastic first-order methods: non-asymptotic and computeraided analyses via potential functions. In Conference on Learning Theory , pages 2934-2992. PMLR, 2019.

Paul Tseng. A modified forward-backward splitting method for maximal monotone mappings. SIAM Journal on Control and Optimization , 38(2):431-446, 2000.

Aad W Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Sharan Vaswani, Francis Bach, and Mark Schmidt. Fast and faster convergence of SGD for over-parameterized models and an accelerated perceptron. In The 22nd international conference on artificial intelligence and statistics , pages 1195-1204. PMLR, 2019.

Sharan Vaswani, Aaron Mishkin, Issam H. Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. CoRR , abs/1905.09997, 2019.

Puyu Wang, Yunwen Lei, Yiming Ying, and Ding-Xuan Zhou. Stability and generalization for markov chain stochastic gradient methods. arXiv preprint arXiv:2209.08005 , 2022.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

Geoffrey Wolfer and Aryeh Kontorovich. Estimating the mixing time of ergodic markov chains. In Conference on Learning Theory , pages 3120-3159. PMLR, 2019.

Blake E Woodworth and Nathan Srebro. An even more optimal stochastic optimization algorithm: minibatching and interpolation learning. Advances in Neural Information Processing Systems , 34:7333-7345, 2021.

Linli Xu, James Neufeld, Bryce Larson, and Dale Schuurmans. Maximum margin clustering. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 17. MIT Press, 2005.

Junchi Yang, Negar Kiyavash, and Niao He. Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1153-1165. Curran Associates, Inc., 2020.

Bin Yu. Assouad, Fano, and Le Cam , pages 423-435. Springer New York, New York, NY, 1997.

Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the saddle point problems. arXiv preprint arXiv:1912.07481 , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study first order stochastic algorithms under a Markovian noise: the noise is no longer i.i.d. as for vanilla SGD, but follows a Markov chain, of finite mixing time $\\tau$. The proposed algorithms are SGD-like algorithms, with batchsizes that can vary. In particular, they use a randomized batchsize strategy, whose aim is to trade-off between the noise variance of the stochastic gradients, and their bias (due to non-iid sampling). Batches are typically taken equal to the mixing time of the Markov chain.\nAn accelerated SGD algorithm is first introduced, and is proved to converge in the smooth strongly convex regime, with a rate that depends  on the mixing time, a maximal gradient noise (max in terms of Markov chain state space and variables), condition number and strong convexity parameter. Various provided lower bounds show that this rate, although not optimal, has an optimal noise dependency.\nThe SGD algorithm is then studied in the smooth non-convex setting, to obtain typical rates for this algorithm under this type of noise assumption.\nFinally, the authors study variational inequalities under Markovian noise, by introducing Randomized Extra Gradient (with also their batchsize strategy).\n\nSTRENGTHS:\nOverall, the paper is well written, and very easy to follow, on a very interesting topic (stochastic approximation/optimization, under Markovian noise), that has re-gained attention lately.\n\nIn this line of work, the proposed algorithm and analysis of accelerated SGD is novel, this is the first accelerated rate for SGD under Markov chain noise. The batch size technique is also nice, and well explained. Together, these two techniques give a nice algorithm with a near-optimal rate, as shown by the lower bounds, that are more general than previous ones in this setting.\n\nAlthough I am not very familiar of variational inequalities, the topic of VA with MC noise seems to be not very well studied in the literature, and provide the first bound with complexity proportional to the mixing time.\n\nWEAKNESSES:\nThere are two major concerns. Questions related to these are below in the dedicated section.\n\n1) **The noise assumption**. The noise parameter $\\sigma^2$ is defined as (forgetting about the $\\delta^2$): $\\sigma^2\\geq \\sup_{z,x} \\|\\nabla_x F(x,z)-f(x)\\|^2$. Here, taking the sup over $x\\in\\mathbb R^d$ is not problematic (and $\\delta^2$ is usually there to compensate for this), but I worry about this sup over $z\\in Z$. Indeed, for the very simple setting of gaussian additive noise, the noise term here is equal to infinity. Previous works (e.g. [28] in the paper) instead consider $\\bar \\sigma^2$ for noise assumption (in the smooth non-convex or smooth-PL settings), where a mean is taken over $z$ instead of a sup. $\\sigma^2$ here is thus not a noise variance, but an almost sure bound, which can be confusing.\nTable 1 should also discuss this noise assumption.\n\n2) **Non-convex rate, Table, and missing comparison**. *(a)* The same non-convex rate is present in [28], under milder noise assumptions. *(b)* The strongly convex rate in this same reference has no mixing time in the transient regime (i.e. in the linearly converging term), and its noise is taken at the *optimum* (and is thus often null). *(c)* *We note that the complexity\n55 bounds which scale linearly with the mixing time of the underlying Markov chain are currently\n56 available only for non-convex minimization problems*: this assertion is false, since rates exists for convex problems, PL problems [23,28].\nThese three points *(a,b,c)* lead the reviewer to have some concerns with comparison related works that are cited in this paper (and that I would not have read if not cited, so that this leads me to believe, are there some other existing works like this in the literature ?). Indeed, the non-convex rates should clearly be compared with those of [28], that seem stronger, if I understand well. Also, their strongly convex rates should be discussed: the rate of Theorem 1 can be faster in some cases but not all I think. These rates should thus be in table 1 also.\n\nQUESTIONS:\nQuestions related to the two major weaknesses/concerns above:\n\n1. Is the accelerated rate in Theorem 1 obtained by adding stronger assumptions than previous works (i.e. a sup over the state space instead of a mean) ? I would like the authors to discuss this assumption, and its relation with other works.\n\n2. Could the author provide a clearer comparison of their work to the related ones ? I think this requires adding some discussions in several parts of the paper.\n\n3. The lower bound in [28] is in term of hitting time, and the authors of this paper provide an algorithm and analysis without noise assumption; could this be related to the current submission ?\n\n\nOverall, I vote for acceptance, despite the concerns noted above, due to the contributions in variational inequalities, for the clarity of the paper, and the accelerated rate with batch randomization, while waiting for the authors' answer in order to provide a final decision.\n\nLIMITATIONS:\nNo such limitation noted.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors design and analyze first-order methods for stochastic minimization and stochastic variational inequalities (VIs), with Markovian noise. Typically in stochastic minimization, the target function is only accessible at point x, defined as $f(x)$, and its gradient $\\nabla f(x)$, through the sampling of a seed $z \\in Z$ using an oracle $F(x,z)$ and its gradient $\\nabla F(x,z)$. Under Markovian noise, the seed $z_t$ sampled at time $t$, is a state in a time-homogeneous Markov chain with a unique invariant distribution $\\pi$. Under this assumption, the goal is to minimize the expected value over the invariant distribution, that is $\\min_x f(x)=E_{z\\sim \\pi}[F(x,z)]$.\n\nThe authors make an assumption about the mixing time of the Markov chain. Beyond this, they make no other fundamental assumptions about the Markov chain. Moreover, for every point $x$ and seed $z$, and for some positive $\\sigma,\\delta$, the noisy gradient $\\nabla F(\\cdot,\\cdot)$ satisfies the following inequality:\n$$\n\\|\\nabla f(x) - \\nabla F(x,z) \\|^2 \\leq \\sigma^2 + \\delta^2 \\|\\nabla f(x)\\|^2,\n$$\n\nThe authors prove the following results:\n1. For strongly-convex minimization, they design and analyze an accelerated algorithm. This algorithm builds upon the accelerated stochastic gradient descent by Nesterov, and they further use randomized batch sizes and momentum. Additionally, they show that their convergence rate is tight in all parameters, except for $\\delta$. They also show that the dependence on $\\delta$ and the parameter that controls the Markov chain's mixing is necessary.\n2. The authors extend their results to non-convex minimization as well as to stochastic variational inequalities (VIs) with Markovian noise. They design and analyze algorithms for both settings by modifying existing algorithms, specifically, the Stochastic Gradient Descent and Stochastic Extra Gradient. They use the randomized batching technique, previously used in their algorithm for strongly-convex minimization.\n\n\n\nAfter the rebuttal, the authors effectively addressed my concerns about the real-world applications of their noise model. Furthermore, in my view, they have successfully addressed the main concerns raised by the other reviewers. Consequently, I maintained my original score.\n\nSTRENGTHS:\nThe assumptions about the Markovian noise make sense especially in setting where the value of the noise sample is only influenced by factors exogenous of the picked action. Moreover, I think the paper is well-written, and the comparison with related work seems extensive. The authors seem to improve either in terms of assumptions, or in terms of convergence in all their settings. The results look reasonable, I checked the statements in the main body.\n\nWEAKNESSES:\nGiven that this is a theory paper, the only minor weaknesses is that for the stochastic strongly-convex minimization, the bound is not tight for parameter $\\delta$, and the fact that they are missing lower bounds for the non-convex setting, as well as the stochastic variational inequality setting. I do not consider this a big weakness.\n\nQUESTIONS:\nCould the authors provide more examples of stochastic minimization or stochastic VIs with Markovian noise?\n\nLIMITATIONS:\nI think that the authors have addressed the limitations of their work, by listing the necessary assumptions for their theorems. Moreover, I do not believe that this work will have negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an optimization study focusing on stochastic optimization with Markovian noise as opposed to i.i.d. noise. The authors explore three distinct scenarios: strongly convex, non-convex, and Variational Inequalities (VI). In comparison to previous research, they relax the boundedness assumptions related to the domain or gradient magnitude for cases 1 and 2. The newly proposed convergence bounds exhibit linear dependence on the mixing time, whereas previous works relied on quadratic or exponential dependence. The authors establish a lower bound to demonstrate the optimality of this linear dependence. Notably, this work is the first to demonstrate the convergence of the stochastic EG algorithm for VI inequalities under Markovian noise.\n\nSTRENGTHS:\nThis paper introduces a relaxation of the restrictive assumptions found in previous analyses by adopting common and mild assumptions. It explores three distinct settings and presents a comprehensive framework for their convenient analysis. The exposition of the paper is accessible and the proofs are clearly articulated.\n\nWEAKNESSES:\nOne major limitation of the method is its lack of obliviousness to the mixing time. This paper addresses this weakness by achieving linear dependence on the mixing time; however, this is accomplished through a dependence on multilevel mini-batching, where the mini-batch sizes are contingent on the mixing time. This particular dependence was absent in some previous works, which may explain their quadratic dependence on the mixing time.\n\nAnother limitation is the comparison of convergence with existing results. While the paper relaxes certain restrictive assumptions, its convergence is limited to the neighborhood. In contrast, for example, [21] achieves asymptotic convergence to the minima. Moreover, when analyzing convergence with fixed step size or momentum parameters, the feasible parameter values in this work rely on σ and other unknown values to achieve an ε error.\n\nFinally, it would be beneficial to augment the paper with experimental results that validate the upper bound. Specifically, it would be valuable to observe the linear dependence on the mixing time and also include numerical results for the Variational Inequalities (VI) setting\n\nQUESTIONS:\n1- There are some typos in your proofs in supplementary materials e.g. in the proof of Lemma 2, in the equation after line 601, it should be Log M instead of log N and x^k_g instead of x^k. \n\n2- There is no discussion in the paper about a range of \\delta^2 appearing in the accelerated rate. It could be large as the condition number and in that case, the rate given in the paper is worse than SGD in terms of dependence on the condition number. Why the accelerated rate doesn’t have backward compatibility in this case? \n\n3- Your convergence proofs are based on fixed momentum and step size parameters. And due to that, you get a convergence to the neighborhood. Could you analyze in the decaying step size regime?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the convergence of first-order methods with Markovian noise for solving stochastic optimization problems and variational inequalities. The authors establish finite-time complexity bounds which scale linearly with the mixing time of the underlying Markov chain.

STRENGTHS:
This paper analyzes SGD (with and without Nesterov acceleration) with Markovian noise without the bounded domain and unifomly bounded stochastic gradient estimate assumptions, for both strongly convex and nonconvex scenarios. Matching lower bounds are also provided for the strongly convex case. Finally, an analysis for variational inequalities with general stochastic Markov oracle, arbitrary optimization set and arbitrary composite term is also provided. The results are interesting and appear to be novel.

WEAKNESSES:
Purely theoretical paper. No numerical experiments are provided.

QUESTIONS:
It is suggested that the authors should provide some simple numerical experiments to demonstrate the effectiveness of the proposed algorithms.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-15