Title: Stochastic Extragradient with Flip-Flop Shuffling & Anchoring: Provable Improvements
Venue: NeurIPS 2024
Accepted: yes
Keywords: minimax optimization

stochastic optimization

extragradient method

without-replacement sampling
Abstract: In minimax optimization, the extragradient (EG) method has been extensively studied because it outperforms the gradient descent-ascent method in convex-concave (C-C) problems. Yet, stochastic EG (SEG) has seen limited success in C-C problems, especially for unconstrained cases. Motivated by the recent progress of shuffling-based stochastic methods, we investigate the convergence of shuffling-based SEG in unconstrained finite-sum minimax problems, in search of convergent shuffling-based SEG. Our analysis reveals that both random reshuffling and the recently proposed flip-flop shuffling alone can suffer divergence in C-C problems. However, with an additional simple trick called anchoring, we develop the SEG with flip-flop anchoring (SEG-FFA) method which successfully converges in C-C problems. We also show upper and lower bounds in the strongly-convex-strongly-concave setting, demonstrating that SEG-FFA has a provably faster convergence rate compared to other shuffling-based methods.

Stochastic Extragradient with Flip-Flop Shuffling & Anchoring: Provable Improvements

Abstract

In minimax optimization, the extragradient (EG) method has been extensively studied because it outperforms the gradient descent-ascent method in convexconcave (C-C) problems. Yet, stochastic EG (SEG) has seen limited success in C-C problems, especially for unconstrained cases. Motivated by the recent progress of shuffling-based stochastic methods, we investigate the convergence of shufflingbased SEG in unconstrained finite-sum minimax problems, in search of convergent shuffling-based SEG. Our analysis reveals that both random reshuffling and the recently proposed flip-flop shuffling alone can suffer divergence in C-C problems. However, with an additional simple trick called anchoring, we develop the SEG with flip-flop anchoring ( SEG-FFA ) method which successfully converges in C-C problems. We also show upper and lower bounds in the strongly-convex-stronglyconcave setting, demonstrating that SEG-FFA has a provably faster convergence rate compared to other shuffling-based methods.

1 Introduction

Minimax problems with a finite-sum structure, which are optimization problems of the form

can be found in many interesting applications, such as generative adversarial networks [19], refining diffusion models [28], adversarial training [37], optimal transport based generative models [48], multi-agent reinforcement learning [53], and so on. Deterministic methods for minimax problems, such as gradient descent-ascent (GDA) [3] and extragradient (EG) [29], have been extensively studied in the literature. It is though known that, unlike gradient descent (GD) for minimization problems, GDA may diverge even when f is convex on x and concave on y . On the other hand, EG employs a two-step update procedure, named extrapolation and update steps (see Section 2 for details), which allows it to find an optimum under this convex-concave setting [29, 51], and moreover, attains a convergence rate faster than GDA [4] when f is strongly convex on x and strongly concave on y .

In contrast, attempts to construct stochastic variants of these algorithms have not been so fruitful. When f is convex-concave, stochastic gradient descent-ascent (SGDA) clearly may diverge, just as

in the deterministic GDA. To make matters worse, stochastic extragradient (SEG) methods have also had limited success on unconstrained convex-concave problems. As we elaborate in Section 2 in more detail, existing versions of SEG and their analyses have limitations that hinder its application to general unconstrained finite-sum convex-concave problems, requiring additional assumptions such as bounded domain, increasing batch size, convex-concavity of each component f i , uniformly bounded gradient variance, and/or absence of convergence rates.

In the context of finite-sum optimization, most of the theoretical studies on stochastic methods have long been based on the with-replacement sampling scheme, where an index i ( t ) is independently and uniformly sampled among { 1 , . . . , n } at each iteration t . Such a sampling scheme is relatively easy to theoretically analyze, because the sampled f i ( t ) is an unbiased estimator of the full objective function f . In practice, however, inspired by the empirical observations of faster convergence in finite-sum minimization [8, 47], the without-replacement sampling schemes have been the de facto standard. Among them, the most popular is the random reshuffling (RR) scheme, where in every epoch consisting of n iterations, the indices are chosen exactly once in a randomly shuffled order.

This gap between theory and practice in minimization problems is being closed by the recent breakthroughs in stochastic gradient descent (SGD), namely that SGD with RR leads to a provably faster convergence compared to with-replacement SGD when the number of epochs is large enough [1, 35, 39, 41, 55, 56]. This has motivated further studies on finding other shuffling-based sampling schemes that can improve upon RR, resulting in the discoveries such as the flip-flop scheme [46] and gradient balancing (GraB) [11, 32]. The flip-flop scheme is a particularly simple yet interesting modification of RR with improved rates in quadratic problems: a random permutation is used twice in a single epoch (i.e., two passes over n components in an epoch), but the order is reversed in the second pass.

The aforesaid progress in minimization also triggered the study of stochastic minimax methods with shuffling. Similar to minimization problems, SGDA with RR indeed converges faster than the withreplacement SGDA, under assumptions such as strongly-convex-strongly-concave objectives [15] or f satisfying the Polyak-Łojasiewicz condition [13]. Despite the superiority of EG over GDA, the SEG with shuffling has not been shown to have a solid theoretical advantage over the SGDA with shuffling yet. This motivated us to study the following question:

Can shuffling schemes provide convergence guarantees for SEG, improved upon SGDA with shuffling, in unconstrained finite-sum (strongly-)convex-(strongly-)concave settings?

There are two types of SEG: same-sample SEG, where a sample chosen is used both for the extrapolation step and the update step, and independent-sample SEG, where two independently chosen samples are used in each step. We will particularly focus on the same-sample SEG because it combines more naturally with shuffling-based schemes than independent-sample SEG. Therefore, to be more specific, we are interested in developing shuffling-based variants of same-sample SEG in unconstrained finite-sum minimax problems with minimal modifications to the algorithm. We show that (a) in convex-concave settings, our new method reaches an optimum with a guarantee on the rate of convergence, overcoming the limitations of existing results; (b) in strongly-convex-stronglyconcave settings, the method converges faster than other SGDA/SEG variants.

1.1 Our Contributions

In this paper, we study various same-sample SEG algorithms under different shuffling schemes, and propose the stochastic extragradient with flip-flop anchoring ( SEG-FFA ) method, which is SEG amended with the techniques of flip-flop shuffling scheme and anchoring . Here, by anchoring we refer to a step of taking a convex combination between the initial and final iterates of an epoch, resembling the celebrated Krasnosel'ski˘ ı-Mann iteration [30, 33] as we discuss in Section 5. With such minimal modifications to SEG, we show that SEG-FFA achieves provably improved convergence guarantees. More precisely, our contributions can be listed as follows (see Table 1 for a summary). For clarity, we use SEG-US to refer to with-replacement SEG, where US stands for uniform sampling.

We first study the same-sample versions of SEG-US , SEG with RR ( SEG-RR ), and SEG with flip-flop ( SEG-FF ). We show that they all can diverge when f is convex-concave, 1

† [17, 20] show upper bounds for SEG-US , but they require increasing batch sizes as well as other assumptions (see Appendix B.1).

‡ [25] shows that independent-sample SEG-US converges for stepsizes α t , β t decaying at different rates, but gives no conv. rate.

§ Unfortunately, the proof of this convergence bound in this recent AISTATS 2024 paper seems to be incorrect: see Appendix B.4.

by constructing an explicit counterexample (Theorem 4.1). This shows that shuffling alone cannot fix the divergence issue of SEG-US .

Wenext investigate the underlying cause for the nonconvergence of SEG-US , SEG-RR , and SEG-FF . In particular, we identify that either they fail to match the update equation of the reference method EG beyond first-order Taylor expansion terms, or attempting to match both the firstand second-order Taylor expansion terms results in divergence (Proposition 5.2).

By adopting a simple technique of anchoring on top of flip-flop shuffling, we devise our algorithm SEG-FFA , whose epoch-wise update deterministically matches EG up to secondorder Taylor expansion terms (Proposition 5.3). We prove that SEG-FFA enjoys improved convergence guarantees, as anticipated by our design principle. Most importantly, we show that SEG-FFA achieves a convergence rate of ˜ O ( 1 / K 1 / 3 ) when f is convex-concave, where K denotes the number of epochs. This is in stark contrast to other baseline algorithms that diverge under this setting (see the last column of Table 1).

Moreover, we show that when f is strongly-convex-strongly-concave, SEG-FFA achieves a convergence rate of ˜ O ( 1 / nK 4 ) (Theorem 5.5). In addition, by proving Ω( 1 / nK 3 ) lower bounds for the convergence rates of SGDA-RR and SEG-RR under the same setting (Theorem 5.6), we show that SEG-FFA has a provable advantage over these baseline algorithms.

2 Related Works

Extragradient and EG+ Extragradient (EG) method [29] is a widely used minimax optimization method, well-known for resolving the nonconvergence issue of GDA on convex-concave problems. In this paper, we also consider EG+ [17], which is a generalization of EG. The update rule of EG+ is defined, for stepsizes { η 1 ,k } k ≥ 0 and { η 2 ,k } k ≥ 0 , as

The first step is called the extrapolation step , and the second step is called the update step . If f is convex-concave, Diakonikolas et al. [17] show that EG+ reaches an optimum when η 1 ,k ≥ η 2 ,k . In particular, when η 1 ,k = η 2 ,k , we recover the standard EG by Korpelevich [29].

Stochastic Variants of Extragradient In (2), if the stochastic estimators of ∇ x f and ∇ y f are used instead of the gradients themselves, we get the standard SEG. If an estimator chosen is used for both the extrapolation and the update steps, we get the same-sample SEG, which we focus on in this paper; see Appendix A for the pseudocode.

While EG improves upon GDA, unfortunately, SEG has not been able to show a clear advantage over SGDA. On one hand, analyses of SEG on strongly-convex-strongly-concave problems have shown some success; see, e.g., [18, 20]. Yet, on the other hand, for general unconstrained convex-concave problems, to the best of our knowledge, the existing stochastic variants of EG and their analyses face several limitations. 23 Assumptions commonly imposed in the existing literature include: (i) the domain is bounded, either explicitly or implicitly [27, 36], (ii) one must increase the batch size to achieve convergence [9, 17, 20], 4 and (iii) each component f i is convex-concave [20, 36], and (iv) the components have uniformly bounded gradient variance [9, 17, 42]. For further details, see Appendix B.1 and Table 2 therein. Notably, Hsieh et al. [25] prove convergence of the independentsample SEG without these four restrictions, but the result lacks an explicit convergence rate.

Our proposed SEG-FFA overcomes all the aforementioned limitations, and reaches an optimum with an explicit rate in unconstrained convex-concave problems, under relatively mild conditions. The readers may also refer to [7] for a comprehensive overview on this topic.

Meanwhile, under the finite-sum setting, variance reduction schemes have also been considered, achieving some promising results [2, 10]. Yet, although theoretically appealing, variance reduction is less widely used in practice due to their curiously inferior performance in training neural networks [16]. On top of this practical issue, variance reduction techniques share the aforementioned limitation (ii), as accessing full gradients can be viewed as increasing the batch size. In contrast, our main goal in this paper is to study how a carefully chosen sampling scheme, with minimal modifications to the algorithm, can improve the convergence of SEG without the need for increased batch size; therefore, we believe that our work is not directly comparable to variance reduction-based EG.

Taylor Expansion Matching and Convergence Guarantees It has been repeatedly reported that the convergence of an optimization method is deeply related to the degree to which the Taylor expansion (with respect to the step size) of its update equation matches with that of an already known convergent method. For example, Mokhtari et al. [38] observed that the advantage of EG over GDA comes from the Taylor expansion of update equations of EG matching that of the proximal point (PP) method [34] up to second-order terms, whereas GDA matches PP only up to first-order terms.

The advantages of the shuffling scheme over the with-replacement sampling can be explained in a similar way. One key property of shuffling-based methods is that, while the individual estimators are biased as they are dependent to other estimators within the same epoch, the overall stochastic error across the epoch decreases dramatically compared to using n independent unbiased estimators. For instance, in SGD with RR [1] and in SGDA with RR [15], the overall progress made within each epoch exactly matches their deterministic counterparts up to the first-order, leaving an error as small as O ( η 2 ) , where η is the stepsize. Rajput et al. [46] observed that, when each component functions are convex quadratics, then using flip-flop on SGD can reduce the error further to O ( η 3 ) , resulting in an even faster convergence. As we further elaborate in Section 5, the motivation behind our design principle of SEG-FFA is also based on this line of observations.

3 Notations and Problem Settings

Let [ n ] ⊂ Z denote the set { 1 , . . . , n } . The set of all permutations on [ n ] will be denoted by S n . For the finite-sum minimax problem (1), we denote the saddle gradient operators by

2

Most of these results are carried out assuming access to a stochastic oracle of

f

, which indeed subsumes the finite-sum setting as a special case. However, it seems unlikely that these limitations of the existing studies will

be easily resolved by simply narrowing the focus down to the finite-sum setting; see Appendix B.3.

The derivative of an operator will be denoted with a prefix D . For example, the derivative of F is denoted by D F . Often a single vector will be used to denote the minimization and the maximization variable at once. For instance, for z ∈ R d 1 + d 2 which is a concatenation of x ∈ R d 1 and y ∈ R d 2 , we simply write Fz to denote F ( x , y ) .

It is well known that, if f is µ -strongly convex on x and µ -strongly concave on y for some µ > 0 (respectively, µ = 0 ), then its saddle gradient F is µ -strongly monotone (respectively, monotone), in the following sense. For a proof of this standard fact, see, e.g. , [22].

Assumption 3.1 (Monotonicity & Strong Monotonicity) . For µ > 0 , we say that an operator F is µ -strongly monotone if, for any z , w ∈ R d 1 + d 2 , it holds that

If (3) holds for µ = 0 , then we say that F is monotone.

Thus, from now on, we will use the term strongly monotone (respectively, monotone ) problems rather than strongly-convex-strongly-concave (respectively, convex-concave) problems. Notice that we only assume that the full saddle gradient F is (strongly) monotone, not the individual F i 's.

In addition, we remark that our convergence analysis under the monotonicity of F , Theorem 5.4, in fact requires only a relaxed version of monotonicity, known as star -monotonicity. This condition imposes the inequality (3) with µ = 0 , but only when w = z ∗ , where z ∗ is a point such that Fz ∗ = 0 . This relaxation allows for a certain degree of nonconvex-nonconcavity in f . For a more detailed discussion on the star-monotonicity condition, see Appendix G.1.

Other three underlying assumptions we make on the problem (1) can be listed as follows.

Assumption 3.2 (Existence of an Optimal Solution) . An optimal solution of the problem (1), which is a point we denote by z ∗ = ( x ∗ , y ∗ ) that satisfies

for any x ∈ R d 1 and y ∈ R d 2 , exists in R d 1 + d 2 .

Because the problem is unconstrained and f is convex-concave, a point z ∗ is an optimum if and only if Fz ∗ = 0 . For strongly monotone problems, Assumption 3.2 is not explicitly required, as it is guaranteed a priori [5, Proposition 22.11]. For monotone problems, we explicitly impose Assumption 3.2 in order to exclude pathological problems such as f ( x, y ) = x -y .

Assumption 3.3 (Smoothness) . Each f i is L -smooth, and each F i is M -smooth. That is, for any z , w ∈ R d 1 + d 2 ,

(i) ∥ F i z -F i w ∥ ≤ L ∥ z -w ∥ ,

(ii) ∥ D F i z -D F i w ∥ ≤ M ∥ z -w ∥ .

It is worth mentioning that the gradient operator F i arising from a quadratic function f i is M -smooth with M = 0 . Notice also that, by the finite-sum structure F = 1 n ∑ n i =1 F i , it is clear that Assumption 3.3 implies f being L -smooth and F being M -smooth.

The L -smoothness assumption on the objective functions is standard in the optimization literature, while the M -smoothness assumption on the saddle gradients may look less standard. This smoothness assumption on the saddle gradient, in other words the Lipschitz Hessian condition, for analyzing SEGFFA stems from the analysis of the flip-flop sampling scheme [46]. In particular, this is needed for bounding the high-order error terms between the (deterministic) EG and SEG-FFA in Section 5.1. The existing analysis of flip-flop sampling [46] is limited to quadratic functions that trivially have 0 -Lipschitz Hessians ( M = 0 ), so our analysis is a step forward.

Assumption 3.4 (Component Variance) . There exist constants ρ ≥ 0 and σ ≥ 0 such that

For strongly monotone problems, Assumption 3.4 is not explicitly required, because it can be obtained as a consequence of the preceding assumptions: see Lemma C.9. Nevertheless, for convenience, we will keep the notations ρ and σ as in (4) for the strongly monotone setting as well.

In many existing works on stochastic optimization methods for minimax problems, Assumption 3.4 with ρ = 0 is imposed. This uniform bound on the variance simplifies the convergence analyses, but it is also fairly restrictive especially in the unconstrained settings. Already for bilinear finite-sum minimax problems f ( x , y ) = 1 n ∑ n i =1 x ⊤ B i y , one can easily check that setting ρ = 0 forces the matrices B i to be exactly equal to each other. For machine learning applications, it has been also reported that the assumption with ρ = 0 often fails to hold [7]. Therefore, allowing the variance to grow with the gradient Fz makes the assumption much more realistic.

The Lipschitz Hessian condition and the component variance assumption for monotone problems may still look rather strong. We leave the study on how one can relax such assumptions to prove upper bounds for convergence rates as an interesting future direction. On the other hand, while our lower bound results in Theorems 4.1 and 5.6 are derived under those strong assumptions, they still serve as lower bound results also for larger function classes that do not have those assumptions. In other words, the value of those results are not limited because of those assumptions being imposed.

4 Shuffling Alone Is Not Enough

Under the settings we have discussed, we study the SEG with shuffling-based sampling schemes. First we describe the precise methods of our consideration, namely the SEG-RR and SEG-FF .

For k ≥ 0 , in the beginning of an epoch, a random permutation τ k is sampled from a uniform distribution over S n . Then, for n iterations, we use each of the component functions once, in the order determined by τ k . That is, for i = 0 , 1 , . . . , n -1 we do

for some stepsizes α k and β k . In case of SEG-RR , the epoch is completed here, and we set z k +1 0 ← z k n as the initial point for the next epoch.

In case of SEG-FF , we additionally perform n more iterations in the epoch, as proposed in Rajput et al. [46]. In these additional iterations, the component functions are each used once more, but in the reverse order. That is, for i = n, n +1 , . . . , 2 n -1 , we do

Then we set z k +1 0 ← z k 2 n as the initial point for the next epoch. The full pseudocode of these methods can be found in Appendix A.

When F is strongly monotone, it is possible to show that both SEG-RR and SEG-FF indeed provide speed-up over SEG-US . The well-known rate of SEG-US under strong monotonicity of F is Θ( 1 / T ) , where T is the total number of iterations [6, 20]. Translating this rate to our shuffling-based setting, where there are Θ( n ) iterations per epoch, this rate amounts to Θ( 1 / nK ) . Recently, Emmanouilidis et al. [18] have shown that SEG-RR , under the same setting as ours, attains a convergence rate of ˜ O ( 1 / nK 2 ) , on par with the rate of SGDA-RR [15]. In Appendix F, we also show that SEG-FF attains a similar rate of convergence.

However, it turns out that the benefit of shuffling does not extend further beyond the strongly monotone setting. In fact, when F is merely monotone, then in the worst case, SEG-RR and SEG-FF suffers from nonconvergence, just as in the case of SEG-US .

Theorem 4.1. For n = 2 , there exists a minimax problem with f ( x, y ) = 1 2 ∑ 2 i =1 f i ( x, y ) having a monotone F , consisting of L -smooth quadratic f i 's satisfying Assumption 3.4 with ( ρ, σ ) = (1 , 0) , such that SEG-US , SEG-RR and SEG-FF diverge in expectation for any positive stepsizes.

We provide the explicit counterexample and the proof of divergence in Appendix H.1. Note that Theorem 4.1 and its proof in Appendix H.1 imply that min t =0 ,...,T E [ ∥ Fz t ∥ 2 ] = Ω(1) for SEG-US and min k =0 ,...,K E [ ∥ ∥ Fz k 0 ∥ ∥ 2 ] = Ω(1) for SEG-RR and SEG-FF , as summarized in Table 1.

5 SEG-FFA : SEG with Flip-Flop Anchoring

In this section, we investigate the underlying cause for nonconvergence of SEG-RR and SEG-FF from the perspective of how accurately they match the convergent EG or PP methods in terms of the Taylor expansions of updates. We then propose adding a simple anchoring step at the end of each epoch of SEG-FF . It turns out that adding the anchoring step, which is a step of taking a convex combination of an iterate with a previously computed iterate, reduces the stochastic noise and leads to a method with improved convergence properties.

5.1 Design Principle: Second-Order Matching

As observed by [38], the key feature of EG behind its superior convergence properties compared to GDA is its update rule closely resembling PP, while the 'error' of GDA as an approximation of PP is so large that it hinders convergence. The difference between the updates of EG and PP, in the Taylor expansion, is as small as O ( η 3 ) per iteration, where η is the stepsize. On the other hand, GDA and PP show a difference of O ( η 2 ) , and this greater 'error' explains why GDA diverges while EG and PP converge. Of course, EG and PP are not the only two algorithms that converge in the monotone setting; let us recall the update rule of EG+ method [17], and Taylor-expand it as the following:

EG+ is known to converge for unconstrained monotone problems if η 1 ≥ η 2 . When η 1 = η 2 , it recovers EG and matches PP up to second-order terms.

Based on these observations, we now state our key principle for designing a convergent version of SEG: second-order matching . We would like to choose proper stepsizes, sampling scheme, and anchoring scheme so that our without-replacement SEG can deterministically match the update equation of a convergent algorithm (EG/PP or EG+) up to the O ( η 2 ) terms (i.e., second-order terms in the Taylor expansion), thereby satisfying a small O ( η 3 ) approximation error. We show that (a) this second-order matching can be achieved with flip-flop anchoring , but not solely by permutation-based sampling such as RR and flip-flop (without anchoring), and (b) second-order matching indeed grants convergence for monotone problems. In particular, we demonstrate that

SEG-RR suffers a poor approximation error of O ( η 2 ) as an approximation of EG/EG+.

SEG-FF can match EG+ up to second-order terms, but it results in a choice of stepsizes ( η 2 = 2 η 1 ) that make EG+ diverge (Proposition 5.2).

SEG-FFA , the method we propose, matches EG up to second-order terms to get an error of O ( η 3 ) (Proposition 5.3), achieving convergence in monotone problems (Theorem 5.4).

To this end, let us consider a general form of SEG that incorporates any arbitrary sampling scheme. More precisely, in the k -th 'epoch' consisted of N iterations, the components are chosen in the order of T k 0 , T k 1 , · · · , T k N -1 , where T k i ∈ { F 1 , . . . , F n } for each i . For our purpose, we assume that N is some multiple of n (e.g., N = n for SEG-RR , N = 2 n for SEG-FF ). Then, given α and β we perform SEG updates, for i = 0 , 1 , . . . , N -1 ,

5.1.1 Necessity of Flip-Flop Sampling

The general method in (8) that sets the initial point for the next epoch as z k +1 0 ← z k N satisfies the following property.

Proposition 5.1. Suppose that Assumption 3.3 holds. For some ϵ k N = o ( ( α + β ) 2 ) , it holds that

See Appendix D.1 for the proof. To make (7) and (9) match up to the second-order, both the equations

̸

must hold. Clearly, without-replacement sampling will make (10) hold. However, it is easy to check that random reshuffling falls short of making (11) hold. This is because, if RR is used, then T k 0 , T k 1 , . . . , T k n -1 is nothing but a reordering of F 1 , . . . , F n into F τ (1) , . . . , F τ ( n ) , so the RHS of (11) can only contain terms D F τ ( j ) ( z k 0 ) F τ ( i ) z k 0 with i ≤ j . This observation motivates the use of flip-flop sampling, because choosing T k i = T k 2 n -1 -i lets all the required terms D F j ( z k 0 ) F i z k 0 to appear in the RHS of (11).

5.1.2 Designing SEG-FFA

Flip-flop does resolve the aforesaid issue, but still another complication remains for plain SEG-FF .

Proposition 5.2. Suppose we use flip-flop sampling (without anchoring). In order to make (10) and (11) hold, we must choose β = η 1 / n and α = β / 2 . However, this leads to η 2 = 2 η 1 , which is the set of parameters that fails to make EG+ converge.

For the proof, see Appendix D.2. This shows that a modification is necessary to develop a stochastic method that achieves second-order matching to convergent EG/EG+ methods.

We thus propose to add an anchoring step:

after finishing the N updates (8), instead of z k +1 0 ← z k N . This is our Stochastic ExtraGradient with Flip-Flop Anchoring ( SEG-FFA ) method, named after the design of combining the flip-flop sampling scheme and the anchoring step. We note that this idea of taking a convex combination has originally appeared in the Krasnosel'ski˘ ı-Mann iteration [30, 33], and also under the name of Lookahead methods [12, 43]. This slightly differs from the more widely used Halpern iteration [23] based anchoring ( cf. [54]), which would have used the initial point z 0 0 instead of z k 0 in (12).

This anchoring step changes (9) accordingly, and essentially amounts to dividing the right-hand sides of (10) and (11) each by 2 (see Appendix D for the detailed derivations). We show that choosing α = β / 2 in fact leads to the second-order matching to EG, i.e., EG+ with η 1 = η 2 .

Proposition 5.3. Suppose that Assumptions 3.3 and 3.4 hold. Then, for β k = η and α k = β k / 2 , SEG-FFA becomes an approximation of EG with error at most O ( η 3 ) . In other words, we achieve

In other words, adding the anchoring step allows us to get a method that well approximates the convergent EG with an error as small as O ( η 3 ) . For a more in-depth discussion, see Appendix E.

5.2 Convergence Analysis of SEG-FFA

As a result of the second-order matching, we obtain SEG-FFA , a stochastic method that has an error of O ( η 3 ) as an approximation of EG. Achieving this order of magnitude for the approximation error turns out to be the key to the exact convergence to an optimum under the monotone setting.

Theorem 5.4. Suppose that F is (star-)monotone, Assumptions 3.2, 3.3, and 3.4 hold, and we are running SEG-FFA . Then, for any K ≥ 1 , by choosing the stepsizes sufficiently small and decaying as β k = O ( 1 / k 1 / 3 log k ) and α k = β k / 2 , the iterates generated by SEG-FFA achieves the bound

For the full statement of the theorem and its proof, see Appendix G. We note that, although Theorem 5.4, and also Theorem 5.5 below, are stated specifically for SEG-FFA , our analyses show that both theorems can be applied to any method that achieves the second-order matching in terms of Proposition 5.3.

The reduced error also shows a gain in the rate of convergence under the strongly monotone setting. This aligns with the intuition that error hinders convergence, hence having a smaller error is beneficial.

Theorem 5.5. Suppose that F is µ -strongly monotone with µ > 0 and Assumption 3.3 holds. Then, there exists a choice of η > 0 such that, when SEG-FFA is run for K epochs with constant stepsizes β k = η and α k = η / 2 , for some constant ω independent of η , the iterates generated by SEG-FFA achieves the bound

Theorem 5.5 actually stems from a unified analysis that encompasses all the shuffling-based SEG methods introduced in this paper, including SEG-RR and SEG-FF . See Appendix F for the details.

Notice the exponent 4 of the number of epochs K in the convergence rate, which is twice as large as the exponent 2 of SGDA-RR and SEG-RR . In fact, this gain in the rate of convergence turns out to be fundamental. As we show in the following theorem, the theoretical lower bounds of convergence for SGDA-RR and SEG-RR with constant stepsize are both Ω( 1 / nK 3 ) . This exhibits that there is a provable gap between those methods and SEG-FFA , which attains ˜ O ( 1 / nK 4 ) .

Theorem 5.6. Suppose n ≥ 2 . For both SGDA-RR with constant stepsize α k = α > 0 and SEG-RR with constant stepsize α k = α > 0 , β k = β > 0 , there exists a µ -strongly monotone minimax problem f ( z ) = 1 n ∑ n i =1 f i ( z ) with µ > 0 such that regardless of stepsizes, we have

Proof. The full statement and the proof are presented in Appendix H.3.

6 Experiments

We consider randomly generated quadratic problems of the form

In particular, we sample the random components so that the full objective is either monotone or strongly monotone, respectively, while each of the components may be nonmonotone. For the exact descriptions on how we constructed the problems, see Appendix I.1.

Monotone Case We ran the experiment on 5 random instances of (13) with the stepsizes scheduled as η k = η 0 / (1+ k/ 10) 0 . 34 where η 0 = min { 0 . 01 , 1 L } for SEG-FFA , and α k = β k = η k for SEG-US , SEG-RR , and SEG-FF . The exponent 0 . 34 is to ensure a sufficient decay rate required by Theorem 5.4, and the convergence of SEG-FFA under such a stepsize scheduling is validated in Remark G.5. The value of η 0 is, however, a heuristically determined small number. The results of the geometric mean over the 5 runs are plotted in Figure 1. As expected by our theory, SEG-FFA successfully shows convergence, while all of SEG-FF , SEG-RR , and SEG-US diverge in the long run.

Strongly monotone case Along with the variants of SEG, we also compare the performances of SGDA-RR and SGDA-US . We ran the experiment on 5 random instances of (13) with stepsizes η k = 0 . 001 , and the results are plotted in Figure 1. Additional results obtained from using other stepsizes can be found in Appendix I.4. We again observe an agreement between the empirical results and our theory; SEG-FFA eventually finds the point with the smallest gradient norm among the methods that are considered.

Further additional experiments and ablation studies we have conducted can be found in Appendix I.

0

20000

40000

60000

80000

100000

number of passes (

t

)

10

5

10

3

10

1

10 1

10 3

10 5

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

0

100

200

300

400

500

600

700

800

number of passes (

t

)

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

SGDA-RR

SGDA-US

7 Conclusion

We proposed SEG-FFA , a new stochastic variant of EG that uses flip-flop sampling and anchoring. While being a minimal modification from the vanilla SEG, SEG-FFA attains the crucial 'secondorder matching' property to the deterministic EG, leading to a two-fold improved convergence. On one hand, SEG-FFA reaches an optimum in the monotone setting, unlike many baseline methods such as SEG-US , SEG-RR , and SEG-FF that diverge. Moreover, in the strongly monotone setting, SEG-FFA shows a faster convergence with a provable gap from the other methods.

An interesting future direction would be to extend our work to more general nonconvex-nonconcave problems, further exploring the potentials of the second-order matching technique. It would also be appealing to further study whether it is possible to devise a new method that achieves second-order (or higher) matching without the anchoring step, potentially enhancing our understanding of the effectiveness of the matching technique.

Acknowledgments and Disclosure of Funding

This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2019-NR040050). JC and DK acknowledge support from the NRF grant (No. RS-2022-NR071715) funded by the Korea government (MSIT), and the Samsung Science & Technology Foundation grant (No. SSTF-BA2101-02). CY acknowledges support from the NRF grant (No. RS-2023-00211352) funded by the Korea government (MSIT).

References

Kwangjun Ahn, Chulhee Yun, and Suvrit Sra. SGD with shuffling: optimal rates without component convexity and large epoch requirements. Advances in Neural Information Processing Systems , 33:17526-17535, 2020.

Ahmet Alacaoglu and Yura Malitsky. Stochastic variance reduction for variational inequality methods. In Conference on Learning Theory , pages 778-816. PMLR, 2022.

Kenneth J. Arrow and Leonid Hurwicz. Reduction of constrained maxima to saddle-point problems. In Proc. Third Berkeley Symp. on Math. Statist. and Prob. , volume 5, pages 1-20, 1956. Univ. of Calif. Press.

Waïss Azizian, Ioannis Mitliagkas, Simon Lacoste-Julien, and Gauthier Gidel. A tight and unified analysis of gradient-based methods for a whole spectrum of differentiable games. In

International Conference on Artificial Intelligence and Statistics , pages 2863-2873. PMLR, 2020.

Heinz H. Bauschke and Patrick L. Combettes. Convex Analysis and Monotone Operator Theory in Hilbert Spaces . Springer, 2nd edition, 2017.

Aleksandr Beznosikov, Valentin Samokhin, and Alexander Gasnikov. Distributed saddle-point problems: Lower bounds, near-optimal and robust algorithms. arXiv preprint arXiv:2010.13112 , 2020.

Aleksandr Beznosikov, Boris Polyak, Eduard Gorbunov, Dmitry Kovalev, and Alexander Gasnikov. Smooth monotone stochastic variational inequalities and saddle point problems: A survey. European Mathematical Society Magazine , 127:15-28, 2023.

Léon Bottou. Curiously fast convergence of some stochastic gradient descent algorithms. In Proceedings of the symposium on learning and data science, Paris , volume 8, pages 2624-2633. Citeseer, 2009.

Xufeng Cai, Chaobing Song, Cristóbal Guzmán, and Jelena Diakonikolas. Stochastic Halpern iteration with variance reduction for stochastic monotone inclusions. Advances in Neural Information Processing Systems , 35:24766-24779, 2022.

Yair Carmon, Yujia Jin, Aaron Sidford, and Kevin Tian. Variance reduction for matrix games. Advances in Neural Information Processing Systems , 32, 2019.

Jaeyoung Cha, Jaewook Lee, and Chulhee Yun. Tighter lower bounds for shuffling SGD: Random permutations and beyond. In International Conference on Machine Learning , pages 3855-3912. PMLR, 2023.

Tatjana Chavdarova, Matteo Pagliardini, Sebastian U Stich, François Fleuret, and Martin Jaggi. Taming GANs with Lookahead-minmax. In The Ninth International Conference on Learning Representations , 2021.

Hanseul Cho and Chulhee Yun. SGDA with shuffling: faster convergence for nonconvex-PŁ minimax optimization. In The Eleventh International Conference on Learning Representations , 2023.

Sayantan Choudhury, Eduard Gorbunov, and Nicolas Loizou. Single-call stochastic extragradient methods for structured non-monotone variational inequalities: Improved analysis under weaker conditions. Advances in Neural Information Processing Systems , 36:64918-64956, 2023.

Aniket Das, Bernhard Schölkopf, and Michael Muehlebach. Sampling without replacement leads to faster rates in finite-sum minimax optimization. Advances in Neural Information Processing Systems , 35:6749-6762, 2022.

Aaron Defazio and Léon Bottou. On the ineffectiveness of variance reduced optimization for deep learning. Advances in Neural Information Processing Systems , 32, 2019.

Jelena Diakonikolas, Constantinos Daskalakis, and Michael I. Jordan. Efficient methods for structured nonconvex-nonconcave min-max optimization. In International Conference on Artificial Intelligence and Statistics , pages 2746-2754. PMLR, 2021.

Konstantinos Emmanouilidis, René Vidal, and Nicolas Loizou. Stochastic extragradient with random reshuffling: Improved convergence for variational inequalities. In International Conference on Artificial Intelligence and Statistics , pages 3682-3690. PMLR, 2024.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Eduard Gorbunov, Hugo Berard, Gauthier Gidel, and Nicolas Loizou. Stochastic extragradient: General analysis and improved rates. In International Conference on Artificial Intelligence and Statistics , pages 7865-7901. PMLR, 2022.

Eduard Gorbunov, Nicolas Loizou, and Gauthier Gidel. Extragradient method: O (1 /K ) lastiterate convergence for monotone variational inequalities and connections with cocoercivity. In International Conference on Artificial Intelligence and Statistics , pages 366-402. PMLR, 2022.

Benjamin Grimmer, Haihao Lu, Pratik Worah, and Vahab Mirrokni. The landscape of the proximal point method for nonconvex-nonconcave minimax optimization. Mathematical Programming , 201(1-2):373-407, 2023.

Benjamin Halpern. Fixed points of nonexpanding maps. Bulletin of the American Mathematical Society , 73(6):957-961, 1967.

Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature , 585(7825):357-362, September 2020.

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. Explore aggressively, update conservatively: Stochastic extragradient methods with variable stepsize scaling. Advances in Neural Information Processing Systems , 33:16223-16234, 2020.

J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering , 9 (3):90-95, 2007.

Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems , 1(1):17-58, 2011.

Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In International Conference on Learning Representations , 2024.

Galina M. Korpelevich. The extragradient method for finding saddle points and other problems. Matecon , 12:747-756, 1976.

M. A. Krasnosel'ski˘ ı. Two remarks on the method of successive approximations. Uspekhi

Matematicheskikh Nauk , 10:123-127, 1955.

Nicolas Loizou, Hugo Berard, Gauthier Gidel, Ioannis Mitliagkas, and Simon Lacoste-Julien. Stochastic gradient descent-ascent and consensus optimization for smooth games: Convergence analysis under expected co-coercivity. Advances in Neural Information Processing Systems , 34: 19095-19108, 2021.

Yucheng Lu, Wentao Guo, and Christopher De Sa. GraB: Finding provably better data permutations than random reshuffling. Advances in Neural Information Processing Systems , 35: 8969-8981, 2022.

W. Robert Mann. Mean value methods in iteration. Proceedings of the American Mathematical Society , 4(3):506-510, 1953.

Bernard Martinet. Regularisation d'inéquations variationelles par approximations succesives. Revue Française d'informatique et de Recherche opérationelle , 1970.

Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Random reshuffling: Simple analysis with vast improvements. Advances in Neural Information Processing Systems , 33: 17309-17320, 2020.

Konstantin Mishchenko, Dmitry Kovalev, Egor Shulgin, Peter Richtárik, and Yura Malitsky. Revisiting stochastic extragradient. In International Conference on Artificial Intelligence and Statistics , pages 4573-4582. PMLR, 2020.

Aleksander M ˛ adry, Aleksandar Makelov, Ludwig Schmdit, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

Aryan Mokhtari, Asuman Ozdaglar, and Sarath Pattathil. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In International Conference on Artificial Intelligence and Statistics , pages 1497-1507. PMLR, 2020.

Dheeraj Nagaraj, Prateek Jain, and Praneeth Netrapalli. SGD without replacement: Sharper rates for general smooth convex functions. In International Conference on Machine Learning , pages 4703-4711. PMLR, 2019.

Yurii Nesterov. Lectures on convex optimization , volume 137 of Springer Optimization and Its Applications . Springer, second edition, 2018.

Lam M. Nguyen, Quoc Tran-Dinh, Dzung T. Phan, Phuong Ha Nguyen, and Marten Van Dijk. A unified convergence analysis for shuffling-type gradient methods. The Journal of Machine Learning Research , 22(1):9397-9440, 2021.

Thomas Pethick, Olivier Fercoq, Puya Latafat, Panagiotis Patrinos, and Volkan Cevher. Solving stochastic weak Minty variational inequalities without increasing batch size. In International Conference on Learning Representations , 2023.

Thomas Pethick, Wanyun Xie, and Volkan Cevher. Stable nonconvex-nonconcave training via linear interpolation. Advances in Neural Information Processing Systems , 37, 2023.

L. D. Popov. A modification of the Arrow-Hurwitz method of search for saddle points. Matematicheskie Zametki , 28(5):777-784, 1980.

Shashank Rajput, Anant Gupta, and Dimitris Papailiopoulos. Closing the convergence gap of SGDwithout replacement. In International Conference on Machine Learning , pages 7964-7973. PMLR, 2020.

Shashank Rajput, Kangwook Lee, and Dimitris Papailiopoulos. Permutation-based SGD: Is random optimal? In International Conference on Learning Representations , 2022.

Benjamin Recht and Christopher Ré. Parallel stochastic gradient algorithms for large-scale matrix completion. Mathematical Programming Computation , 5(2):201-226, 2013.

Litu Rout, Alexander Korotin, and Evgeny Burnaev. Generative modeling with optimal transport maps. In International Conference on Learning Representations , 2022.

Itay Safran and Ohad Shamir. How good is SGD with random shuffling? In Conference on Learning Theory , pages 3250-3284. PMLR, 2020.

Itay Safran and Ohad Shamir. Random shuffling beats SGD only after many epochs on illconditioned problems. Advances in Neural Information Processing Systems , 34:15151-15161, 2021.

Mikhail V. Solodov and Benar F. Svaiter. A hybrid approximate extragradient-proximal point algorithm using the enlargement of a maximal monotone operator. Set-Valued Analysis , 7(4): 323-345, 1999.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020.

Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, and Mingyi Hong. Multi-agent reinforcement learning via double averaging primal-dual optimization. Advances in Neural Information Processing Systems , 31, 2018.

TaeHo Yoon and Ernest K. Ryu. Accelerated Algorithms for Smooth Convex-Concave Minimax Problems with O (1 /k 2 ) Rate on Squared Gradient Norm. In International Conference on Machine Learning , pages 12098-12109. PMLR, 2021.

Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Open problem: Can single-shuffle SGD be better than reshuffling SGD and GD? In Conference on Learning Theory , pages 4653-4658. PMLR, 2021.

Chulhee Yun, Shashank Rajput, and Suvrit Sra. Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond. In International Conference on Learning Representations , 2022.

Contents

A Pseudocode of the Algorithms

We present the pseudocode of the algorithms we consider in this paper in Algorithms 2, 3 and 4, with the pseudocode of the with-replacement stochastic methods in Algorithm 1.

Algorithm 1 SEG-US / SGDA-US

Algorithm 2 SEG-RR / SGDA-RR

Algorithm 3 SEG-FF

B Further Details and Discussions on the Related Works

B.1 A Summary of the Limitations of the Existing Works in the Monotone Setting

In Table 2, we have summarized the settings considered in each of the previous works on stochastic variants of EG discussed in Section 2, and compare them with our settings. Please note that we focus on the monotone F setting in the table. Entries that are worth further discussions are marked, with the corresponding explanations below.

( * ) The methods proposed in these works are not stochastic variants of EG in a strict sense. The method introduced by Cai et al. [9] is rather a hybrid of EG and the Halpern iteration [23], while the method by Choudhury et al. [14] is a stochastic version of the so-called optimistic gradient method [44]. Hence, determining whether these methods fall into the category of same-sample methods or not is unnecessary. Nonetheless, as these works focus on solving a similar problem to ours, we include them as references.

(†) Under the assumptions that Gorbunov et al. [20] make in their paper, one can show that each of the components must necessarily be (star-)monotone when the full F is (star-)monotone. For further explanations on why this is the case, see the following Appendix B.2.

(‡) Yet, to be precise, what Gorbunov et al. [20] have shown in the monotone case is that SEG-US can find an optimal solution if we increase the batch size each iteration. If the batch size is fixed as a constant, then they were only able to show that the iterates will be bounded in the (star-)monotone setting. In particular, they did not provide a guarantee that the iterates will be necessarily convergent.

In fact, as we demonstrate with an explicit counterexample in Appendix H.2, if we do not increase the batch size each iteration, then it is possible to show that SEG-US in the worst case will never

converge to an optimal point. This nonconvergence result in fact holds for any SEG-US whose extrapolation and update stepsizes differ by a constant factor. Hence, it not only applies to [20], but also to [17].

(§) Hsieh et al. [25] show that independent-sample SEG-US converges for stepsizes α t , β t decaying at certain different rates, but gives no convergence rates.

(¶) Mishchenko et al. [36] assume a uniformly bounded gradient variance in the strongly monotone case. In the monotone case, the bound they derived depends on the supremum of the gradient variance over the domain that is under consideration. Hence, in the monotone case, either the domain has to be (implicitly) bounded, or the uniform gradient variance assumption should be imposed.

B.2 On the Assumptions Made by Gorbunov et al. [20]

We would like to first clarify that in [20], the requirement to increase the batch size is utilized only in the monotone setting: see, e.g. , Corollary E.4 therein.

Gorbunov et al. [20] use a generalized notion of µ -strong monotonicity, namely the µ -quasi strong monotonicity , which requires the operator F to satisfy

In the notion of µ -quasi strong monotonicity they also allow µ ≤ 0 . In particular, if (14) holds with µ = 0 , then F is called a star-monotone operator. In Appendix G.1 we further discuss on star-monotone operators.

Meanwhile, let us further elaborate on why in the (star-)monotone setting, the assumptions made by the authors of [20] lead to each component being star-monotone. In their work the authors require, as equation (10) therein, that

Observe that this amounts to

However, if any of µ i is strictly negative, then the rightmost sum in (16) becomes strictly positive, hence cannot be less than or equal to µ if µ = 0 . Therefore, the only possible case is when the rightmost sum is an empty sum. In other words, (15) can hold with µ = 0 only when µ i ≥ 0 for all i , so that each F i is star-monotone. We would like to remind the readers that our analyses, on the other hand, do not have any restrictions on the individual components.

B.3 Finite Sum Structure vs. General Stochastic Setting

The works mentioned in Section 2 usually assume that we have access to a stochastic oracle that returns a stochastic estimator of F . Indeed, having a finite sum structure is a special case of having a stochastic oracle, as each F i can be seen as an estimator of F . One might then ask whether assuming the finite sum structure can help the works mentioned in Section 2 overcome the mentioned limitations. We strongly believe that this is not the case. Recall Theorem 4.1, where we have constructed an explicit counterexample that SEG-US , SEG-RR , and SEG-FF all diverge. Because the set of problems with a finite sum structure is a subset of the set of problems with a stochastic oracle, the (counter-)example in Theorem 4.1 also works as an example that displays the nonconvergence of SEG-US , SEG-RR , and SEG-FF in the general stochastic setting. That is, a variant of SEG that only modifies the stepsizes and/or the sampling scheme into a without-replacement based one will suffer from nonconvergence, due to the counterexample in Theorem 4.1. It is also true that there are some methods that cannot exactly be classified as one of SEG-US , SEG-RR , or SEG-FF , but this counterexample demonstrates that, unless explicitly proven otherwise, there is not a good reason to believe that the existing convergence analyses will be easily extended beyond the assumptions they are each based on.

B.4 On the Claimed Convergence of SEG-RR in the Monotone Setting by Emmanouilidis et al. [18]

Recently, a paper focusing on the study of SEG-RR [18] has been published. As we have briefly introduced in Table 1 with a discussion in Section 4, the authors have established a convergence rate ˜ O ( 1 / nK 2 ) of SEG-RR in the strongly monotone setting, using an independent analysis of ours.

On the other hand, the authors of [18] furthermore claim that SEG-RR is capable of finding an optimum in the monotone setting, which is seemingly contradictory to our analyses. We assert that this is not the case, as their proof, at least in their AISTATS 2024 version, seems to have a flaw.

In establishing equation (85) in [18], the authors claim that the inequality

holds by Jensen's inequality, where G ≥ 6 is a fixed constant. However, Jensen's inequality cannot be applied here, because not only ∥ F ( · ) ∥ 2 is possibly nonconvex, but also the weights multiplied to the iterates, namely 1 / KG k , do not sum up to 1 . Hence, the 'averaged' iterate is not in the form of a convex combination. So, even if ∥ F ( · ) ∥ 2 was convex, if we were to properly apply Jensen's inequality, at least the averaged iterate should be multiplied by 1 ∑ K k =0 1 /G k instead of 1 K . Yet then, the sum ∑ K k =0 1 G k ≤ G G -1 is bounded above by a constant independent of K , and the right hand side of the equation right above (85) in [18] shall no longer be divided by K . Therefore, their claimed convergence is unobtainable.

We would also like to remark that the linear decay rate of 1 /G k can make the series ∑ ∞ k =0 1 G k E [ ∥ F ( z k 0 ) ∥ 2 ] convergent even when E [ ∥ F ( z k 0 ) ∥ 2 ] grows exponentially as k → ∞ , as long as its rate of exponential growth is less than G . In particular, once their (85) is corrected, there is no contradiction with our divergence result in Theorem 4.1.

C Useful Lemmata

Lemma C.1 (Polarization identity) . For any two vectors a and b , it holds that

Proof. The identities are immediate from ∥ a ± b ∥ 2 = ∥ a ∥ 2 ± 2 ⟨ a , b ⟩ + ∥ b ∥ 2 .

Lemma C.2 (Weighted AM-GM inequality) . For any γ > 0 and two vectors a and b in R d ,

Proof. Notice that

Lemma C.3 (Young's inequality) . For any γ > 0 and two vectors a and b ,

In particular, as a special case where γ = 1 , it holds that

Proof. The left hand side of (17) is ∥ a ∥ 2 +2 ⟨ a , b ⟩ + ∥ b ∥ 2 . Applying Lemma C.2 then suffices.

Lemma C.4. For any two vectors a and b , it holds that

Proof. From (18) it follows that

Simply rearranging the terms gives us the result.

Lemma C.5 (Generalized Young's inequality) . For any nonnegative scalars p 1 , . . . , p n such that p 1 + · · · + p n = 1 and vectors a 1 , . . . , a n , it holds that

In particular, setting p 1 = · · · = p n = 1 n and multiplying both sides by n 2 yields

Proof. We use induction on n . If n = 1 then p 1 = 1 , so there is nothing to show. For the inductive step, suppose that the statement holds for some n ≥ 1 . Say we are given nonnegative scalars p 1 , . . . , p n +1 such that p 1 + · · · + p n +1 = 1 , and vectors a 1 , . . . , a n +1 . For the moment, suppose that p n +1 < 1 . Applying Lemma C.3 with γ = p n +1 1 -p n +1 and using the induction hypothesis, we get

where in last line we used that p 1 + · · · + p n = 1 -p n +1 . Now, if p n +1 = 1 , then we must have p 1 = · · · = p n = 0 , so the claimed inequality holds in this case also. This completes the proof.

Lemma C.6. Suppose that F is M -smooth. Then for any z and w it holds that

Proof. The proof closely follows the arguments used for Lemma 1.2.4 in [40], by replacing the gradients therein by saddle gradients. The fundamental theorem of calculus with the M -smoothness of F gives us

Lemma C.7. Let F be a µ -strongly monotone operator. Let z ∗ be a point such that Fz ∗ = 0 , and let η > 0 . Then, for any point z in the domain of F and w := z -η Fz , it holds that

Proof. By the µ -strong monotonicity of F and Lemma C.4 it holds that

so we are done.

The following lemma generalizes Lemma 3.2 in [21] shown for monotone F to µ -strongly monotone F with µ > 0 .

Lemma C.8. Let F be a µ -strongly monotone L -Lipschitz operator, and let z be any point in the domain of F . Then for any 0 < η < 1 L √ 2 , it holds that

Proof. For convenience, let us define w := z -η Fz and z + := z -η F ( z -η Fz ) = z -η Fw . Because F is µ -strongly monotone, we have

Also from the µ -strong monotonicity of F we get

Meanwhile, from the L -Lipschitzness of F we have

Summing up the inequalities (19), (20), (21) with weights 2 / η , 1 / 2 η , and 3 / 2 respectively, we obtain

From this inequality, we can exactly follow the arguments used in the proof of Lemma D.4 in [21] to derive that

Meanwhile, Young's inequality (Lemma C.3) tells us that

Using this to lower bound the left hand side of (22), we get that

It remains to simply rearrange the terms.

Lemma C.9. Suppose that F i is L -Lipschitz for all i = 1 , . . . , n , and that F := 1 n ∑ n i =1 F i is µ -strongly monotone with µ > 0 . Define κ := L / µ and σ 2 ∗ := 1 n ∑ n i =1 ∥ F i z ∗ ∥ 2 . Then, for any z ∈ R d 1 + d 2 it holds that

Proof. For any z , w ∈ R d 1 + d 2 , as Assumption 3.1 holds with µ > 0 , by Cauchy-Schwarz inequality

and as a consequence, ∥ z -w ∥ ≤ 1 / µ ∥ Fz -Fw ∥ . Thus, for any i ∈ [ n ] , it holds that

Summing this inequality over i = 1 , . . . , n and then dividing by n leads to

The conclusion follows from the basic inequality a 2 + b 2 ≤ ( a + b ) 2 which holds for any a, b ≥ 0 .

Lemma C.10 (Nonexpansiveness of the EG operator) . Let F be a monotone L -Lipschitz operator, and z ∗ be a point such that Fz ∗ = 0 . Then, for any point z in the domain of F and η > 0 ,

Proof. This classical result dates back to the original paper on EG by Korpelevich [29]. Here, for completeness, we replicate the proof using our notations.

Expanding the left hand side of the inequality stated, we obtain

For the first inner product term, we have

because F is monotone. For the second inner product term, we use the polarization identity (Lemma C.1) and the L -Lipschitzness of F to get

Applying these two bounds on (23) completes the proof.

Lemma C.11. Let { a k } k ≥ 0 , { b k } k ≥ 0 , { c k } k ≥ 0 , and { d k } k ≥ 0 be sequences of nonnegative numbers satisfying the recurrence relation

Then for any k ≥ 0 it holds that

Proof. Because a k ≥ 0 , it suffices to show that

as this implies

So, we show that (24) holds, by induction on k . For the base case k = 0 , the recurrence relation tells us that

which is exactly (24) when k = 0 . Now suppose that (24) holds for some k ≥ 0 . Using the induction hypothesis and the recurrence relation we get

This shows that (24) holds also for k +1 , and we are done.

The subsequent lemma is technical, but it can be derived from elementary calculus.

Proof. Consider the function h ( x ) := 1 x 2 / 3 (log x ) 2 over the interval [2 , K +3] . As

h is decreasing. Hence, an upper Riemann sum becomes an upper bound for the integral, so we have

Now consider a function g : [1 , ∞ ) → R , defined as

Differentiating, we get

whenever y ≥ 1 . That is, g is increasing on y ≥ 1 . We then show that g (1) ≥ 0 . To this end, let us begin with observing that

from which we get that h is convex. In particular, it holds that

and similarly, ∫ 4 3 h ( x ) d x ≥ h ( 7 / 2 ) . Thus we indeed have

Recalling that g is increasing, we have g ( K ) ≥ g (1) ≥ 0 for all K ≥ 1 . This, with (25), implies that

holds whenever K ≥ 1 , which is exactly the claimed.

D Missing Proofs for Section 5

D.1 Unravelling the Recurrence of the Generalized SEG in (8) and (12)

In Section 5.1, we considered the method where, in a single epoch (hence omitting all superscripts that are used to denote the epoch number for convenience), the iterates are generated following the recurrence

for i = 0 , 1 , . . . , N -1 , where each T i are sampled from the set { F 1 , . . . , F n } , and an additional anchoring step

is performed so that z ♯ is used as the initial point of the next epoch. Notice that (27) is a generalized anchoring step that incorporates all the settings we are considering, as the versions of SEG where anchoring is not used correspond to taking θ = 0 , and the anchoring step (12) that is used in SEGFFA corresponds to taking θ = 1 . In this section we would like to prove the following statement regarding this update rule.

Proposition D.1 (Proposition 5.1) . It holds that

for some ϵ N = o ( ( α + β ) 2 ) .

Proof. Equation (28) immediately follows from Proposition D.2, with (30) giving us the precise definition of ϵ N . To show that ϵ N = o ( ( α + β ) 2 ) , we begin with noting that both ∥ z j -z 0 ∥ and ∥ w j -z 0 ∥ are of O ( α + β ) , because both z j and w j are obtained from z 0 by performing at most j updates following (26). Thus, the first term in the right hand side of (30) is of O ( β ( α + β ) 2 ) by Lemma C.6, and the remaining terms are of O (( α + β ) 3 ) by the L -smoothness of the operators F 1 , . . . , F n .

Proposition D.2. For any i = 0 , 1 , . . . , N , it holds that

where we denote

Proof. We use induction on i . There is nothing to show for the base case i = 0 . Now, suppose that (29) and (30) hold for some i < N , and write

Here, notice that by the update rule we have

Using this identity and the induction hypothesis we get

which asserts that (29) also holds for i +1 .

D.2 Insufficiency of Only Using Flip-Flop Sampling

Here we prove the following.

Proposition D.3 (Proposition 5.2) . Suppose we use flip-flop sampling only. In order to make (10) and (11) hold, we must choose β = η 1 / n and α = β / 2 . However, this leads to η 2 = 2 η 1 , which is the set of parameters that fails to make EG+ converge.

Proof. Suppose that we have already established the upcoming Lemma D.4. Then, we can see by setting θ = 0 in the result of Lemma D.4 that for (11) to hold, the following system of equations should be satisfied:

Solving this system of equations, we get η 1 = nβ , η 2 = 2 nβ , and α = β / 2 .

For the latter part of the statement on the divergence of EG+ with η 2 = 2 η 1 , consider the (1 + 1) -dimensional bilinear problem

whose unique optimum is z ∗ = (0 , 0) . A simple computation shows that

Consequently, for any η > 0 , the update rule of EG+ with η 1 = η and η 2 = 2 η amounts to

It follows that

Therefore, the distance from the optimal solution strictly increases every iterate.

It remains to actually prove Lemma D.4.

Lemma D.4. When flip-flop sampling is used with the generalized anchoring step (27) , it holds that

̸

Proof. As we are using flip-flop sampling, we have N = 2 n , and it is clear that

For the second term, as T i = T 2 n -1 -i , we have

The claimed identity can be obtained by taking the weighted sum of the two results.

E Within-Epoch Error Analysis for Upper Bounds

All the upper bounds for SEG-RR , SEG-FF , and SEG-FFA in this paper are established by following the two steps below.

The first step is to decompose the cumulative updates made within an epoch by using the method into a sum of an exact EG update and a within-epoch error term, which we denote by r k . In particular, we show that the error term r k occurring from any of SEG-RR , SEG-FF , and SEG-FFA can be expressed in a specific unified form (described in Theorem E.1). This will be the main focus of this section.

The second step is establishing a convergence rate that can be applied to any method whose update can be decomposed into a sum of an exact EG update and an error term that is of the specific unified form mentioned above. By doing so, the convergence rates of SEG-RR , SEG-FF , and SEG-FFA will automatically follow as special cases of the general convergence result. This step will be dealt in Appendices F and G.

To this end, for any of SEG-RR , SEG-FF , and SEG-FFA , let us decompose the cumulative updates made within an epoch into a sum of an exact EG update and a within-epoch error term r k , as

The quality of the method will depend on how small the 'noise' term r k is, as the noise will in general hinder the convergence. As mentioned above, it turns out that, regardless of the method that is in use, the noise term can be bounded in a unified format, as follows.

Theorem E.1. Suppose that Assumptions 3.3 and 3.4 hold. Then, for each of SEG-RR , SEG-FF , and SEG-FFA , there exists a choice of stepsizes that makes the following hold: for an exponent a that depends on the method, there exist constants C 1 , D 1 , V 1 , C 2 , D 2 , and V 2 , all independent of η k and n , such that the error term r k satisfies a deterministic bound

and a bound that holds on expectation

Furthermore, the exponent is a = 2 for SEG-RR and SEG-FF , and a = 3 for SEG-FFA .

In other words, SEG-FFA has an error that is an order of magnitude smaller than other methods. Thus, it is now intuitively clear that SEG-FFA should have an advantage in the convergence. The proof of Theorem E.1 is quite long and technical, so we defer it to Appendix E.2.

Within the remaining of this section only, although it is an abuse of notation, for convenience we will write F i to denote the saddle gradient of the component function chosen in the i th iteration. More precisely, for indices i = 0 , 1 , . . . , n -1 we denote F τ ( i +1) by F i . Similarly, in cases of considering SEG-FF or SEG-FFA , for i ≥ n we denote F τ (2 n -i ) by F i . Also, we omit the superscripts and subscripts denoting the epoch number k unless strictly necessary, as all the iterates that we consider will be from the same epoch.

Let us reformulate the update rule (8) into

Note that ξ = 1 / 2 for SEG-FFA , and ξ = 1 for SEG-RR and SEG-FF .

E.1 Auxiliary Lemmata

For j = 1 , . . . , 2 n we define

We set Σ 0 = Ψ 0 = 0 , as they are empty sums. Notice that δ j is a random variable that depends on the permutation τ .

Meanwhile, by triangle inequality it is immediate that

and by Young's inequality it holds that

Lemma E.2. For any index i ≥ 1 , it holds that

Proof. By the fundamental theorem of calculus for line integrals and the update rule (33), we have

and similarly

Hence, by defining

the update rule can be rewritten using these quantities as

Subtracting z 0 from both sides of (41) we get

and plugging this into (42) gives us

For convenience let us define

and P i,i := I as it denotes an empty product. Observe that for any j we have

Also note that for any ℓ it holds that and hence

Unravelling the recurrence relation (44) we get

and therefore

In order to compute the bound for ∥ z i -z 0 ∥ , we use summation by parts to get

Here, observe that

so by using (45) and (46) we obtain

Therefore, we conclude that

Meanwhile, substituting (47) back to (43) gives us

For ℓ < i let us define

and for convenience R i,i := I so that (48) can be rewritten as

Applying summation by parts on the above, we obtain

and as a consequence we get

It remains to bound ∥ R i,ℓ +1 -R i,ℓ ∥ . For the special case where ℓ = i -1 , a direct computation leads to and thus we have

For the other cases; that is, when ℓ < i -1 , we have

Proposition E.3. Suppose that SEG-FFA is used, η < 1 nL , and let ν := 1 + 1 2 n . Then for any i = 1 , . . . , 2 n -1 we have the bounds

Proof. Using elementary calculus one can show that x ↦→ (1 + 1 x + 1 2 x 2 ) x increases on x > 0 and is bounded above by e . Hence for all 0 ≤ ℓ < i ≤ 2 n we have

Applying the definitions (35) and (36) on (38) and then substituting ξ = 1 / 2 we get

Similarly, from (39) we get

Finally, applying generalized Young's inequality on (39) we get

Using generalized Young's inequality once more on the last term gives us

Plugging this back yields

Now the claimed inequalities can be obtained simply by rearranging the terms appropriately.

Proposition E.4. Suppose that either SEG-RR or SEG-FF is used with α = β = η < 1 nL , and let ˜ ν := 1 + 1 n . Then for any i = 1 , . . . , 2 n -1 we have the bounds

Proof. One can verify that x ↦→ (1 + 4 3 x ) x increases on x ≥ 3 and is bounded above by e 4 / 3 < 4 . With noting that (1 + 1 1 + 1 1 2 ) 1 = 3 < 4 , (1 + 1 2 + 1 2 2 ) 2 = 49 16 < 4 , and 1 + 1 x + 1 x 2 ≤ 1 + 4 3 x whenever x ≥ 3 , we see that for all 0 ≤ ℓ < i ≤ 2 n it holds that

Also, we have

Applying the definitions (35) and (36) on (38) and then substituting ξ = 1 we get

Similarly, from (39) we get

Finally, applying Young's inequality on (39) we get

Using Young's inequality once more on the last term gives us

Plugging this back yields

Now the claimed inequalities can be obtained simply by rearranging the terms appropriately.

Let us now derive the upper bounds for the quantities related to δ j and Σ j , defined in (35) and (36) respectively, using the upper bound of the variance of saddle gradients (4).

Lemma E.5. For any j = 1 , . . . , 2 n , it deterministically holds that

Proof. For any set of indices J ⊂ { 0 , . . . , n -1 } , by Assumption 3.4 it holds that

Hence, for any j = 1 , . . . , n we have

and for any j = n +1 , . . . , 2 n we have

Therefore, in any case we have

Taking square roots on both sides gives us the desired bound.

Lemma E.6. For any j = 1 , . . . , 2 n , it holds that

Proof. If n = 1 then the left hand side is always 0 , so there is nothing to show. So, we may assume that n ≥ 2 . Then, for any j = 1 , . . . , n , using Lemma 1 in [35] we obtain

Multiplying both sides by j 2 and applying AM-GM inequality leads to

Meanwhile, for j = n +1 , . . . , 2 n , following the first few steps in (54) we get

Here, once more applying Lemma 1 of [35], we get

Using AM-GM inequality on the last line gives us

E τ ∥ g j -j Fz 0 ∥ 2 ≤ ( j -n )(2 n -j ) n -1 ( ρ ∥ Fz 0 ∥ + σ ) 2 ≤ n 2 4( n -1) ( ρ ∥ Fz 0 ∥ + σ ) 2 ≤ n 2 ( ρ ∥ Fz 0 ∥ + σ ) 2 . Thus, for any case, we have (55).

Lemma E.7. For any k, ℓ ∈ { 0 , 1 , . . . , 2 n } , it holds that

Proof. Expanding the product Σ k Σ ℓ and writing in terms of δ , we get

where the last line follows from the AM-GM inequality. Taking the expectation with respect to τ and using the bound from Lemma E.6, we obtain

which is exactly the claimed.

Lemma E.8. For any k, ℓ ∈ { 0 , 1 , . . . , 2 n } , it holds that

Proof. Expanding the product in the left hand side of (57) and applying (56), we get

E.2 Upper Bounds of the Within-Epoch Errors

The full proof of Theorem E.1 is quite long and technical, so we divide it into several parts. First we show that (31) and (32) holds with a = 3 when SEG-FFA is in use. Then we show that Theorem E.1 also holds for SEG-FF in Appendix E.2.3, and for SEG-RR in Appendix E.2.4.

Throughout the remaining of this section, we always assume that the variance of the saddle gradients satisfies (4).

E.2.1 Proof of Equation (31) for SEG-FFA

In this section we prove the following.

Theorem E.9. Say we use SEG-FFA . Then, as long as the stepsize used in an epoch satisfies η < 1 nL , it holds that

for constants where we denote

We first list the intermediate results. The actual proof of Theorem E.9 is in page 43, at the end of this section.

Proposition E.10. For using SEG-FFA , the within-epoch update z ♯ as given by (12) satisfies

Proof. Setting α = η / 2 , β = η , and θ = 1 in (28), we get

where ϵ 2 n is defined as in (30). Recall that F i = F 2 n -1 -i for all i = 0 , 1 , . . . , 2 n -1 , and moreover, ∑ n -1 i =0 F i = ∑ 2 n -1 i = n F i = n F . Thus, the first sum in the above is equal to 2 n Fz 0 , and the second sum is equal to 2 ∑ n -1 j =0 D F j ( z 0 ) F j z 0 . For the last sum, observe that

̸

Hence, (63) is equivalent to

̸

Observing that the terms (62b), (62c), and (62d) add up to 1 2 ϵ 2 n completes the proof.

Proposition E.11. Suppose that η < 1 nL , and let ν := 1 + 1 2 n . Then the noise term satisfies the bound

Proof. We bound each line in equation (62). For (62a), we use Lemma C.6 to get

In bounding the remaining three lines we repeatedly use the bounds obtained in Proposition E.3. We will also use the following bounds, which follows from (33), (35), and Young's inequality:

For (62b), observe that Lemma C.6 gives us

Thus, by using the bound obtained in Proposition E.3, we get

where along the derivation we used the inequality

which holds for all n ≥ 1 . From now on, we will keep on using similar techniques to reduce the exponents of ν , without explicitly stating the inequalities used, but recovering the inequalities that are used should be clear from context.

For (62c), we use L -smoothness of F j , and also the fact that it implies ∥ D F j ( z 0 ) ∥ ≤ L , to get

By the same logic, each summand in (62d) with j > 0 can be bounded as

and when j = 0 the sum with respect to k becomes an empty sum. Thus, (62d) in total satisfies the bound

Simply collecting all the inequalities and rearranging the terms leads to the claimed bound.

Before we proceed, let us write

so that the bound on ∥ r ∥ obtained in Proposition E.11 can be written as

Theorem E.12. Suppose that η < 1 nL , and let ν := 1 + 1 2 n . Then the noise term deterministically satisfies the bound

Proof. From (36), (37), and Lemma E.5, it holds that

Plugging the bound for Ψ j into (64) we get

By Young's inequality, it holds that

from which we get

Meanwhile, plugging the bound for Σ j into (65) we get

where in the last line we used that η < 1 nL . Because the inequality

holds for all n ≥ 1 , continuing from above we obtain

Rearranging (66) with applying the bounds (69) and (70) gives us the claimed result.

Proof of Theorem E.9. As n ≥ 1 , we notice that 1 / n ≤ 1 and ν ≤ 3 / 2 where ν = 1 + 1 2 n following the notation of Theorem E.12. Then the bound (58) is immediate from Theorem E.12.

E.2.2 Proof of Equation (32) for SEG-FFA

In this section, we prove the following.

Theorem E.13. Say we use SEG-FFA . Then, as long as the stepsize used in an epoch satisfies η < 1 nL , it holds that

for constants

Proof. The bound is then immediate from the following Theorem E.14, as n ≥ 1 implies 1 / n ≤ 1 and ν ≤ 3 / 2 for ν defined in the statement of Theorem E.14.

Theorem E.14. Suppose that η < 1 nL , and let ν := 1 + 1 2 n . Then, in expectation , the noise term satisfies the bound

Proof. Notice that, when conditioned on z 0 , the only source of randomness included in Ψ j is the random permutation τ selected for the epoch. Hence, we can use Lemma E.6 to get

Applying Young's inequality on (66) we get

When conditioned on z 0 , the first two lines are not random quantities. Thus, it suffices to derive the bounds for E [ X 2 i ∣ ∣ z 0 ] , i = 1 , 2 .

Recall that the bound (69) on X 1 holds deterministically . Hence, it holds that

Now, to compute E [ X 1 | z 0 ] , we apply the linearity of expectation on (64) to get

Young's inequality gives us the bound

which, with the inequality derived above, leads to

As a consequence, with using Young's inequality once again, we obtain

To get the bound of E [ X 2 2 ∣ ∣ z 0 ] , we begin by using

which holds for all 1 ≤ k ≤ 2 n -2 , to (65) to obtain

Then we directly square both sides and expand them to get

Here, using Lemma E.7 and Lemma E.8 on the right hand side leads to

As a consequence, with using (76) once again, we obtain

Taking the conditional expectation on (75), applying the bounds (77) and (78), and then rearranging the terms leads to the claimed inequality.

E.2.3 Upper Bounds of the Within-Epoch Errors for SEG-FF

Theorem E.15. Say we use SEG-FF with α = β = η / 2 . Then, as long as the stepsize used in an epoch satisfies η < 1 nL , it holds that

for constants C 1F , D 1F , V 1F , C 2F , D 2F , and V 2F to be determined later in (81) and (82) .

Proof. As we have discussed in Section 5.1, we already know that aiming to achieve O ( η 3 ) error without anchoring is futile. Instead, we show that error of magnitude O ( η 2 ) is possible with the chosen stepsizes.

By Proposition D.2 and Lemma D.4 we have For any i = 0 , 1 , . . . , N , it holds that

̸

̸

where we denote

Comparing z 2 n to a point that would have been the result of a deterministic EG update with stepsize ηn we get

̸

Let us define

Noticing the resemblence between (62) and the equations in (79) and (80), we can repeat the same reasoning used for Theorem E.9 and Theorem E.13, but with replacing the bounds given by Proposition E.3 to those in Proposition E.4 (and plugging in η / 2 in place of η in the statement of Proposition E.4) to conclude that

for some constants ˜ C 1A , ˜ D 1A , ˜ V 1A , ˜ C 2A , ˜ D 2A , and ˜ V 2A . Meanwhile, we also have

̸

where in the second to the last line we used the Cauchy-Schwarz inequality. Therefore, as η ≤ 1 / nL , we conclude that

for constants

Moreover, using Young's inequality, we see that

̸

so we also conclude that

[

E

∥

z

2

n

-

(

z

0

-

ηn

F

(

holds for constants

E.2.4 Upper Bounds of the Within-Epoch Errors for SEG-RR

Theorem E.16. Say we use SEG-RR with α = β = η . Then, as long as the stepsize used in an epoch satisfies η < 1 nL , it holds that

for constants C 1R , D 1R , V 1R , C 2R , D 2R , and V 2R to be determined later in (86) and (87) .

Proof. As we have discussed in Section 5.1, we already know that aiming to achieve O ( η 3 ) error with only using random reshuffling is futile. Instead, we show that error of magnitude O ( η 2 ) is possible with the chosen stepsizes.

By Proposition D.2 and Lemma D.4 we have For any i = 0 , 1 , . . . , N , it holds that

where we denote

Comparing z n to a point that would have been the result of a deterministic EG update with stepsize ηn we get

Let us define

Comparing the sums (62b)-(62d) to (83), we can repeat the same reasoning used for Theorem E.9 and Theorem E.13, but with replacing the bounds given by Proposition E.3 to those in Proposition E.4, to conclude that

z

0

-

ηn

Fz

0

))

∥

2

∣

∣

∣

z

0

]

η

4

n

4

C

2F

∥

Fz

0

∥

2

+

η

4

n

4

D

2F

∥

Fz

0

∥

4

+

η

4

3

n

V

2F

for some constants ˇ C 1A , ˇ D 1A , ˇ V 1A , ˇ C 2A , ˇ D 2A , and ˇ V 2A . Meanwhile, we also have

which leads to

Therefore, from η ≤ 1 / nL and Lemma E.5, on one hand we obtain

for constants

On the other hand, applying Young's inequality on (85) we get

Taking the expectation conditioned on z 0 and applying Lemma E.6, we conclude that

holds for constants

F Convergence Bounds in the Strongly Monotone Setting

In this section, we focus only on the iterates { z k 0 } k ≥ 0 . So, we omit the subscript 0 unless necessary, and simply write z k instead of z k 0 .

F.1 Unified Analysis of the Upper Bounds for Shuffling-Based SEG Methods

When F is µ -strongly monotone with µ > 0 , all of SEG-RR , SEG-FF , and SEG-FFA do not diverge. In fact, it is possible to establish the following unified analysis of the methods.

Theorem F.1 (Theorem F.5, simplified) . Suppose that F is µ -strongly monotone with µ > 0 , Assumption 3.3 holds, and an optimization method whose within-epoch error satisfies (31) and (32) for some constant a > 0 is run for K epochs. Then, for a sufficiently small constant ω that does not depend on K , we achieve the bound

The goal of this section is to prove this theorem, whose precise statement is in Theorem F.5. As the polynomial decay will dominate the exponential decay for large enough K , the bound we get is essentially ˜ O ( 1 / nK 2 a -2 ) . Recall that for SEG-FF and SEG-RR we have a = 2 (by Theorems E.15 and E.16) which leads to an upper bound of ˜ O ( 1 / nK 2 ) , whereas for SEG-FFA we have a = 3 (by Theorems E.9 and E.13) which gives an upper bound of ˜ O ( 1 / nK 4 ) .

As also mentioned in the beginning of Appendix E, for any of SEG-RR , SEG-FF , and SEG-FFA , we can decompose the update across the epoch into a deterministic EG update plus a noise. In this section, letting w k † := z k -η k n Fz k , we define ̂ F k by the relation η k n ̂ F k = η k n Fw k † + r k so that

Proposition F.2. Let F be µ -strongly monotone with µ > 0 . Then, for any η k > 0 , it holds that

Proof. From (88), using Lemma C.7 we get

Meanwhile, using the polarization identity (Lemma C.1) and the L -smoothness of F we get

Combining the two inequalities and using the definition of ̂ F we obtain

Let us consider the inner product term in the last line above. By Lemma C.2 and the nonexpansiveness of the EG update (Lemma C.10), for any γ k > 0 we have

Plugging this back we get

Choosing γ k = µη k n 2 completes the proof.

Proposition F.3. Let F be a µ -strongly monotone and L -Lipschitz operator. Then, whenever η k ≤ 1 nL √ 2 , it holds that

Proof. Let z k +1 † := z k -η k n F ( z k -η k n Fz k ) , so that we have ∥ ∥ ∥ z k +1 -z k +1 † ∥ ∥ ∥ = ∥ ∥ r k ∥ ∥ . Then, the L -smoothness of F and Lemma C.8 implies

where in the last line we apply a simple inequality 1 -2 x ≤ (1 -x ) 2 which holds for all x ∈ R .

Lemma F.4. Suppose that (31) holds. Say we use a constant stepsize η k = η , where η satisfies η ≤ 1 nL √ 2 and

Then for any k = 0 , 1 , . . . , the following two inequalities both hold:

Proof. For the case k = 0 , the inequality (93) clearly holds. For the remaining cases, we use strong induction on k . More precisely, assuming that (93) holds for all 0 , 1 , . . . , k , we will show that (92) holds, and from that the inequality

follows. To this end, let us begin from noting that Proposition F.3, (31), and the induction hypothesis (93) implies

Here, from the choice of the stepsize (91), we have

Hence, from (95) we get

which is exactly (92). Now, considering that we are assuming (93) holds for all 0 , 1 , . . . , k , we must also have (92) for all 0 , 1 , . . . , k . Thus we can unravel the recurrence to get

As (91) also implies 10 η a -1 n a -1 L ≤ 1 / L , we obtain (94), as claimed. This completes the proof.

Theorem F.5 (Theorem F.1) . Suppose that F is µ -strongly monotone with µ > 0 , Assumption 3.3 holds, and an optimization method whose within-epoch error satisfies (31) and (32) for some constant a > 0 is run for K epochs. Let us define a constant

Say we use a constant stepsize η k = η , where η is chosen as

Then for ω denoting the minimum among (97a) -(97d) , it holds that

As a reminder, for SEG-FF and SEG-RR we have a = 2 , and for SEG-FFA we have a = 3 .

Proof. Notice that (97b) and (97c) together implies (91), and that η k = η ≤ 2 5 nL ≤ 1 nL √ 2 < 1 nL . So, we can utilize (32) and Lemma F.4 to get

Taking the conditional expectation on (89) and applying the bound just derived, we obtain

A simple rearrangement of the terms leads to

Notice that by assuming (97a) and (97d), it holds that

so we can guarantee that the left hand side of (99) is nonnegative. It then follows that

Applying the law of total expectation, from the above we obtain

We can now unravel this recurrence over k = 0 , 1 , . . . , K -1 as done in (96) to get

where in the last line we used the basic inequality 1 + x ≤ e x which holds for all x ∈ R . With the choice of the stepsize (97e), we arrive at

Now, recall that η is chosen to be the smallest one among (97a)-(97e). Notice that the options (97a)-(97d) are independent with respect to K , and (97e) is the only one that depends on K . Let us consider these two cases separately.

(i) η is chosen to be the minimum among (97a)-(97d).

This is the case where we have η = ω . Notice that the constant ω that does not depend on K . The inequality (100) then takes the form

(ii) η is chosen to be (97e), that is, η = 4( a -1) log( n 1 / (2 a -2) K )

µnK .

In this case, the exponential factor of the first term in the right hand side of (100) reduces to

Thus, the second term in (100) dominates the first term, and in total (100) becomes

Therefore, in both cases we have

which is exactly (98). This completes the proof.

Remark F.6 . To compare the convergence rate of SEG-FFA in the strongly monotone setting with that of SEG-RR by Emmanouilidis et al. [18] more in depth, let us make an estimation on the size of ω appearing in Theorem F.5 when a = 3 .

To this end, we need estimates on the constants C 1A , D 1A , V 1A , C 2A , and D 2A . From their definitions in (59)-(61), (72), and (73) we have C 1A ≍ L 2 , D 1A ≍ M , V 1A ≍ M + L 2 , C 2A ≍ L 4 , and D 2A ≍ M 2 . In general, there is not a direct relation between L and M . For example, recall that if all components are quadratic, then M = 0 . Meanwhile, Gorbunov et al. [21] has argued that M can be much larger than L in certain cases, by providing an example where M ≍ L 3 / 2 . For our purposes, however, let us allow M to be even as large as M ≍ L 2 , so that the situation is simplified into C 1A ≍ D 1A ≍ V 1A ≍ L 2 and C 2A ≍ D 2A ≍ L 4 .

Then, we get the estimate of (97c),

Meanwhile, as for the constant Φ it holds that

As (97a) while (97b) are both Θ( 1 / nL ) and µ ≤ L , we essentially have ω ≍ µ / nL 2 . Or equivalently, for some b = Θ(1) , the convergence rate (98) reads

On the other hand, Theorem 2.1 of [18] states that, for some b ′ = Θ(1) , SEG-RR exhibits a rate of

Comparing (101) with (102), the exponents in the exponentially decaying term are of the same order of -µ 2 K L 2 , so SEG-FFA having a faster polynomially decaying term ˜ O ( 1 / nK 4 ) enjoys an improved convergence rate.

for (97d) we have

G Convergence Rate of SEG-FFA in the Monotone Setting

G.1 Star-monotonicity

Notice that we only used Assumptions 3.3 and 3.4 in deriving the results in Appendices D and E, and in particular, the monotonicity assumption on F was not necessary. Moreover, among the lemmata listed in Appendix C, Lemma C.10 is the only one that possibly uses the (non-strongly) monotone assumption, but that lemma is not used in this section.

In fact, as it turns out in Appendix G.2, in the convergence analysis of SEG-FFA , we need not fully exploit the inequality (3) provided by the monotonicity assumption. Rather, all the results on the performance of SEG-FFA can be established with only assuming the following condition (which has been also briefly mentioned in Appendix B.2).

Assumption G.1 (Star-monotonicity) . Given an operator F with a point z ∗ ∈ R d 1 + d 2 such that Fz ∗ = 0 , we say that F is star-monotone if, for any z ∈ R d 1 + d 2 , it holds that

Monotone and strongly-monotone operators are clearly star-monotone, as they satisfy (3). On the other hand, there exist operators that are star-monotone but not monotone: see, e.g. , [31, Appendix A.6].

Recall that when F is monotone, Assumption 3.2 is equivalent to assuming the existence of a point z ∗ that satisfies Fz ∗ = 0 . Hence, after simply replacing the optimality condition in Assumption 3.2 with Fz ∗ = 0 , our convergence analyses not only will show that our SEG-FFA finds an optimum on monotone problems, but also that it can be also used to find stationary points in 'star-monotone' problems, allowing the objective function f to be nonconvex-nonconcave.

Star-monotonicity is also known as the variational stability condition [25], and has much been studied in the literature. For further details on star-monotonicity, we refer to [25, 31] and the references therein.

G.2 Convergence Analysis of SEG-FFA in the (Star-)Monotone Setting

Let us in particular consider SEG-FFA . As in the previous section, we focus only on the iterates { z k 0 } k ≥ 0 , so again, we omit the subscript 0 unless necessary, and simply write z k instead of z k 0 .

Decompose the update across the epoch into a deterministic EG update plus a noise, as

Lemma G.2. Let F be a (star-)monotone operator with a point z ∗ that satisfies Fz ∗ = 0 , and suppose that Assumption 3.3 holds. Then for any η k > 0 and γ k > 0 , it holds that

Proof. By (104) and (105) we get

for ̂ F k defined by the equation

We now bound the inner products. On one hand, by the polarization identity (Lemma C.1) and the L -smoothness of f , we have

On the other hand, by the weighted AM-GM inequality (Lemma C.2), for any number a k ∈ (0 , 1) it holds that

Using these two bounds, we get

Choosing a k = γ k 1+ γ k and rearranging the terms, we obtain

The left hand side of (107) is nonnegative by the star-monotonicity of F (103), and the claimed inequality follows.

Now we show that choosing the appropriate stepsizes leads to ∥ ∥ Fz k ∥ ∥ being bounded uniformly over k .

Proposition G.3. Let F be a (star-)monotone operator with a point z ∗ that satisfies Fz ∗ = 0 , and suppose that Assumptions 3.3 and 3.4 hold. Say we are using SEG-FFA , or any optimization method whose within-epoch error satisfies (58) and (71) . Let the sequence of stepsizes { η k } k ≥ 0 be nonincreasing, with

Suppose that initial stepsize η 0 is chosen sufficiently small so that

for constants C 1A , D 1A , and V 1A defined in (59) -(61) . Then for all k ≥ 0 ,

Proof. We use induction on k , to establish a stronger inequality

To see that (111) indeed implies (110), notice that by the L -smoothness of f it holds that

For the case when k = 0 , as S > 0 , it is clear that (111) holds. Now suppose that (111) holds for some k ≥ 0 . Applying Young's inequality on (31) leads to

Applying this bound on ∥ ∥ r k ∥ ∥ 2 on (106), we obtain

Choose γ k = η 3 k n 3 L 3 . Notice that (109) implies η 0 nL ≤ 1 , henceforth η k ≤ η 0 ≤ 1 / nL . This, with the induction hypothesis (110), implies

That is, the left hand side of (112) becomes nonnegative. Then it is immediate from (112) that

Using Lemma C.11 to unravel this recurrence relation, we obtain

which shows that (111) also holds when k is replaced by k +1 . This completes the proof.

Theorem G.4 (Formal version of Theorem 5.4) . Let F be a (star-)monotone operator with a point z ∗ that satisfies Fz ∗ = 0 , and suppose that Assumptions 3.3 and 3.4 hold. Say that we are using SEG-FFA , or any optimization method whose within-epoch error satisfies (58) and (71) , with β k = η k = η 0 3 √ 2 log 2 ( k +2) 1 / 3 log( k +2) and α k = β k / 2 for k = 0 , 1 , . . . , where, for S := ∑ ∞ k =0 η 3 k n 3 L 3 , the initial stepsize η 0 is chosen so that

for constants C 1A , D 1A , and V 1A defined in (59) -(61) , and there exists a positive constant λ > 0 such that

for constants C 2A , D 2A , and V 2A defined in (72) -(74) . Then for any K ≥ 1 , it holds that

Proof. As the sequence of stepsizes { η k } k ≥ 0 is nonincreasing and (113) asserts that η 0 ≤ 1 / nL , we can use the bounds established in Theorem E.9 and Theorem E.13. Also, the premises required for Proposition G.3 are also satisfied, so the bound (110) holds.

Setting γ k = η 3 k n 3 L 3 in (106) and then taking the conditional expectation given z k , with using (71) and (114), we obtain

By the law of total expectation, and that γ k = η 3 k n 3 L 3 < 1 , from the above we get

This recurrence can be unraveled using Lemma C.11, giving us

For the left hand side of (116), we have

From Lemma C.12, we know that whenever K ≥ 1 ,

Meanwhile, as x ↦→ 2(log 2) 3 ( x +2)(log( x +2)) 3 is a decreasing function, we have

and thus

Thus, for the right hand side of (116), it holds that

Therefore, from (116) we get

Simply rearranging the terms gives us the desired inequality.

Remark G.5 . While η 0 should be chosen so that both (113) and (114) hold, in practice, there is a way to circumvent this complication. Notice that in deriving the upper bound (118) of the right hand side of (116), it suffices to have η k ≤ η 0 3 √ 2 log 2 ( k +2) 1 / 3 log( k +2) , and the lower bound (117) of the left hand side holds for any η k ≥ 0 . In other words, if we have had chosen η k = Θ( 1 / ( k +1) q ) for q > 1 3 so that S = ∑ ∞ k =0 η 3 k n 3 L 3 < ∞ , as long as η 0 satisfies (113) and (114), we would still have obtained the inequality

In particular, if we additionally assume that q < 1 2 then

so from (119) we would have obtained the convergence rate

We now claim that, if one accepts a slight sacrifice of the convergence rate from ˜ O ( 1 / K 1 / 3 ) to O ( 1 / K 1 -2 q ) for 1 / 3 < q < 1 / 2 , one can simply choose the stepsizes as η k = η 00 / ( k +1) q for a sufficiently small η 00 . To see why this is the case, let us fix η 0 to be a number that satisfies the inequalities (113) and (114). Then, because η 00 / ( k +1) q = o ( 1 ( k +2) 1 / 3 log( k +2) ) , there will exist a nonnegative integer k 0 such that η k ≤ η 0 3 √ 2 log 2 ( k +2) 1 / 3 log( k +2) for all k ≥ k 0 . So, by ignoring the first k 0 terms if necessary-that is, considering as if the k 0 th iteration is the 0 th iteration-it follows from the discussions made above in obtaining (120) that we get the rate of convergence O ( 1 / K 1 -2 q ) .

This discussion also justifies the choice of stepsizes η k = Θ( 1 / (1+ k/ 10) 0 . 34 ) used in the experiments for the monotone setting.

H Proof of Lower Bounds

H.1 Proof of the Divergence of SEG-US , SEG-RR and SEG-FF

We prove the divergence of SEG-US , SEG-RR and SEG-FF in each proposition below, using the same worst-case problem for n = 2 . These constitute the proof of Theorem 4.1.

Proposition H.1 (Part of Theorem 4.1) . For n = 2 , there exists a convex-concave minimax problem f ( x, y ) = 1 2 ∑ 2 i =1 f i ( x, y ) having a monotone F , consisting of L -smooth quadratic f i 's satisfying Assumption 3.4 with ( ρ, σ ) = (1 , 0) such that SEG-US diverges in expectation for any choice of stepsizes { α t } t ≥ 0 and { β t } t ≥ 0 . That is, for all t ≥ 0 ,

Proof. We consider the case of

which result in a bilinear (and hence convex-concave) objective function

One can quickly check from the definitions of the component functions f 1 and f 2 that the corresponding saddle gradient operators are given as

where z = ( x, y ) ∈ R 2 . From the fact that ∥ A i ∥ ≤ L for all i 's, we can confirm that f i 's are indeed L -smooth. As for Assumption 3.4, we can verify that

thus proving that our example f indeed satisfies Assumption 3.4 with ( ρ, σ ) = (1 , 0) .

We now proceed to show that for this particular worst-case example f , SEG-US diverges in expectation. For t ≥ 0 , the ( t +1) -th iteration of SEG-US starts at z t , and the algorithm uniformly chooses an index i ( t ) from [ n ] . The algorithm then makes an update

In our worst-case example f , the updates can be compactly written as

Since we have n = 2 , the update can be summarized as

By the definition of A 1 and A 2 and using A 2 1 = A 2 2 = 0 , we can verify that

From this, we notice that the expectation of ∥ z t +1 ∥ 2 conditional on z t reads

Working out the calculations, we can check that

Since this holds for all t ≥ 0 , SEG-US diverges in expectation, for any positive stepsizes { α t } t ≥ 0 and { β t } t ≥ 0 . The statement on ∥ Fz t ∥ follows by realizing that ∥ Fz ∥ = L 2 ∥ z ∥ .

Proposition H.2 (Part of Theorem 4.1) . For n = 2 , there exists a convex-concave minimax problem f ( x, y ) = 1 2 ∑ 2 i =1 f i ( x, y ) having a monotone F , consisting of L -smooth quadratic f i 's satisfying Assumption 3.4 with ( ρ, σ ) = (1 , 0) such that SEG-RR diverges in expectation for any choice of stepsizes { α k } k ≥ 0 and { β k } k ≥ 0 . That is, for any k ≥ 0 ,

Proof. The proof uses the same example as Proposition H.1, outlined in (121). We show that for this particular worst-case example f , SEG-RR diverges in expectation. For k ≥ 0 , the ( k +1) -th epoch of SEG-RR starts at z k 0 , and the algorithm randomly chooses a permutation τ k : [ n ] → [ n ] . The algorithm then goes through a series of updates

for i = 0 , . . . , n -1 . In our worst-case example f , the updates can be compactly written as

Since we have n = 2 and there are only two possible permutations, the updates over an epoch can be summarized as

By the definition of A 1 and A 2 and using A 2 1 = A 2 2 = 0 , we can verify that

From this, we notice that the expectation of ∥ ∥ z k +1 0 ∥ ∥ 2 conditional on z k 0 reads

Working out the calculations, we can check that

Since this holds for all k ≥ 0 , SEG-RR diverges in expectation, for any positive stepsizes { α k } k ≥ 0 and { β k } k ≥ 0 . The statement on ∥ ∥ Fz k 0 ∥ ∥ follows by realizing that ∥ Fz ∥ = L 2 ∥ z ∥ .

thus resulting in thus resulting in

Proposition H.3 (Part of Theorem 4.1) . For n = 2 , there exists a convex-concave minimax problem f ( x, y ) = 1 2 ∑ 2 i =1 f i ( x, y ) having a monotone F , consisting of L -smooth quadratic f i 's satisfying Assumption 3.4 with ( ρ, σ ) = (1 , 0) such that SEG-FF diverges in expectation for any positive stepsizes { α k } k ≥ 0 and { β k } k ≥ 0 . That is, for any k ≥ 0 ,

Proof. The proof uses the same example as Proposition H.1, outlined in (121). We prove that SEG-FF also diverges for this f . For k ≥ 0 , the ( k +1) -th epoch of SEG-FF starts at z k 0 , and the algorithm randomly chooses a permutation τ k : [ n ] → [ n ] , as in the case of SEG-RR . The algorithm then goes through a series of updates for i = 0 , . . . , n -1 :

z

k

k

i

+1

=

z

i

-

β

k

F

τ

k

(

i

+1)

w

k

i

,

which are the same as SEG-RR ; but then, it performs another series of n updates, in the reverse order. For i = n, . . . , 2 n -1 ,

Using the definition of M 1 and M 2 from (122) and (123), one can verify that the 2 n = 4 updates over an epoch of SEG-FF can be summarized as

From this, we notice that the expectation of ∥ ∥ z k +1 0 ∥ ∥ 2 conditional on z k 0 reads

Working out the calculations, we can check that

thus resulting in

Since this holds for all k ≥ 0 , SEG-FF diverges in expectation, for any positive stepsizes { α k } k ≥ 0 and { β k } k ≥ 0 . The statement on ∥ ∥ Fz k 0 ∥ ∥ follows by realizing that ∥ Fz ∥ = L 2 ∥ z ∥ .

H.2 Proof of Limited Convergence of SEG-US in Monotone Cases

In [17, 20], the authors study the same-sample and independent-sample versions of SEG-US , with step sizes α t and β t satisfying a constant ratio: β t = γα t for γ ∈ (0 , 1] . While the authors show convergence in the monotone F case, there is one important limitation shared by the existing analyses. In order to achieve min t =0 ,...,T E [ ∥ Fz t ∥ 2 ] ≤ ϵ 2 for an arbitrarily chosen ϵ , the algorithms must repeat the same query to the stochastic gradient oracle b = O ( 1 ϵ 2 ) times at every iteration to reduce the gradient variance from σ 2 to σ 2 b . In other words, the convergence bounds for SEG-US in the monotone case have an additive term O ( σ 2 ) that cannot be reduced to zero by proper choices of stepsizes. Below, we prove that such a σ 2 term is in fact inevitable for any choices of stepsizes, if the ratio γ is fixed constant. This indicates that SEG-US considered in the existing results can never converge all the way to the optimum if b = 1 is maintained throughout training. In contrast, our SEG-FFA shows convergence in the monotone case even when b = 1 .

Theorem H.4. For n = 2 , there exists a convex-concave minimax problem f ( x, y ) = 1 2 ∑ 2 i =1 f i ( x, y ) having a monotone F , consisting of L -smooth quadratic f i 's satisfying Assumption 3.4 with ( ρ, σ ) =

(0 , σ ) such that SEG-US with any positive stepsizes { α t } t ≥ 0 and { β t } t ≥ 0 satisfying β t = γα t for γ > 0 cannot converge beyond a certain fixed constant Ω( σ 2 ) . More concretely, for any t ≥ 0 ,

regardless of the stepsizes. This holds for both same-sample and independent-sample SEG-US .

Proof. We consider the case of

which results in a bilinear (and hence convex-concave) objective function

One can quickly check from the definitions of the component functions f 1 and f 2 that the corresponding saddle gradient operators are given as

where z = ( x, y ) ∈ R 2 . From the fact that ∥ A ∥ ≤ L , we can confirm that f i 's are indeed L -smooth. As for Assumption 3.4, we can verify that

Therefore, by choosing ν 2 = σ 2 2 , our example f indeed satisfies Assumption 3.4 with ( ρ, σ ) = (0 , σ ) .

The proof is outlined as follows. For the example constructed above, we will calculate the E [ ∥ z t +1 ∥ 2 ] and show that the expectation is identical for both same-sample and independent sample versions of SEG-US . We will then show that the update on the expected squared distance to equilibrium E [ ∥ z t +1 ∥ 2 ] for given z t can only belong to two categories: either ∥ z t ∥ 2 ≤ E [ ∥ z t +1 ∥ 2 ] (expected squared distance increases) or ∥ z t ∥ 2 ≥ E [ ∥ z t +1 ∥ 2 ] ≥ γσ 2 2 L (expected squared distance shrinks but is bounded from below by a constant). Since the two cases hold for any t ≥ 0 and any choices of α t and β t = γα t , we show that the 'convergence' can happen only up to a neighborhood of equilibrium.

At iteration t , SEG-US samples component indices i ( t ) , j ( t ) ∈ { 1 , 2 } for its extrapolation step and update step, respectively. In the independent-sample version i ( t ) and j ( t ) are independently sampled from Unif ( { 1 , 2 } ) , and in the same-sample version i ( t ) is sampled uniformly at random and j ( t ) is set to be equal to i ( t ) . With the indices sampled as above, SEG-US then makes an update

In our worst-case example f , the updates can be written as

where we defined s 1 = +1 and s 2 = -1 for simplicity of notation.

We now calculate the expected value of ∥ z t +1 ∥ 2 .

For the independent-sample case, since s i ( t ) and s j ( t ) are independent mean-zero random variables,

In the same-sample case, s = s is a mean-zero random variable, so

but once we realize that ⟨ A 1 , 1 ⟩ = 0 , the expectation becomes identical to (124); hence, the rest of the analysis is the same for the two versions.

We now expand and arrange the RHS of (124). It is easy to check that

and hence

From this, we get

where we used the choice ν 2 = σ 2 2 as above.

The rest of the proof proceeds as follows: we show that, regardless of t ≥ 0 and the choices of α t and β t = γα t , the expected value of ∥ z t +1 ∥ 2 given z t can be categorized into only two cases:

∥ z t ∥ 2 ≤ E [ ∥ z t +1 ∥ 2 ] . That is, the iterate moves away from the equilibrium in expectation.

∥ z t ∥ 2 ≥ E [ ∥ z t +1 ∥ 2 ] ≥ γσ 2 2 L 2 . That is, the expected squared distance shrinks but is lower bounded by a certain constant independent of the stepsizes.

Showing this immediately finishes the proof, because there is no way that any E [ ∥ z t ∥ 2 ] can get smaller than min {∥ z 0 ∥ 2 , γσ 2 2 L 2 } , and ∥ Fz ∥ = L ∥ z ∥ for our example f .

The remaining proof is simple, by noticing that ∥ z t ∥ 2 ≤ E [ ∥ z t +1 ∥ 2 ] is equivalent to ( 2 α t β t L 2 -β 2 t L 2 (1 + α 2 t L 2 ) ) ∥ z t ∥ 2 ≤ β 2 t σ 2 (1 + α 2 t L 2 ) . (125)

Hence, if α t , β t , and z t satisfies (125), we belong to the first category. Otherwise, we are in the second category, for which we need to additionally show E [ ∥ z t +1 ∥ 2 ] ≥ γσ 2 2 L 2 . When the inequality (125) is satisfied with the opposite sign, we must have 2 α t β t L 2 -β 2 t L 2 (1 + α 2 t L 2 ) > 0 and

Also, notice that

Using 2 α t β t L 2 -β 2 t L 2 (1 + α 2 t L 2 ) ∈ (0 , 1) and substituting the lower bound on ∥ z t ∥ 2 into the update equation, we find that

Lastly, substituting β t = γα t into the RHS gives

This finishes the proof.

Remark H.5 . We remark that, while Theorem H.4 successfully shows that SEG-US as studied in [17, 20] cannot converge to an optimal point unless the batch size is increased every iteration, it does not contradict the (almost sure) convergence result of independent-sample SEG by Hsieh et al. [25]. Indeed, in [25], the stepsizes { α t } t ≥ 0 and { β t } t ≥ 0 are chosen so that they decay to 0 with a different rate and hence the corresponding ratio γ approaches 0 , while Theorem H.4 considers the case where α t and β t differ by a constant factor γ .

H.3 Proof of SGDA-RR and SEG-RR Lower Bounds

Theorem H.6. Suppose n ≥ 2 and L, µ > 0 satisfies L/µ ≥ 2 . There exists a µ -strongly-convexstrongly-concave minimax problem f ( z ) = 1 n ∑ n i =1 f i ( z ) consisting of L -smooth quadratic f i 's satisfying Assumption 3.4 with ( ρ, σ ) = (0 , σ ) and initialization z 0 0 such that SEG-RR with any constant stepsize α k = α > 0 , β k = β > 0 satisfies

where z ∗ is the unique equilibrium point of f . For a similar choice of problem f (this time with ( ρ, σ ) = (1 , σ ) ), SGDA-RR with any constant stepsize α k = α > 0 satisfies

Remark H.7 . In Theorem H.6, we adopt techniques from the existing lower bounds for SGD-RR to prove lower bounds for the minimax algorithms SGDA-RR and SEG-RR . In the literature, there are two types of lower bounds for SGD-RR when K ≳ L/µ : namely, Ω( 1 n 2 K 2 + 1 nK 3 ) bounds for strongly convex quadratic functions [49, 50] and Ω( 1 nK 2 ) bounds for strongly convex non-quadratic functions [11, 45, 56]. Upper bounds that match the lower bounds in n and K are also known, indicating that SGD-RR is one of the rare examples of minimization algorithms whose tight convergence rates for quadratic vs. non-quadratic functions differ, within the narrow scope of strongly convex and smooth functions. While it is tempting to aim for a tighter Ω( 1 nK 2 ) lower bound for our algorithms of interest, we note that the existing Ω( 1 nK 2 ) bounds for SGD-RR are proven for piecewise-quadratic functions whose Hessian is discontinuous . Since the discontinuous Hessian violates our Assumption 3.3, we instead adhere to the quadratic case to prove lower bounds Ω( 1 nK 3 ) for both SGDA-RR and SEG-RR (when K ≥ L/µ ). These bounds may not be the tightest possible (since they are restricted to quadratics), but they still suffice to demonstrate that SEG-FFA is provably superior to both SGDA-RR and SEG-RR .

H.3.1 Existing Lower Bound for SGD-RR

For the proof of lower bounds for SGDA-RR and SEG-RR , we utilize the results and techniques from the lower bounds proven for SGD-RR ; thus, it would be profitable to summarize the existing result.

In case of SGD-RR , it is known from Theorem 2 of Safran and Shamir [50] that there exists a minimization problem g ( x ) such that SGD-RR satisfies a lower bound of Ω( 1 n 2 K 2 + 1 nK 3 ) for large enough values of K . We rewrite the theorem in a version in accordance with our notation and assumptions:

Theorem H.8 (Theorem 2 of Safran and Shamir [50]) . For any n ≥ 2 and L, µ > 0 satisfying L/µ ≥ 2 , there exists a µ -strongly convex minimization problem g ( x ) = 1 n ∑ n i =1 g i ( x ) consisting of L -smooth quadratic g i 's satisfying Assumption 3.4 with ( ρ, σ ) = (1 , σ ) such that SGD-RR using any constant stepsize α k = α > 0 satisfies

The statement is equivalent to saying that for SGD-RR with constant stepsize α > 0 , the bound Ω( σ 2 LµnK ) holds for K ≲ L/µ and Ω( σ 2 µ 2 n 2 K 2 + Lσ 2 µ 3 nK 3 ) for K ≳ L/µ .

The function g = 1 n ∑ n i =1 g i used in the theorem is defined by the following component functions:

thus making the objective function

One can notice that the linear terms in g i (126) change signs depending on i ≤ n 2 or not, and handling these sign flips is the key to the proof of the lower bound.

H.3.2 Proof of Lower Bound for SGDA-RR

For the SGDA-RR lower bound, we consider the following minimax optimization problem:

where g i 's are from (126). We need to first check if the problem instance satisfies the assumptions listed in the theorem statement. Since f ( x , y ) = g ( x ) -µ 2 y 2 and g is a µ -strongly convex function, f is µ -strongly-convex-strongly-concave as claimed. Also, it is easy to check from the definition of g i that each component f i ( x , y ) is L -smooth quadratic.

Lastly, to verify Assumption 3.4, we first define s 1 , . . . , s n as s i = 1 for i ≤ n 2 and s i = 0 for i > n 2 . Using this notation, The function g i can be compactly written as the following:

Therefore, the saddle gradient operators F i of f i and F of f evaluate to

which in turn yields

for all i = 1 , . . . , n . This confirms that the function f = 1 n ∑ i f i satisfies Assumption 3.4 with ( ρ, σ ) = (1 , σ ) .

If we run SGDA-RR on this problem, the updates on x done by SGDA-RR is exactly identical to what SGD-RR would perform for the minimization problem g ( x ) = 1 n ∑ i g i ( x ) with the same choices of random permutations. Therefore, after K epochs of SGDA-RR , it follows from Theorem H.8 that

which is in fact a tighter lower bound for SGDA-RR than what is stated in Theorem H.6. This finishes the proof.

H.3.3 Proof of Lower Bound for SEG-RR

In this subsection, we prove the lower bound for SEG-RR . We will first define a new problem instance f to be used here, and verify that the assumptions in the theorem statement are indeed satisfied by this new f . We will then spell out the update equation of SEG-RR for this example, which will serve as a basis for the case analysis that follows: we will divide the choices of stepsizes

α, β > 0 to four regimes and prove a lower bound for each of them. Combining the regimes will result in the desired lower bound.

For SEG-RR , we use a slightly different problem from (127). This time, we consider

where s i = 1 for i ≤ n 2 and s i = 0 for i > n 2 , as defined above.

We first check if the problem (128) satisfies the assumptions in the theorem statement. Since

and L/ 2 ≥ µ by assumption, f is µ -strongly-convex-strongly-concave. Also, it is straightforward to see that each f i is an L -smooth quadratic function. It is left to check Assumption 3.4. The saddle gradient operators F i of f i and F of f evaluate to

for all i = 1 , . . . , n . This confirms that the function f = 1 n ∑ i f i satisfies Assumption 3.4 with ( ρ, σ ) = (0 , σ ) , as required by the theorem.

For k ≥ 0 , the ( k +1) -th epoch of SEG-RR starts at z k 0 = ( x k 0 , y k 0 ) and the algorithm chooses a random permutation τ k . The algorithm then goes through a series of updates

for i = 0 , . . . , n -1 . For our example f (128), it can be checked that a single iteration by SEG-RR reads

Aggregating the SEG-RR updates over an entire epoch ( i = 0 , . . . , n -1 ) results in

We will now square both sides of these equations above and take expectations over τ k . In doing so, there is a useful identity:

Also, it is worth mentioning that τ k is independent of z k 0 = ( x k 0 , 1 , x k 0 , 2 , y k 0 ) . Using these facts, we can arrange the terms to obtain

which in turn yields

Based on these three per-epoch update equations above, we now divide the choices of SEG-RR stepsizes α, β > 0 into the following four cases and handle them separately:

α > 1 L , in which case we show that SEG-RR makes ( x k +1 0 , 1 ) 2 > ( x k 0 , 1 ) 2 hold deterministically, so that if we initialize at x 0 0 , 1 = σ √ Lµ then we have

2. α ≤ 1 L and β ≤ 1 µnK , in which case we show that SEG-RR initialized at y 0 0 = σ √ Lµ suffers

α ≤ 1 L and 1 µnK < β < 1 nL , in which case we show that SEG-RR initialized at x 0 0 , 2 = 0 suffers

α ≤ 1 L , β > 1 µnK , and β ≥ 1 nL in which case we show that SEG-RR initialized at x 0 0 , 2 = 0 suffers

Notice that the third case 1 µnK < β < 1 nL only makes sense when K > L/µ ; otherwise, the third case just disappears. Hence, for the 'large epoch' regime where K > L/µ , the third case achieves the minimum error possible, so it holds that

For the 'small epoch' regime ( K ≤ L/µ ), the third case does not exist and the fourth case achieves the minimum, so

Combining the two cases yields the desired lower bound in the theorem statement. It now remains to carry out the case analysis.

Case 1: α > 1 L . For this case, we use (129) to prove divergence. Notice from α > 1 L that

regardless of β > 0 . Hence, from (129), we get

If we initialize at x 0 0 , 1 = σ √ Lµ , then this proves

Case 2: α ≤ 1 L and β ≤ 1 µnK . For this case, we employ (131) to show that the 'contraction rate' is too small to make enough 'progress.' Notice from our stepsizes that

Applying this inequality to (131), we have

which in turn means that the progress over K epoch is bounded from below by

where we used our assumption that n ≥ 2 and K ≥ 1 . Hence, if our initialization was given as y 0 0 = σ √ Lµ , then this proves

Case 3: α ≤ 1 L and 1 µnK < β < 1 nL . For stepsizes in this interval, we use (130) to derive the desired bound. Here, it is important to characterize a lower bound on the quantity

To this end, we can use a lemma from Safran and Shamir [49], stated below:

Lemma H.9 (Lemma 1 of Safran and Shamir [49]) . Let π 1 , . . . , π n (for even n ) be a random permutation of (1 , 1 , . . . , 1 , -1 , -1 , . . . , -1) where both 1 and -1 appear exactly n/ 2 times. Then there is a numerical constant c > 0 such that for any ν > 0 ,

One can notice that Lemma H.9 is directly applicable to E [Φ 2 ] , with ν ← βL 2 -αβL 2 4 . Since

we have n 3 ν 2 ≤ 1 8 ν , thereby

where the last inequality used α ≤ 1 L . Applying (132) to (130) and also using (1 -αL 2 ) 2 ≥ 1 4 ,

Unrolling the inequality for k = 0 , . . . , K -1 gives

Now note that our initialization x 0 0 , 2 can be set to zero, which eliminates the need to think about the first term in the RHS. It is now left to bound the second term. First, by the stepsize range α ≤ 1 L , β > 1 µnK and our assumption L/µ ≥ 2 , we have

Next, by Bernoulli's inequality

Plugging in the two inequalities to above, we obtain

for a numerical constant c ′ > 0 . Plugging in the lower bound β > 1 µnK yields

Case 4: α ≤ 1 L , β > 1 µnK , and β ≥ 1 nL . We again use (130). By noticing that the initialization x 0 0 , 2 = 0 , we can unroll (130) for k = 0 , . . . , K -1 to get

where the last inequality holds regardless of β because each summand with j ≥ 1 is nonnegative. We then invoke Lemma H.9 to lower bound E [Φ 2 ] , again with ν ← βL 2 -αβL 2 4 . Since

we have n 3 ν 2 ≥ 1 64 ν , thereby

Combining (134) with (133) gives

where the last step used β > 1 µnK . This finishes the case analysis, hence the proof of Theorem H.6.

I Additional Experiments

To evaluate our algorithm SEG-FFA as well as other baseline algorithms, we conduct numerical experiments on monotone and strongly monotone problems. Specifically, as we have mentioned in Section 6, we consider random quadratic problems of the form

We choose d x = d y = 20 and n = 40 for all the experiments. Numerical computations are done using NumPy [24] and SciPy [52], and the plots are drawn using Matplotlib [26].

I.1 Problem Constructions for Experiments in Section 6

For an experiment for the monotone case, the random components are sampled as follows. We choose B i so that each element is an i.i.d. sample from a uniform distribution over the interval [0 , 1] , and t i so that each element is an i.i.d. sample from a standard normal distribution. We chose A i to be diagonal matrices in the following procedure: for each j = 1 , . . . , 20 we randomly chose a subset I j of n 2 = 20 indices from [ n ] = { 1 , . . . , 40 } , and set the ( j, j ) -entry of A i to be

We repeat the exact same procedure for C i as well. Notice that ∑ n i =1 A i = ∑ n i =1 C i = 0 by design. Hence, each of the component functions will be a nonconvex-nonconcave quadratic function in general, but the objective function itself becomes a convex-concave function.

For the experiment in the strongly monotone case, we sample B i and t i in the same way as in the monotone case, but we use different choices of A i and C i to ensure the objective function to be strongly-convex-strongly-concave. In particular, for each i = 1 , . . . , n , we sample A i by computing A i = Q i D i Q ⊤ i , where D i is a random diagonal matrix whose diagonal entries are i.i.d. samples from a uniform distribution over the interval [ 1 2 , 1] , and Q i is a random orthogonal matrix obtained by computing a QR decomposition of a 20 × 20 random matrix whose elements are i.i.d. samples from a standard normal distribution. We sample C i by the exact same method.

I.2 Monotone Case & Ablation Study on the Anchoring Step

In Section 6, we compared the empirical performance of various SEGs, namely SEG-FFA , SEG-FF , SEG-RR , and SEG-US . Here, as an ablation study on the anchoring technique, we additionally compare SEG-RRA and SEG-USA , which are each SEG-RR and SEG-US with an additional anchoring step, respectively. For these two methods, we take the anchoring step after every n iterations. We ran those methods on the same 5 random instances used in Section 6. For both SEGRRA and SEG-USA, we ran the method with two different stepsize choices, namely α k = β k = η k (inspired by the stepsize used in deterministic EG) and α k = β k / 2 = η k / 2 (the stepsize used for SEG-FFA ) where we again set η k = η 0 / (1+ k/ 10) 0 . 34 with η 0 = min { 0 . 01 , 1 L } .

The results are plotted in Figure 2. As SEG-RRA and SEG-USA are designed to take one pass per epoch, for those methods, we compute the ratio ∥ Fz t 0 ∥ 2 ∥ Fz 0 0 ∥ 2 where t denotes the number of passes, and plot the geometric mean over the 5 runs.

From the performance of SEG-RRA with α k = β k and the two variants of SEG-USA, it is possible to observe that adding the anchoring step does improve the performance of the method up to a certain level, but it alone does not fully resolve the nonconvergence issue. On the other hand, quite interestingly, SEG-RRA with α k = β k / 2 shows a hint of convergence. While its performance is slightly worse compared to SEG-FFA , it is nonetheless still notable as it is the only other method from SEG-FFA that seems to be capable of converging to an optimum.

We conjecture that this intriguing performance of SEG-RRA with α k = β k / 2 is because it achieves an 'expected' second order matching to the (deterministic) EG. Indeed, following the notations of Proposition D.1, one can deduce from Proposition D.1 that using SEG-RRA with α = β / 2 will result in an epoch-level update of

with ϵ n = o ( β 2 ) . Here, notice that ( T 0 , T 1 , . . . , T n -1 ) = ( F τ (1) , F τ (2) , . . . , F τ ( n ) ) for some randomly chosen permutation τ ∈ S n . Now, observe that for any two distinct i, j ∈ [ n ] , there are exactly n ! 2 permutations in S n such that i comes before j in the sequence τ (1) , τ (2) , . . . , τ ( n ) , and also exactly n ! 2 permutations such that j comes before i . Thus, in taking the expectation over the

0

20000

40000

60000

80000

100000

number of passes (

t

)

10

5

10

3

10

1

10 1

10 3

10 5

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

SEG-RRA,

α

=

β

SEG-RRA,

α

=

β/

2

SEG-USA,

α

=

β

SEG-USA,

α

=

β/

2

randomness of choosing the permutation τ , we get

̸

̸

where in getting the third line we have used the previously made observation that for any fixed i and j with i = j , the term D F j ( z 0 ) F i z 0 appears exactly n ! 2 times in the sum on the second line. Hence, taking the expectation with respect to the random permutation on (135) we get

̸

Comparing this to (7) when η 1 = η 2 = nβ / 2 , we indeed see that the update rule of SEG-RRA with α = β / 2 achieves a second-order matching on expectation to the (deterministic) EG update with stepsize nβ / 2 .

We also conjecture that the relatively worse performance of SEG-RRA with α = β / 2 compared to SEG-FFA is because the error over an epoch is O ( η 3 ) only on expectation, and thus the actual error occurring in each epoch can be larger than O ( η 3 ) . Unfortunately, our convergence analysis on SEG-FFA relies on the error over an epoch being O ( η 3 ) deterministically ( cf. Proposition 5.3), hence cannot be directly applied to SEG-RRA with α = β / 2 . We leave the search for a theoretical explanation on this alluring performance of SEG-RRA with α = β / 2 as a stimulating direction for future work.

I.3 Monotone Case: Comparison with Hsieh et al. [25]

Let us also compare the performance of SEG-FFA with the independent-sample double stepsize SEG ( DSEG ) by Hsieh et al. [25]. Writing in terms of the finite-sum structure, the update rule of DSEG

0

2000

4000

6000

8000

10000

number of passes (

t

)

10

5

10

4

10

3

10

2

10

1

10 0

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

DSEG, exponents (0, 1)

DSEG, exponents (1/3, 2/3)

can be written as

where i (1 , k ) and i (2 , k ) are random indices that are independently drawn from [ n ] for each k . The stepsizes are chosen in the form of η 1 ,k = Θ( 1 / k r 1 ) and η 2 ,k = Θ( 1 / k r 2 ) , where setting r 1 ≤ r 2 is the key point of DSEG. Two choices of the exponent pair ( r 1 , r 2 ) proposed in [25] are (1 / 3 , 2 / 3) for general monotone problems and (0 , 1) exclusively for the case when F is affine.

We again use the same component functions as in the previous experiment. The setup for running SEG-FFA are kept the same. For DSEG, we use the default choices suggested by Hsieh et al. [25], namely η 1 ,k = γ 0 / ( k +19) r 1 and η 2 ,k = η 0 / ( k +19) r 2 , where ( γ 0 , η 0 ) = (1 , 0 . 1) for the bilinear case with ( r 1 , r 2 ) = (0 , 1) and ( γ 0 , η 0 ) = (0 . 1 , 0 . 05) for the general case with ( r 1 , r 2 ) = (1 / 3 , 2 / 3) .

The results are displayed in Figure 3, where the details on how the plots are drawn are the same as Figure 2. Here we can clearly see that SEG-FFA outperforms both versions of DSEG.

I.4 Strongly Monotone Case Again, with Various Stepsizes

We also ran the experiment on strongly monotone problems described in Section 6, but with changing the stepsizes. We tested six different values of η k ; we have tested with η k = a × 10 b where a ∈ { 1 , 2 , 5 } and b ∈ {-4 , -3 } . Notice that the case η k = 10 -3 is exactly the experiment conducted in Section 6.

The results are plotted in Figure 4. The overall details are the same as described in Section 6, as the only difference is the stepsize choice. We can observe that, while the initial speed of convergence may not be the fastest depending on the stepsize, SEG-FFA is always the method that eventually finds the point with the smallest gradient. In other words, as predicted by our theoretical analyses, the supremacy of SEG-FFA is in general not affected by the choice of the stepsize, as long as the chosen stepsize is reasonably small.

0

2000

4000

6000

8000

10000

number of passes (

t

)

10

11

10

9

10

7

10

5

10

3

10

1

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

SGDA-RR

SGDA-US

(a)

η

k

= 0

.

0001

0

1000

2000

3000

4000

5000

number of passes (

t

)

10

10

10

8

10

6

10

4

10

2

10 0

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

SGDA-RR

SGDA-US

(b)

η

k

= 0

.

0002

0

250

500

750

1000

1250

1500

1750

2000

number of passes (

t

)

10

8

10

6

10

4

10

2

10 0

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

SGDA-RR

SGDA-US

(c)

η

k

= 0

.

0005

0

100

200

300

400

500

600

700

800

number of passes (

t

)

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

SGDA-RR

SGDA-US

(d)

η

k

= 0

.

001

0

100

200

300

400

500

600

number of passes (

t

)

10

6

10

5

10

4

10

3

10

2

10

1

10 0

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

SGDA-RR

SGDA-US

(e)

η

k

= 0

.

002

0

50

100

150

200

250

300

350

400

number of passes (

t

)

10

4

10

3

10

2

10

1

10 0

Fz t

0

2

Fz

0

0

2

or

Fz t

/2

0

2

Fz

0

0

2

SEG-FFA

SEG-FF

SEG-RR

SEG-US

SGDA-RR

SGDA-US

(f)

η

k

= 0

.

005

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and the introduction well summarizes our theoretical results and the problem settings we are considering.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: While we do not have a separate "Limitations" sections, in Section 3 we thoroughly discuss about the assumptions we have imposed. The paper is highly theoretical, hence the other factors listed in the guidelines below are either not applicable to this paper, or apparent from the statements of the theorems/lemmata/propositions and the discussions that follow.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Section 3 is devoted for the discussions on the assumptions. Full proofs of the theorems/lemmata/propositions can be found in the appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Appendix I, we provide full explanations on how the experiments have been conducted. We have also submitted the exact code that we used for our experiments as a supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have submitted the exact code that we used for our experiments as a supplemental material, so that it becomes revealed to the public once our paper gets accepted. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The overall settings are discussed in Appendix I. The code we submit along with the paper is an exact copy of the one we used in the reported experiments, so the details not included in the paper shall be found in the code itself.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our paper is mainly theoretical, and the experiments are to demonstrate that our analyses are correct. Hence, we claim that error bars or information about the statistical significance are not necessary, and rather, the interpretations we made regarding our experiments in the relevant section(s) are enough.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The experiments are numerical validations of our theoretical analyses using simple quadratic functions, so they should be executable on any modern computer with a reasonable CPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read through the Code of Ethics, but due to the theoretical nature of the paper, there are no risks regarding ethical issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are no societal impacts of this paper, as it is a theory paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is highly theoretical, hence poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Packages used in the experiments, NumPy , SciPy , and Matplotlib , are cited. No existing data nor models are used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper provides novel theoretical results rather than datasets or models, hence this question is not applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies unconstrained (strongly)-monotone finite sum minimization problems. They introduce a new scheme called SEG-FFA which:\n\ni) for a given epoch runs stochastic extragradient (SEG) with possibly two different stepsizes\nii) uses flip-flop shuffling per epoch\niii) uses the average of the epoch initialization and the last iterate as the initialization for the next epoch\n\nWith $K$ being the number of epochs and $n$ being the number of finite sum elements, they show:\n\n- For the monotone case a $\\tilde{\\mathcal O}(1/K^{1/3})$ rate for the squared norm of the operator.\n- For the $\\mu$-strongly-monotone case a $\\mathcal O(\\exp{(-\\mu K^\\epsilon)\\Vert z^0_0 - z^\\star \\lVert^2} + \\tfrac{1}{nK^{4-5\\epsilon}})$ rate where $\\epsilon \\in (0,2/3)$.\n\nSTRENGTHS:\nThe paper reads very well in terms of laying out what problem is being addressed and the current landscape of the literature (including the extended discussion in the appendix). The reasoning behind their algorithmic construction is also very clear.\n\nWEAKNESSES:\nMy main concern is with the strength of the results:\n\n- Regarding the monotone case (Thm. 5.4): The argument seems to be that one _full epoch_ of SEG-FFA can approximate one step of the deterministic EG accurately. In other words, the rate has no dependency on $n$. Why not instead simply run full batch EG which would get the much faster $\\mathcal O(1/K)$ rate? (e.g. using gradient accumulation to make it memory efficient)\n- Regarding the strongly-monotone case (Thm. 5.5): There is a tradeoff in the rate. If we want fast rate for the stochastic term $\\mathcal O(\\tfrac{1}{nK^{4-5\\epsilon}})$ we need to take $\\varepsilon \\rightarrow 0$ in which case the linear rate $\\mathcal O(\\exp{(-\\mu K^\\epsilon)\\Vert z^0_0 - z^\\star \\lVert^2})=\\mathcal O(\\exp{(-\\mu)\\Vert z^0_0 - z^\\star \\lVert^2})$ cannot be made small. Even if we ask to just match the lowerbound $\\mathcal O(\\tfrac{1}{nK^{3}})$ on SGDA-RR and SEG-RR we run into trouble.  Pick $\\epsilon=1/5$ to get $\\mathcal O(\\exp{(-\\mu K^{1/5})\\Vert z^0_0 - z^\\star \\lVert^2} + \\tfrac{1}{nK^{3}})$. If we plot the two terms (ignoring the constants) we will see that the linear rate dominates the polynomial term for even very large $K$ (100M+). In other words, it is unfair to compare only the polynomial terms in this case, and the benefit of SEG-FFA over e.g.  SEG-RR is much less clear when the linear term is taken into account.\n\nOther concerns:\n\n- Regarding Thm. 4.1: It seems surprising that it is possible to show divergence for e.g. SEG-US (with two stepsizes) since this is the scheme that [24] shows converges almost surely. How is it possible for these two results to coexist?\n- Currently the rates do not state dependencies on noise parameters. It is good practice to include $\\rho$ and $\\sigma$ in the rates as we should expect different dependencies.\n- Table 1 currently does not include the dependency on the initialization in the rate for SEG-FFA, which can be misleading. I suggest also including the rate $\\mathcal O(\\exp{(-\\mu^2 K)\\Vert z^0_0 - z^\\star \\lVert^2} + \\tfrac{1}{nK^2})$ of e.g. SEG-RR [17, Thm. 2.1] to make the smaller dependency on $K$ in the linear rate apparent.\n\nMinor:\n\n- SEG-RR/SEG-FF is already introduced using two stepsizes, so it is slightly confusing when EG+ is introduced in (7) and contrasted with EG as being a version with two stepsizes. You might also want to shortly comment that strictly speaking the method you consider is not an instance of [16] as you consider a _larger_ second stepsize as oppose to a smaller second stepsize.\n\nRelated work:\n\n- Apart from mention KM iterations it might be worth referencing the Lookahead algorithm for minimax https://arxiv.org/pdf/2006.14567 (LA–ExtraGrad, for which also two different stepsizes are considered, is essential equivalent to SEG-FFA but without flipflip)\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe paper lacks appropriate discussion on the rates obtained in the main results of Thm. 5.4-5.5 (see weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors study stochastic extragradient methods for solving unconstrained minimax convex-concave problems with a finite sum structure. In particular, various shuffling schemes (random reshuffling without replacement, flip-flop, uniform sampling with replacement) are investigated and it was shown that stochastic extragradient with them can lead to divergent behavior when $f$ is merely convex-concave. The authors proceed to propose stochastic extragradient with flip-flop + anchoring which successfully converges in convex-concave problems with rate $O(1/K^{1/3})$ and rate $O(1/nK^{4-5\\epsilon})$ for $\\epsilon < 2/3$ when $f$ is strongly-convex-strongly-concave. This is supplemented with a lower bound on random-reshuffling based stochastic gradient descent-ascent and stochastic extragradient in the same setup, demonstrating advantage of the proposed algorithm. The analysis is based on controlling the degree of approximation to a known convergent method. Some numerical experiments are presented to illustrate the practical performance.\n\nSTRENGTHS:\nThe results in this paper removed several limitations from the previous work and shed light on algorithm design for the appropriate sampling scheme on stochastic min-max problem. I believe the contributions are of great interest for the community and the resulting proposal is conceptually simple, easy to implement with minimum modification from known algorithms. The exposition is clear with nice explanation of the design principle (second-order matching to a known deterministic convergent method) and the relevant literature is thoroughly surveyed and compared against. The technical claims are sound with supporting numerical evidence.\n\nWEAKNESSES:\nI've listed a few questions in the section below but I don't think there are major weakness with the paper.\n\nMinor comments: \n\n- In the second set of equations in (2), I'd recommend changing the subscript in nabla to u and v instead, or some other notation to avoid confusion.\n\n- In the definition of Assumption 3.3, it might make more sense to call each $F_i$ $L$-lipschitz (corresponding to (i)) and each $F_i$ $M$-smooth (corresponding to (ii)) - this would possibly make things more consistent with convex optimization literature. In Line 178, should it be $f$ being $L$-smooth or $F$?\n\nQUESTIONS:\n- It seems like this gap of $1/nK^3$ for SEG-RR vs. $1/nK^{4-5\\epsilon}$ for SEG-FFA only exists in a very narrow range of $\\epsilon$. Is $\\epsilon$ tied to other parts of the results? If not, I'd recommend just optimizing over $\\epsilon$ and state the final complexity instead.\n\n- Is there any special meaning behind the $1/2$ in (12)? Would other convex combination of the iterates work as well? Or possibly with other anchor points or even more than $1$ point help for higher-order matching?\n\n- In Theorem 5.4, can one afford a constant (and larger) stepsize if instead of asking guarantee in $\\min_k ...$ we do tail averaging?\n\n- The parameter $\\rho$ or $\\sigma$ from Assumption 3.4 don't make an appearance in Theorem 5.4 and 5.5?\n\n- What's the difficulty of generalizing this framework from same sample to independent sample extragradient methods?\n\nLIMITATIONS:\nYes it's adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm, SEG-FFA, which converges for the convex-concave minimax problem, while existing algorithms like SEG-RR and SEG-FF diverge for the same class of problem. Moreover, the authors show that SEG-FFA can better approximate EG in comparison to SEG-RR and SEG-FF.\n\nSTRENGTHS:\n- It proposes a new stochastic algorithm, SEG-FFA, which simultaneously incorporates flip-flop shuffling and anchoring to prove convergence. To the best of my knowledge, this is the first work to successfully combine these two techniques.  \n- Usual Halpern iterations take a convex combination with the initial iterate, while SEG-FFA takes a convex combination with the initial iterate of the epoch. This idea is interesting and certainly paves the way for further research. \n- I find section 5.1 very interesting, where the authors explain how well different algorithms can approximate EG.\n\nWEAKNESSES:\n- Lipschitz Hessian in Assumption 3.3 is too restrictive, and the authors agree. I don't know if practical ML applications satisfy such assumptions. \n- In each epoch, SEG-FFA and SEG-FF make $4n$ oracle calls compared to $2n$ oracle calls made by SEG-RR and SEG-US and $n$ oracle calls by SGDA-US and SGDA-RR. Therefore, the comparison made in the second plot of Figure 1 is unfair. I recommend having the number of oracle calls on the $x$-axis to compare these algorithms fairly. \n- In Theorem 5.4, authors propose a decreasing step size of the order $\\mathcal{O} \\left( 1/ k^{1/3} \\log k \\right)$ while in plot 1 of the experiment, they use a different step size. Why not use the step size proposed in the Theorem?\n\nQUESTIONS:\n- What is $\\eta_k$ in Theorem 5.4? The step sizes of SEG-FFA are $\\alpha_k$ and $\\beta_k$ in Eq 5. \n- Is it possible to relax Assumption 3.3? \n- Multiple papers show the convergence of SEG and Stochastic Past Extragradient [1] method for monotone problems with increasing batch sizes. Moreover, a work [2] proposes BCSEG (bias-corrected SEG), which proves convergence without increasing batch sizes. I recommend comparing these works with your Theorem 5.4 in terms of the number of oracle calls to achieve a given accuracy $\\epsilon$.\n\n\n[1] \"Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditions.\"\n \n[2] \"Solving stochastic weak Minty variational inequalities without increasing batch size.\"\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies various same-sample SEG algorithms under different shuffling schemes, including SEG-US, SEG-RR and SEG-FF. The three algorithms all can diverge when $f$ is convex-concave. Furthermore, the authors discuss the underlying cause for the nonconvergence of the three algorithms. Moreover, the authors propose a novel stochastic extragradient method named as SEG-FFA, which is SEG amended with flip-flop shuffling scheme and anchoring. The proposed algorithm enjoys improved convergence guarantees and the convergence rate is calculated under different conditions of $f$. Finally, the authors conduct numerical experiments to verify the convergence of SEG-FFA.\n\nSTRENGTHS:\n1. The paper proposes a novel algorithm SEG-FFA for minimax problems, which achieves a better convergence rate when $f$ is strongly-convex-strongly-concave. Furthermore, the algorithm enjoys convergence when $f$ is convex-concave while other baseline algorithms diverge under this setting.\n2. The paper provides comprehensive proof for divergence of other baseline algorithms when $F$ is merely monotone.\n3. The proposed algorithm SEG-FFA can match extragradient up to second-order terms and get an error of $\\mathcal{O}(\\eta)$ where $\\eta$ is the stepsize. As a result, SEG-FFA achieves convergence in monotone problems.\n4. Sufficient numerical experiments verify the convergence result and superiority of SEG-FFA.\n\nWEAKNESSES:\n1. Due to the presence of anchoring step, the initial point seems to be more important, which implies that the algorithm is more dependent on the selection of the initial points.\n2. How to select the parameter $\\epsilon$ concerning convergence in Table 1 and Theorem 5.5 for practical applications and specific problems.\n3. Table 1 does not provide the lower bound of SEG-FFA under convex-concave setting. Is there such a lower bound on the iterative complexity for SEG-FFA?\n\nQUESTIONS:\n1. Section 4 explains the reasons why simple shuffling is not enough for convergence for monotone problems. However,  why the added anchoring step can make the algorithm SEG-FFA converge is not sufficiently demonstrated in the proof, especially compared with SEG-FF.\n2. What is the difference between the stepsizes $\\alpha_k, \\beta_k$ of SEG-FF and $\\eta_k$ of SEG-FFA.\n3. In Figure 4(a) and 4(b), the convergence result of SEG-FFA is better than other baseline algorithms. However, the convergence rate appears slower.\n4. There are few typos in the main text and appendix.\n\nLIMITATIONS:\n1. The length of the paper is excessive, especially the appendix section.\n2. The experiments could consider demonstrating the performance of SEG-FFA on more large-scale problems.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}