Title: Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Optimization

Machine Learning

Extragradient Methods

Min-Max Optimization
Abstract: Single-call stochastic extragradient methods, like stochastic past extragradient (SPEG) and stochastic optimistic gradient (SOG),  have gained a lot of interest in recent years and are one of the most efficient algorithms for solving large-scale min-max optimization and variational inequalities problems (VIP) appearing in various machine learning tasks. However, despite their undoubted popularity, current convergence analyses of SPEG and SOG require strong assumptions like bounded variance or growth conditions. In addition, several important questions regarding the convergence properties of these methods are still open, including mini-batching, efficient step-size selection, and convergence guarantees under different sampling strategies. In this work, we address these questions and provide convergence guarantees for two large classes of structured non-monotone VIPs: (i) quasi-strongly monotone problems (a generalization of strongly monotone problems) and (ii) weak Minty variational inequalities (a generalization of monotone and Minty VIPs). We introduce the expected residual condition, explain its benefits, and show how it allows us to obtain a strictly weaker bound than previously used growth conditions, expected co-coercivity, or bounded variance assumptions. Finally, our convergence analysis holds under the arbitrary sampling paradigm, which includes importance sampling and various mini-batching strategies as special cases.

Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditions

Abstract

Single-call stochastic extragradient methods, like stochastic past extragradient ( SPEG ) and stochastic optimistic gradient ( SOG ), have gained a lot of interest in recent years and are one of the most efficient algorithms for solving large-scale minmax optimization and variational inequalities problems (VIP) appearing in various machine learning tasks. However, despite their undoubted popularity, current convergence analyses of SPEG and SOG require strong assumptions like bounded variance or growth conditions. In addition, several important questions regarding the convergence properties of these methods are still open, including mini-batching, efficient step-size selection, and convergence guarantees under different sampling strategies. In this work, we address these questions and provide convergence guarantees for two large classes of structured non-monotone VIPs: (i) quasistrongly monotone problems (a generalization of strongly monotone problems) and (ii) weak Minty variational inequalities (a generalization of monotone and Minty VIPs). We introduce the expected residual condition, explain its benefits, and show how it allows us to obtain a strictly weaker bound than previously used growth conditions, expected co-coercivity, or bounded variance assumptions. Finally, our convergence analysis holds under the arbitrary sampling paradigm, which includes importance sampling and various mini-batching strategies as special cases.

1 Introduction

Differentiable game formulations where several parameterized models/players compete to minimize their respective objective functions have recently gained much attention from the machine learning community. Some landmark advances in machine learning that are framed as games (or in their simplified form as min-max optimization problems) are Generative Adversarial Networks (GANs) [19, 2], adversarial training of neural networks [46, 72], reinforcement learning [9, 64], and distributionally robust learning [51, 73].

In this work, we consider a more abstract formulation of the problem and focus on solving the following unconstrained stochastic variational inequality problem (VIP):

where each F i : R d → R d is a Lipschitz continuous operator. Problem (1) generalizes the solution of several types of stochastic smooth games [16, 44, 20, 7]. The simplest example is the unconstrained min-max optimization problem (also called a zero-sum game):

where each component function g i : R d 1 × R d 2 → R is assumed to be smooth. In this scenario, operator F i of (1) represents the appropriate concatenation of the block-gradients of g i : F i ( x ) := ( ∇ x 1 g i ( x 1 , x 2 ); -∇ x 2 g i ( x 1 , x 2 )) , where x := ( x 1 ; x 2 ) . Solving (1) then amounts to finding a stationary point x ∗ = ( x ∗ 1 ; x ∗ 2 ) for (2), which under a convex-concavity assumption for g i , implies that it is a global solution for the min-max problem.

However, in modern machine learning applications, game-theoretical formulations that are special cases of problem (1) are rarely monotone. That is, the min-max optimization problem (2) does not satisfy the popular and well-studied convex-concave setting. For this reason, the ML community started focusing on non-monotone problems with extra structural properties. 1 In this work, we focus on such settings (structured non-monotone operators) for which we are able to provide tight convergence guarantees and avoid the standard issues (like cycling and divergence of the methods) appearing in the more general non-monotone regime. In particular, we focus on understanding and efficiently analyze the performance of single-call extragradient methods for solving (i) µ -quasistrongly monotone VIPs [44, 6] and (ii) weak Minty variational inequalities [14, 33].

Classes of structured non-monotone VIPs. Throughout this work we assume that operator F in (1) is L - Lipschitz i.e. ∀ x, y ∈ R d operator F satisfy ∥ F ( x ) -F ( y ) ∥ ≤ L ∥ x -y ∥ .

As we have already mentioned, in this work, we deal with two classes of structured non-monotone problems: the µ -quasi strongly monotone VIPs and the weak Minty variational inequalities.

Definition 1.1. F is said to be µ -quasi strongly monotone if there is µ > 0 such that:

Condition (3) is a relaxation of µ -strong monotonicity, and it includes several non-monotone games as special cases [44]. Inequality (3) can be seen as an extension of the popular quasi-strong convexity assumption from optimization literature [53, 25] to the VIPs [44]. In the literature of variational inequality problems, quasi strongly monotone problems are also known as strong coherent VIPs [66] or VIPs satisfying the strong stability condition [47], or strong Minty variational inequality [14].

One of the weakest possible assumptions on the structure of non-monotone VIPs is the weak Minty variational inequality [14].

Definition 1.2. We say weak Minty Variational Inequality (MVI) holds for F if for some ρ > 0

To the best of our knowledge, the weak Minty variational inequality (4) as an assumption was first introduced in [14]. The more popular and extensively studied Minty variational inequality [12, 37, 38, 48] is a particular case of (4) with ρ = 0 . In addition, the weak MVI condition is implied by the negative comonotonicity [4] or, equivalently, the positive cohypomonotonicity [11]. Finally, when we focus on min-max optimization problems (2), weak MVI condition (with ρ = 0 ) is satisfied for several non-convex non-concave families of min-max objectives, including quasi-convex quasi-concave or star convex- star concave [20]. Extragradient-type methods for solving VIPs satisfying the weak MVI have been proposed in [14, 54] and [8].

1.1 Main Contributions

Our main contributions are summarized below.

Expected Residual. We propose the expected residual (ER) condition for stochastic variational inequality problems (1). We explain the benefits of ER and show how it can be used to derive an upper bound on E ∥ g ( x ) ∥ 2 (see Lemma 3.2) that it is strictly weaker than the bounded variance assumption and 'growth conditions' previously used for the analysis of stochastic algorithms for solving (1). We prove that ER holds for a large class of operators, i.e., whenever F i of (1) are Lipschitz continuous.

Novel Convergence Guarantees. We prove the first convergence guarantees for SPEG (7) in the quasi-strongly monotone (3) and weak MVI (4) cases without using the bounded variance

The result is derived for ρ < 3 / 8 L . Here a and c are assumed to satisfy aL ≤ 7 -√ 1+48 c 2 8(1+ c ) , c > 0 and a > ρ .

The result is derived for ρ < 1 / 2 L . Here we assume that max { 2 ρ, 1 / (2 L ) } < γ < 1 / L and 0 < ω < min { γ -2 ρ, (4 -γL ) / 4 L } .

assumption . We achieve that by using the proposed (ER) condition. In particular, for the class of quasi-strongly monotone VIPs, we show a linear convergence rate to a neighborhood of x ∗ when constant step-sizes are used. We also provide theoretically motivated step-size switching rules that guarantee exact convergence of SPEG to x ∗ . In the weak MVI case, we prove the convergence of SPEG for ρ < 1 / 2 L , improving the existing restrictions on ρ . We compare our results with the existing literature in Table 1.

Arbitrary Sampling. Via a stochastic reformulation of the variational inequality problem (1) we explain how our convergence guarantees of SPEG hold under the arbitrary sampling paradigm. This allows us to cover a wide range of samplings for SPEG that were never considered in the literature before, including mini-batching, uniform sampling, and importance sampling as special cases. In this sense, our analysis of SPEG is unified for different sampling strategies. Finally, to highlight the tightness of our analysis, we show that the best-known convergence guarantees of deterministic PEG for strongly monotone and weak MVI can be obtained as special cases of our main theorems.

2 Stochastic Reformulation of VIPs & Single-Call Extragradient Methods

In this work, we provide a theoretical analysis of single-call stochastic extragradient methods that allows us to obtain convergence guarantees of any minibatch and reasonable sampling selection. We achieve that by using the recently proposed 'stochastic reformulation' of the variational inequality problem (1) from [44]. That is, to allow for any form of minibatching, we use the arbitrary sampling notation

where v ∈ R n + is a random sampling vector drawn from a user-defined distribution D such that E D [ v i ] = 1 , for i = 1 , . . . , n . In this setting, the original problem (1) can be equivalently written as,

where the equivalence trivially holds since E D [ F v ( x )] = 1 n ∑ n i =1 E D [ v i ] F i ( x ) = F ( x ) .

In this work, we consider Stochastic Past Extragradient Method ( SPEG ) applied to (6):

where ˆ x -1 = x 0 and v k ∼ D is sampled i.i.d at each iteration and γ k > 0 and ω k > 0 are the extrapolation step-size and update step-size respectively. We note that in our convergence analysis, we allow selecting any distribution D that satisfies E D [ v i ] = 1 ∀ i . This means that for a different selection of D , (7) yields different interpretations of SPEG for solving the original problem (1).

One example of distribution D is τ -minibatch sampling, which is defined as follows.

Definition 2.1 ( τ -Minibatch sampling) . Let τ ∈ [ n ] . Wesay that v ∈ R n is a τ -minibatch sampling if for every subset S ∈ [ n ] with | S | = τ , we have that P [ v = n τ ∑ i ∈ S e i ] := 1 ( n τ ) = τ !( n -τ )! n ! .

By using a double counting argument, one can show that if v is a τ -minibatch sampling, it is also a valid sampling vector ( E D [ v i ] = 1 ) [25]. We highlight that our analysis holds for every form of minibatching and for several choices of sampling vectors v . Later in Section 5, we provide more details related to non-uniform sampling. In addition, by Definition 2.1, it is clear that if τ = n , then v i = 1 for all i ∈ [ n ] . Later in Section 4, we prove how our analysis captures the deterministic Past Extragradient Method as a special case.

In [44], an analysis of stochastic gradient descent-ascent ( x k +1 = x k -ω k F v k ( x k ) ) under the arbitrary sampling paradigm was proposed for solving star-co-coercive VIPs. Later [20], extended this approach and provided general convergence guarantees for stochastic extragradient method ( SEG ) (a stochastic variant of the popular extragradient method [32, 30]) for solving quasi-strongly monotone and monotone VIPs. Despite its popularity, SEG requires two oracle calls per iteration which makes it prohibitively expensive in many large-scale applications and not easily applicable to the online learning problems [18]. This motivates us to explore in detail the convergence guarantees of single-call variants of extragradient methods (extragradient methods that require only a single oracle call per iteration).

On Single-Call Extragradient Methods. The seminal work of [56] is the first paper that proposes the deterministic Past Extagradient method. In the stochastic setting, [28] provides an analysis of several stochastic single-call extragradient methods for solving strongly monotone VIPs. In [28], it was also shown that in the unconstrained setting, the update rules of Past Extragradient and Optimistic Gradient are exactly equivalent (see also Proposition B.6 in appendix). Through this connection, and via our stochastic reformulation (6) our theoretical results hold also for the Stochastic Optimistic Gradient Method ( SOG ): x k +1 = x k -ω k F v k ( x k ) -γ k ( F v k ( x k ) -F v k -1 ( x k -1 )) . [8] provides the convergence guarantees of SOG for weak MVI. To the best of our knowledge, our work is the first that provides convergence guarantees for SOG under the arbitrary sampling paradigm (captures sampling beyond uniform sampling) and also without using the bounded variance assumption.

3 Expected Residual

In our theoretical results, we rely on Expected Residual (ER) condition. In this section, we define ER and explain how it is connected with similar conditions used in optimization literature. We further provide sufficient conditions for ER to hold and prove how it can be used to obtain a strictly weaker upper bound of E ∥ g ( x ) ∥ 2 than previously used growth conditions, expected co-coercivity, or bounded variance assumptions.

Assumption 3.1. We say the Expected Residual (ER) condition holds if there is a parameter δ > 0 such that for an unbiased estimator g ( x ) of the operator F , we have

The ER condition bounds how far the stochastic estimator g ( x ) = F v ( x ) (5) used in SPEG is from the true operator F ( x ) . ER depends on both the properties of the operator F ( x ) and of the selection of sampling (via g ( x ) ). Conditions similar to ER appeared before in optimization literature but they

have never been used in operator theory and the analysis of SPEG . In particular, [24] used a similar condition for analyzing SGD in stochastic optimization problems but with the right-hand side of ER to be the function suboptimality f ( x ) -f ( x ∗ ) (such concept is not available in VIPs). In [68] and [22], similar conditions appear under the name 'Hessian variance' assumption for distributed minimization problems. In the context of distributed VIPs, a similar but stronger condition to ER is used by [5].

Bound on Operator Noise. A common approach for proving the convergence of stochastic algorithms for solving the VIPs is assuming uniform boundedness of the stochastic operator or uniform boundedness of the variance. However, as we explain below, these assumptions either do not hold or are true only for a restrictive set of problems. In our work, we do not assume such bounds. Instead, we use the following direct consequence of ER.

Lemma 3.2. Let σ 2 ∗ := E ∥ g ( x ∗ ) ∥ 2 < ∞ (operator noise at the optimum is finite). If ER holds, then

Sufficient Conditions for ER. Let us now provide sufficient conditions which guarantee that the ER condition holds and give a closed-form expression for the expected residual parameter δ and σ 2 ∗ = E ∥ g ( x ∗ ) ∥ 2 for the case of τ -minibatch sampling (Def. 2.1).

Proposition 3.3. Let F i of problem (1) be L i -Lipschitz operators, then ER holds. If, in addition, vector v ∈ R n is a τ -minibatch sampling (Def. 2.1) then: δ = 2 nτ n -τ n -1 ∑ n i =1 L 2 i , and σ 2 ∗ = 1 nτ n -τ n -1 ∑ n i =1 ∥ F i ( x ∗ ) ∥ 2 .

Similar results to Prop. 3.3 but under different sufficient conditions have been obtained for τ -minibatch sampling under expected smoothness and a variant of expected residual for solving minimization problems in [25] and [24] respectively. In [44], a similar proposition was derived but for the much more restrictive class of co-coercive operators.

Connection to Other Assumptions. In the proofs of our convergence results, we use the bound (8), which, as we explained above, is a direct consequence of ER. In this paragraph, we place this bound in a hierarchy of common assumptions used for the analysis of stochastic algorithms for solving VIPs. In the literature on stochastic algorithms for solving the VIPs and min-max optimization problems, previous works assume either bounded operator ( E ∥ g ( x ) ∥ 2 ≤ c ) [1, 52], bounded variance ( E ∥ g ( x ) -F ( x ) ∥ 2 ≤ c ) [35, 69, 30] (in Appendix C we provide a simple example where bounded variance assumption does not hold) or growth condition ( E ∥ g ( x ) ∥ 2 ≤ c 1 ∥ F ( x ) ∥ 2 + c 2 ) [36]. In all of these conditions, the parameters c , c 1 , and c 2 are usually constants that do not have a closed-form expression. The closer works to our results are [44, 6] which assumes existence of l F > 0 such that the expected co-coercivity condition ( E ∥ g ( x ) -g ( x ∗ ) ∥ 2 ≤ l F ⟨ F ( x ) , x -x ∗ ⟩ ) holds. Their convergence guarantees provide an efficient analysis for several variants of SGDA for solving cocoercive VIPs. In the proposition below, we prove how these conditions are related to the bound (8) obtained using ER.

Proposition 3.4. Suppose F is a L -Lipschitz operator. Then we have the following hierarchy of assumptions:

Bounded Operator

Bounded Variance

Growth Condition

(8)

F

i

are

L

i

-Lipschitz

(ER)

Expected Cocoercivity

Let us also mention that [29] provided convergence guarantee of double-oracle stochastic extragradient ( SEG ) method under the variance control condition E ∥ g ( x ) -F ( x ) ∥ 2 ≤ ( a ∥ x -x ∗ ∥ + b ) 2 where a, b ≥ 0 . In their work, they focus on solving VIPs satisfying the error-bound condition, and they did not provide closed-form expressions of parameters a and b . Although the analysis of [29]

can be conducted with a > 0 , the authors only provide rates for the case a = 0 . The main difference between their results (for SEG ) and our results (for SPEG ) is that our bound (8) is not really an assumption, but it holds for free when F i are L i -Lipschitz. In addition, the values of parameters δ and σ 2 ∗ in (8) could have different values based on the sampling used in the update rule of SPEG .

4 Convergence Analysis

In this section, we present and discuss the main convergence results of this work. In the first part, we focus on the ones derived for µ -quasi strongly monotone problems (3) (both for constant and decreasing step-sizes), and in the second part on the Weak Minty VIP (4).

4.1 Quasi-Strongly Monotone Problems

Constant Step-size: We start with the case of µ -quasi strongly monotone problems and consider the convergence of SPEG with constant step-size.

Theorem 4.1. Let F be L -Lipschitz, µ -quasi strongly monotone, and let ER hold. Choose stepsizes γ k = ω k = ω such that

for all k . Then the iterates produced by SPEG , given by (7) satisfy

where R 2 k := E [ ∥ x k -x ∗ ∥ 2 + ∥ x k -ˆ x k -1 ∥ 2 ] . Hence, given any ε > 0 , and choosing ω = min { µ 18 δ , 1 4 L , εµ 48 σ 2 ∗ } , SPEG achieves E ∥ x K -x ∗ ∥ 2 ≤ ε after K ≥ max { 8 L µ , 36 δ µ 2 , 96 σ 2 ∗ εµ 2 } log ( 2 R 2 0 ε ) iterations.

To the best of our knowledge, the above theorem is the first result on the convergence of SPEG that does not rely on the bounded variance assumption. Theorem 4.1 recovers the same rate of convergence with the Independent-Samples SEG ( I-SEG ) under assumption (8) [20], although [20] simply assume (8), while we show that it follows from Assumption 3.1 holding whenever all summands F i are Lipschitz. However, in the case when all F i are µ -quasi strongly monotone and L i -Lipschitz (on average), one can use Same-Sample SEG ( S-SEG ). The existing results for S-SEG have better exponentially decaying term [49, 20] then Theorem 4.1, e.g., in the case when L i = L for all i ∈ [ n ] , we have δ = O ( L 2 ) meaning that the exponentially decaying term in (10) is O ( R 2 0 exp( -µ 2 k / L 2 ))) , while S-SEG has much better exponentially decaying term O ( R 2 0 exp( -µk / L ))) .

Such a discrepancy can be partially explained by the following fact: S-SEG can be seen as one step of deterministic Extragradient for stochastic operator F v k allowing to use one-iteration analysis of Extragradient without controlling the variance. In contrast, there is no version of SPEG that uses the same sample for extrapolation and update steps. This forces to use different samples for these steps and this is a key reason why SPEG cannot be seen as one iteration of deterministic Past-Extragradient for some operator. Due to this, we need to rely on some bound on the variance to handle the stochasticity in the updates; see also [20, Appendix F.1]. Therefore, in our analysis, we use Assumption 3.1, implying (8). Nevertheless, it is still an open question whether it is possible to improve the rate of SPEG in the case of µ -quasi strongly monotone and Lipschitz operators F i .

To highlight the generality of Theorem 4.1, we note that for the deterministic PEG , δ = 0 and σ 2 ∗ = 0 (by selecting τ = n in the definition 2.1 of minibatch sampling). In this case, Theorem 4.1 recovers the well-known result (up to 1 / 2 factor in the rate) for deterministic PEG proposed in [17] as shown in the following corollary.

Corollary 4.2. Let the assumptions of Theorem 4.1 hold and a deterministic version of SPEG is considered, i.e., δ = 0 , σ 2 ∗ = 0 . Then, Theorem 4.1 implies that for all k ≥ 0 the iterates produced

by SPEG with step-sizes γ k = ω k = ω such that 0 < ω ≤ 1 4 L satisfy R 2 k ≤ ( 1 -ωµ 2 ) k R 2 0 , where R 2 k := ∥ x k -x ∗ ∥ 2 + ∥ x k -ˆ x k -1 ∥ 2 .

Decreasing Step-size: In this section, we consider two different decreasing step-sizes policies for SPEG applied to solve quasi-strongly monotone problems.

Theorem 4.3. Let F be L -Lipschitz, µ -quasi strongly monotone, and Assumption 3.1 hold. Let

where ¯ ω := min { 1 / (4 L ) , µ / (18 δ ) } and k ∗ = ⌈ 4 / ( µ ¯ ω ) ⌉ . Then for all K ≥ k ∗ the iterates produced by SPEG with step-sizes (11) satisfy

where R 2 K := E [ ∥ x K -x ∗ ∥ 2 + ∥ x K -ˆ x K -1 ∥ 2 ] .

SPEG with step-size policy 2 (11) has two stages of convergence: during first k ∗ iterations it uses constant step-size to reach some neighborhood of the solution and then the method switches to the decreasing O ( 1 / k ) step-size allowing to reduce the size of the neighborhood.

For the case of strongly monotone problems (a special case of our quasi-strongly monotone setting) [28] also analyze SPEG with decreasing O ( 1 / k ) step-size 3 under bounded variance assumption, i.e., when (8) holds with δ = 0 and some σ 2 ∗ ≥ 0 , which is equivalent to the uniformly bounded variance assumption. In particular, Theorem 5 [28] states E [ ∥ x K -x ∗ ∥ 2 ] ≤ Cσ 2 ∗ µ 2 K + o ( 1 K ) where C is some numerical constant. If the problem is strongly monotone, the result of [28] is closely related to what is obtained in Theorem 4.3: the main difference in the upper-bound is that we provide an explicit form of o ( 1 / K ) term. Moreover, in contrast to the result from [28], Theorem 4.3 holds even when δ > 0 in (8), which covers a larger class of problems.

Following [67, 20, 6], we also consider another decreasing step-size policy.

Theorem 4.4. Let F be L -Lipschitz, µ -quasi strongly monotone, and Assumption 3.1 hold. Let ¯ ω := min { 1 / (4 L ) , µ / (18 δ ) } . If for K ≥ 0 step-sizes { γ k } k ≥ 0 , { ω k } k ≥ 0 satisfy γ k = ω k and

where k 0 = ⌈ K / 2 ⌉ , then the iterates produced by SPEG with the step-sizes defined above satisfy

In contrast to (12), the rate from (14) has much better (exponentially decaying) o ( 1 / K ) term. When σ 2 ∗ is large and one needs to achieve very good accuracy of the solution, this difference is negligible, since the dominating O ( 1 / K ) term is the same for both bounds (up to numerical factors). However, when σ 2 ∗ is small enough, e.g., the model is close to over-parameterized, or it is sufficient to achieve low accuracy of the solution, the dominating term in (14) is typically much smaller than the one from (12). Finally, it is worth mentioning, that the improvement of o ( 1 / K ) is not achieved for free: unlike the policy from (11), step-size rule (13) relies on the knowledge of the total number of steps K , which can be inconvenient for the practical use in some cases.

4.2 Weak Minty Variational Inequality Problems

In this subsection we will discuss convergence of Stochastic Past Extragradient method for Minty Variational Inequality problem. To solve the Minty variational inequality problem we use different step-sizes for SPEG iterates (7).

Theorem 4.5. Let F be L -Lipschitz and satisfy Weak Minty condition with parameter ρ < 1 / (2 L ) . Let Assumption 3.1 hold. Assume that γ k = γ , ω k = ω such that max { 2 ρ, 1 2 L } < γ < 1 L and 0 < ω < min { γ -2 ρ, 1 4 L -γ 4 } . Then, for all K ≥ 2 the iterates produced by mini-batched SPEG with batch-size

The above result establishes O ( 1 / K ) convergence with O ( K ) batchsizes for SPEG applied to problems satisfying Weak Minty condition. 4 The closest result is obtained by [8], for the same method under bounded variance assumption, i.e., when δ = 0 . In particular, the result of [8] also gives O ( 1 / K ) rate and requires O ( K ) batchsizes at each step. We extend this result to the case of non-zeroth δ and we also improve the assumption on ρ : [8] assumes that ρ < 3 / 8 L , while Theorem 4.5 holds for ρ < 1 / 2 L . The bound on ρ cannot be improved even in the deterministic case [23]. Moreover, it is worth mentioning that the proof of Theorem 4.5 noticeably differs from the one obtained by [8].

In the case of a deterministic oracle, we recover the best-known result for Optimistic Gradient in the Weak Minty setup [8, 23].

Corollary 4.6. Let the assumptions of Theorem 4.5 hold and deterministic version of SPEG is considered, i.e., δ = 0 , σ 2 ∗ = 0 . Then, Theorem 4.5 implies that for all k ≥ 0 the iterates produced by SPEG with step-sizes max { 2 ρ, 1 2 L } < γ < 1 L and 0 < ω < min { γ -2 ρ, 1 4 L -γ 4 } satisfy min 0 ≤ k ≤ K -1 ∥ F (ˆ x k ) ∥ 2 ≤ C ∥ x 0 -x ∗ ∥ 2 K -1 , where C = 48 ωγ (1 -L ( γ +4 ω )) .

5 Beyond Uniform Sampling

In this section, we illustrate the generality of our analysis by focusing on the non-uniform sampling. In particular, we focus on single-element sampling in which only the singleton sets { i } for i = { 1 , . . . , n } have a non-zero probability of being sampled; that is, P [ | S | = 1] = 1 . We have P [ v = e i /p i ] = p i . [25] proved that if v is a single-element sampling, it is also a valid sampling vector ( E D [ v i ] = 1 ). With the following proposition, we provide closed-form expressions for the ER parameter δ and σ 2 ∗ = E ∥ g ( x ∗ ) ∥ 2 for the case of (non-uniform) single-element sampling.

Proposition 5.1. Let F i of problem (1) be L i -Lipschitz operators. If, vector v ∈ R n is a single element sampling then δ = 2 n 2 ∑ n i =1 L 2 i p i and σ 2 ∗ = 1 n 2 ∑ n i =1 1 p i ∥ F i ( x ∗ ) ∥ 2 .

Importance Sampling. In importance sampling we aim to choose the probabilities p i that optimize the iteration complexity. [25] and [20] analyze importance sampling for SGD and SEG respectively. In this work, we provide the first convergence guarantees of SPEG with importance sampling. In particular, we optimize the expected residual parameter δ with respect to p i , which in turn affects the iteration complexity. Note that, by using Cauchy-Schwarz inequality (20), we have ∑ n i =1 L 2 i p i ≥ ( ∑ n i =1 L i ) 2 , and this lower bound can be achieved for p δ i = L i / ∑ n j =1 L j . In case of importance sampling, we will use these probabilities p δ i which optimizes δ and define the corresponding δ as δ IS := 2 n 2 ( ∑ n i =1 L i ) 2 . For uniform sampling ( i.e. p i = 1 n ) , the value of the parameter is δ US = 2 n ∑ n i =1 L 2 i . Note that, δ IS equals δ US when all L i are equal, however δ IS can be much smaller than δ US when L i are very different from each other, e.g., when all L i are relatively small (close to

zero) and one L i is large, δ IS is almost n times smaller than δ US. In this latter scenario (when δ IS is much smaller than δ US), importance sampling could be useful and can significantly improve the performance of SPEG . For example, note that the exponentially decaying term in (14) decreases with δ . Hence, this term will decrease much faster with importance sampling than with uniform sampling.

6 Numerical Experiments

To verify our theoretical results, we run several experiments on two classes of problems, i.e., strongly monotone problems (a special case of the quasi-strongly monotone VIPs) and weak MVI problems. The code to reproduce our results can be found at https://github.com/isayantan/Single-Call-StochasticExtragradient-Methods.

6.1 Strongly Monotone Problems

Our experiments consider the quadratic strongly-convex strongly-concave min-max problem from [20]. That is, we implement SPEG on quadratic games of the form min x ∈ R d max y ∈ R d 1 n ∑ n i =1 f i ( x, y ) where

Here A i , B i , C i are generated such that the quadratic game is strongly monotone and smooth. In all our experiments, we take n = 100 and d = 30 . We generate positive semi-definite matrices A i , B i , C i such that their eigenvalues lie in the interval [ µ A , L A ] , [ µ B , L B ] and [ µ C , L C ] respectively. In all our experiments, we consider L A = L B = L C = 1 and µ A = µ C = 0 . 1 , µ B = 0 unless otherwise mentioned. The vectors a i and c i are generated from N d (0 , I d ) . Here, the i th operator is given by

∥

x

∥

k

x

0

-

-

x

x

∗

∗

∥

∥

2

2

.

0

20000

40000

60000

80000

100000

Number of Iterations

10

2

10

1

10 0

Relative Error

Decreasing Stepsize (Theorem 4.4)

Change of step-size

In Figures 1, 2, and 3, we plot the relative error on the y -axis i.e.

Constant vs Switching Step-size Rule. In Fig. 1, we illustrate the step-size switching rule of Theorem 4.3. We place the dotted line to mark when we switch from constant step-size to decreasing step-size. In Fig. 1, the trajectory of switching stepsize rule (11) matches that of constant step-size (9) in the first phase ( where SPEG runs with constant step-size following (11) ) . However, it becomes stagnant when the constant stepsize SPEG reaches a neighbourhood of optimality. In contrast, the step-size of Theorem 4.3 helps the method to converge to better accuracy.

Comparison to Hsieh et al. [28]. In this experiment, we compare SPEG step-sizes proposed in Theorems 4.1 and 4.3

with step-sizes from [28]. To implement SPEG with the step-sizes from [28], we choose γ and b such that 1 µ < γ ≤ b 4 L and set ω k = γ k = γ k + b . For Fig. 2a, we generate A i , B i , C i as before. First, we sample optimal points x ∗ , y ∗ from N d (0 , I d ) and then generate a i , c i such that F ( x ∗ , y ∗ ) = 0 .

In Fig. 2a, we run the algorithms on interpolated model ( F i ( x ∗ ) = 0 for all i ∈ [ n ] ) . Since the model is interpolated, we have σ 2 ∗ = 0 in Theorem 4.1 and linear convergence to the exact optimum asymptotically. In this setting, as shown in Fig. 2a, our proposed step-size results in major improvement compared to the decreasing step-size selection analyzed in [28]. In Fig. 2b, we compare the switching step-size rule with step-size from [28]. In Fig. 2b, we generate a i , c i from the normal distribution. In this plot, we manually switch the step-size from constant to decreasing after 305 steps. We observe that such a semi-empirical rule has comparable performance to the step-size selection of Hsieh et al. [28].

0

2000

4000

6000

8000

10000

Number of Iterations

10

27

10

23

10

19

10

15

10

11

10

7

10

3

10 1

Relative Error

Constant stepsize (Theorem 4.1)

Decreasing stepsize (Hsieh et al. 2019)

(a) Interpolated Model

0

10000

20000

30000

40000

50000

Number of Iterations

10

2

10

1

10 0

10 1

Relative Error

Decreasing stepsize (Theorem 4.4)

Decreasing stepsize (Hsieh et al, 2019)

Change of step-size

(b) Non-Interpolated Model

0

200

400

600

800

1000

Iterations

10

2

10

1

10 0

Relative Error

Important

Uniform

(a)

Λ = 2

0

200

400

600

800

1000

Iterations

10

2

10

1

10 0

Relative Error

Important

Uniform

(b)

Λ = 5

0

200

400

600

800

1000

Iterations

10

2

10

1

10 0

Relative Error

Important

Uniform

(c)

Λ = 10

0

200

400

600

800

1000

Iterations

10

2

10

1

10 0

Relative Error

Important

Uniform

(d)

Λ = 20

Uniform vs. Importance Sampling. In this experiment, we highlight the advantage of using importance sampling over uniform sampling. The eigenvalues of A 1 , C 1 are uniformly generated from the interval [0 . 1 , Λ] while the rest of the matrices are generated as mentioned before. We vary the value of Λ ∈ { 2 , 5 , 10 , 20 } and run and compare SPEG with both uniform and importance sampling (see Fig. 3). For importance sampling, we use the probabilities p i = L i / ∑ n j =1 L j . In Fig. 3, it is clear that as the value of Λ increases, the trajectories under uniform sampling get worse, while the trajectory under importance sampling remains almost identical. This behavior aligns well with our discussion in Section 5.

6.2 Weak Minty Variational Inequality Problems

This experiment verifies the convergence guarantees of SPEG in Theorem 4.5. Following the min-max problem mentioned in [8], we consider the objective function

In this experiment, we generate ξ i , ζ i such that L = 8 and ρ = 1 / 32 for the above min-max problem [8]. We implement SPEG with extrapolation step γ k = 0 . 08 and update step ω k = 0 . 01 which satisfies the conditions on step-size in Theorem 4.5. In Fig. 4, we use a batchsize of 6 . This plot illustrates that for some weak MVI problems the requirement on the stepsize from Theorem 4.5 can be too pessimistic and SPEG with relatively small batchsize achieves reasonable accuracy of the solution. The choice of batchsize ensures that bound (15) holds and δ is small enough to guarantee convergence of SPEG . We

0

200

400

600

800

1000

Number of Iterations

10

8

10

6

10

4

10

2

10 0

10 2

Squared Operator Norm Error

SPEG on WMVI

SPEG

also tried to compare SPEG with SEG+ from [54], however, the authors do not mention their choice of update step-size. We examined several decreasing update step-size for which SEG+ failed to converge. Further details on experiments can be found in Appendix G.1.

References

J. Abernethy, K. A. Lai, and A. Wibisono. Last-iterate convergence rates for min-max optimization: Convergence of hamiltonian gradient descent and consensus optimization. In Algorithmic Learning Theory , pages 3-47. PMLR, 2021.

M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In ICML , 2017.

S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. NeurIPS , 2019.

H. H. Bauschke, W. M. Moursi, and X. Wang. Generalized monotone operators and their averaged resolvents. Mathematical Programming , 189(1):55-74, 2021.

A. Beznosikov and A. Gasnikov. Compression and data similarity: Combination of two techniques for communication-efficient solving of distributed variational inequalities. In Optimization and Applications: 13th International Conference, OPTIMA 2022, Petrovac, Montenegro, September 26-30, 2022, Revised Selected Papers , pages 151-162. Springer, 2023.

A. Beznosikov, E. Gorbunov, H. Berard, and N. Loizou. Stochastic gradient descent-ascent: Unified theory and new efficient methods. AISTATS , 2023.

A. Beznosikov, B. Polyak, E. Gorbunov, D. Kovalev, and A. Gasnikov. Smooth monotone stochastic variational inequalities and saddle point problems-survey. European Mathematical Society Magazine , 2023.

A. Böhm. Solving nonconvex-nonconcave min-max problems exhibiting weak minty solutions. arXiv preprint arXiv:2201.12247 , 2022.

N. Brown, A. Bakhtin, A. Lerer, and Q. Gong. Combining deep reinforcement learning and search for imperfect-information games. NeurIPS , 2020.

Y. Cai, A. Oikonomou, and W. Zheng. Tight last-iterate convergence of the extragradient method for constrained monotone variational inequalities. arXiv preprint arXiv:2204.09228 , 2022.

P. L. Combettes and T. Pennanen. Proximal methods for cohypomonotone operators. SIAM journal on control and optimization , 43(2):731-742, 2004.

C. D. Dang and G. Lan. On the convergence properties of non-euclidean extragradient methods for variational inequalities with generalized monotone operators. Computational Optimization and applications , 60(2):277-310, 2015.

C. Daskalakis, S. Skoulakis, and M. Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 1466-1478, 2021.

J. Diakonikolas, C. Daskalakis, and M. Jordan. Efficient methods for structured nonconvexnonconcave min-max optimization. In AISTATS , 2021.

R. D'Orazio, N. Loizou, I. Laradji, and I. Mitliagkas. On stochastic mirror descent: Convergence analysis and adaptive variants. ICML , 2021.

F. Facchinei and C. Kanzow. Generalized Nash equilibrium problems. 4OR , 5(3):173-210, 2007.

G. Gidel, H. Berard, G. Vignoud, P. Vincent, and S. Lacoste-Julien. A variational inequality perspective on generative adversarial networks. ICLR , 2019.

N. Golowich, S. Pattathil, and C. Daskalakis. Tight last-iterate convergence rates for no-regret learning in multi-player games. NeurIPS , 2020.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NeurIPS , 2014.

E. Gorbunov, H. Berard, G. Gidel, and N. Loizou. Stochastic extragradient: General analysis and improved rates. In AISTATS , 2022.

E. Gorbunov, N. Loizou, and G. Gidel. Extragradient method: O (1/k) last-iterate convergence for monotone variational inequalities and connections with cocoercivity. In AISTATS , 2022.

E. Gorbunov, S. Horváth, P. Richtárik, and G. Gidel. Variance reduction is an antidote to byzantines: Better rates, weaker assumptions and communication compression as a cherry on the top. ICLR , 2023.

E. Gorbunov, A. Taylor, S. Horváth, and G. Gidel. Convergence of proximal point and extragradient-based methods beyond monotonicity: the case of negative comonotonicity. ICML , 2023.

R. Gower, O. Sebbouh, and N. Loizou. Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation. In AISTATS , 2021.

R. M. Gower, N. Loizou, X. Qian, A. Sailanbayev, E. Shulgin, and P. Richtárik. Sgd: General analysis and improved rates. In ICML , 2019.

F. Hanzely and P. Richtárik. Accelerated coordinate descent with arbitrary sampling and best rates for minibatches. In AISTATS , 2019.

S. Horváth and P. Richtárik. Nonconvex variance reduced optimization with arbitrary sampling. In ICML , 2019.

Y.-G. Hsieh, F. Iutzeler, J. Malick, and P. Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. NeurIPS , 2019.

Y.-G. Hsieh, F. Iutzeler, J. Malick, and P. Mertikopoulos. Explore aggressively, update conservatively: Stochastic extragradient methods with variable stepsize scaling. NeurIPS , 2020.

A. Juditsky, A. Nemirovski, and C. Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems , 1(1):17-58, 2011.

A. Khaled, O. Sebbouh, N. Loizou, R. M. Gower, and P. Richtárik. Unified analysis of stochastic gradient methods for composite convex and smooth optimization. Journal of Optimization Theory and Applications , 2020.

G. M. Korpelevich. The extragradient method for finding saddle points and other problems. Matecon , 12:747-756, 1976.

S. Lee and D. Kim. Fast extra gradient methods for smooth structured nonconvex-nonconcave minimax problems. NeurIPS , 2021.

C. J. Li, Y . Yu, N. Loizou, G. Gidel, Y . Ma, N. Le Roux, and M. Jordan. On the convergence of stochastic extragradient for bilinear games using restarted iteration averaging. In AISTATS , 2022.

T. Lin, C. Jin, and M. Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In ICML , 2020.

T. Lin, Z. Zhou, P. Mertikopoulos, and M. Jordan. Finite-time last-iterate convergence for multi-agent learning in games. In ICML , 2020.

M. Liu, Y. Mroueh, J. Ross, W. Zhang, X. Cui, P. Das, and T. Yang. Towards better understanding of adaptive gradient algorithms in generative adversarial nets. ICLR , 2020.

M. Liu, H. Rafique, Q. Lin, and T. Yang. First-order convergence theory for weakly-convexweakly-concave min-max problems. J. Mach. Learn. Res. , 22:169-1, 2021.

N. Loizou and P. Richtárik. A new perspective on randomized gossip algorithms. In 2016 IEEE global conference on signal and information processing (GlobalSIP) , pages 440-444. IEEE, 2016.

N. Loizou and P. Richtárik. Convergence analysis of inexact randomized iterative methods. SIAM Journal on Scientific Computing , 42(6):A3979-A4016, 2020.

N. Loizou and P. Richtárik. Momentum and stochastic momentum for stochastic gradient, newton, proximal point and subspace descent methods. Computational Optimization and Applications , 77(3):653-710, 2020.

N. Loizou and P. Richtárik. Revisiting randomized gossip algorithms: General framework, convergence rates and novel block and accelerated protocols. IEEE Transactions on Information Theory , 67(12):8300-8324, 2021.

N. Loizou, H. Berard, A. Jolicoeur-Martineau, P. Vincent, S. Lacoste-Julien, and I. Mitliagkas. Stochastic hamiltonian gradient methods for smooth games. In ICML , 2020.

N. Loizou, H. Berard, G. Gidel, I. Mitliagkas, and S. Lacoste-Julien. Stochastic gradient descent-ascent and consensus optimization for smooth games: Convergence analysis under expected co-coercivity. NeurIPS , 2021.

N. Loizou, S. Vaswani, I. Hadj Laradji, and S. Lacoste-Julien. Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In AISTATS , 2021.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

P. Mertikopoulos and Z. Zhou. Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming , 173(1):465-507, 2019.

P. Mertikopoulos, B. Lecouat, H. Zenati, C.-S. Foo, V. Chandrasekhar, and G. Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. ICLR , 2019.

K. Mishchenko, D. Kovalev, E. Shulgin, P. Richtárik, and Y. Malitsky. Revisiting stochastic extragradient. In AISTATS , 2020.

A. Mokhtari, A. Ozdaglar, and S. Pattathil. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In AISTATS , 2020.

H. Namkoong and J. C. Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. NeurIPS , 2016.

A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4):1574-1609, 2009.

Y. Nesterov. Linear convergence of first order methods for non-strongly convex optimization. Mathematical Programming , 175:69-107, 2019.

T. Pethick, P. Patrinos, O. Fercoq, V . Cevherå, et al. Escaping limit cycles: Global convergence for constrained nonconvex-nonconcave minimax problems. In ICLR , 2022.

T. Pethick, O. Fercoq, P. Latafat, P. Patrinos, and V. Cevher. Solving stochastic weak minty variational inequalities without increasing batch size. ICLR , 2023.

L. D. Popov. A modification of the arrow-hurwicz method for search of saddle points. Mathematical notes of the Academy of Sciences of the USSR , 28(5):845-848, 1980.

X. Qian, Z. Qu, and P. Richtárik. Saga with arbitrary sampling. In ICML , 2019.

Z. Qu and P. Richtárik. Coordinate descent with arbitrary sampling i: Algorithms and complexity. Optimization Methods and Software , 31(5):829-857, 2016.

P. Richtárik and M. Takáˇ c. On optimal probabilities in stochastic coordinate descent methods. arXiv preprint arXiv:1310.3438 , 2013.

P. Richtárik and M. Takác. Stochastic reformulations of linear systems: algorithms and convergence theory. SIAM Journal on Matrix Analysis and Applications , 41(2):487-524, 2020.

M. Rolinek and G. Martius. L4: Practical loss-based stepsize adaptation for deep learning. NeurIPS , 2018.

E. K. Ryu, K. Yuan, and W. Yin. Ode analysis of stochastic gradient methods with optimism and anchoring for minimax problems. arXiv preprint arXiv:1905.10899 , 2019.

O. Sebbouh, N. Gazagnadou, S. Jelassi, F. Bach, and R. Gower. Towards closing the gap between the theory and practice of svrg. NeurIPS , 2019.

S. Sokota, R. D'Orazio, J. Z. Kolter, N. Loizou, M. Lanctot, I. Mitliagkas, N. Brown, and C. Kroer. A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games. ICLR , 2023.

M. V. Solodov and B. F. Svaiter. A hybrid approximate extragradient-proximal point algorithm using the enlargement of a maximal monotone operator. Set-Valued Analysis , 7(4):323-345, 1999.

C. Song, Z. Zhou, Y. Zhou, Y. Jiang, and Y. Ma. Optimistic dual extrapolation for coherent non-monotone variational inequalities. In NeurIPS , 2020.

S. U. Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232 , 2019.

R. Szlendak, A. Tyurin, and P. Richtárik. Permutation compressors for provably faster distributed nonconvex optimization. ICLR , 2022.

Q. Tran Dinh, D. Liu, and L. Nguyen. Hybrid variance-reduced sgd algorithms for minimax problems with nonconvex-linear function. NeurIPS , 2020.

S. Vaswani, F. Bach, and M. Schmidt. Fast and faster convergence of sgd for over-parameterized models and an accelerated perceptron. In AISTATS , 2019.

S. Vaswani, A. Mishkin, I. Laradji, M. Schmidt, G. Gidel, and S. Lacoste-Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. NeurIPS , 2019.

J. Wang, T. Zhang, S. Liu, P.-Y. Chen, J. Xu, M. Fardad, and B. Li. Adversarial attack generation empowered by min-max optimization. NeurIPS , 2021.

Y. Yu, T. Lin, E. V. Mazumdar, and M. Jordan. Fast distributionally robust learning with variance-reduced min-max optimization. In AISTATS , 2022.

Supplementary Material

We organize the Supplementary Material as follows: Section A discusses the existing literature related to our work. In Section B, we present some technical lemmas required for our analysis, while in Section C, we provide a simple problem where the bounded variance assumption does not hold. Then, in Section D, we provide the proofs of propositions related to Expected Residual. Next, Section E presents the proofs of the main theorems, while a proposition related to arbitrary sampling is proved in Section F. Finally, additional numerical experiments are presented in Section G.

Contents

A Further Related Work

The references necessary to motivate our work and connect it to the most relevant literature are included in the appropriate sections of the main body of the paper. In this section, we present a broader view of the literature, including more details on closely related work and more references to papers that are not directly related to our main results.

Classes of Structured Non-monotone Operators. With an increasing interest in improved computational speed, first-order methods are the primary choice for solving VIPs. However, computation of an approximate first-order locally optimal solution of a general non-monotone VIP is intractable [13, 33]. It motivates us to exploit the additional structures prevalent in large classes of non-monotone VIPs. Recently [20, 28] provide convergence guarantees of stochastic methods for solving quasi-strongly monotone VIPs, while [29] for problems satisfying error-bound conditions. [14] defined the notion of a weak MVI (4) covering classes of non-monotone VIPs.

Assumptions on Operator Noise. The standard analysis of stochastic methods for solving VIPs relies on bounded variance assumption. [8, 14, 28, 17] use bounded variance assumption (i.e. E ∥ F i ( x ) -F ( x ) ∥ 2 ≤ σ 2 for all x ) while [52, 1] assume bounded operators for their analysis. However, there are examples of simple quadratic games that do not satisfy these conditions. It has motivated researchers to look for alternative/relaxed assumptions on distributions. [44] provides convergence of Stochastic Gradient Descent Ascent Method under Expected Cocoercivity. [29, 49] considered alternative assumptions for analyzing Stochastic Extragradient Methods that do not imply boundedness of the variance. However, there is no analysis of single-call extragradient methods without bounded variance assumption.

Weak Minty Variational Inequalities. Numerous contemporary studies look to identify first-order methods for efficiently solving min-max optimization problems. It varies from simple convexconcave to nontrivial nonconvex nonconcave objectives. Though there has been a significant development in the convex-concave setting, [13] demonstrates that even finding local solutions are intractable for general nonconvex nonconcave objectives. Therefore, researchers seek to identify the structure of objective functions for which it is possible to resolve the intractability issues. [14] proposes the notion of non-monotonicity, which generalizes the existence of a Minty solution (i.e., ρ = 0 in (4)). This problem is known as weak Minty variational inequality in the literature. [14, 54] provides convergence guarantees of the Extragradient Method for weak Minty variational inequality. They establish a convergence rate of O (1 /k ) for the squared operator norm. [33] shows that it is possible to have an accelerated extragradient method even for non-monotone problems. Furthermore, [8] provides a convergence guarantee for the SOG with a complexity bound of O ( ε -2 ) . However, all papers exploring stochastic extragradient methods for solving weak Minty variational inequality consider bounded variance assumption [8, 14]. Moreover, all algorithms solving Weak Minty variational inequality require increasing batchsize. Recently, [55] introduced BCSEG+ which can solve weak minty variational inequality without increasing batchsize. BCSEG+ involves three oracle calls per iteration and addition of a bias-corrected term in the extrapolation step.

Arbitrary Sampling Paradigm. As we mentioned in the main paper, the stochastic reformulation (6) of the original problem (1) allows us to analyze single-call extragradient methods under the arbitrary sampling paradigm. That is, provide a unified analysis for SPEG that captures multiple sampling strategies, including τ -minibatch and importance samplings. An arbitrary sampling analysis of a stochastic optinmization method was first proposed in the context of the randomized coordinate descent method for solving strongly convex functions in [59]. Since then, several other stochastic methods were studied in this regime, including accelerated coordinate descent algorithms [58, 26], randomized iterative methods for solving consistent linear systems [60, 41, 40], randomized gossip algorithms [39, 42], stochastic gradient descent ( SGD ) [25, 24], and variance reduced methods [57, 27, 31]. The first analysis of stochastic algorithms under the arbitrary sampling paradigm for solving variational inequality problems was proposed in [43, 44]. In [43, 44], the authors focus on algorithms like the stochastic Hamiltonian method, the stochastic gradient descent ascent, and the stochastic consensus optimization. These ideas were later extended to the case of Stochastic Extragradient by [20]. To the best of our knowledge, our work is the first that provides an analysis of single-call extragradient methods under the arbitrary sampling paradigm.

Overparameterized Models and Interpolation. For a function f ( x ) := 1 n ∑ n i =1 f i ( x ) we say that interpolation condition holds if there exists x ∗ such that min x f i ( x ) = f i ( x ∗ ) for all i ∈ [ n ] (or equivalently ∇ f i ( x ∗ ) = 0 for smooth convex functions) [24]. The interpolation condition is satisfied when the underlying models are sufficiently overparameterized [70]. Some known examples include deep matrix factorization and classification using neural networks [3, 61, 70]. The interpolated model structure enables SGD and other optimization algorithms to have faster convergence [24, 45, 15]. Inspired by this, one can extend the notion of the interpolation condition to operators. In this scenario, we say that the VIP (1) is interpolated if there exists solution x ∗ of (1) such that F i ( x ∗ ) = 0 for all i ∈ [ n ] . This concept has been explored for analyzing the stochastic extragradient method in [71, 34]. We highlight that our proposed theorems show fast convergence of SPEG in this interpolated regime (when σ 2 ∗ = 0 ). To the best of our knowledge, our work is the first that proves such convergence for SPEG . In Fig. 2a, we experimentally verify the fast convergence for solving a strongly monotone interpolated problem.

Deterministic Extragradient Methods. The Extragradient method ( EG ) [32] and its single-call variant, Optimistic Gradient ( OG ) [56], were proposed to overcome the convergence issues of gradient descent-ascent method for solving monotone problems. Since their introduction, these methods have been revisited and explored in various ways. [50] analyzed EG and OG as an approximation of the Proximal Point method to solve bilinear and strongly convex-strongly concave min-max problems. [65] and [62] provide the best-iterate convergence guarantees of EG and OG with a rate of O ( 1 / K ) for solving monotone problems. However, providing a last-iterate convergence rate of EG and OG for monotone VIPs has been a long-lasting open problem that was only recently resolved. The works of [18, 21, 10] prove a last-iterate O ( 1 / K ) convergence rate for these methods. Finally, in the deterministic setting, some recent works provide convergence analysis of EG and OG for solving weak MVI (4) [14, 54, 8, 23].

B Technical Preliminaries

Throughout our work, we assume

Assumption B.1. Operator F in (1) is L Lipschitz, i.e., ∀ x, y ∈ R d operator F satisfies

Operators F i : R d → R d of problem (1) are L i - Lipschitz, i.e., ∀ x, y ∈ R d operator F i satisfies

In our proofs, we often use the following simple inequalities.

Lemma B.2. For all a, b, a 1 , a 2 , · · · a n ∈ R d , n ≥ 1 , α > 0 , we have the following inequalities:

Inequality (22) is well known as Young's Inequality. Now, we present a simple property of unbiased estimators.

Next, we present the following lemma from [67], which plays a vital role in proving the convergence guarantee of Theorem 4.4.

Lemma B.4. (Simplified Verison of Lemma 3 from [67]) Let the non-negative sequence { r k } k ≥ 0 satisfy the relation r k +1 ≤ (1 -aγ k ) r k + cγ 2 k for all k ≥ 0 , parameters a, c ≥ 0 and any nonnegative sequence { γ k } k ≥ 0 such that γ k ≤ 1 h for some h ≥ a, h > 0 . Then for any K ≥ 0 one can choose { γ k } k ≥ 0 as follows:

where κ = 2 h a and k 0 = ⌈ K 2 ⌉ . For this choice of γ k the following inequality holds:

We use the next lemma to bound the trace of matrix products.

Lemma B.5. For positive semidefinite matrices A,B ∈ R d × d we have

where λ max ( B ) denotes the maximum eigenvalue of B .

Next lemma proves equivalence of SPEG and SOG :

Proposition B.6 ( Equivalence of SPEG and SOG ) . Consider the iterates of SPEG { x k , ˆ x k } ∞ k =1 with constant step-sizes ω k = ω, γ k = γ in (7). Then ˆ x k follows the iteration rule of SOG i.e.

Proof. From the update rule of SPEG (7) we get

This shows that SPEG iterations are equivalent to SOG , with ˆ x k being the k -th iterate of SOG .

C Example: A Problem where the Bounded Variance Condition not Hold

Here, we provide a simple problem that does not satisfy the bounded variance assumption. Consider the linear regression problem

where x ∈ R . Here f 1 ( x ) = ( a 1 x -b 1 ) 2 and f 2 ( x ) = ( a 2 x -b 2 ) 2 . Now consider the estimator g ( x ) of ∇ f ( x ) under uniform sampling i.e. g ( x ) takes the value ∇ f 1 ( x ) with probability 1 2 and ∇ f 2 ( x ) with probability 1 2 . Then we have

Therefore, E ∥ g ( x ) -∇ f ( x ) ∥ 2 is a quadratic function of x with the coefficient of x being positive. Hence, as x →∞ , we have E ∥ g ( x ) -∇ f ( x ) ∥ 2 →∞ , which means that a constant can not bound the variance.

D Proofs of Results on Expected Residual

D.1 Proof of Lemma 3.2

Proof. Using Young's Inequality (22), we get

Then breaking down the RHS, we obtain

Now we rearrange the terms and set σ 2 ∗ = E ∥ g ( x ∗ ) ∥ 2 to complete the proof of this Lemma.

Proposition D.1. If F i are L i -lipschitz then Expected Residual condition (ER) holds. In that case

In addition, if F is µ -quasi strongly monotone (3) then we have

Proof. Note that

The first part of the lemma follows by ignoring the positive term ∥ F ( x ) ∥ 2 . For the second part we assume F is µ -quasi strongly monotone. Then we have

Cancelling ∥ x -x ∥ from both sides we get

Therefore we have the following bound for µ -quasi strongly monotone operator F :

This proves the second part of the lemma. This lemma ensures that the Lipschitz property is sufficient to guarantee Expected Residual (ER) condition.

D.2 Proof of Proposition 3.3

Proof. Proposition D.1 implies that Lipschitzness of all operators F i is enough to ensure that ER holds. For τ - minibatch sampling, denote the matrix R = ( F 1 ( x ) -F 1 ( x ∗ ) , · · · , F n ( x ) -F n ( x ∗ ) ) ∈ R d × n . Then we obtain the following bound:

From the proof details of Lemma F.3 in [63] we have λ max ( Var [ v ]) = n ( n -τ ) τ ( n -1) for τ -minibatch sampling. Thus we obtain

Now we focus on the derivation of σ 2 ∗ = E ∥ F v ( x ∗ ) ∥ 2 for τ -minibatch sampling. We expand E ∥ F v ( x ∗ ) ∥ 2 as follows:

where P ij = P ( i, j ∈ S ) and p i = P ( i ∈ S ) . For τ -minibatch sampling, we obtain P ij = τ ( τ -1) n ( n -1) and p i = τ n . Plugging in these values of P ij and p i in (30) we get the closed-form expression of σ 2 ∗ . This completes the proof of Proposition 3.3.

D.3 Proof of Proposition 3.4

Here we enlist the assumptions made on operators. Suppose g is an estimator of operator F .

Bounded Operator: E ∥ g ( x ) ∥ 2 ≤ σ 2

Bounded Variance: E ∥ g ( x ) -F ( x ) ∥ 2 ≤ σ 2

Growth Condition: E ∥ g ( x ) ∥ 2 ≤ α ∥ F ( x ) ∥ 2 + β

Expected Co-coercivity: E ∥ g ( x ) -g ( x ∗ ) ∥ 2 ≤ l F ⟨ F ( x ) , x -x ∗ ⟩

Expected Residual: E ∥ ( g ( x ) -g ( x ∗ )) -( F ( x ) -F ( x ∗ )) ∥ 2 ≤ δ 2 ∥ x -x ∗ ∥ 2

Bound from Lemma 3.2: E ∥ g ( x ) ∥ 2 ≤ δ ∥ x -x ∗ ∥ 2 + ∥ F ( x ) ∥ 2 +2 σ 2 ∗

F i are Lipschitz: ∥ F i ( x ) -F i ( y ) ∥ ≤ L i ∥ x -y ∥ ∀ i = 1 , . . . , n

Proof. Here we will prove Proposition 3.4

1 = ⇒ 2 . Note that E ∥ g ( x ) ∥ 2 ≤ σ 2 ≤ ∥ F ( x ) ∥ 2 + σ 2 = ⇒ E ∥ g ( x ) -F ( x ) ∥ ≤ σ 2 .

2 = ⇒ 3 . Here E ∥ g ( x ) -F ( x ) ∥ 2 ≤ σ 2 = ⇒ E ∥ g ( x ) ∥ 2 ≤ ∥ F ( x ) ∥ 2 + σ 2 as g is an unbiased for estimator of F . Then take α = 1 and β = σ 2 .

3 = ⇒ 6 . Note that E ∥ g ( x ) ∥ 2 ≤ α ∥ F ( x ) ∥ 2 + β ≤ αL 2 ∥ x -x ∗ ∥ 2 + β . The last inequality follows from lipschitz property of F and F ( x ∗ ) = 0 . Then choose δ = αL 2 and σ 2 ∗ = β / 2 to get the result.

4 = ⇒ 5 . Note that expected cocoercivity and L -Lipschitzness of F imply E ∥ ( g ( x ) -g ( x ∗ )) -( F ( x ) -F ( x ∗ )) ∥ 2 = E ∥ g ( x ) -g ( x ∗ ) ∥ 2 - ∥ F ( x ) -F ( x ∗ ) ∥ 2 ≤ E ∥ g ( x ) -g ( x ∗ ) ∥ 2 ≤ l F ⟨ F ( x ) , x -x ∗ ⟩ (B.2) ≤ l F 2 L ∥ F ( x ) ∥ 2 + l F L 2 ∥ x -x ∗ ∥ 2 ≤ l F L ∥ x -x ∗ ∥ 2 .

7 = ⇒ 5 . This follows from Proposition D.1.

5 = ⇒ 6 . This follows from Lemma 3.2

E Main Convergence Analysis Results

First, we present some results followed by iterates of SPEG . These will play a key role in proving the Theorems later in this section. Recall that iterates of SPEG are

Lemma E.1. For SPEG iterates with step-size ω k = γ k = ω , we have

Proof. We have

Lemma E.2. Let F be L -Lipschitz, and let ER hold. Then SPEG iterates satisfy

Proof.

Proof. Note that for ω ∈ [ 0 , 1 4 L ] , we have

This proves the first condition. The second condition is equivalent to ω ( µ -ωδ ) + 8 ω 2 L 2 -1 ≤ 0 , which is again true using similar arguments.

E.1 Proof of Theorem 4.1

Proof. For ω ∈ [ 0 , µ 18 δ ] we have ω ( µ -9 ωδ ) ≥ 0 and 1 -ω ( µ -9 ωδ ) ≤ 1 -ωµ 2 . Then we derive

Then using (33), (34) we have

Then we take total expectation with respect to the algorithm to obtain the following recurrence:

Using the inequality 1 -ω ( µ -9 ωδ ) ≤ 1 -ωµ 2 , we have

The theorem follows by unrolling the above recurrence. In order to compute the iteration complexity of SPEG , we consider any arbitrary ε > 0 . Then we choose the step-size ω such that 24 ωσ 2 ∗ µ ≤ ε 2 i.e. ω ≤ εµ 48 σ 2 ∗ . Next we will choose the number of iterations k such that (1 -ωµ 2 ) k R 2 0 ≤ ε 2 . It is equivalent to choosing k such that

Now using the fact log ( 1 ρ ) ≥ 1 -ρ for 0 < ρ ≤ 1 , we get log ( 2 R 2 0 ε ) ≤ kωµ 2 , or equivalently k ≥ 2 ωµ log ( 2 R 2 0 ε ) . Therefore, with step-size ω = min { µ 18 δ , 1 4 L , εµ 48 σ 2 ∗ } we get the following lower bound on the number of iterations

E.2 Proof of Theorem 4.3

Proof. For ω ≤ min { 1 4 L , µ 18 δ } , from Theorem 4.1 we obtain

Let the step-size ω k = 2 k +1 ( k +1) 2 2 µ and k ∗ be an integer that satisfies ω k ∗ ≤ ¯ ω . In particular this holds when k ∗ ≥ ⌈ 4 µ ¯ ω -1 ⌉ . Note that ω k is decreasing in k and consequently ω k ≤ ¯ ω for all k ≥ k ∗ . Therefore, from (36) we derive

for all k ≥ k ∗ . Then we replace ω k with 2 k +1 ( k +1) 2 2 µ to obtain

Multiplying both sides by ( k +1) 2 we get

where in the last line follows from 2 k +1 k +1 < 2 . Rearranging and summing the last expression for t = k ∗ , · · · , k we obtain

Using telescopic sum and dividing both sides by ( k +1) 2 we obtain

Suppose for k ≤ k ∗ , we have ω k = ¯ ω = min { 1 4 L , µ 18 δ } i.e. constant step-size. Then from (10), we obtain R 2 k ∗ ≤ ( 1 -µ ¯ ω 2 ) k ∗ R 2 0 + 24¯ ωσ 2 ∗ µ . This bound on R 2 k ∗ , combined with (37) yields

Now we want to choose k ∗ which minimizes the expression ( k ∗ k +1 ) 2 24¯ ωσ 2 ∗ µ + 192 σ 2 ∗ ( k -k ∗ ) µ 2 ( k +1) 2 . Note that, it is minimized at 4 µ ¯ ω , hence we choose k ∗ = ⌈ 4 µ ¯ ω ⌉ . Therefore, using this value of k ∗ , we obtain

The last line follows from ( 1 -1 x ) x ≤ e -1 for all x ≥ 1 . This completes the proof.

E.3 Proof of Theorem 4.4

Proof. For 0 < ω k ≤ { 1 4 L , µ 18 δ } we obtain the following bound from Theorem 4.1:

Then using Lemma B.4 with a = µ 2 , h = 1 ¯ ω and c = 12 σ 2 ∗ we complete the proof of this Theorem.

E.4 Proof of Theorem E.4

Theorem E.4. Let F be L -Lipschitz and satisfy Weak Minty condition with parameter ρ < 1 / (2 L ) . Assume that inequality (8) holds (e.g., it holds whenever Assumption 3.1 holds, see Lemma 3.2). Assume that γ k = γ , ω k = ω and

Then, for all K ≥ 2 the iterates produced by SPEG satisfy

Proof. The proof closely follows the proof of Lemma C.3 and Theorem C.4 from [23]. The update rule of SPEG implies for k ≥ 1

where in the last step we apply 2 ⟨ a, b ⟩ = ∥ a ∥ 2 + ∥ b ∥ 2 -∥ a -b ∥ 2 , which holds for all a, b ∈ R d . Taking the full expectation and using E [ E v k [ · ]] = E [ · ] and Weak Minty condition, we derive

where we apply Jensen's inequality ∥ F (ˆ x k ) ∥ 2 = ∥ E v k [ F v k (ˆ x k )] ∥ 2 ≤ E v k [ ∥ F v k (ˆ x k ) ∥ 2 ] and γ > 2 ρ + ω . For k = 0 we have x 1 = x 0 -ωF v 0 (ˆ x 0 ) = x 0 -ωF v 0 ( x 0 ) and

Applying Weak Minty condition, we get

The next step of our proof is in estimating the last term from (39). Using Young's inequality ∥ a + b ∥ 2 ≤ (1 + α ) ∥ a ∥ 2 +(1 + α -1 ) ∥ b ∥ 2 , which holds for any a, b ∈ R d , α > 0 , we get for all k ≥ 2

Then using (8) we get,

Let { w k } K -1 k =0 be a non-increasing sequence of positive numbers that will be specified later and W K = ∑ K -1 w k . Summing up the above two inequalities with weights { w k } K -1 , we derive

Next, we rearrange the terms using w k ≥ w k +1 and new notation ∆ k = E [ ∥ F v k (ˆ x k ) -F v k -1 (ˆ x k -1 ) ∥ 2 ] :

To simplify the above inequality we choose α = 1 2 L 2 γ ( γ + ω ) -1 2 , which is positive due to γ < 1 / L and γ + ω < 1 / L . In this case, we have

where we also use 1 / 2 L < γ < 1 / L and γ + ω < 1 / L . Using these relations, we can continue our derivation as follows:

Dividing both sides by 1 2 ( 1 -L 2 γ ( γ + ω ) ) , we derive

where C 1 = 3 Lω 1 -L 2 γ ( γ + ω ) + 32 (1 -L 2 γ ( γ + ω )) 2 δγ 2 , C 2 = 3 1 -L 2 γ ( γ + ω ) + 32 (1 -L 2 γ ( γ + ω )) 2 δ ( γ + ω ) 2 , and C 3 = 16 (1 -L 2 γ ( γ + ω )) 2 . Summing up inequalities (39) for k = 1 , . . . , K -1 with weights w 1 , . . . , w K -1 and (40) with weight w 0 , we get

Since w k ≥ w k +1 , we can continue the derivation as follows:

Now we need to specify the weights w -1 , w 0 , w 1 , . . . , w K -1 . Let w K -2 = 1 and w k -1 = (1 + 3 C 3 ωγδ ) w k . Then, rearranging the terms, dividing both sides by ωγ (1 -C 1 ) W K -1 , we get

It remains to simplify the right-hand side of the above inequality. First, we notice that W K -1 = ∑ K -2 k =0 w k ≥ ( K -1) w K -2 = K -1 since w k ≥ w k +1 . Moreover, w -1 = (1 + 3 C 3 ωγδ ) K -1 . Next,

where we use δ ≤ (1 -Lγ ) L 3 ω / 16 and γ + ω < 1 / L . Using these inequalities, we simplify the bound as follows:

where we use W K = W K -1 + w K -1 ≤ W K -1 + w K -2 ≤ 2 W K -1 . Finally, we use (8) to upper-bound E [ ∥ F v 0 (ˆ x 0 ) ∥ 2 ] :

Plugging this inequality in (42), we derive

which concludes the proof.

E.5 Proof of Theorem 4.5

Theorem E.5. Let F be L -Lipschitz and satisfy Weak Minty condition with parameter ρ < 1 / (2 L ) . Assume that inequality (8) holds (e.g., it holds whenever Assumption 3.1 holds, see Lemma 3.2). Assume that γ k = γ , ω k = ω and

Then, for all K ≥ 2 the iterates produced by mini-batched SPEG with batch-size

satisfy

Proof. Mini-batched SPEG uses estimator

where F v k, 1 (ˆ x k ) , . . . , F v k,τ (ˆ x k ) are independent samples satisfying (8) with parameters δ and σ 2 ∗ . Using variance decomposition and independence of F v k, 1 (ˆ x k ) , . . . , F v k,τ (ˆ x k ) , we get

That is, mini-batched estimator F v k (ˆ x k ) satisfies (8) with parameters δ / τ and σ 2 ∗ / τ . Therefore, Theorem E.4 implies

Since τ satisfies (43) and γ ≤ 1 / L , ω ≤ 1 / 4 L , we have

Using this, we can simplify (45) as follows:

This concludes the proof.

On Oracle Complexity of Theorem 4.5. Let us now express the result of Theorem 4.5 via oracle complexity.

Oracle complexity captures the computational requirements required to solve a specific optimization problem. That is, given a prespecified accuracy ε > 0 , it measures the number of oracle calls needed to solve the problem to this ε accuracy. In our setting, an oracle call indicates the computation of one operator, F i (for some i ∈ [ n ] ). Therefore, in Theorem 4.5, where a mini-batch of size τ is required in each iteration of the update rule, we have τ many oracle calls per iteration. In that scenario, the total number of oracle calls required to obtain specific accuracy ε > 0 is given by Kτ (multiplication of K iterations with τ oracle calls).

Note that according to Theorem 4.5 to achieve an ε accuracy, we need K ≥ C ∥ x 0 -x ∗ ∥ 2 ϵ iterations. This follows trivially by

Therefore, using K ≥ C ∥ x 0 -x ∗ ∥ 2 ϵ in combination with the lower bound on τ from (15), the total number of oracle calls to satisfy (46) is given by:

F Further Results on Arbitrary Sampling

F.1 Proof of Proposition 5.1

Expanding the left hand side of Expected Residual (ER) condition we have

For any x and y with v i = 1 p i we obtain

Then taking expectation on both sides we get

̸

Now we consider the case, where the ratio P ij p i p j = c 2 i.e. constant for i = j and P ii = p i . Then from the above computations we derive

̸

Thus replacing y = x ∗ and combining with (47) we get the following bound on the Expected Residual:

For single-element sampling c 2 = 0 (as probability of two points appearing in same sample is zero for single element sampling i.e. P ij = 0 ). Then we obtain

from (48). This completes the derivation of δ for single element sampling. To compute σ 2 ∗ for single element sampling, we replace

in (30) to get

G Numerical Experiments

In Appendix G.1, we add more details on the experiments discussed in the main paper. Furthermore, in Appendix G.2, we run more experiments to evaluate the performance of SPEG on quasi-strongly monotone and weak MVI problems.

G.1 More Details on the Numerical Experiments of Section 6

On Constant vs Switching Stepsize Rule. We run the experiments on two synthetic datasets. In Fig. 1 of the main paper, we take µ A = µ C = 0 . 6 . Here we include one more plot with a similar flavor but in a different setting. For Fig. 5, we generate the data such that eigenvalues of A 1 , B 1 , C 1 are generated uniformly from the interval [0 . 1 , 10] . In the new plot, similar to the main paper, we can see the benefit of switching the step-size rule of Theorem 4.3.

0

20000

40000

60000

80000

100000

Number of Iterations

10

2

10

1

10 0

Relative Error

Decreasing Stepsize (Theorem 4.4)

Change of step-size

On Weak Minty VIPs. In this experiment, we generate ξ i , ζ i such that 1 n ∑ n i =1 ξ i = √ 63 and 1 n ∑ n i =1 ζ i = -1 . This choice of ξ i , ζ i ensures that L = 8 and ρ = 1 / 32 for the min-max problem we considered in Section 6.2. In Fig. 6, we again implement the SPEG on (17) with batchsize = 0 . 15 × n (different batchsize compare to the plot of the main paper).

0

200

400

600

800

1000

Number of Iterations

10

33

10

28

10

23

10

18

10

13

10

8

10

3

10 2

Squared Operator Norm Error

SPEG on WMVI

SPEG

G.2 Additional Experiments

In this subsection, we include more experiments to evaluate the performance of SPEG on quasistrongly monotone and weak MVI problems. First, we run the experiment comparing constant and switching step-size rules on a different setup than the one we included in the main paper to analyze the performance of SPEG under different condition numbers. Then, we implement SPEG on the weak MVI of (17). To evaluate the performance in this experiment, we plot ∥ F (ˆ x k ) ∥ 2 / ∥ F ( x 0 ) ∥ 2 on the y -axis.

G.2.1 Strongly Monotone Quadratic Game:

In this experiment, we compare the proposed constant step-size (9) and the switching step-size rule (11). We implement our algorithm on operator F : R 4 → R 4 given by

where M 1 , M 2 and M 3 are the diagonal matrices,

and

This choice of M i and x ∗ i ensures that the Lipschitz constant of operator F is ∆+2 3 while quasi-strong monotonicity parameter (3) is µ = 1 . Hence the condition number of F is given by ∆+2 3 . This allows us to vary the condition number of operator F by changing the value of ∆ . For Fig. 7a we take ∆ = 3 (condition number = 1 . 67 ) while for Fig. 7b we choose ∆ = 10 (condition number = 10 . 67 ). The vertical dotted line in plots of Fig. 7 marks the transition point from constant to switching step-size rule as predicted by our theoretical result in Theorem 4.3.

0

20000

40000

60000

80000

100000

Number of Iterations

10

5

10

4

10

3

10

2

10

1

10 0

Relative Error

Decreasing Stepsize (Theorem 4.4)

Change of step-size

(a) Condition Number

L

µ

= 1

.

67

.

0.0

0.2

0.4

0.6

0.8

1.0

Number of Iterations

1e6

10

6

10

5

10

4

10

3

10

2

10

1

10 0

Relative Error

Decreasing Stepsize (Theorem 4.4)

Change of step-size

(b) Condition Number L µ = 10 . 67 .

G.2.2 Weak Minty VIPs Continued

In this experiment, we reevaluate the performance of SPEG on weak MVI example of (17). That is, we generate the data in exactly the same way as the ones in section 6.2 with n = 100 . In Fig. 8a and 8b, we implement SPEG with batchsize 10 and 15 , respectively (we note that in this setting the full-gradient evaluation requires a batchsize of 100 ). For these plots, we use the relative operator norm on the y -axis, i.e. ∥ F (ˆ x k ) ∥ 2 / ∥ F ( x 0 ) ∥ 2 , where x 0 denotes the starting point of SPEG . As expected, the plots illustrate that SPEG performs better as we increase the batchsize. From Fig. 8 it is clear that with batchsize 15 SPEG reaches an accuracy close to 10 -10 while when we use a batchsize of 10 for the same number of iterations we are only able to converge to an accuracy of 10 -4 .

0

10000

20000

30000

40000

50000

Number of Iterations

10

4

10

3

10

2

10

1

10 0

10 1

10 2

10 3

Relative Operator Norm

SPEG

(a) Batchsize =

0

.

1

×

n.

0

10000

20000

30000

40000

50000

Number of Iterations

10

8

10

6

10

4

10

2

10 0

Relative Operator Norm

SPEG

(b) Batchsize =

0

.

15

×

n.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes new convergence results for single-call stochastic extragradient methods under weaker conditions. More specifically, the authors consider quasi-strongly monotone and weak Minty VI problems, both under an unconstrained finite-sum (or arbitrary sampling) setting. The authors propose the expected residual (ER) condition, which extends similar conditions used in stochastic optimization (minimization) to the VI setting. ER is more general than the boundedness assumption on operator noise, and is sufficient for establishing their convergence results. The authors also give sufficient conditions for ER and explain its connections to other widely used technical conditions/assumptions. Using ER, the authors establish convergence results for single-call stochastic extragradient methods for quasi-strongly monotone and weak Minty VI problems. For quasi-strongly monotone problems, two results are given for constant and decreasing step size rules. The authors then give expressions for the ER parameters (which factor into convergence rates) under non-uniform sampling. Numerical experiments complement the theoretical findings.\n\nSTRENGTHS:\n- Clear presentation of results.\n- Sufficient discussion of the background and context of the new ER condition.\n- First convergence guarantee for SOG under arbitrary sampling.\n- First convergence guarantee for SPEG without any bounded variance assumption.\n\nWEAKNESSES:\nSee **Questions**.\n\nQUESTIONS:\nMy main question is around the practicality of the proposed step size rules. In previous work, most (static, non-adaptive) step sizes used in convergence theorems tend to be highly conservative, and more aggressive or adaptive step size rules (which may weaken/break convergence guarantees) need to be used in order to make the algorithms converge in experiments.\n\nIn this work, to achieve desirable convergence rates, the step sizes (constant or decreasing) depend on $\\delta$, $\\mu$, and $L$. My concern is that (i) these constants may be hard to find, or (ii) only conservative, safe estimates can be given. Therefore, the step sizes given by the theorems may be too small. It seems that in all experiments, theoretically safe constants are used. Have the authors considered using more aggressive step sizes which do not follow the theorems entirely (but partially, such as the $1/k^2$ decreasing trend in (11) but with different constants)? If so, it would be helpful to point these out. If the theorems' suggested step size rules are already \"near-optimal\" in experiments (meaning that further increasing them would lead to non-convergence), it would also be worth pointing out this favorable observation. In fact, if true, this would be such a rarity based on my experience and other work, and should be worth highlighting.\n\nLIMITATIONS:\nThis is a theoretical and methodological work. The limitations are mainly technical: many practical problems mentioned as a motivation of the work do not (or cannot be easily shown to) satisfy the exact technical assumptions. However, this is typical and not a concern.\n\nI do not see potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the single-call stochastic extragradient methods for solving two classes of structured variational inequality (VI) problems, i.e., (i) quasi-strongly monotone problems and (ii) weak Minty variational problems. These two classes generalize the assumptions of strong monotonicity and comonotonicity to be only satisfied w.r.t. the solution point, respectively. The authors consider the stochastic reformulation of VIs, which allows for mini-batching with arbitrary sampling. The convergence results are built upon the expected residue condition, which can be explicitly implied by the component Lipschitzness and can imply bounded variance. Convergence are analyzed in both settings and with different stepsize strategies. The authors also provide numerical experiments to support their theoretical discussion.\n\nSTRENGTHS:\n1. The authors generalize the idea from stochastic minimization and propose the expected residue condition, which can be explicitly implied by the Lipschitzness of the component operator, and can be used to provide a variance bound. Thus, this paper does not require a bounded variance assumption nor growth conditions.\n2. The authors provide a thorough discussion for the convergence results and detailed comparison with prior works, such as Lines 183--200.\n\nWEAKNESSES:\n1. Although the authors state the stochastic problem in term of the **finite-sum** structure (see Eq. 1 and 5), the convergence analysis is more likely for **infinite-sum** problems, especially when the authors also consider mini-batch in this paper.\n2. The expected residue condition proposed in the paper looks restricted because the authors make this assumption directly on the stochastic estimator $g(x)$ instead of the stochastic oracle queries. It seems non-trivial to assume it for the stochastic estimators beyond the mini-batching ones, such as SVRG or SARAH.\n\nQUESTIONS:\n1. Although the weak Minty variational problems look more general than comonotonicity assumption, but I am not sure whether it is more of a theoretical artifact. Could the authors provide some practical examples which satisfy Def 1.2 but are not comonotone?\n2. Could the authors clarify the stochastic settings this paper focused on, finite-sum or infinite-sum? I feel confused because $n$ never appears in the convergence results, and the choice of the batch size $\\tau$ in Theorem 4.5 (Eq. 15) can essentially be larger than $n$.\n\nLIMITATIONS:\nPlease see the weaknesses and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies single-call stochastic extra-gradient method for quasi strongly monotone and weak Minty Variational Inequality (VI). They relax the commonly-used bounded noise variance assumption and used the expected residual condition.\n\nSTRENGTHS:\nThe paper is well-written and the problem is relevant.\n\nWEAKNESSES:\nIn short, I think the technical novelty in this paper is minimal. I substantiate my claim below. \n\n\n1. It introduces expected residual condition which has been studied before in SGD literature as the authors correctly state. So the condition itself is not a contribution. **As the authors agree (line 173-175), the main difference between this work and [29] is that (8) is an Assumption in [29] whereas Assumption 3.1 implies (8) in this work. But it is easy to see that the proof of Lemma 3.2 just requires a single use of Young's inequality (and indeed, that's how it's done in the Appendix).**\n\n2. **Authors in [29] use another condition** $ E || g(x) − F(x) ||^2 \\leq (a||x − x^*|| + b)^2 $ **which is extremely similar to ER condition. But in line $171$, the authors claim that the constants $a$ and $b$ are not available in closed form. But this is a trivial result (Proposition 3.3). These calculations are routinely done in literature involving  stochastic gradient as the authors themselves agree (line 153, [24-25]).**\n\n3. **Use of ER condition relaxes the bounded condition on the noise variance but the additional technical innovation needed, compared to the bounded noise variance, is minimal.** Familiarity with the stochastic extra-gradient proof techniques right away reveals that the RHS of ER condition, $||x-x^*||_2^2$, is designed to be subsumed in other terms appearing in the proof which effectively leaves the proof almost same as the bounded variance case.\n\nSpecifically, **note that ER condition only introduces the term** $\\omega^2\\delta || \\hat{x}_k - x^* ||_2^2$ **in the 5th line of the display appearing after line 664. But there is another term** $ -2 \\omega \\mu ||\\hat{x}_k-x^*||_2^2$. **So the convergence can be guaranteed if** $\\omega $ **is chosen small enough such that** $2\\omega\\mu ||\\hat{x}_k-x^*||_2^2> \\omega^2\\delta ||\\hat{x}_k-x^*||_2^2$. **This is almost trivial.**\n\nIn fact, one can easily think of allowing for bigger bounds for the noise variance keeping the proof unimpacted. For example, convergence can be easily shown if one allows for a noise variance bound like $\\omega^{-1/2}\\delta ||\\hat{x}_k-x^*||_2^2$. **All the proofs depend on this simple and trivial extension.**\n\nQUESTIONS:\nPlease see the weakness section. If you could elaborate on the novelty that would be great.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores single-call stochastic extragradient methods like stochastic past extragradient (SPEG) and stochastic optimistic gradient (SOG), which are increasingly popular and efficient algorithms for tackling large-scale min-max optimization and variational inequalities problems (VIP) commonly found in diverse machine learning tasks. Notwithstanding their growing popularity, the current convergence analyses of SPEG and SOG demand strong assumptions like bounded variance or growth conditions. Moreover, numerous key questions related to the convergence properties of these methods, such as mini-batching, effective step-size selection, and convergence guarantees under various sampling strategies, remain unaddressed.\n\nThis research endeavors to answer these questions, presenting convergence guarantees for two extensive classes of structured non-monotone VIPs: (i) quasi-strongly monotone problems, which extend strongly monotone problems, and (ii) weak Minty variational inequalities, which expand upon monotone and Minty VIPs. The paper introduces the expected residual condition and discusses its advantages, demonstrating how it enables a strictly weaker bound than those achieved by previously employed growth conditions, expected co-coercivity, or bounded variance assumptions. Lastly, the presented convergence analysis is applicable under the arbitrary sampling paradigm, encompassing special cases like importance sampling and various mini-batching strategies.\n\nSTRENGTHS:\nThe main contributions of the paper include the following:\n\n- **Expected Residual:** The authors introduce the Expected Residual (ER) condition for stochastic variational inequality problems. The ER condition is used to derive an upper bound on $E\\|g(x)\\|^2$ (as detailed in Lemma 3.2), offering a strictly weaker alternative to the bounded variance assumption and previously used \"growth conditions\" for the analysis of stochastic algorithms. The paper shows that the ER condition holds for a large class of operators, specifically when the $F_i$ of the problem are Lipschitz continuous.\n\n- **Novel Convergence Guarantees:** The paper presents novel convergence guarantees for Stochastic Past Extragradient (SPEG) without the need for a bounded variance assumption in the cases of quasi-strongly monotone and weak Minty Variational Inequalities (MVI). This is achieved through the use of the proposed ER condition. For the class of quasi-strongly monotone Variational Inequalities Problems (VIPs), the paper demonstrates a linear convergence rate to a neighborhood of $x^*$ when constant step-sizes are utilized. Furthermore, theoretically motivated step-size switching rules are provided that guarantee exact convergence of SPEG to $x^*$. In the weak MVI case, the convergence of SPEG is proved for $\\rho < 1/2L$, improving existing restrictions on $\\rho$. The authors compare their results with the existing literature in Table 1.\n\n- **Arbitrary Sampling:** By reformulating the variational inequality problem stochastically, the authors explain how their convergence guarantees for SPEG hold under the arbitrary sampling paradigm. This approach allows them to cover a wide range of sampling strategies for SPEG not previously considered, including mini-batching, uniform sampling, and importance sampling. Therefore, their analysis of SPEG is unified for different sampling strategies. The authors also demonstrate the tightness of their analysis by showing that the best-known convergence guarantees of deterministic Past Extragradient (PEG) for strongly monotone and weak MVI can be obtained as special cases of their main theorems.\n\nWEAKNESSES:\n- **Redundant Main Result:** The primary outcome of this research could essentially be seen as a single-call version of \"Stochastic Extragradient: General Analysis and Improved Rates,\" but under simpler guarantees. This could potentially question the novelty of this research.\n\n- **Uncertainty of Expected Residual (ER):** The Expected Residual (ER) is proposed as a new and beneficial condition, however, it is unclear whether this condition is indeed intuitive and interesting in practice. It would be beneficial if the authors could provide more justification or insights on the relevance of the ER condition in practical scenarios.\n\n- **Importance of Single-Call Methods:** While the paper puts substantial focus on single-call methods, it is unclear whether these methods are significantly important in practical scenarios. Theoretical advantages are evident, but it would be constructive if the authors could provide more practical motivations or empirical examples where single-call methods bring notable improvements.\n\n- **Comparison with Existing Methods:** If Stochastic Extragradient (SEG) methods already have satisfactory rates and guarantees, it is intuitive to assume that Stochastic Optimistic Gradient (SOG) methods would offer similar advantages. Therefore, a clear delineation between the contributions of the proposed method and existing ones would be beneficial for the reader.\n\nQUESTIONS:\nAddress the weaknesses' section\n\nLIMITATIONS:\nN/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}