SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To the best of our knowledge, the above theorem is the first result on the convergence of SPEG that does not rely on the bounded variance assumption. Theorem 4.1 recovers the same rate of convergence with the Independent-Samples SEG (l-SEG) under assumption (8) [20], although [20] simply assume (8), while we show that it follows from Assumption 3.1 holding whenever all summands  $F_{i}$  are Lipschitz. However, in the case when all  $F_{i}$  are  $\mu$ -quasi strongly monotone and  $L_{i}$ -Lipschitz (on average), one can use Same-Sample SEG (S-SEG). The existing results for S-SEG have better exponentially decaying term [49, 20] then Theorem 4.1, e.g., in the case when  $L_{i} = L$  for all  $i \in [n]$ , we have  $\delta = \mathcal{O}(L^2)$  meaning that the exponentially decaying term in (10) is  $\mathcal{O}(R_0^2 \exp(-\mu^2 k / L^2))$ , while S-SEG has much better exponentially decaying term  $\mathcal{O}(R_0^2 \exp(-\mu^k / L))$ .
Such a discrepancy can be partially explained by the following fact: S-SEG can be seen as one step of deterministic Extragradian for stochastic operator  $F_{v_k}$  allowing to use one-iteration analysis of Extragradian without controlling the variance. In contrast, there is no version of SPEG that uses the same sample for extrapolation and update steps. This forces us to use different samples for these steps and this is a key reason why SPEG cannot be seen as one iteration of deterministic Past-Extragradian for some operator. Due to this, we need to rely on some bound on the variance to handle the stochasticity in the updates; see also [20, Appendix F.1]. Therefore, in our analysis, we use Assumption 3.1, implying (8). Nevertheless, it is still an open question whether it is possible to improve the rate of SPEG in the case of  $\mu$ -quasi strongly monotone and Lipschitz operators  $F_i$ .
To highlight the generality of Theorem 4.1, we note that for the deterministic PEG,  $\delta = 0$  and  $\sigma_{*}^{2} = 0$  (by selecting  $\tau = n$  in the definition 2.1 of minibatch sampling). In this case, Theorem 4.1 recovers the well-known result (up to  $1/2$  factor in the rate) for deterministic PEG proposed in [17] as shown in the following corollary.
Corollary 4.2. Let the assumptions of Theorem 4.1 hold and a deterministic version of SPEG is considered, i.e.,  $\delta = 0$ ,  $\sigma_{*}^{2} = 0$ . Then, Theorem 4.1 implies that for all  $k \geq 0$  the iterates produced
by SPEG with step-sizes  $\gamma_{k} = \omega_{k} = \omega$  such that  $0 < \omega \leq \frac{1}{4L}$  satisfy  $R_{k}^{2} \leq \left(1 - \frac{\omega\mu}{2}\right)^{k}R_{0}^{2}$ , where  $R_{k}^{2} \coloneqq \| x_{k} - x^{*}\|^{2} + \| x_{k} - \hat{x}_{k - 1}\|^{2}$ .
Decreasing Step-size: In this section, we consider two different decreasing step-sizes policies for SPEG applied to solve quasi-strongly monotone problems.
Theorem 4.3. Let  $F$  be  $L$ -Lipschitz,  $\mu$ -quasi strongly monotone, and Assumption 3.1 hold. Let
$$
\gamma_ {k} = \omega_ {k} := \left\{ \begin{array}{l l} \bar {\omega}, & \text {i f} k \leq k ^ {*}, \\ \frac {2 k + 1}{(k + 1) ^ {2}} \frac {2}{\mu}, & \text {i f} k > k ^ {*}, \end{array} \right. \tag {11}
$$
where  $\bar{\omega} \coloneqq \min \left\{1 / (4L), \frac{\mu}{(18\delta)}\right\}$  and  $k^* = \lceil 4 / (\mu \bar{\omega}) \rceil$ . Then for all  $K \geq k^*$  the iterates produced by SPEG with step-sizes (11) satisfy
$$
R _ {K} ^ {2} \leq \left(\frac {k ^ {*}}{K}\right) ^ {2} \frac {R _ {0} ^ {2}}{\exp (2)} + \frac {1 9 2 \sigma_ {*} ^ {2}}{\mu^ {2} K}, \tag {12}
$$
where  $R_K^2 \coloneqq \mathbb{E}\left[\|x_K - x^*\|^2 + \|x_K - \hat{x}_{K-1}\|^2\right]$ .
SPEG with step-size policy $^2$  (11) has two stages of convergence: during first  $k^*$  iterations it uses constant step-size to reach some neighborhood of the solution and then the method switches to the decreasing  $\mathcal{O}(1 / k)$  step-size allowing to reduce the size of the neighborhood.
For the case of strongly monotone problems (a special case of our quasi-strongly monotone setting) [28] also analyze SPEG with decreasing  $\mathcal{O}(1 / k)$  step-size $^3$  under bounded variance assumption, i.e., when (8) holds with  $\delta = 0$  and some  $\sigma_{*}^{2}\geq 0$ , which is equivalent to the uniformly bounded variance assumption. In particular, Theorem 5 [28] states  $\mathbb{E}\left[\| x_K - x^*\| ^2\right]\leq \frac{C\sigma_*^2}{\mu^2K} +o\left(\frac{1}{K}\right)$  where  $C$  is some numerical constant. If the problem is strongly monotone, the result of [28] is closely related to what is obtained in Theorem 4.3: the main difference in the upper-bound is that we provide an explicit form of  $o(1 / K)$  term. Moreover, in contrast to the result from [28], Theorem 4.3 holds even when  $\delta >0$  in (8), which covers a larger class of problems.
Following [67, 20, 6], we also consider another decreasing step-size policy.
Theorem 4.4. Let  $F$  be  $L$ -Lipschitz,  $\mu$ -quasi strongly monotone, and Assumption 3.1 hold. Let  $\bar{\omega} \coloneqq \min \left\{1/(4L), \frac{\mu}{(18\delta)}\right\}$ . If for  $K \geq 0$  step-sizes  $\{\gamma_k\}_{k \geq 0}$ ,  $\{\omega_k\}_{k \geq 0}$  satisfy  $\gamma_k = \omega_k$  and
$$
\omega_ {k} := \left\{ \begin{array}{l l} \bar {\omega}, & \text {i f} K \leq \frac {2}{\mu \bar {\omega}}, \\ \bar {\omega}, & \text {i f} K > \frac {2}{\mu \bar {\omega}} \text {a n d} k \leq k _ {0}, \\ \frac {2}{\bar {\omega} + \frac {\mu}{2} (k - k _ {0})}, & \text {i f} K > \frac {2}{\mu \bar {\omega}} \text {a n d} k > k _ {0} \end{array} \right. \tag {13}
$$
where  $k_{0} = \lceil K / 2\rceil$ , then the iterates produced by SPEG with the step-sizes defined above satisfy
$$
R _ {K} ^ {2} \leq \frac {6 4 R _ {0} ^ {2}}{\bar {\omega} \mu} \exp \left\{- \min  \left\{\frac {\mu}{1 6 L}, \frac {\mu^ {2}}{7 2 \delta} \right\} K \right\} + \frac {1 7 2 8 \sigma_ {*} ^ {2}}{\mu^ {2} K}, \tag {14}
$$
In contrast to (12), the rate from (14) has much better (exponentially decaying)  $o\left(1 / \kappa\right)$  term. When  $\sigma_{*}^{2}$  is large and one needs to achieve very good accuracy of the solution, this difference is negligible, since the dominating  $\mathcal{O}(1 / \kappa)$  term is the same for both bounds (up to numerical factors). However, when  $\sigma_{*}^{2}$  is small enough, e.g., the model is close to over-parameterized, or it is sufficient to achieve low accuracy of the solution, the dominating term in (14) is typically much smaller than the one from (12). Finally, it is worth mentioning, that the improvement of  $o\left(1 / \kappa\right)$  is not achieved for free: unlike the policy from (11), step-size rule (13) relies on the knowledge of the total number of steps  $K$ , which can be inconvenient for the practical use in some cases.
2Similar step-size policy is used for SGD [25] and SGDA [44].
<sup>3</sup>We point out the proof by [28] can be generalized to the case of constant step-size, though the authors do not consider this step-size schedule explicitly.
4.2 Weak Minty Variational Inequality Problems
In this subsection we will discuss convergence of Stochastic Past Extragradient method for Minty Variational Inequality problem. To solve the Minty variational inequality problem we use different step-sizes for SPEG iterates (7).
Theorem 4.5. Let  $F$  be  $L$ -Lipschitz and satisfy Weak Minty condition with parameter  $\rho < \frac{1}{(2L)}$ . Let Assumption 3.1 hold. Assume that  $\gamma_k = \gamma$ ,  $\omega_k = \omega$  such that  $\max \left\{2\rho, \frac{1}{2L}\right\} < \gamma < \frac{1}{L}$  and  $0 < \omega < \min \left\{\gamma - 2\rho, \frac{1}{4L} - \frac{\gamma}{4}\right\}$ . Then, for all  $K \geq 2$  the iterates produced by mini-batched SPEG with batch-size
$$
\tau \geq \max  \left\{1, \frac {3 2 \delta}{(1 - L \gamma) L ^ {3} \omega}, \frac {4 8 \omega \gamma \delta (K - 1)}{(1 - L \gamma) ^ {2}}, \frac {2 \omega \gamma \sigma_ {*} ^ {2} (K - 1)}{(1 - L \gamma) \| x _ {0} - x ^ {*} \| ^ {2}} \right\} \tag {15}
$$
satisfy  $\min_{0\leq k\leq K - 1}\mathbb{E}\left[||F(\hat{x}_k)||^2\right]\leq \frac{C\|x_0 - x^*\|^2}{K - 1}$ , where  $C = \frac{48}{\omega\gamma(1 - L(\gamma + 4\omega))}$ .
The above result establishes  $\mathcal{O}(1 / K)$  convergence with  $\mathcal{O}(K)$  batchesizes for SPEG applied to problems satisfying Weak Minty condition. The closest result is obtained by [8], for the same method under bounded variance assumption, i.e., when  $\delta = 0$ . In particular, the result of [8] also gives  $\mathcal{O}(1 / K)$  rate and requires  $\mathcal{O}(K)$  batchesizes at each step. We extend this result to the case of non-zeroth  $\delta$  and we also improve the assumption on  $\rho$ : [8] assumes that  $\rho < 3 / 8L$ , while Theorem 4.5 holds for  $\rho < 1 / 2L$ . The bound on  $\rho$  cannot be improved even in the deterministic case [23]. Moreover, it is worth mentioning that the proof of Theorem 4.5 noticeably differs from the one obtained by [8].
In the case of a deterministic oracle, we recover the best-known result for Optimistic Gradient in the Weak Minty setup [8, 23].
Corollary 4.6. Let the assumptions of Theorem 4.5 hold and deterministic version of SPEG is considered, i.e.,  $\delta = 0$ ,  $\sigma_{*}^{2} = 0$ . Then, Theorem 4.5 implies that for all  $k \geq 0$  the iterates produced by SPEG with step-sizes  $\max \left\{2\rho, \frac{1}{2L}\right\} < \gamma < \frac{1}{L}$  and  $0 < \omega < \min \left\{\gamma - 2\rho, \frac{1}{4L} - \frac{\gamma}{4}\right\}$  satisfy  $\min_{0 \leq k \leq K-1} \|F(\hat{x}_k)\|^2 \leq \frac{C \|x_0 - x^*\|^2}{K-1}$ , where  $C = \frac{48}{\omega\gamma(1-L(\gamma+4\omega))}$ .
5 Beyond Uniform Sampling
In this section, we illustrate the generality of our analysis by focusing on the non-uniform sampling. In particular, we focus on single-element sampling in which only the singleton sets  $\{i\}$  for  $i = \{1,\dots ,n\}$  have a non-zero probability of being sampled; that is,  $\mathbb{P}[|S| = 1] = 1$ . We have  $\mathbb{P}[v = e_i / p_i] = p_i$ . [25] proved that if  $v$  is a single-element sampling, it is also a valid sampling vector  $(\mathbb{E}_{\mathcal{D}}[v_i] = 1)$ . With the following proposition, we provide closed-form expressions for the ER parameter  $\delta$  and  $\sigma_{*}^{2} = \mathbb{E}\| g(x^{*})\|^{2}$  for the case of (non-uniform) single-element sampling.
Proposition 5.1. Let  $F_{i}$  of problem (1) be  $L_{i}$ -Lipschitz operators. If, vector  $\boldsymbol{v} \in \mathbb{R}^n$  is a single element sampling then  $\delta = \frac{2}{n^2} \sum_{i=1}^{n} \frac{L_i^2}{p_i}$  and  $\sigma_*^2 = \frac{1}{n^2} \sum_{i=1}^{n} \frac{1}{p_i} \| F_i(x^*) \|^2$ .
Importance Sampling. In importance sampling we aim to choose the probabilities  $p_i$  that optimize the iteration complexity. [25] and [20] analyze importance sampling for SGD and SEG respectively. In this work, we provide the first convergence guarantees of SPEG with importance sampling. In particular, we optimize the expected residual parameter  $\delta$  with respect to  $p_i$ , which in turn affects the iteration complexity. Note that, by using Cauchy-Schwarz inequality (20), we have  $\sum_{i=1}^{n} \frac{L_i^2}{p_i} \geq \left( \sum_{i=1}^{n} L_i \right)^2$ , and this lower bound can be achieved for  $p_i^\delta = L_i / \sum_{j=1}^{n} L_j$ . In case of importance sampling, we will use these probabilities  $p_i^\delta$  which optimizes  $\delta$  and define the corresponding  $\delta$  as  $\delta_{\mathrm{IS}} := \frac{2}{n^2} \left( \sum_{i=1}^{n} L_i \right)^2$ . For uniform sampling (i.e.,  $p_i = \frac{1}{n}$ ), the value of the parameter is  $\delta_{\mathrm{US}} = \frac{2}{n} \sum_{i=1}^{n} L_i^2$ . Note that,  $\delta_{\mathrm{IS}}$  equals  $\delta_{\mathrm{US}}$  when all  $L_i$  are equal, however  $\delta_{\mathrm{IS}}$  can be much smaller than  $\delta_{\mathrm{US}}$  when  $L_i$  are very different from each other, e.g., when all  $L_i$  are relatively small (close to
See also Appendix E.5 for a discussion related to the oracle complexity of Theorem 4.5.
zero) and one  $L_{i}$  is large,  $\delta_{\mathrm{IS}}$  is almost  $n$  times smaller than  $\delta_{\mathrm{US}}$ . In this latter scenario (when  $\delta_{\mathrm{IS}}$  is much smaller than  $\delta_{\mathrm{US}}$ ), importance sampling could be useful and can significantly improve the performance of SPEG. For example, note that the exponentially decaying term in (14) decreases with  $\delta$ . Hence, this term will decrease much faster with importance sampling than with uniform sampling.
To verify our theoretical results, we run several experiments on two classes of problems, i.e., strongly monotone problems (a special case of the quasi-strongly monotone VIPs) and weak MVI problems. The code to reproduce our results can be found at https://github.com/isayantan/Single-Call-Stochastic-Extragradient-Methods.
6.1 Strongly Monotone Problems
Our experiments consider the quadratic strongly-convex strongly-concave min-max problem from [20]. That is, we implement SPEG on quadratic games of the form  $\min_{x\in \mathbb{R}^d}\max_{y\in \mathbb{R}^d}\frac{1}{n}\sum_{i = 1}^{n}f_i(x,y)$  where
$$
f _ {i} (x, y) := \frac {1}{2} x ^ {\intercal} A _ {i} x + x ^ {\intercal} B _ {i} y - \frac {1}{2} y ^ {\intercal} C _ {i} y + a _ {i} ^ {\intercal} x - c _ {i} ^ {\intercal} y. \tag {16}
$$
Here  $A_{i}, B_{i}, C_{i}$  are generated such that the quadratic game is strongly monotone and smooth. In all our experiments, we take  $n = 100$  and  $d = 30$ . We generate positive semi-definite matrices  $A_{i}, B_{i}, C_{i}$  such that their eigenvalues lie in the interval  $[\mu_A, L_A], [\mu_B, L_B]$  and  $[\mu_C, L_C]$  respectively. In all our experiments, we consider  $L_{A} = L_{B} = L_{C} = 1$  and  $\mu_A = \mu_C = 0.1$ ,  $\mu_B = 0$  unless otherwise mentioned. The vectors  $a_{i}$  and  $c_{i}$  are generated from  $\mathcal{N}_d(0, I_d)$ . Here, the  $i$ th operator is given by
$$
F _ {i} \left( \begin{array}{c} x \\ y \end{array} \right) = \left( \begin{array}{c} \nabla_ {x} f _ {i} (x, y) \\ - \nabla_ {y} f _ {i} (x, y) \end{array} \right) = \left( \begin{array}{c} A _ {i} x + B _ {i} y + a _ {i} \\ C _ {i} y - B _ {i} ^ {\mathsf {T}} x + c _ {i} \end{array} \right)
$$
In Figures 1, 2, and 3, we plot the relative error on the  $y$ -axis i.e.  $\frac{\|x_k - x^*\|^2}{\|x_0 - x^*\|^2}$ .
Constant vs Switching Step-size Rule. In Fig. 1, we illustrate the step-size switching rule of Theorem 4.3. We place the dotted line to mark when we switch from constant step-size to decreasing step-size. In Fig. 1, the trajectory of switching step-size rule (11) matches that of constant step-size (9) in the first phase (where SPEG runs with constant step-size following (11)). However, it becomes stagnant when the constant step-size SPEG reaches a neighbourhood of optimality. In contrast, the step-size of Theorem 4.3 helps the method to converge to better accuracy.
Figure 1: Constant vs Switching
Comparison to Hsieh et al. [28]. In this experiment, we compare SPEG step-sizes proposed in Theorems 4.1 and 4.3
with step-sizes from [28]. To implement SPEG with the step-sizes from [28], we choose  $\gamma$  and  $b$  such that  $\frac{1}{\mu} < \gamma \leq \frac{b}{4L}$  and set  $\omega_{k} = \gamma_{k} = \frac{\gamma}{k + b}$ . For Fig. 2a, we generate  $A_{i}, B_{i}, C_{i}$  as before. First, we sample optimal points  $x^{*}, y^{*}$  from  $\mathcal{N}_d(0, I_d)$  and then generate  $a_{i}, c_{i}$  such that  $F(x^{*}, y^{*}) = 0$ .
$$
\left( \begin{array}{c} a _ {i} \\ c _ {i} \end{array} \right) = \left( \begin{array}{c c} A _ {i} & B _ {i} \\ - B _ {i} ^ {\intercal} & C _ {i} \end{array} \right) ^ {- 1} \left( \begin{array}{c} x ^ {*} \\ y ^ {*} \end{array} \right).
$$
In Fig. 2a, we run the algorithms on interpolated model  $\left(F_{i}(x^{*}) = 0\text{for all} i\in [n]\right)$ . Since the model is interpolated, we have  $\sigma_{*}^{2} = 0$  in Theorem 4.1 and linear convergence to the exact optimum asymptotically. In this setting, as shown in Fig. 2a, our proposed step-size results in major improvement compared to the decreasing step-size selection analyzed in [28]. In Fig. 2b, we compare the switching step-size rule with step-size from [28]. In Fig. 2b, we generate  $a_{i},c_{i}$  from the normal distribution. In this plot, we manually switch the step-size from constant to decreasing after 305 steps. We observe that such a semi-empirical rule has comparable performance to the step-size selection of Hsieh et al. [28].
(a) Interpolated Model
(b) Non-Interpolated Model
Figure 2: Comparison of our SPEG using our step-size against decreasing step-size of Hsieh et al. [28]. In plot (a), for constant step-size of SPEG we use the upper bound of (9). In plot (b), we run our switching step-size SPEG (11).
(a)  $\Lambda = 2$
(b)  $\Lambda = 5$
(c)  $\Lambda = 10$
(d)  $\Lambda = 20$
Figure 3: Comparison of SPEG with Uniform and Importance Sampling for different  $\Lambda \in \{2,5,10,20\}$ , where the eigenvalues of matrices  $A_{1},C_{1}$  are uniformly generated from the interval  $[0.1,\Lambda ]$
Uniform vs. Importance Sampling. In this experiment, we highlight the advantage of using importance sampling over uniform sampling. The eigenvalues of  $A_{1}, C_{1}$  are uniformly generated from the interval [0.1,  $\Lambda$ ] while the rest of the matrices are generated as mentioned before. We vary the value of  $\Lambda \in \{2, 5, 10, 20\}$  and run and compare SPEG with both uniform and importance sampling (see Fig. 3). For importance sampling, we use the probabilities  $p_{i} = L_{i} / \sum_{j=1}^{n} L_{j}$ . In Fig. 3, it is clear that as the value of  $\Lambda$  increases, the trajectories under uniform sampling get worse, while the trajectory under importance sampling remains almost identical. This behavior aligns well with our discussion in Section 5.
6.2 Weak Minty Variational Inequality Problems
This experiment verifies the convergence guarantees of SPEG in Theorem 4.5. Following the min-max problem mentioned in [8], we consider the objective function
$$
\min  _ {x \in \mathbb {R}} \max  _ {y \in \mathbb {R}} \frac {1}{n} \sum_ {i = 1} ^ {n} \xi_ {i} x y + \frac {\zeta_ {i}}{2} \left(x ^ {2} - y ^ {2}\right). \tag {17}
$$
In this experiment, we generate  $\xi_{i},\zeta_{i}$  such that  $L = 8$  and  $\rho = 1 / 32$  for the above min-max problem [8]. We implement SPEG with extrapolation step  $\gamma_{k} = 0.08$  and update step  $\omega_{k} = 0.01$  which satisfies the conditions on step-size in Theorem 4.5. In Fig. 4, we use a batchsize of 6. This plot illustrates that for some weak MVI problems the requirement on the stepsize from Theorem 4.5 can be too pessimistic and SPEG with relatively small batchsize achieves reasonable accuracy of the solution. The choice of batchsize ensures that bound (15) holds and  $\delta$  is small enough to guarantee convergence of SPEG. We
also tried to compare SPEG with  $\mathrm{SEG + }$  from [54], however, the authors do not mention their choice of update step-size. We examined several decreasing update step-size for which  $\mathrm{SEG + }$  failed to converge. Further details on experiments can be found in Appendix G.1.
Figure 4: Trajectory of SPEG for solving weak MVI. "Squared Operator Norm Error" in vertical axis denotes the  $\min_{0\leq k\leq K - 1}\mathbb{E}\left[\| F(\hat{x}_k)\|^2\right]$  of Theorem 4.5.
Sayantan Choudhury acknowledges support from the Acheson J. Duncan Fund for the Advancement of Research in Statistics. Nicolas Loizou acknowledges support from CISCO Research.
[1] J. Abernethy, K. A. Lai, and A. Wibisono. Last-iterate convergence rates for min-max optimization: Convergence of hamiltonian gradient descent and consensus optimization. In Algorithmic Learning Theory, pages 3–47. PMLR, 2021.
[2] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
[3] S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. NeurIPS, 2019.
[4] H. H. Bauschke, W. M. Moursi, and X. Wang. Generalized monotone operators and their averaged resolvents. Mathematical Programming, 189(1):55-74, 2021.
[5] A. Beznosikov and A. Gasnikov. Compression and data similarity: Combination of two techniques for communication-efficient solving of distributed variational inequalities. In Optimization and Applications: 13th International Conference, OPTIMA 2022, Petrovac, Montenegro, September 26–30, 2022, Revised Selected Papers, pages 151–162. Springer, 2023.
[6] A. Beznosikov, E. Gorbunov, H. Berard, and N. Loizou. Stochastic gradient descent-ascent: Unified theory and new efficient methods. AISTATS, 2023.
[7] A. Beznosikov, B. Polyak, E. Gorbunov, D. Kovalev, and A. Gasnikov. Smooth monotone stochastic variational inequalities and saddle point problems—survey. European Mathematical Society Magazine, 2023.
[8] A. Böhm. Solving nonconvex-nonconcave min-max problems exhibiting weak minty solutions. arXiv preprint arXiv:2201.12247, 2022.
[9] N. Brown, A. Bakhtin, A. Lerer, and Q. Gong. Combining deep reinforcement learning and search for imperfect-information games. NeurIPS, 2020.
[10] Y. Cai, A. Oikonomou, and W. Zheng. Tight last-iterate convergence of the extragradient method for constrained monotone variational inequalities. arXiv preprint arXiv:2204.09228, 2022.
[11] P. L. Combettes and T. Pennanen. Proximal methods for cohypomonotone operators. SIAM journal on control and optimization, 43(2):731-742, 2004.
[12] C. D. Dang and G. Lan. On the convergence properties of non-euclidean extragradient methods for variational inequalities with generalized monotone operators. Computational Optimization and applications, 60(2):277-310, 2015.
[13] C. Daskalakis, S. Skoulakis, and M. Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1466-1478, 2021.
[14] J. Diakonikolas, C. Daskalakis, and M. Jordan. Efficient methods for structured nonconvex-nonconcave min-max optimization. In AISTATS, 2021.
[15] R. D'Orazio, N. Loizou, I. Laradji, and I. Mitliagkas. On stochastic mirror descent: Convergence analysis and adaptive variants. ICML, 2021.
[16] F. Facchinei and C. Kanzow. Generalized Nash equilibrium problems. 4OR, 5(3):173-210, 2007.
[17] G. Gidel, H. Berard, G. Vignoud, P. Vincent, and S. Lacoste-Julien. A variational inequality perspective on generative adversarial networks. *ICLR*, 2019.