SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huizhuo Yuan $^{\diamond}$  Chris Junchi Li $^{\dagger}$  Gauthier Gidel $^{\ddagger}$
Michael I. Jordan†,‡ Quanquan Gu‡ Simon S. Du*
$^{\diamond}$  Department of Computer Science, University of California, Los Angeles
{hzyuan, qgu}@cs.ucla.edu
† Department of Electrical Engineering and Computer Sciences, University of California, Berkeley
{junchili, jordan}@cs.berkeley.edu
‡ DIRO, Université de Montréal and Mila
gauthier.gidel@umontreal.ca
Department of Statistics, University of California, Berkeley
* Paul G. Allen School of Computer Science and Engineering, University of Washington
ssdu@cs.washington.edu
We consider the problem of solving stochastic monotone variational inequalities with a separable structure using a stochastic first-order oracle. Building on standard extragradient for variational inequalities we propose a novel algorithm—stochastic accelerated gradient-extragradient (AG-EG)—for strongly monotone variational inequalities (VIs). Our approach combines the strengths of extragradient and Nesterov acceleration. By showing that its iterates remain in a bounded domain and applying scheduled restarting, we prove that AG-EG has an optimal convergence rate for strongly monotone VIs. Furthermore, when specializing to the particular case of bilinearly coupled strongly-convex-strongly-concave saddle-point problems, including bilinear games, our algorithm achieves fine-grained convergence rates that match the respective lower bounds, with the stochasticity being characterized by an additive statistical error term that is optimal up to a constant prefactor.
The variational inequality (VI) problem plays a central role in a wide range of optimization problems with convex structure, including convex minimization, saddle-point problems, and games [Facchine] and Pang, 2003; Nemirovski, 2004; Nemirovski et al., 2009; Juditsky et al., 2011; Jordan et al., 2023].
A general VI problem aims to find a solution  $z^{*}\in \mathcal{Z}$  that satisfies:
$$
\langle \mathcal {W} (z ^ {*}), z ^ {*} - z \rangle \leq 0, \quad \forall z \in \mathcal {Z}, \tag {1}
$$
where  $\mathcal{Z}$  is a finite-dimensional closed and convex feasible set and  $\mathcal{W}(\cdot)$  is a monotone operator in the following form:
$$
\mathcal {W} (z) = \nabla \mathcal {F} (z) + \mathcal {H} (z) + J ^ {\prime} (z) \equiv \mathbb {E} _ {\xi} [ \nabla \widetilde {\mathcal {F}} (z; \xi) ] + \mathbb {E} _ {\zeta} [ \widetilde {\mathcal {H}} (z; \zeta) ] + J ^ {\prime} (z), \tag {2}
$$
where  $\mathcal{F}$  is continuously differentiable with  $L$ -Lipschitz continuous gradient and is  $\mu$ -strongly convex,  $\mathcal{H}$  is an  $M$ -Lipschitz monotone operator,  $J' \in \partial J$  is the subgradient of a simple and convex function,  $\xi$  and  $\zeta$  are drawn from distributions  $\mathcal{D}_{\xi}$  and  $\mathcal{D}_{\zeta}$ , respectively. This formulation captures a separable structure in which  $\mathcal{H}$  usually models the competing forces in a system, and  $J$  models
a nonsmooth factor. In addition, we consider the stochastic setting where we can only access  $\nabla \mathcal{F}$  and  $\mathcal{H}$  through their unbiased estimators  $\nabla \widetilde{\mathcal{F}}(z; \xi)$  and  $\widetilde{\mathcal{H}}(z; \zeta)$  respectively.
A notable instance of the VI problem (1) with separable structure (2) is the widely studied bilinearly coupled strongly-convex-strongly-concave saddle-point problem:
$$
\min  _ {\boldsymbol {x} \in \mathbb {R} ^ {n}} \max  _ {\boldsymbol {y} \in \mathbb {R} ^ {m}} \mathcal {F} (\boldsymbol {x}, \boldsymbol {y}) = F (\boldsymbol {x}) + H (\boldsymbol {x}, \boldsymbol {y}) - G (\boldsymbol {y}) \equiv \mathbb {E} _ {\xi} [ f (\boldsymbol {x}; \xi) ] + \mathbb {E} _ {\zeta} [ h (\boldsymbol {x}, \boldsymbol {y}; \zeta) ] - \mathbb {E} _ {\xi} [ g (\boldsymbol {y}; \xi) ], \tag {3}
$$
where  $H(\pmb{x},\pmb{y}) \equiv \pmb{x}^{\top}\mathbf{B}\pmb{y} - \pmb{x}^{\top}\mathbf{u}_{\pmb{x}} + \mathbf{u}_{\pmb{y}}^{\top}\pmb{y}$  is the bilinear coupling function with the coupling matrix  $\mathbf{B} \in \mathbb{R}^{n\times m}$ . Note that (3) is a special instance of (1) when taking  $\mathcal{F}(\pmb{z}) = F(\pmb{x}) + G(\pmb{y})$ ,  $\mathcal{H}(\pmb{z}) = [\nabla_{\pmb{x}}H(\pmb{x},\pmb{y}); - \nabla_{\pmb{y}}H(\pmb{x},\pmb{y})]$  and  $J = 0$ . In addition to a wide range of applications in economics, problems of form (3) are becoming increasingly important in machine learning. For instance, (3) appears in reinforcement learning, differentiable games, regularized empirical risk minimization, and robust optimization formulations. It can also be seen as a local approximation of the nonconvex-nonconcave minimax games—e.g., the generative adversarial network (GAN) [Goodfellow et al., 2020]—around a local Nash equilibrium [Mescheder et al., 2017], [Nagarajan and Kolter, 2017].
In this paper, we aim to improve the efficiency of solving (1) by utilizing the structural information of the monotone operator in (2). More specifically, we consider the case when  $\mathcal{F}$  is strongly monotone, or zero. Although optimal convergence results have been obtained for the monotone VI problem (1) [Chen et al., 2017] as well as the special case of convex-concave saddle-point problem with bilinear coupling (3) [Chen et al., 2014], it remains open how to design an optimal algorithm for the strongly monotone VI problem. Notably, for the special case (3) when  $F$  and/or  $G$  are strongly convex, several concurrent works have independently obtained the optimal convergence rates [Kovalev et al., 2022, Thekumparampil et al., 2022, Jin et al., 2022, Metelev et al., 2022, Li et al., 2022b]. On the other hand, when both  $F$  and  $G$  are zero, optimal convergence results have been obtained by Li et al. [2022a] and the accelerated-gradient optimistic gradient approach [Li et al., 2022b]. We defer a more complete overview of related work to the appendix.
We start with the strongly monotone VI problem in an unbounded feasible set, extending the scope of recent work such as Jin et al. [2022] and going beyond earlier studies that focus on nonstrongly monotone VIs in a bounded feasible set [Juditsky et al. 2011 Chen et al. 2017]. We propose a class of algorithms named stochastic accelerated gradient-extragradient (AG-EG), which combine Nesterov acceleration with the extragradient method. By employing either a strong-convexity shifting technique or a scheduled restarting scheme, our algorithm achieves convergence rates that match the lower bounds for the general strongly monotone VI problem [1], the special SC-SC saddle-point problem [3], and bilinear games, in both deterministic and stochastic settings, thus providing a unified optimal solution. In sharp contrast to the accelerated mirror-prox (AMP) algorithm proposed by Chen et al. [2017], Jordan et al. [2023], our analysis does not rely on the boundedness of the feasible set  $\mathcal{Z}$ , which makes our algorithm projection-free. We also extend our algorithm to VIs with bounded feasible set and/or nondifferentiable convex regularization through proximal mapping. We summarize our contributions as follows:
(1) We present a direct approach for separable strongly monotone VIs, where the iteration complexity lower bound due to Zhang et al. [2022] is matched as  $\mathcal{O}\left(\sqrt{\frac{L}{\mu}} + \frac{M}{\mu} + \frac{\sigma^2}{\mu^2\varepsilon^2}\right)\log \left(\frac{L}{\mu}\frac{1}{\varepsilon}\right)$ , which admits a sharp near-unity coefficient [§2.3] Theorem [2.3]. Here  $\sigma^2$  is the weighted, uniform variance bound on the stochastic gradient and stochastic operator.
(2) We also present a stochastic AG-EG algorithm equipped with scheduled restarting, which achieves the sharpest possible iteration complexity of  $\mathcal{O}\left(\left(\sqrt{\frac{L}{\mu}} + \frac{M}{\mu}\right)\log\left(\frac{1}{\varepsilon}\right) + \frac{\sigma^2}{\mu^2\varepsilon^2}\right)$  for finding an  $\varepsilon$ -optimal point. The deterministic part matches the complexity lower bound in Zhang et al. [2022], while the stochastic part matches the optimal statistical error.
<sup>1</sup>VIs in an unbounded feasible set is more difficult to solve because existing algorithms and analyses crucially rely on the boundedness of the feasible set.
When specializing the VI problem to bilinearly coupled SC-SC saddle-point problems, our results have the following implications:
Strongly-convex-strongly-concave (SC-SC) Saddle-Point Problem. For the class of SC-SC saddle-point problems, the stochastic AG-EG descent-ascent Algorithm equipped with scaling reduction, achieves an iteration complexity of
$$
\mathcal {O} \left(\left(\sqrt {\frac {L _ {F}}{\mu_ {F}} \vee \frac {L _ {G}}{\mu_ {G}}} + \sqrt {\frac {\lambda_ {\max } (\mathbf {B} ^ {\top} \mathbf {B})}{\mu_ {F} \mu_ {G}}}\right) \log \left(\frac {1}{\varepsilon}\right) + \frac {\sigma^ {2}}{\mu_ {F} ^ {2} \varepsilon^ {2}}\right), \tag {4}
$$
where  $F: \mathbb{R}^n \to \mathbb{R}$  is  $L_F$ -smooth and  $\mu_F$ -strongly convex,  $G: \mathbb{R}^m \to \mathbb{R}$  is  $L_G$ -smooth and  $\mu_G$ -strongly convex. When the optimization problem is deterministic, the complexity upper bound matches the lower bound of Zhang et al. [2022][§3.1] Corollary 2.8].
Bilinear Games. For bilinear games  $(\nabla f(\pmb{x};\xi) = \mathbf{0}$  and  $\nabla g(\pmb{y};\xi) = \mathbf{0}$  almost surely), Algorithm equipped with scheduled restarting achieves an iteration complexity of
$$
\mathcal {O} \left(\sqrt {\frac {\lambda_ {\operatorname* {m a x}} (\mathbf {B} ^ {\top} \mathbf {B})}{\lambda_ {\operatorname* {m i n}} (\mathbf {B B} ^ {\top})}} \log \left(\frac {\sqrt [ 4 ]{\lambda_ {\operatorname* {m i n}} (\mathbf {B B} ^ {\top}) \lambda_ {\operatorname* {m a x}} (\mathbf {B} ^ {\top} \mathbf {B})}}{\sigma_ {\mathrm {B i l}}}\right) + \frac {\sigma_ {\mathrm {B i l}} ^ {2}}{\lambda_ {\operatorname* {m i n}} (\mathbf {B B} ^ {\top}) \varepsilon^ {2}}\right), \tag {5}
$$
where  $\sigma_{\mathrm{Bil}}^2$  is the variance of the stochastic gradient on the bilinear coupling term. When there is no randomness, this complexity result reduces to  $\mathcal{O}\left(\sqrt{\frac{\lambda_{\max}(\mathbf{B}^\top\mathbf{B})}{\lambda_{\min}(\mathbf{BB}^\top)}}\log \left(\frac{1}{\varepsilon}\right)\right)$  for bilinear games, matching the lower bound of [Ibrahim et al. 2020] [§3.2, Corollary 3.3].
Organization. The rest of this paper is organized as follows. Section 2 proposes the Accelerated Gradient-Extragradient Descent-Ascent algorithm for strongly monotone VIs, showing that it achieves an accelerated convergence rate, and extending to VIs with bounded domains with proximal operator. Section 3 discusses two specific instances of saddle-point problems, where our proposed AG-EG algorithm has a convergence rate that matches the corresponding lower bounds. Finally, Section 4 summarizes our results and suggests future directions.
Notation. Let  $\lambda_{\mathrm{max}}(\mathbf{M})$  (resp.  $\lambda_{\mathrm{min}}(\mathbf{M})$  be the largest (resp. smallest) eigenvalue of a real symmetric matrix  $\mathbf{M}$ . Let  $a \vee b \equiv \max(a, b)$  (resp.  $a \wedge b \equiv \min(a, b)$ ) denote the maximum (resp. minimum) value of two reals  $a, b$ . For two nonnegative real sequences  $(a_n)$  and  $(b_n)$ , we write  $a_n = \mathcal{O}(b_n)$  or  $a_n \lesssim b_n$  (resp.  $a_n = \Omega(b_n)$  or  $a_n \gtrsim b_n$ ) to denote  $a_n \leq C b_n$  (resp.  $a_n \geq C b_n$ ) for all  $n \geq 1$  for a positive, numerical constant  $C$ , and let  $a_n \asymp b_n$  if both  $a_n \lesssim b_n$  and  $a_n \gtrsim b_n$  hold. We also let  $a_n = \tilde{\mathcal{O}}(b_n)$  denote  $a_n \leq C b_n$  where  $C$  hides a polylogarithmic factor in problem-dependent constants. We let  $[\pmb{x}; \pmb{y}] \in \mathbb{R}^{n+m}$  concatenate two vectors  $\pmb{x} \in \mathbb{R}^n$  and  $\pmb{y} \in \mathbb{R}^m$ . Finally for two real symmetric matrices  $\mathbf{A}$  and  $\mathbf{B}$ , we denote  $\mathbf{A} \preceq \mathbf{B}$  (resp.  $\mathbf{A} \succeq \mathbf{B}$ ) when  $\mathbf{v}^\top (\mathbf{A} - \mathbf{B})\mathbf{v} \leq 0$  (resp.  $\mathbf{v}^\top (\mathbf{A} - \mathbf{B})\mathbf{v} \geq 0$ ) holds for all vectors  $\mathbf{v}$ .
2 Accelerated Gradient-Extragradients Descent-Ascent Algorithm
In this section, we focus on accelerating the extragradient algorithm for the strongly monotone VI problem in (1) with separable structure (2). Our algorithm design draws inspiration from the work of Chen et al. [2017] on the stochastic Accelerated MirrorProx (AMP) algorithm for nonstrongly monotone VIs. The AMP algorithm applies Nesterov-type acceleration on top of the mirror-prox method [Korpelevich, 1976, Nemirovski, 2004] and attains the optimal iteration complexity of  $\mathcal{O}\left(\sqrt{\frac{L}{\varepsilon} + \frac{M}{\varepsilon}}\right)$ . However, the big-O notation hides the diameter of the feasible set, and the existing theory for the AMP algorithm can only deal with VIs with bounded domain. Our algorithm not only achieves the optimal convergence rates for the strongly monotone VI problem with separable structure but we also remove the dependency on the diameter of the feasible set. Therefore, our algorithm can deal with VIs with unbounded domains.
2. For the function class of bilinear games, we assume that  $n = m$  where  $\mathbf{B}$  is a nonsingular square matrix, so that  $\lambda_{\min}(\mathbf{BB}^\top) > 0$  and the complexity makes sense. See §3.2 for more on this.
Throughout §2 we maintain conceptual simplicity by presenting all our algorithm designs in the deterministic setting, while presenting the convergence results in the more general stochastic setting. These results can be easily reduced to the deterministic setting when the stochastic noise vanishes.
2.1 Setting and Assumptions
In this section, we formally introduce our assumptions. We first state the smoothness and monotonicity assumptions that we impose on  $\mathcal{F}$  and  $\mathcal{H}$ .
Assumption 2.1 (Monotonicity, strong convexity and smoothness) We assume that function  $\mathcal{F}(\cdot)$  is continuously differentiable with  $L$ -Lipschitz continuous gradient and is  $\mu$ -strongly convex. That is, for any  $z, z' \in \mathcal{Z}$ ,
$$
\frac {\mu}{2} \| \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \| ^ {2} \leq \mathcal {F} (\boldsymbol {z}) - \mathcal {F} (\boldsymbol {z} ^ {\prime}) - \nabla \mathcal {F} (\boldsymbol {z} ^ {\prime}) ^ {\top} (\boldsymbol {z} - \boldsymbol {z} ^ {\prime}) \leq \frac {L}{2} \| \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \| ^ {2}.
$$
Furthermore, operator  $\mathcal{H}(\cdot)$  is monotone and  $M$ -Lipschitz in the sense that for any  $z, z' \in \mathcal{Z}$ ,
$$
\langle \mathcal {H} (z) - \mathcal {H} \left(z ^ {\prime}\right), z - z ^ {\prime} \rangle \geq 0, \| \mathcal {H} (z) - \mathcal {H} \left(z ^ {\prime}\right) \| \leq M \| z - z ^ {\prime} \|.
$$
Second, we impose assumptions on the noise variance.
Assumption 2.2 (Unbiased gradients and variance bounds) We assume that  $\mathbf{z} \in \mathcal{Z}$ , samples  $\xi \sim \mathcal{D}_{\xi}$  and  $\zeta \sim \mathcal{D}_{\zeta}$  are drawn from given distributions such that the following conditions hold:  $\mathbb{E}_{\xi}[\nabla \tilde{\mathcal{F}}(\mathbf{z}; \xi)] = \nabla \mathcal{F}(\mathbf{z})$ ,  $\mathbb{E}_{\zeta}[\tilde{\mathcal{H}}(\mathbf{z}; \zeta)] = \mathcal{H}(\mathbf{z})$ , and
$$
\mathbb {E} _ {\xi} \left[ \| \nabla \tilde {\mathcal {F}} (\boldsymbol {z}; \xi) - \nabla \mathcal {F} (\boldsymbol {z}) \| ^ {2} \right] \leq \sigma_ {\mathrm {S t r}} ^ {2}, \quad \mathbb {E} _ {\zeta} \left[ \| \tilde {\mathcal {H}} (\boldsymbol {z}; \zeta) - \mathcal {H} (\boldsymbol {z}) \| ^ {2} \right] \leq \sigma_ {\mathrm {B i l}} ^ {2}. \tag {6}
$$
For all results in this work, we suppose that Assumptions 2.1 and 2.2 hold with appropriate parameter settings. Given a desired accuracy  $\varepsilon >0$ , our goal is to find an  $\varepsilon$ -optimal point defined as:
Definition 2.1 ( $\varepsilon$ -Optimal point) A point  $z \in \mathcal{Z}$  is called an  $\varepsilon$ -optimal point for the VI problem in (1) if  $\| z - z^{*} \| \leq \varepsilon$ .
2.2 The ExtraGradient (EG) Algorithm
We first consider the case where  $\mathcal{Z}$  is the entire space  $\mathbb{R}^n$  and the objective is smooth  $(J = 0)$ . The extragradient (EG) algorithm, introduced by Korpelevich [1976], is designed to address cyclic behavior in saddle-point problems by introducing an extrapolated point for gradient evaluation. In the context of VI problems (1), let  $z_{t}$  represents the  $t$ -th iterate of the EG algorithm. The update rule of EG is as follows:
$$
\boldsymbol {z} _ {t + 1} = \boldsymbol {z} _ {t} - \eta \mathcal {W} \left(\boldsymbol {z} _ {t} - \eta \mathcal {W} \left(\boldsymbol {z} _ {t}\right)\right), \tag {7}
$$
where  $\eta > 0$  is the step size. For a  $L$ -smooth and  $\mu$ -strongly monotone operator  $\mathcal{W}$ , [Tseng [1995], Mokhtari et al. [2020], Gidel et al. [2019a] have shown that the EG algorithm achieves an iteration complexity of  $\mathcal{O}(\kappa \log(1/\varepsilon))$ , where  $\kappa = L / \mu$  denotes the condition number of the problem.
2.3 Accelerating the ExtraGradient Algorithm, Direct Approach
The convergence rate of the EG algorithm is far from optimal for the strongly monotone VI problem in (1) with separable structure (2). Firstly, the update rule in (7) takes  $\mathcal{W}$  as a whole without utilizing the separable structure. This prevents us from exploiting the properties of  $\nabla \mathcal{F}$ . Secondly, in the case of bilinear games, the established lower bound for EG is  $\Omega (\sqrt{\kappa}\log (1 / \varepsilon))$  rather than  $\Omega (\kappa \log (1 / \varepsilon))$ . This discrepancy highlights the potential for accelerating the EG algorithm in various directions. We first rewrite the EG update rule in (7) as follows:
$$
\begin{array}{l} \boldsymbol {z} _ {t - \frac {1}{2}} = \boldsymbol {z} _ {t - 1} - \eta \mathcal {W} (\boldsymbol {z} _ {t - 1}) = \boldsymbol {z} _ {t - 1} - \eta \left(\mathcal {H} (\boldsymbol {z} _ {t - 1}) + \nabla \mathcal {F} (\boldsymbol {z} _ {t - 1})\right), \\ \boldsymbol {z} _ {t} = \boldsymbol {z} _ {t - 1} - \eta \mathcal {W} \left(\boldsymbol {z} _ {t - \frac {1}{2}}\right) = \boldsymbol {z} _ {t - 1} - \eta \left(\mathcal {H} \left(\boldsymbol {z} _ {t - \frac {1}{2}}\right) + \nabla \mathcal {F} \left(\boldsymbol {z} _ {t - \frac {1}{2}}\right)\right). \tag {8} \\ \end{array}
$$
To accelerate the process based on  $\nabla \mathcal{F}$ , we consider Nesterov's second acceleration scheme on minimizing a single convex function  $\mathcal{F}$  [Tseng, 2008; Lan and Zhou, 2018; Lin et al., 2020c]:
$$
\boldsymbol {z} _ {t - 1} ^ {\mathrm {m d}} = \left(1 - \alpha_ {t}\right) \boldsymbol {z} _ {t - 1} ^ {\mathrm {a g}} + \alpha_ {t} \boldsymbol {z} _ {t - 1}, \boldsymbol {z} _ {t} = \boldsymbol {z} _ {t - 1} - \frac {\eta}{\alpha_ {t}} \nabla \mathcal {F} \left(\boldsymbol {z} _ {t - 1} ^ {\mathrm {m d}}\right), \boldsymbol {z} _ {t} ^ {\mathrm {a g}} = \left(1 - \alpha_ {t}\right) \boldsymbol {z} _ {t - 1} ^ {\mathrm {a g}} + \alpha_ {t} \boldsymbol {z} _ {t}, \tag {9}
$$
where  $\alpha_{t}$  is the extrapolation step size in the standard three-line Nesterov scheme. Here we adopt the notation  $z^{\mathrm{md}}$  and  $z^{\mathrm{ag}}$  to indicate the middle point and the aggregated point [Chen et al., 2017], respectively. Next, to achieve acceleration, we replace the gradient of  $\nabla \mathcal{F}$  evaluated at both  $z_{t-1}$  and  $z_{t-\frac{1}{2}}$  in (8) by the gradient evaluated at the extrapolated point  $z_{t-1}^{\mathrm{md}}$  in (9). Furthermore, we shift the index of  $z^{\mathrm{ag}}$  by  $\frac{1}{2}$  to indicate the use of  $z_{t-\frac{1}{2}}$  instead of  $z_{t}$  in the  $z^{\mathrm{ag}}$  update in (9). In addition, we take into account the  $\mu$ -strong convexity of  $\mathcal{F}$  and shift the gradient of the strongly convex part  $\nabla z\left[\frac{\mu}{2}\|z-z_0\|^2\right] = \mu(z-z_0)$  from  $\nabla \mathcal{F}(z)$  to  $\mathcal{H}(z)$  as  $\mathcal{W}(z) = (\nabla \mathcal{F}(z) - \mu(z-z_0)) + (\mathcal{H}(z) + \mu(z-z_0))$ , we obtain the following update rule for a direct version of an accelerated EG algorithm (different step size schemes for  $\eta_t$  are required for different algorithmic designs):
$$
\left\{ \begin{array}{l} z _ {t - 1} ^ {\mathrm {m d}} = \left(1 - \alpha_ {t}\right) z _ {t - \frac {3}{2}} ^ {\mathrm {a g}} + \alpha_ {t} z _ {t - 1}, \\ z _ {t - \frac {1}{2}} = z _ {t - 1} - \eta_ {t} \left(\mathcal {H} \left(z _ {t - 1}\right) + \nabla \mathcal {F} \left(z _ {t - 1} ^ {\mathrm {m d}}\right) - \mu \left(z _ {t - 1} ^ {\mathrm {m d}} - z _ {t - 1}\right)\right), \\ z _ {t} = z _ {t - 1} - \eta_ {t} \left(\mathcal {H} \left(z _ {t - \frac {1}{2}}\right) + \nabla \mathcal {F} \left(z _ {t - 1} ^ {\mathrm {m d}}\right) - \mu \left(z _ {t - 1} ^ {\mathrm {m d}} - z _ {t - \frac {1}{2}}\right)\right), \\ z _ {t - \frac {1}{2}} ^ {\mathrm {a g}} = \left(1 - \alpha_ {t}\right) z _ {t - \frac {3}{2}} ^ {\mathrm {a g}} + \alpha_ {t} z _ {t - \frac {1}{2}}. \end{array} \right. \tag {10}
$$
We call the algorithm in [10] the accelerated gradient-extragradient, direct approach (AG-EG-Direct), and postpone its full description to Algorithm 2 in §C.1. The final output of the direct approach is  $z_{T}$  after  $T$  iterates. The following theorem records the convergence rate and iteration complexity of AG-EG (direct approach).
Theorem 2.3 (Convergence of stochastic AG-EG, direct approach) Suppose Assumptions [2.1] and [2.2] hold. Fix any  $r \in (0,1)$ ,  $\beta \in (0,\infty)$ , let  $\kappa_{\beta} = \frac{L}{\mu} + \frac{(1 + \beta)M^2}{\mu^2}$  and set the step size upper bound  $\bar{\alpha} \equiv \frac{r}{1 + \sqrt{1 + r\kappa_{\beta}}}$ . For any sequence of step sizes  $\alpha_t \in (0,\bar{\alpha}]$  and  $\eta_t = \frac{\alpha_t}{\mu}$ , the iterates of stochastic AG-EG (direct approach) satisfy that for all  $t = 1,\dots,T$ , we have
$$
\mathbb {E} \left\| \boldsymbol {z} _ {t} - \boldsymbol {z} ^ {*} \right\| ^ {2} \leq \left\| \boldsymbol {z} _ {0} - \boldsymbol {z} ^ {*} \right\| ^ {2} \left(\frac {L}{\mu} + 1\right) \prod_ {s = 1} ^ {t} \left(1 - \alpha_ {s}\right) + \frac {3 \sigma^ {2}}{\mu^ {2}} \sum_ {s = 1} ^ {t} \alpha_ {s} ^ {2} \prod_ {\tau = s + 1} ^ {t} \left(1 - \alpha_ {\tau}\right), \tag {11}
$$
where we define  $\sigma = \frac{1}{\sqrt{3}}\sqrt{\frac{1}{1 - r}\sigma_{\mathrm{Str}}^2 + (2 + \frac{1}{\beta})\sigma_{\mathrm{Bil}}^2}$ .
In the rest of the paper, we use the same definition  $\sigma$  as in Theorem 2.3. The proof of Theorem 2.3 is provided in §D.4. We further note that one possible choice of step size is to let  $\alpha_{t} \equiv \alpha$ , such that (11) reduces to
$$
\mathbb {E} \left\| \boldsymbol {z} _ {t} - \boldsymbol {z} ^ {*} \right\| ^ {2} \leq \left\| \boldsymbol {z} _ {0} - \boldsymbol {z} ^ {*} \right\| ^ {2} \left(\frac {L}{\mu} + 1\right) e ^ {- \alpha t} + \frac {3 \sigma^ {2}}{\mu^ {2}} \alpha .
$$
For any given  $T \geq 1$ , by choosing the optimal  $\alpha = \frac{1}{T}\left(1 + \log \left(\frac{\mu^2T}{3\sigma^2}\left(\frac{L}{\mu} + 1\right)\|z_0 - z^*\|^2\right)\right) \wedge \bar{\alpha},$  (11) implies
$$
\mathbb {E} \| \boldsymbol {z} _ {T} - \boldsymbol {z} ^ {*} \| ^ {2} \leq \| \boldsymbol {z} _ {0} - \boldsymbol {z} ^ {*} \| ^ {2} \left(\frac {L}{\mu} + 1\right) e ^ {- \bar {\alpha} T} + \frac {3 \sigma^ {2}}{\mu^ {2} T} \left(1 + \log \left(\frac {\mu^ {2} T}{3 \sigma^ {2}} \left(\frac {L}{\mu} + 1\right) \| \boldsymbol {z} _ {0} - \boldsymbol {z} ^ {*} \| ^ {2}\right)\right).
$$
Prescribing the desired accuracy  $\varepsilon > 0$ , the iteration complexity to output an  $\varepsilon$ -optimal minimax point is  $\boxed{3}$
$$
\mathcal {O} \left(\left(\sqrt {\frac {L}{\mu}} + \frac {M}{\mu} + \frac {\sigma^ {2}}{\mu^ {2} \varepsilon^ {2}}\right) \log \left(\left(\frac {L}{\mu} + 1\right) \| \boldsymbol {z} _ {0} - \boldsymbol {z} ^ {*} \| ^ {2} / \varepsilon^ {2}\right)\right).
$$