SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conjecture that the logarithmic factor in the optimal statistical rate  $\frac{\sigma^2}{\mu^2\varepsilon^2}$  is removable using a proper diminishing step size, a possibility that we reserve for future study. In the setting of deterministic optimization, setting  $\sigma = 0$  and  $r \to 1^{-}$ ,  $\beta \to 0^{+}$  in Theorem 2.3 we obtain the optimal iteration complexity bound as follows:
$$
\left(1 + \sqrt {1 + \frac {L}{\mu} + \frac {M ^ {2}}{\mu^ {2}}}\right) \log \left(\left(\frac {L}{\mu} + 1\right) / \varepsilon^ {2}\right). \tag {12}
$$
3 Throughout this work, we focus on the iteration complexity whereas the required number of queries to the stochastic gradient oracle is three times the iteration complexity.
Algorithm 1 Stochastic AcceleratedGradient-ExtraGradient (AG-EG) Descent-Ascent Algorithm, with Scheduled Restarting
Require: Initialization  $z_0^{[0]}$ , total number of epochs  $\mathcal{S} \geq 1$ , total number of per-epoch iterates  $(T_s : s = 1, \dots, \mathcal{S})$ , stepsizes  $(\alpha_t, \eta_t : t = 1, 2, \dots)$ .
for  $s = 1,2,\ldots ,\mathcal{S}$  do
$$
\operatorname {S e t} z _ {- \frac {1}{2}} ^ {\mathrm {a g}} \leftarrow z _ {0} ^ {[ s - 1 ]}, z _ {0} \leftarrow z _ {0} ^ {[ s - 1 ]}, z _ {0} ^ {\mathrm {m d}} \leftarrow z _ {0} ^ {[ s - 1 ]}
$$
for  $t = 1,2,\dots ,T_s$  do
Draw samples  $\xi_{t - \frac{1}{2}}\sim \mathcal{D}_{\xi}$  from oracle, and also  $\zeta_{t - \frac{1}{2}},\zeta_t\sim \mathcal{D}_\zeta$  independently from oracle
$$
\boldsymbol {z} _ {t - \frac {1}{2}} \leftarrow \boldsymbol {z} _ {t - 1} - \eta_ {t} \left(\tilde {\mathcal {H}} \left(\boldsymbol {z} _ {t - 1}; \zeta_ {t - \frac {1}{2}}\right) + \nabla \tilde {\mathcal {F}} \left(\boldsymbol {z} _ {t - 1} ^ {\mathrm {m d}}; \xi_ {t - \frac {1}{2}}\right)\right)
$$
$$
\pmb {z} _ {t - \frac {1}{2}} ^ {\mathrm {a g}} \gets (1 - \alpha_ {t}) \pmb {z} _ {t - \frac {3}{2}} ^ {\mathrm {a g}} + \alpha_ {t} \pmb {z} _ {t - \frac {1}{2}}
$$
$$
\boldsymbol {z} _ {t} \leftarrow \boldsymbol {z} _ {t - 1} - \eta_ {t} \left(\tilde {\mathcal {H}} \left(\boldsymbol {z} _ {t - \frac {1}{2}}; \zeta_ {t}\right) + \nabla \tilde {\mathcal {F}} \left(\boldsymbol {z} _ {t - 1} ^ {\mathrm {m d}}; \xi_ {t - \frac {1}{2}}\right)\right)
$$
$$
\pmb {z} _ {t} ^ {\mathrm {m d}} \gets (1 - \alpha_ {t + 1}) \pmb {z} _ {t - \frac {1}{2}} ^ {\mathrm {a g}} + \alpha_ {t + 1} \pmb {z} _ {t}
$$
Set  $z_0^{[s]}\gets z_{T_s - \frac{1}{2}}^{\mathrm{ag}}$
{//Warm-start using the output of the previous epoch}
Output:  $z_0^{[\mathcal{S}]}$
Remark 2.4 Our complexity bounds fundamentally differs from the previous analysis [Chen et al. 2017; Jordan et al. 2023] for separable smooth (strongly) monotone VIs. The convergence results in previous studies are dependent on the diameter of the domain, whereas our convergence rate is independent of the domain parameters and eliminates the need for projection onto a bounded domain. Moreover, our contributions go beyond those of [Chen et al. 2017] by extending the analysis to the strongly monotone case. In comparison with Jordan et al. [2023], we design an algorithm where  $\nabla \mathcal{F}$  is strongly monotone and resolve the open problem of extending the analysis to the stochastic case. Additionally, our complexity bound in [12] indicates a near-unity coefficient on the condition-number exponent, improving the corresponding coefficient in [Chen et al. 2017] Theorem 15] by an asymptotic factor of 4.
The direct approach, which reduces to EG when  $\nabla \mathcal{F} = 0$  and  $\mu = 0$ , falls short of attaining optimality within the specific regime of bilinear games. In the next subsection, we will introduce a new algorithm that can overcome this limitation.
2.4 Accelerating the ExtraGradient Algorithm with Scheduled Restarting
In this subsection, we solve problem (1) by further accelerating the stochastic EG algorithm. Rather than directly relying on the strong monotonicity of  $\nabla \mathcal{F}$ , the inner updates of our new algorithm are identical to the updates in (10) with  $\mu = 0$ . Due to the domain-independent nature of our analysis, we can apply the scheduled restarting technique [O'donoghue and Candes, 2015, Roulet and d'Aspremont, 2017, Renegar and Grimmer, 2022] to the outer loop, accelerating the algorithm from sublinear convergence to linear convergence. In addition, the output of our algorithm is the aggregated point  $z_{T - \frac{1}{2}}^{\mathrm{ag}}$  after  $T$  iterates. We present the full algorithm in Algorithm 1.
We first present the convergence rate of a single epoch (i.e., the inner loop) of Algorithm  $\mathbb{L}$  in Theorem 2.5. To accommodate more flexibility in the choice of parameters, we introduce three constants  $r,\beta$ , and  $C$  in the theorem statement.
Theorem 2.5 (Convergence of stochastic AG-EG, one epoch) Suppose Assumptions [2.1] and [2.2] hold. For any fixed epoch length  $T \geq 1$ , any constant  $r \in (0,1)$ ,  $\beta \in (0,\infty)$ ,  $C \in (0,\infty)$ , choose step sizes  $\alpha_{t} = \frac{2}{t + 1}$  and  $\eta_t$  such that
$$
\frac {t}{\eta_ {t}} = \frac {2}{r} L \vee B + \sqrt {\frac {1 + \beta}{r}} M t, \tag {13}
$$
where  $B = \frac{\sigma\sqrt{T}(T + 1)}{C\sqrt{\mathbb{E}\|\pmb{z}_0 - \pmb{z}^*\|^2}}$ . The output  $\pmb{z}_{T - \frac{1}{2}}^{ag}$  of a single epoch of Algorithm satisfies
$$
\mathbb {E} \left\| \boldsymbol {z} _ {T - \frac {1}{2}} ^ {a g} - \boldsymbol {z} ^ {*} \right\| ^ {2} \leq \frac {2}{\mu (T + 1)} \left(\frac {2 L}{r T} + A \sqrt {\frac {1 + \beta}{r}} M\right) \mathbb {E} \| \boldsymbol {z} _ {0} - \boldsymbol {z} ^ {*} \| ^ {2} + \frac {2 (\frac {1}{C} + C) \sigma}{\mu \sqrt {T}} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {0} - \boldsymbol {z} ^ {*} \| ^ {2}}, \tag {14}
$$
where the prefactor  $A \equiv 1 + C^2 B\eta_1 \leq 1 + C^2$  reduces to 1 when  $\sigma = 0$ .
The proof of Theorem 2.5 is provided in §D.3. We make a few remarks on Theorem 2.5 as follows:
Remark 2.6 In the setting of deterministic optimization, by taking  $\sigma = 0$ ,  $r \to 1^{-}$ ,  $\beta \to 0^{+}$  in our analysis, with step size choice  $\eta_{t} = \frac{t}{2L + Mt}$ , we obtain that
$$
\left\| z _ {T - \frac {1}{2}} ^ {a g} - z ^ {*} \right\| ^ {2} \leq \frac {2}{\mu (T + 1)} \left(\frac {2 L}{T} + M\right) \left\| z _ {0} - z ^ {*} \right\| ^ {2}, \tag {15}
$$
In this setting, the algorithm is independent of  $B$  and requires no knowledge of  $\| z_0 - z^*\|^2$ . In the face of stochasticity, we choose  $C = 1$  when the initial distance to the optimal point is known. Alternatively, when only an over-estimate  $\Gamma_0$  of  $\sqrt{\mathbb{E}\|z_0 - z^*\|^2}$  is available, we can set (large enough)  $C = \frac{\Gamma_0}{\sqrt{\mathbb{E}\|z_0 - z^*\|^2}} \geq 1$  to obtain
$$
\mathbb {E} \left\| \boldsymbol {z} _ {T - \frac {1}{2}} ^ {a g} - \boldsymbol {z} ^ {*} \right\| ^ {2} \leq \frac {2}{\mu (T + 1)} \left(\frac {2 L}{r T} + 2 \sqrt {\frac {1 + \beta}{r}} M\right) \Gamma_ {0} ^ {2} + \frac {4 \sigma}{\mu \sqrt {T}} \Gamma_ {0}. \tag {16}
$$
Remark 2.7 When the constants are not a concern, the coarse-grained choices of  $r = \frac{1}{2}$  and  $\beta = 1$  would suffice. Nevertheless, to optimize the constants, the tradeoff between the deviation of  $r$  from 1 and  $\beta$  from 0 is crucial, as it determines a balance between the stochastic gradient noise variance and the convergence rate coefficients.
To prepare for our multi-epoch result with the help of scheduled restarting, we perform an induction based on (16) as follows. Supposing that  $\mathbb{E}\| z_0^{[s - 1]} - z^*\|^2\leq \Gamma_0^2 e^{1 - s}$  hold, by taking  $r = \frac{1}{2}$  and  $\beta = 1$ , we have
$$
\mathbb {E} \| \pmb {z} _ {0} ^ {[ s ]} - \pmb {z} ^ {*} \| ^ {2} \lesssim \frac {L}{\mu T _ {s} ^ {2}} \Gamma_ {0} ^ {2} e ^ {1 - s} + \frac {M}{\mu T _ {s}} \Gamma_ {0} ^ {2} e ^ {1 - s} + \frac {\sigma}{\mu \sqrt {T _ {s}}} \Gamma_ {0} e ^ {\frac {1 - s}{2}}.
$$
Setting the right-hand side of the above inequality to satisfy  $\leq \Gamma_0^2 e^{-s}$ , and solving for  $T_{s}$ , we need the epoch length satisfies  $T_{s} \asymp \sqrt{\frac{L}{\mu}} + \frac{M}{\mu} + \frac{\sigma^{2}}{\mu^{2}\Gamma_{0}^{2}e^{1 - s}}$ . Thus, we can obtain the total iteration complexity as
$$
\sum_ {s = 1} ^ {S} \left[ \sqrt {\frac {L}{\mu}} + \frac {M}{\mu} + \frac {\sigma^ {2}}{\mu^ {2} \Gamma_ {0} ^ {2} e ^ {1 - s}} \right] = \left(\sqrt {\frac {L}{\mu}} + \frac {M}{\mu}\right) S + \frac {\sigma^ {2}}{\mu^ {2} \Gamma_ {0} ^ {2}} \cdot \frac {e ^ {S} - 1}{e - 1},
$$
where  $S \equiv \left\lceil \log \frac{\Gamma_0^2}{\varepsilon^2} \right\rceil$ . This yields the following multi-epoch iteration complexity bound:
Corollary 2.8 (Iteration complexity of stochastic AG-EG with scheduled restarting) Under the same condition of Theorem [2.5] the stochastic AG-EG with scheduled restarting in Algorithm [1] with epoch length  $T_{s} \asymp \sqrt{\frac{L}{\mu}} + \frac{M}{\mu} + \frac{\sigma^{2}}{\mu^{2}\Gamma_{0}^{2}e^{1 - s}}$  has a total iteration complexity of
$$
\mathcal {O} \left(\left(\sqrt {\frac {L}{\mu}} + \frac {M}{\mu}\right) \log \left(\frac {1}{\varepsilon}\right) + \frac {\sigma^ {2}}{\mu^ {2} \varepsilon^ {2}}\right). \tag {17}
$$
Note that the hard instance constructed by Zhang et al. [2022] can be modified in a straightforward way to establish a lower bound of  $\Omega\left(\left(\sqrt{\frac{L}{\mu}} + \frac{M}{\mu}\right) \log \left(\frac{1}{\varepsilon}\right)\right)$  for our monotone VI [1], demonstrating the optimality of Corollary 2.8 in the deterministic separable setting. An alternative optimality argument proceeds as follows: the first term  $\sqrt{\frac{L}{\mu}}$  matches the lower bound for the minimization of a strongly convex function  $\mathcal{F}$  [Nesterov, 2004], and the second term  $\frac{M}{\mu}$  matches the lower bound for VI for non-strongly monotone operator when  $\nabla \mathcal{F} = 0$  [Ouyang and Xu, 2021]. This together gives a lower bound for solving monotone VI [1] via a similar argument by Thekumparampil et al. [2022].
It is worth noting that while both complexity bounds in Corollary 2.8 and Theorem 2.3 match the lower bound in Zhang et al. [2022] for strongly monotone VIs with separable structure, the direct approach in §2.3 reduces to the last-iterate independent-sample stochastic extragradient (SEG) algorithm in bilinear games. Consequently, the deterministic part ( $\sigma = 0$ ) fails to match the lower bound in Ibrahim et al. [2020]. In the stochastic case with noise variance bounded away from zero, the direct approach in §2.3 can exhibit nonconvergence behavior [Hsieh et al., 2020, §3]. The AG-EG algorithm in §2.4 resolves this issue by restarting the average-iterate SEG, matching the lower bound results (see §3.2 for more details). In addition, the complexity bound in (17) also eliminates the log prefactor of the statistical error term  $\frac{\sigma^2}{\mu^2\varepsilon^2}$  compared to Theorem 2.3. The optimality of our algorithm lies in not only the optimization complexity but also the statistical error rate  $\frac{\sigma^2}{\mu^2\varepsilon^2}$ . Here the  $\varepsilon$ -optimal point  $z$  is defined as  $\| z - z^{*} \| \leq \varepsilon$ .
2.5 Extension of AG-EG to Proximal Algorithms
In previous subsections, we have focused on the case where the feasible set  $\mathcal{Z}$  represents the entire space and the nondifferentiable convex function  $J$  is dropped. We now extend the AG-EG algorithm and its analysis to the more general setting that has a bounded feasible set (via Euclidean projection onto the feasible set) as well as a nondifferentiable convex regularization term (via a proximal operator). These settings are useful in various applications, such as the variational inequality on the Lorentz cone where projection onto  $\mathcal{Z} = \{(x,t)\in \mathbb{R}^{(n + 1)}:\| x\| \leq t\}$  is required [Chen et al. 2017], and the two-player game that involves projection onto the probability simplex, among others. To deal with bounded feasible set  $\mathcal{Z}$ , we adopt a variant of the EG algorithm, where we project the extrapolated point and the main iterates back onto the feasible set  $\mathcal{Z}$  of  $\mathcal{W}$ :
$$
\begin{array}{l} \boldsymbol {z} _ {t - \frac {1}{2}} = P _ {\mathcal {Z}} \left[ \boldsymbol {z} _ {t - 1} - \eta \mathcal {W} (\boldsymbol {z} _ {t - 1}) \right] = \arg \min _ {\boldsymbol {z} \in \mathcal {Z}} \left\langle \boldsymbol {z} - \boldsymbol {z} _ {t - 1}, \eta \mathcal {W} (\boldsymbol {z} _ {t - 1}) \right\rangle + \frac {1}{2} \left\| \boldsymbol {z} - \boldsymbol {z} _ {t - 1} \right\| ^ {2}, \\ \boldsymbol {z} _ {t} = P _ {\mathcal {Z}} \left[ \boldsymbol {z} _ {t - 1} - \eta \mathcal {W} \left(\boldsymbol {z} _ {t - \frac {1}{2}}\right) \right] = \arg \min  _ {\boldsymbol {z} \in \mathcal {Z}} \left\langle \boldsymbol {z} - \boldsymbol {z} _ {t - 1}, \eta \mathcal {W} \left(\boldsymbol {z} _ {t - \frac {1}{2}}\right) \right\rangle + \frac {1}{2} \| \boldsymbol {z} - \boldsymbol {z} _ {t - 1} \| ^ {2}, \tag {18} \\ \end{array}
$$
where  $P_{\mathcal{Z}}(\boldsymbol {z}) = \arg \min_{\boldsymbol{z}^{\prime}\in \mathcal{Z}}\| \boldsymbol {z} - \boldsymbol{z}^{\prime}\|^{2}$  is the Euclidean projection operator. To handle the nondifferentiable simple convex function  $J$  , we replace the projection operator in (18) by the following proximal mapping defined via a Bregman divergence  $\mathcal{B}(\cdot ,\cdot)$  ..
$$
\operatorname {p r o x} _ {\boldsymbol {z}} ^ {J} (\mathbf {v}) \equiv \underset {\mathbf {u} \in \mathcal {Z}} {\operatorname {a r g m i n}} \langle \mathbf {v}, \mathbf {u} - \boldsymbol {z} \rangle + \mathcal {B} (\boldsymbol {z}, \mathbf {u}) + J (\mathbf {u}). \tag {19}
$$
In fact, (18) can be seen as a special case of (19) when choosing the Bregman divergence  $\mathcal{B}(z,\mathbf{u}) = \frac{1}{2}\| z - \mathbf{u}\|^2$  and  $J(\mathbf{u})$  as the set indicator function of the feasible set  $\mathcal{Z}$ . Therefore, by substituting the prox-mapping (19) into the AG-EG updates introduced in §2.4, we obtain the more general proximal AG-EG algorithm in Algorithm 3 (See in §C.2), which reduces to Algorithm 1 when  $J = 0$ ,  $\mathcal{B}(z,\mathbf{u}) = \frac{1}{2}\| z - \mathbf{u}\|^2$  and  $\mathcal{Z} = \mathbb{R}^n$ . Moreover, we assume that  $\mathcal{B}(\cdot ,\cdot)$  is  $\mu_{\mathcal{B}}$ -strongly convex. Without loss of generality, in contrast to the previous assumption of  $\mu$ -strong convexity for  $\mathcal{F}$ , we instead assume that  $\mathcal{F}$  is  $\mu$ -strongly convex with respect to the Bregman divergence  $\mathcal{B}(\cdot ,\cdot)$  (See, for example, Hazan and Kale [2014], Xu et al. [2018]). Similar to Corollary 2.8 we have the following iteration complexity result, whose proof is deferred to §D.5:
Corollary 2.9 (Iteration complexity of stochastic proximal AG-EG with scheduled restarting) Under the same condition of Theorem [2.5] the stochastic proximal AG-EG with scheduled restarting in Algorithm [3] with epoch length  $T_{s} \asymp \sqrt{\frac{L}{\mu\mu_{\mathcal{B}}}} + \frac{M}{\mu\mu_{\mathcal{B}}} + \frac{\sigma^{2}\mathcal{B}(z_{0},z^{*})}{\mu^{2}\mu_{\mathcal{B}}\Gamma_{0}^{2}e^{1 - s}}$ , has a total iteration complexity of
$$
\mathcal {O} \left(\left(\sqrt {\frac {L}{\mu \mu_ {\mathcal {B}}}} + \frac {M}{\mu \mu_ {\mathcal {B}}}\right) \log \left(\frac {1}{\varepsilon}\right) + \frac {\sigma^ {2} \mathcal {B} \left(z _ {0} , z ^ {*}\right)}{\mu^ {2} \mu_ {\mathcal {B}} \varepsilon^ {2}}\right).
$$
For the deterministic case, proximal AG-EG with scheduled restarting has a total iteration complexity of  $\mathcal{O}\left(\left(\sqrt{\frac{L}{\mu\mu_{\mathcal{B}}}} + \frac{M}{\mu\mu_{\mathcal{B}}}\right)\log \left(\frac{1}{\varepsilon}\right)\right)$  to output an  $\varepsilon$ -optimal point of (I).
4The optimal statistical error rate  $\frac{\sigma^2}{\mu^2T}$  has been achieved by a multistage algorithm in [Fallah et al. 2020], where the  $\varepsilon$  -optimal point is defined by  $\| z - z^{*}\|^{2}\leq \varepsilon$  . In our paper, the  $\varepsilon$  -optimal point is defined by  $\left\| z - z^{*}\right\| \leq \varepsilon$  . Therefore, our statistical error rate can be translated into  $\frac{\sigma^2}{\mu^2T}$  using their definition, which matches their result.
3 Implications for Specific Instances
In this section, we discuss the implications of our AG-EG algorithm and its convergence rates when applying to two instances of saddle-point problems.
3.1 Strongly-Convex-Strongly-Concave Saddle-Point Problem
For the stochastic bilinearly-coupled SC-SC saddle-point problem (3), we note that the smoothness and strong convexity parameters  $L_{F}, L_{G}, \mu_{F}$ , and  $\mu_{G}$  of  $F$  and  $G$  may differ. To accommodate these variations in curvature information, we employ a scaling reduction technique. This technique enables us to convert the SC-SC with equal strong convexity parameters for  $F$  and  $G$  by reparametrizing the objective function. The same argument is also applicable to the direct approach.
In lieu of (3), we consider
$$
\min _ {\hat {\boldsymbol {x}}} \max _ {\hat {\boldsymbol {y}}} \hat {\mathcal {F}} (\hat {\boldsymbol {x}}, \hat {\boldsymbol {y}}) = F (\hat {\boldsymbol {x}}) + \hat {H} (\hat {\boldsymbol {x}}, \hat {\boldsymbol {y}}) - \hat {G} (\hat {\boldsymbol {y}}),
$$
where  $\hat{\mathcal{F}} (\hat{x},\hat{y}) = \mathcal{F}(\pmb {x},\pmb {y})$  with the symbolic reparametrization  $\hat{\pmb{x}} = \pmb {x}$ $\hat{\pmb{y}} = \sqrt{\frac{\mu_G}{\mu_F}}\pmb {y}$ $\hat{H} (\hat{x},\hat{y}) = H(\pmb {x},\pmb {y}),\hat{G} (\hat{\pmb{y}}) = G(\pmb {y})$  and also their derivatives  $\nabla_{\hat{\pmb{y}}}\hat{H} (\hat{\pmb{x}},\hat{\pmb{y}}) = \sqrt{\frac{\mu_F}{\mu_G}}\nabla_{\pmb{y}}H(\pmb {x},\pmb {y}),\nabla \hat{G} (\hat{\pmb{y}}) = \sqrt{\frac{\mu_F}{\mu_G}}\nabla G(\pmb {y})$  (the stochastic oracles  $\hat{h},\hat{g}$  follow the same rule). It is straightforward to verify that  $\hat{\mathcal{F}} (\hat{x},\hat{y})$  is  $\mu$  -strongly-convex-  $\mu$  -strongly-concave. The essence of our update rules can be summarized by the rescaled updates on  $\pmb{y}$ :
$$
\begin{array}{l} \hat {\boldsymbol {y}} _ {t} = \hat {\boldsymbol {y}} _ {t - 1} - \eta_ {t} \left(- \nabla_ {\hat {\boldsymbol {y}}} h \left(\hat {\boldsymbol {x}} _ {t - \frac {1}{2}}, \hat {\boldsymbol {y}} _ {t - \frac {1}{2}}; \zeta_ {t}\right) + \nabla g \left(\hat {\boldsymbol {y}} _ {t - 1} ^ {\mathrm {m d}}; \xi_ {t - \frac {1}{2}}\right)\right) \\ \Leftrightarrow \boldsymbol {y} _ {t} = \boldsymbol {y} _ {t - 1} - \eta_ {t} \cdot \frac {\mu_ {F}}{\mu_ {G}} \left(- \nabla_ {\boldsymbol {y}} h \left(\boldsymbol {x} _ {t - \frac {1}{2}}, \boldsymbol {y} _ {t - \frac {1}{2}}; \zeta_ {t}\right) + \nabla g \left(\hat {\boldsymbol {y}} _ {t - 1} ^ {\mathrm {m d}}; \xi_ {t - \frac {1}{2}}\right)\right). \\ \end{array}
$$
Therefore, it suffices to analyze Algorithm 3 for  $\hat{\mathcal{F}}(\hat{x}, \hat{y})$  and due to this scaling reduction, we only need to prove all results for the case of  $\mu_F = \mu_G = \mu$ . It is also straightforward to justify corresponding scaling changes as:  $L = L_F \vee \frac{\mu_F}{\mu_G} L_G$ ,  $M = \sqrt{\frac{\mu_F}{\mu_G} \lambda_{\max}(\mathbf{B}^\top \mathbf{B})}$ , and  $\mu = \mu_F$ . The following corollary is recovered by reverting the scaling reduction from  $\hat{\mathcal{F}}(\hat{x}, \hat{y})$  to  $\mathcal{F}(\boldsymbol{x}, \boldsymbol{y})$ .
Corollary 3.1 (Iteration complexity of stochastic AG-EG on SC-SC saddle-point problem)
For solving (3), Algorithm 1 with an epoch length  $T_{s} \asymp \sqrt{\frac{L_{F}}{\mu_{F}} \vee \frac{L_{G}}{\mu_{G}}} + \sqrt{\frac{\lambda_{\max}(\mathbf{B}^{\top}\mathbf{B})}{\mu_{F}\mu_{G}}} + \frac{\sigma^{2}}{\mu_{F}^{2}\Gamma_{0}^{2}e^{1 - s}}$  has a total iteration complexity of
$$
\mathcal {O} \left(\left(\sqrt {\frac {L _ {F}}{\mu_ {F}} \vee \frac {L _ {G}}{\mu_ {G}}} + \sqrt {\frac {\lambda_ {\max} (\mathbf {B} ^ {\top} \mathbf {B})}{\mu_ {F} \mu_ {G}}}\right) \log \left(\frac {1}{\varepsilon}\right) + \frac {\sigma^ {2}}{\mu_ {F} ^ {2} \varepsilon^ {2}}\right).
$$
In the deterministic case, the iteration complexity in Theorem 2.8 matches the lower bound established by Zhang et al. [2022], i.e.,  $\Omega\left(\left(\sqrt{\frac{L_F}{\mu_F} \vee \frac{L_G}{\mu_G}} + \sqrt{\frac{\lambda_{\max}(\mathbf{B}^\top \mathbf{B})}{\mu_F \mu_G}}\right) \log \left(\frac{1}{\varepsilon}\right)\right)$ . Moreover, our algorithm achieves the optimal statistical rate of  $\frac{\sigma^2}{\mu_F^2 \varepsilon^2}$  up to a constant prefactor.
Remark 3.2 A well-known finding regarding the second scheme of Nesterov acceleration is its connection to the primal-dual method [Lan and Zhou 2018 Lin et al., 2020c]. This finding has been incorporated into the design of the LPD algorithm [Thekumparamil et al. 2022], where a Chambolle-Pock-style primal-dual method is utilized as an approximation of proximal point methods, instead of the extragradient used in this paper. The LPD algorithm [Thekumparamil et al. 2022] also achieves the optimal complexity for the deterministic bilinearly-coupled saddle-point problem.
3.2 Bilinear Games
In this subsection, we consider the particular case of bilinear games. We assume  $n = m$  such that  $\mathbf{B}$  is a nonsingular square matrix,  $\nabla f(\boldsymbol{x};\xi) = \mathbf{0}$  and  $\nabla g(\boldsymbol{y};\xi) = \mathbf{0}$  a.s., so (3) reduces to
$$
\min  _ {\boldsymbol {x}} \max  _ {\boldsymbol {y}} \mathcal {F} (\boldsymbol {x}, \boldsymbol {y}) = \mathbb {E} _ {\zeta} [ h (\boldsymbol {x}, \boldsymbol {y}; \zeta) ] = H (\boldsymbol {x}, \boldsymbol {y}) = \boldsymbol {x} ^ {\top} \mathbf {B} \boldsymbol {y} - \boldsymbol {x} ^ {\top} \mathbf {u} _ {\boldsymbol {x}} + \mathbf {u} _ {\boldsymbol {y}} ^ {\top} \boldsymbol {y}, \tag {20}
$$
and Algorithm 3 reduces to the independent-sample extragradient descent-ascent algorithm for (20). The saddle point  $[z^{*};\omega_{y}^{\star}]$  in this case is the unique solution to the linear equation
$$
\left[ \begin{array}{c c} \mathbf {0} & \mathbf {B} \\ - \mathbf {B} ^ {\top} & \mathbf {0} \end{array} \right] \left[ \begin{array}{c} z ^ {*} \\ \omega_ {y} ^ {\star} \end{array} \right] = \left[ \begin{array}{c} \mathbf {u} _ {x} \\ \mathbf {u} _ {y} \end{array} \right], \quad \text {w h i c h h a s a c l o s e d - f o r m s o l u t i o n} \left[ \begin{array}{c} z ^ {*} \\ \omega_ {y} ^ {\star} \end{array} \right] = \left[ \begin{array}{c} - (\mathbf {B} ^ {\top}) ^ {- 1} \mathbf {u} _ {y} \\ \mathbf {B} ^ {- 1} \mathbf {u} _ {x} \end{array} \right].
$$
Our results imply the following iteration complexity for solving stochastic bilinear games.
Corollary 3.3 (Iteration complexity of stochastic AG-EG, bilinear games) For solving (20), choose the step sizes  $\alpha_{t} = \frac{2}{t + 1}$  and  $\eta_t\equiv \frac{1}{\sqrt{\lambda_{\mathrm{max}}(\mathbf{B}^\top\mathbf{B})}}$  in which case Algorithm I with an epoch length  $T_{s}\asymp \sqrt{\frac{\lambda_{\mathrm{max}}(\mathbf{B}^{\top}\mathbf{B})}{\lambda_{\mathrm{min}}(\mathbf{BB}^{\top})}}$  has the total iteration complexity of
$$
\mathcal {O} \left(\sqrt {\frac {\lambda_ {\operatorname* {m a x}} (\mathbf {B} ^ {\top} \mathbf {B})}{\lambda_ {\operatorname* {m i n}} (\mathbf {B B} ^ {\top})}} \log \left(\frac {\sqrt [ 4 ]{\lambda_ {\operatorname* {m i n}} (\mathbf {B B} ^ {\top}) \lambda_ {\operatorname* {m a x}} (\mathbf {B} ^ {\top} \mathbf {B})}}{\sigma_ {\mathrm {B i l}}}\right) + \frac {\sigma_ {\mathrm {B i l}} ^ {2}}{\lambda_ {\operatorname* {m i n}} (\mathbf {B B} ^ {\top}) \varepsilon^ {2}}\right). \tag {21}
$$