SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \| \left(F _ {v} (x) - F _ {v} \left(x ^ {*}\right)\right) - \left(F (x) - F \left(x ^ {*}\right)\right) \| ^ {2} \leq \left(c _ {2} L ^ {2} + \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \frac {1 - p _ {i} c _ {2}}{p _ {i}} L _ {i} ^ {2}\right) \| x - x ^ {*} \| ^ {2}. \tag {48}
$$
For single-element sampling  $c_{2} = 0$  (as probability of two points appearing in same sample is zero for single element sampling i.e.  $P_{ij} = 0$ ). Then we obtain
$$
\delta \leq \frac {2}{n ^ {2}} \sum_ {i = 1} ^ {n} \frac {L _ {i} ^ {2}}{p _ {i}}
$$
from (48). This completes the derivation of  $\delta$  for single element sampling. To compute  $\sigma_{*}^{2}$  for single element sampling, we replace
$$
P _ {i j} = \left\{ \begin{array}{l l} p _ {i} & \text {i f} i = j \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
in (30) to get
$$
\sigma_ {*} ^ {2} = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \frac {1}{p _ {i}} \| F _ {i} (x ^ {*}) \| ^ {2}.
$$
G Numerical Experiments
In Appendix G.1, we add more details on the experiments discussed in the main paper. Furthermore, in Appendix G.2, we run more experiments to evaluate the performance of SPEG on quasi-strongly monotone and weak MVI problems.
G.1 More Details on the Numerical Experiments of Section 6
On Constant vs Switching Stepsize Rule. We run the experiments on two synthetic datasets. In Fig. 1 of the main paper, we take  $\mu_{A} = \mu_{C} = 0.6$ . Here we include one more plot with a similar flavor but in a different setting. For Fig. 5, we generate the data such that eigenvalues of  $A_{1}, B_{1}, C_{1}$  are generated uniformly from the interval [0.1, 10]. In the new plot, similar to the main paper, we can see the benefit of switching the step-size rule of Theorem 4.3.
Figure 5: Comparison of the constant step-size rule (9) with the switching step-sizes (11) on the strongly monotone quadratic game.
On Weak Minty VIPs. In this experiment, we generate  $\xi_{i},\zeta_{i}$  such that  $\frac{1}{n}\sum_{i = 1}^{n}\xi_{i} = \sqrt{63}$  and  $\frac{1}{n}\sum_{i = 1}^{n}\zeta_{i} = -1$ . This choice of  $\xi_{i},\zeta_{i}$  ensures that  $L = 8$  and  $\rho = 1 / 32$  for the min-max problem we considered in Section 6.2. In Fig. 6, we again implement the SPEG on (17) with batchsize  $= 0.15\times n$  (different batchsize compare to the plot of the main paper).
Figure 6: Trajectory of SPEG for solving weak MVI using a batchsize  $= 0.15 \times n$ .
G.2 Additional Experiments
In this subsection, we include more experiments to evaluate the performance of SPEG on quasi-strongly monotone and weak MVI problems. First, we run the experiment comparing constant and switching step-size rules on a different setup than the one we included in the main paper to analyze the performance of SPEG under different condition numbers. Then, we implement SPEG on the weak MVI of (17). To evaluate the performance in this experiment, we plot  $\| F(\hat{x}_k)\|^2 / \| F(x_0)\|^2$  on the  $y$ -axis.
G.2.1 Strongly Monotone Quadratic Game:
In this experiment, we compare the proposed constant step-size (9) and the switching step-size rule (11). We implement our algorithm on operator  $F: \mathbb{R}^4 \to \mathbb{R}^4$  given by
$$
F (x) := \frac {1}{3} \left(M _ {1} (x - x _ {1} ^ {*}) + M _ {2} (x - x _ {2} ^ {*}) + M _ {3} (x - x _ {3} ^ {*})\right),
$$
where  $M_1, M_2$  and  $M_3$  are the diagonal matrices,
$$
M _ {1} = \left( \begin{array}{c c c c} \Delta & & & \\ & 1 & & \\ & & 1 & \\ & & & 1 \end{array} \right), \quad M _ {2} = \left( \begin{array}{c c c c} 1 & \Delta & & \\ & & 1 & \\ & & & 1 \end{array} \right), \quad M _ {3} = \left( \begin{array}{c c c c} 1 & 1 & & \\ & \Delta & & \\ & & 1 \end{array} \right)
$$
$$
x _ {1} ^ {*} = \left( \begin{array}{c} \Delta \\ 0 \\ 0 \\ \Delta \end{array} \right), \quad x _ {2} ^ {*} = \left( \begin{array}{c} 0 \\ \Delta \\ 0 \\ 0 \end{array} \right), \quad x _ {3} ^ {*} = \left( \begin{array}{c} 0 \\ 0 \\ \Delta \\ 0 \end{array} \right).
$$
This choice of  $M_{i}$  and  $x_{i}^{*}$  ensures that the Lipschitz constant of operator  $F$  is  $\frac{\Delta + 2}{3}$  while quasi-strong monotonicity parameter (3) is  $\mu = 1$ . Hence the condition number of  $F$  is given by  $\frac{\Delta + 2}{3}$ . This allows us to vary the condition number of operator  $F$  by changing the value of  $\Delta$ . For Fig. 7a we take  $\Delta = 3$  (condition number  $= 1.67$ ) while for Fig. 7b we choose  $\Delta = 10$  (condition number  $= 10.67$ ). The vertical dotted line in plots of Fig. 7 marks the transition point from constant to switching step-size rule as predicted by our theoretical result in Theorem 4.3.
(a) Condition Number  $\frac{L}{\mu} = 1.67$
(b) Condition Number  $\frac{L}{\mu} = 10.67$
Figure 7: Illustration of switching rule (11) in Theorem 4.3. The dotted line marks the transition from phase 1 (where we use constant step-size) to phase 2 (where we use decreasing step-size).
G.2.2 Weak Minty VIPs Continued
In this experiment, we reevaluate the performance of SPEG on weak MVI example of (17). That is, we generate the data in exactly the same way as the ones in section 6.2 with  $n = 100$ . In Fig. 8a and 8b, we implement SPEG with batchsize 10 and 15, respectively (we note that in this setting the full-gradient evaluation requires a batchsize of 100). For these plots, we use the relative operator norm on the  $y$ -axis, i.e.  $\| F(\hat{x}_k) \|^2 / \| F(x_0) \|^2$ , where  $x_0$  denotes the starting point of SPEG. As expected, the plots illustrate that SPEG performs better as we increase the batchsize. From Fig. 8 it is clear that with batchsize 15 SPEG reaches an accuracy close to  $10^{-10}$  while when we use a batchsize of 10 for the same number of iterations we are only able to converge to an accuracy of  $10^{-4}$ .
(a) Batchsize  $= 0.1\times n$
(b) Batchsize  $= 0.15 \times n$ .
Figure 8: Performance of SPEG for solving weak MVI with different batchesizes. In plot (a) we use a batchsize of 10 while in plot (b) we use 15.