SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \| F _ {v} (x) - F _ {v} \left(x ^ {*}\right) - \left(F (x) - F \left(x ^ {*}\right)\right) \| ^ {2} = \mathbb {E} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \left(F _ {i} (x) - F _ {i} \left(x ^ {*}\right)\right) - \left(F _ {i} (x) - F _ {i} \left(x ^ {*}\right)\right) \right\| ^ {2} \\ = \frac {1}{n ^ {2}} \mathbb {E} \left\| \sum_ {i = 1} ^ {n} \left(v _ {i} - 1\right) \left(F _ {i} (x) - F _ {i} \left(x ^ {*}\right)\right) \right\| ^ {2} \\ = \frac {1}{n ^ {2}} \mathbb {E} \left\| \mathbf {R} (v - \mathbf {1}) \right\| ^ {2} \\ = \frac {1}{n ^ {2}} \mathbb {E} (v - \mathbf {1}) ^ {\intercal} \mathbf {R} ^ {\intercal} \mathbf {R} (v - \mathbf {1}) \\ = \frac {1}{n ^ {2}} \mathbb {E} \left(\operatorname {t r} \left(\mathbf {R} ^ {\mathsf {T}} \mathbf {R} (v - \mathbf {1}) (v - \mathbf {1}) ^ {\mathsf {T}}\right)\right) \\ = \frac {1}{n ^ {2}} \operatorname {t r} \left(\mathbf {R} ^ {\intercal} \mathbf {R} \mathbb {E} \left((v - \mathbf {1}) (v - \mathbf {1}) ^ {\intercal}\right)\right) \\ = \frac {1}{n ^ {2}} \operatorname {t r} \left(\mathbf {R} ^ {\top} \mathbf {R} \mathbf {V} \mathbf {a r} [ v ]\right) \\ \stackrel {(2 6)} {\leq} \frac {\lambda_ {\max } (\mathbf {V a r} [ v ])}{n ^ {2}} \operatorname {t r} (\mathbf {R} ^ {\intercal} \mathbf {R}) \\ = \frac {\lambda_ {\max } (\mathbf {V a r} [ v ])}{n ^ {2}} \sum_ {i = 1} ^ {n} \| F _ {i} (x) - F _ {i} \left(x ^ {*}\right) \| ^ {2} \\ \stackrel {(1 9)} {\leq} \frac {\lambda_ {\max } (\mathbf {V a r} [ v ]) \| x - x ^ {*} \| ^ {2}}{n ^ {2}} \sum_ {i = 1} ^ {n} L _ {i} ^ {2}. \\ \end{array}
$$
From the proof details of Lemma F.3 in [63] we have  $\lambda_{\max}(\mathbf{Var}[v]) = \frac{n(n - \tau)}{\tau(n - 1)}$  for  $\tau$ -minibatch sampling. Thus we obtain
$$
\mathbb {E} \left\| F _ {v} (x) - F _ {v} \left(x ^ {*}\right) - \left(F (x) - F \left(x ^ {*}\right)\right) \right\| ^ {2} \leq \frac {2 (n - \tau)}{n \tau (n - 1)} \sum_ {i = 1} ^ {n} L _ {i} ^ {2} \| x - x ^ {*} \| ^ {2}.
$$
Now we focus on the derivation of  $\sigma_{*}^{2} = \mathbb{E}\| F_{v}(x^{*})\|^{2}$  for  $\tau$ -minibatch sampling. We expand  $\mathbb{E}\| F_v(x^*)\|^2$  as follows:
$$
\begin{array}{l} \mathbb {E} \| F _ {v} (x ^ {*}) \| ^ {2} = \frac {1}{n ^ {2}} \mathbb {E} \left\| \sum_ {i = 1} ^ {n} v _ {i} F _ {i} (x ^ {*}) \right\| ^ {2} \\ { = } { \frac { 1 } { n ^ { 2 } } \mathbb { E } \left\| \sum _ { i \in S } \frac { 1 } { p _ { i } } F _ { i } ( x ^ { * } ) \right\| ^ { 2 } } \\ = \frac {1}{n ^ {2}} \mathbb {E} \left\| \sum_ {i = 1} ^ {n} \mathbf {1} _ {i \in S} \frac {1}{p _ {i}} F _ {i} (x ^ {*}) \right\| ^ {2} \\ = \frac {1}{n ^ {2}} \mathbb {E} \left\langle \sum_ {i = 1} ^ {n} \mathbf {1} _ {i \in S} \frac {1}{p _ {i}} F _ {i} \left(x ^ {*}\right), \sum_ {j = 1} ^ {n} \mathbf {1} _ {j \in S} \frac {1}{p _ {j}} F _ {j} \left(x ^ {*}\right) \right\rangle \\ = \frac {1}{n ^ {2}} \sum_ {i, j = 1} ^ {n} \frac {P _ {i j}}{p _ {i} p _ {j}} \left\langle F _ {i} \left(x ^ {*}\right), F _ {j} \left(x ^ {*}\right) \right\rangle , \tag {30} \\ \end{array}
$$
where  $P_{ij} = P(i,j \in S)$  and  $p_i = P(i \in S)$ . For  $\tau$ -minibatch sampling, we obtain  $P_{ij} = \frac{\tau(\tau - 1)}{n(n - 1)}$  and  $p_i = \frac{\tau}{n}$ . Plugging in these values of  $P_{ij}$  and  $p_i$  in (30) we get the closed-form expression of  $\sigma_*^2$ . This completes the proof of Proposition 3.3.
D.3 Proof of Proposition 3.4
Here we enlist the assumptions made on operators. Suppose  $g$  is an estimator of operator  $F$ .
1. Bounded Operator:  $\mathbb{E}\| g(x)\| ^2\leq \sigma^2$
2. Bounded Variance:  $\mathbb{E}\| g(x) - F(x)\| ^2\leq \sigma^2$
3. Growth Condition:  $\mathbb{E}\| g(x)\| ^2\leq \alpha \| F(x)\| ^2 +\beta$
4. Expected Co-coercivity:  $\mathbb{E}\| g(x) - g(x^{*})\|^{2}\leq l_{F}\langle F(x),x - x^{*}\rangle$
5. Expected Residual:  $\mathbb{E}\| (g(x) - g(x^{*})) - (F(x) - F(x^{*}))\| ^2\leq \frac{\delta}{2}\| x - x^{*}\| ^2$
6. Bound from Lemma 3.2:  $\mathbb{E}\| g(x)\| ^2\leq \delta \| x - x^*\| ^2 +\| F(x)\| ^2 +2\sigma_*^2$
7.  $F_{i}$  are Lipschitz:  $\| F_i(x) - F_i(y)\| \leq L_i\| x - y\|\quad \forall i = 1,\ldots ,n$
Proof. Here we will prove Proposition 3.4
-  $1 \Rightarrow 2$ . Note that  $\mathbb{E}\| g(x)\|^2 \leq \sigma^2 \leq \| F(x)\|^2 + \sigma^2 \Rightarrow \mathbb{E}\| g(x) - F(x)\| \leq \sigma^2$ .
-  $2 \Rightarrow 3$ . Here  $\mathbb{E}\|g(x) - F(x)\|^2 \leq \sigma^2 \Rightarrow \mathbb{E}\|g(x)\|^2 \leq \|F(x)\|^2 + \sigma^2$  as  $g$  is an unbiased for estimator of  $F$ . Then take  $\alpha = 1$  and  $\beta = \sigma^2$ .
-  $3 \Rightarrow 6$ . Note that  $\mathbb{E}\| g(x)\|^2 \leq \alpha \| F(x)\|^2 + \beta \leq \alpha L^2\| x - x^*\|^2 + \beta$ . The last inequality follows from lipschitz property of  $F$  and  $F(x^*) = 0$ . Then choose  $\delta = \alpha L^2$  and  $\sigma_*^2 = \beta / 2$  to get the result.
-  $4 \Rightarrow 5$ . Note that expected cocoercivity and  $L$ -Lipschitzness of  $F$  imply  $\mathbb{E} \| (g(x) - g(x^{*})) - (F(x) - F(x^{*})) \|^{2} = \mathbb{E} \| g(x) - g(x^{*}) \|^{2} - \| F(x) - F(x^{*}) \|^{2} \leq \mathbb{E} \| g(x) - g(x^{*}) \|^{2} \leq l_{F} \langle F(x), x - x^{*} \rangle \overset{(B.2)}{\leq} \frac{l_{F}}{2L} \| F(x) \|^{2} + \frac{l_{F} L}{2} \| x - x^{*} \|^{2} \leq l_{F} L \| x - x^{*} \|^{2}$ .
-  $7 \Rightarrow 5$ . This follows from Proposition D.1.
-  $5 \Rightarrow 6$ . This follows from Lemma 3.2
E Main Convergence Analysis Results
First, we present some results followed by iterates of SPEG. These will play a key role in proving the Theorems later in this section. Recall that iterates of SPEG are
$$
\hat {x} _ {k} = x _ {k} - \gamma_ {k} F _ {v _ {k - 1}} (\hat {x} _ {k - 1}),
$$
$$
x _ {k + 1} = x _ {k} - \omega_ {k} F _ {v _ {k}} (\hat {x} _ {k}).
$$
Lemma E.1. For SPEG iterates with step-size  $\omega_{k} = \gamma_{k} = \omega$ , we have
$$
\left\| x _ {k + 1} - x ^ {*} \right\| ^ {2} = \left\| x _ {k + 1} - \hat {x} _ {k} \right\| ^ {2} + \left\| x _ {k} - x ^ {*} \right\| ^ {2} - \left\| \hat {x} _ {k} - x _ {k} \right\| ^ {2} - 2 \omega \left\langle F _ {v _ {k}} (\hat {x} _ {k}), \hat {x} _ {k} - x ^ {*} \right\rangle . \tag {31}
$$
$$
\begin{array}{l} \left\| x _ {k + 1} - x ^ {*} \right\| ^ {2} = \left\| x _ {k + 1} - \hat {x} _ {k} + \hat {x} _ {k} - x _ {k} + x _ {k} - x ^ {*} \right\| ^ {2} \\ = \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} + \| \hat {x} _ {k} - x _ {k} \| ^ {2} + \| x _ {k} - x ^ {*} \| ^ {2} + 2 \langle \hat {x} _ {k} - x _ {k}, x _ {k} - x ^ {*} \rangle \\ + 2 \left\langle x _ {k + 1} - \hat {x} _ {k}, \hat {x} _ {k} - x _ {k} \right\rangle + 2 \left\langle x _ {k + 1} - \hat {x} _ {k}, x _ {k} - x ^ {*} \right\rangle \\ = \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} + \| \hat {x} _ {k} - x _ {k} \| ^ {2} + \| x _ {k} - x ^ {*} \| ^ {2} + 2 \left\langle x _ {k + 1} - \hat {x} _ {k}, \hat {x} _ {k} - x ^ {*} \right\rangle \\ + 2 \left\langle \hat {x} _ {k} - x _ {k}, x _ {k} - x ^ {*} \right\rangle \\ = \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} + \| \hat {x} _ {k} - x _ {k} \| ^ {2} + \| x _ {k} - x ^ {*} \| ^ {2} + 2 \left\langle x _ {k + 1} - \hat {x} _ {k}, \hat {x} _ {k} - x ^ {*} \right\rangle \\ + 2 \left\langle \hat {x} _ {k} - x _ {k}, x _ {k} - \hat {x} _ {k} + \hat {x} _ {k} - x ^ {*} \right\rangle \\ = \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} + \| \hat {x} _ {k} - x _ {k} \| ^ {2} + \| x _ {k} - x ^ {*} \| ^ {2} + 2 \left\langle x _ {k + 1} - \hat {x} _ {k}, \hat {x} _ {k} - x ^ {*} \right\rangle \\ + 2 \left\langle \hat {x} _ {k} - x _ {k}, \hat {x} _ {k} - x ^ {*} \right\rangle - 2 \| \hat {x} _ {k} - x _ {k} \| ^ {2} \\ = \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} - \| \hat {x} _ {k} - x _ {k} \| ^ {2} + \| x _ {k} - x ^ {*} \| ^ {2} + 2 \left\langle x _ {k + 1} - \hat {x} _ {k}, \hat {x} _ {k} - x ^ {*} \right\rangle \\ + 2 \left\langle \hat {x} _ {k} - x _ {k}, \hat {x} _ {k} - x ^ {*} \right\rangle \\ = \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} - \| \hat {x} _ {k} - x _ {k} \| ^ {2} + \| x _ {k} - x ^ {*} \| ^ {2} + 2 \langle x _ {k + 1} - x _ {k}, \hat {x} _ {k} - x ^ {*} \rangle \\ = \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} - \| \hat {x} _ {k} - x _ {k} \| ^ {2} + \| x _ {k} - x ^ {*} \| ^ {2} - 2 \omega \left\langle F _ {v _ {k}} (\hat {x} _ {k}), \hat {x} _ {k} - x ^ {*} \right\rangle . \\ \end{array}
$$
Lemma E.2. Let  $F$  be  $L$ -Lipschitz, and let ER hold. Then SPEG iterates satisfy
$$
\mathbb {E} _ {\mathcal {D}} \| F _ {v _ {k}} (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \leq \delta \| \hat {x} _ {k} - x ^ {*} \| ^ {2} + 2 \delta \| \hat {x} _ {k - 1} - x ^ {*} \| ^ {2} + 2 L ^ {2} \| \hat {x} _ {k} - \hat {x} _ {k - 1} \| ^ {2} + 6 \sigma_ {*} ^ {2}. \tag {32}
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \| F _ {v _ {k}} (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} = \mathbb {E} _ {\mathcal {D}} \| F _ {v _ {k}} (\hat {x} _ {k}) - F (\hat {x} _ {k}) \| ^ {2} + \mathbb {E} _ {\mathcal {D}} \| F (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \\ + 2 \mathbb {E} _ {\mathcal {D}} \left\langle F _ {v _ {k}} (\hat {x} _ {k}) - F (\hat {x} _ {k}), F (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \right\rangle \\ = \mathbb {E} _ {v _ {k}} \| F _ {v _ {k}} (\hat {x} _ {k}) - F (\hat {x} _ {k}) \| ^ {2} + \mathbb {E} _ {\mathcal {D}} \| F (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(2 2)} {\leq} \mathbb {E} _ {\mathcal {D}} \| F _ {v _ {k}} (\hat {x} _ {k}) - F (\hat {x} _ {k}) \| ^ {2} + 2 \mathbb {E} _ {\mathcal {D}} \| F (\hat {x} _ {k}) - F (\hat {x} _ {k - 1}) \| ^ {2} \\ + 2 \mathbb {E} _ {\mathcal {D}} \| F (\hat {x} _ {k - 1}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \\ = \mathbb {E} _ {\mathcal {D}} \| F _ {v _ {k}} (\hat {x} _ {k}) \| ^ {2} - \| F (\hat {x} _ {k}) \| ^ {2} + 2 \| F (\hat {x} _ {k}) - F (\hat {x} _ {k - 1}) \| ^ {2} \\ + 2 \mathbb {E} _ {\mathcal {D}} \| F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} - 2 \| F (\hat {x} _ {k - 1}) \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(8)} {\leq} \delta \| \hat {x} _ {k} - x ^ {*} \| ^ {2} + 2 \delta \| \hat {x} _ {k - 1} - x ^ {*} \| ^ {2} + 6 \sigma_ {*} ^ {2} \\ + 2 \| F (\hat {x} _ {k}) - F (\hat {x} _ {k - 1}) \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(1 8)} {\leq} \delta \| \hat {x} _ {k} - x ^ {*} \| ^ {2} + 2 \delta \| \hat {x} _ {k - 1} - x ^ {*} \| ^ {2} + 6 \sigma_ {*} ^ {2} \\ + 2 L ^ {2} \| \hat {x} _ {k} - \hat {x} _ {k - 1} \| ^ {2}. \\ \end{array}
$$
Lemma E.3. For  $\omega \in \left[0, \frac{1}{4L}\right]$  the following two conditions hold:
$$
2 \omega (\mu - \omega \delta) + 8 \omega^ {2} L ^ {2} - 1 \leq 0, \tag {33}
$$
$$
\text {a n d} \quad 8 \omega^ {2} (\delta + L ^ {2}) \leq 1 - \omega \mu + 9 \omega^ {2} \delta . \tag {34}
$$
Proof. Note that for  $\omega \in \left[0, \frac{1}{4L}\right]$ , we have
$$
2 \omega (\mu - \omega \delta) + 8 \omega^ {2} L ^ {2} - 1 \stackrel {\omega^ {2} \delta \geq 0} {\leq} 2 \omega \mu + 8 \omega^ {2} L ^ {2} - 1 \stackrel {\omega \leq \frac {1}{4 L}} {\leq} \frac {\mu}{2 L} + \frac {1}{2} - 1 \stackrel {\mu \leq L} {\leq} 0.
$$
This proves the first condition. The second condition is equivalent to  $\omega (\mu -\omega \delta) + 8\omega^2 L^2 -1\leq 0$  which is again true using similar arguments.
Proof. For  $\omega \in \left[0, \frac{\mu}{18\delta}\right]$  we have  $\omega(\mu - 9\omega\delta) \geq 0$  and  $1 - \omega(\mu - 9\omega\delta) \leq 1 - \frac{\omega\mu}{2}$ . Then we derive
Then using (33), (34) we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \left[ \| x _ {k + 1} - x ^ {*} \| ^ {2} + \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} \right] \leq (1 - \omega \mu + 9 \omega^ {2} \delta) \left(\| x _ {k} - x ^ {*} \| ^ {2} + \| x _ {k} - \hat {x} _ {k - 1} \| ^ {2}\right) \\ + 1 2 \omega^ {2} \sigma_ {*} ^ {2}. \\ \end{array}
$$
Then we take total expectation with respect to the algorithm to obtain the following recurrence:
$$
R _ {k + 1} ^ {2} \leq (1 - \omega \mu + 9 \omega^ {2} \delta) R _ {k} ^ {2} + 1 2 \omega^ {2} \sigma_ {*} ^ {2}. \tag {35}
$$
Using the inequality  $1 - \omega (\mu -9\omega \delta)\leq 1 - \frac{\omega\mu}{2}$  , we have
$$
\mathbb {E} \left[ \| x _ {k + 1} - x ^ {*} \| ^ {2} + \| x _ {k + 1} - \hat {x} _ {k} \| ^ {2} \right] \leq \left(1 - \frac {\omega \mu}{2}\right) \mathbb {E} \left[ \| x _ {k} - x ^ {*} \| ^ {2} + \| x _ {k} - \hat {x} _ {k - 1} \| ^ {2} \right] + 1 2 \omega^ {2} \sigma_ {*} ^ {2}. \tag {36}
$$
The theorem follows by unrolling the above recurrence. In order to compute the iteration complexity of SPEG, we consider any arbitrary  $\varepsilon > 0$ . Then we choose the step-size  $\omega$  such that  $\frac{24\omega\sigma_{*}^{2}}{\mu} \leq \frac{\varepsilon}{2}$  i.e.  $\omega \leq \frac{\varepsilon\mu}{48\sigma_{*}^{2}}$ . Next we will choose the number of iterations  $k$  such that  $(1 - \frac{\omega\mu}{2})^{k}R_{0}^{2} \leq \frac{\varepsilon}{2}$ . It is equivalent to choosing  $k$  such that
$$
\log \left(\frac {2 R _ {0} ^ {2}}{\varepsilon}\right) \leq k \log \left(\frac {1}{1 - \frac {\omega \mu}{2}}\right).
$$
Now using the fact  $\log \left(\frac{1}{\rho}\right) \geq 1 - \rho$  for  $0 < \rho \leq 1$ , we get  $\log \left(\frac{2R_0^2}{\varepsilon}\right) \leq \frac{k\omega\mu}{2}$ , or equivalently  $k \geq \frac{2}{\omega\mu} \log \left(\frac{2R_0^2}{\varepsilon}\right)$ . Therefore, with step-size  $\omega = \min \left\{\frac{\mu}{18\delta}, \frac{1}{4L}, \frac{\varepsilon\mu}{48\sigma_*^2}\right\}$  we get the following lower bound on the number of iterations
$$
k \geq \max  \left\{\frac {8 L}{\mu}, \frac {3 6 \delta}{\mu^ {2}}, \frac {9 6 \sigma_ {*} ^ {2}}{\varepsilon \mu^ {2}} \right\} \log \left(\frac {2 R _ {0} ^ {2}}{\varepsilon}\right).
$$
E.2 Proof of Theorem 4.3
Proof. For  $\omega \leq \min \left\{\frac{1}{4L},\frac{\mu}{18\delta}\right\}$ , from Theorem 4.1 we obtain
$$
R _ {k + 1} ^ {2} \leq \left(1 - \frac {\omega \mu}{2}\right) ^ {k + 1} R _ {0} ^ {2} + \frac {2 4 \omega \sigma_ {*} ^ {2}}{\mu}.
$$
Let the step-size  $\omega_{k} = \frac{2k + 1}{(k + 1)^{2}}\frac{2}{\mu}$  and  $k^{*}$  be an integer that satisfies  $\omega_{k^*}\leq \bar{\omega}$ . In particular this holds when  $k^{*}\geq \left[\frac{4}{\mu\bar{\omega}} -1\right]$ . Note that  $\omega_{k}$  is decreasing in  $k$  and consequently  $\omega_{k}\leq \bar{\omega}$  for all  $k\geq k^{*}$ . Therefore, from (36) we derive
$$
R _ {k + 1} ^ {2} \leq \left(1 - \omega_ {k} \frac {\mu}{2}\right) R _ {k} ^ {2} + 1 2 \omega_ {k} ^ {2} \sigma_ {*} ^ {2}
$$
for all  $k \geq k^*$ . Then we replace  $\omega_k$  with  $\frac{2k + 1}{(k + 1)^2} \frac{2}{\mu}$  to obtain
$$
\begin{array}{l} R _ {k + 1} ^ {2} \leq \left(1 - \frac {2 k + 1}{(k + 1) ^ {2}}\right) R _ {k} ^ {2} + 4 8 \sigma_ {*} ^ {2} \frac {(2 k + 1) ^ {2}}{\mu^ {2} (k + 1) ^ {4}} \\ { = } { \frac { k ^ { 2 } } { ( k + 1 ) ^ { 2 } } R _ { k } ^ { 2 } + 4 8 \sigma _ { * } ^ { 2 } \frac { ( 2 k + 1 ) ^ { 2 } } { \mu ^ { 2 } ( k + 1 ) ^ { 4 } } . } \\ \end{array}
$$
Multiplying both sides by  $(k + 1)^2$  we get
$$
\begin{array}{l} (k + 1) ^ {2} R _ {k + 1} ^ {2} \leq k ^ {2} R _ {k} ^ {2} + \frac {4 8 \sigma_ {*} ^ {2}}{\mu^ {2}} \left(\frac {2 k + 1}{k + 1}\right) ^ {2} \\ \leq k ^ {2} R _ {k} ^ {2} + \frac {1 9 2 \sigma_ {*} ^ {2}}{\mu^ {2}}, \\ \end{array}
$$
where in the last line follows from  $\frac{2k + 1}{k + 1} < 2$ . Rearranging and summing the last expression for  $t = k^{*}, \dots, k$  we obtain
$$
\sum_ {t = k ^ {*}} ^ {k} (t + 1) ^ {2} R _ {t + 1} ^ {2} - t ^ {2} R _ {t} ^ {2} \leq \frac {1 9 2 \sigma_ {*} ^ {2}}{\mu^ {2}} (k - k ^ {*}).
$$
Using telescopic sum and dividing both sides by  $(k + 1)^2$  we obtain
$$
R _ {k + 1} ^ {2} \leq \left(\frac {k ^ {*}}{k + 1}\right) ^ {2} R _ {k *} ^ {2} + \frac {1 9 2 \sigma_ {*} ^ {2} (k - k ^ {*})}{\mu^ {2} (k + 1) ^ {2}}. \tag {37}
$$
Suppose for  $k \leq k^*$ , we have  $\omega_k = \bar{\omega} = \min \left\{ \frac{1}{4L}, \frac{\mu}{18\delta} \right\}$  i.e. constant step-size. Then from (10), we obtain  $R_{k^*}^2 \leq \left( 1 - \frac{\mu\bar{\omega}}{2} \right)^{k^*} R_0^2 + \frac{24\bar{\omega}\sigma_*^2}{\mu}$ . This bound on  $R_{k^*}^2$ , combined with (37) yields
$$
R _ {k + 1} ^ {2} \leq \left(\frac {k ^ {*}}{k + 1}\right) ^ {2} \left(1 - \frac {\mu \bar {\omega}}{2}\right) ^ {k ^ {*}} R _ {0} ^ {2} + \left(\frac {k ^ {*}}{k + 1}\right) ^ {2} \frac {2 4 \bar {\omega} \sigma_ {*} ^ {2}}{\mu} + \frac {1 9 2 \sigma_ {*} ^ {2} (k - k ^ {*})}{\mu^ {2} (k + 1) ^ {2}}.
$$
Now we want to choose  $k^*$  which minimizes the expression  $\left(\frac{k^*}{k + 1}\right)^2\frac{24\bar{\omega}\sigma_*^2}{\mu} +\frac{192\sigma_*^2(k - k^*)}{\mu^2(k + 1)^2}$ . Note that, it is minimized at  $\frac{4}{\mu\bar{\omega}}$ , hence we choose  $k^* = \left\lceil \frac{4}{\mu\bar{\omega}}\right\rceil$ . Therefore, using this value of  $k^*$ , we obtain
$$
\begin{array}{l} R _ {k + 1} ^ {2} \leq \left(\frac {k ^ {*}}{k + 1}\right) ^ {2} \left(1 - \frac {2}{k ^ {*}}\right) ^ {k ^ {*}} R _ {0} ^ {2} + \frac {2 4 \sigma_ {*} ^ {2}}{\mu^ {2} (k + 1) ^ {2}} (8 k - 4 k ^ {*}) \\ \leq \left(\frac {k ^ {*}}{k + 1}\right) ^ {2} \left(1 - \frac {2}{k ^ {*}}\right) ^ {k ^ {*}} R _ {0} ^ {2} + \frac {1 9 2 k \sigma_ {*} ^ {2}}{\mu^ {2} (k + 1) ^ {2}} \\ \leq \left(\frac {k ^ {*}}{k + 1}\right) ^ {2} \frac {R _ {0} ^ {2}}{e ^ {2}} + \frac {1 9 2 \sigma_ {*} ^ {2}}{\mu^ {2} (k + 1)}. \\ \end{array}
$$
The last line follows from  $\left(1 - \frac{1}{x}\right)^x \leq e^{-1}$  for all  $x \geq 1$ . This completes the proof.
E.3 Proof of Theorem 4.4
Proof. For  $0 < \omega_{k} \leq \left\{\frac{1}{4L}, \frac{\mu}{18\delta}\right\}$  we obtain the following bound from Theorem 4.1:
$$
R _ {k} ^ {2} \leq \left(1 - \frac {\mu \omega_ {k}}{2}\right) R _ {k - 1} ^ {2} + 1 2 \omega_ {k} ^ {2} \sigma_ {*} ^ {2}.
$$
Then using Lemma B.4 with  $a = \frac{\mu}{2}, h = \frac{1}{\bar{\omega}}$  and  $c = 12\sigma_{*}^{2}$  we complete the proof of this Theorem.
E.4 Proof of Theorem E.4
Theorem E.4. Let  $F$  be  $L$ -Lipschitz and satisfy Weak Minty condition with parameter  $\rho < 1/(2L)$ . Assume that inequality (8) holds (e.g., it holds whenever Assumption 3.1 holds, see Lemma 3.2). Assume that  $\gamma_k = \gamma$ ,  $\omega_k = \omega$  and
$$
\max \left\{2 \rho , \frac {1}{2 L} \right\} <   \gamma <   \frac {1}{L}, \quad 0 <   \omega <   \min \left\{\gamma - 2 \rho , \frac {1}{4 L} - \frac {\gamma}{4} \right\}, \quad \delta \leq \frac {(1 - L \gamma) L ^ {3} \omega}{3 2}.
$$
Then, for all  $K \geq 2$  the iterates produced by SPEG satisfy
$$
\begin{array}{l} \min _ {0 \leq k \leq K - 1} \mathbb {E} \left[ \| F (\hat {x} _ {k}) \| ^ {2} \right] \leq \frac {(1 + 8 \omega \gamma (\delta + L ^ {2}) - L \gamma) \left(1 + \frac {4 8 \omega \gamma \delta}{(1 - L \gamma) ^ {2}}\right) ^ {K - 1} \| x _ {0} - x ^ {*} \| ^ {2}}{\omega \gamma (1 - L (\gamma + 4 \omega)) (K - 1)} \\ + \frac {8 \left(8 + \frac {(1 - L \gamma) ^ {2}}{K - 1} \left(1 + \frac {4 8 \omega \gamma \delta}{(1 - L \gamma) ^ {2}}\right) ^ {K - 1}\right) \sigma_ {*} ^ {2}}{(1 - L \gamma) ^ {2} (1 - L (\gamma + 4 \omega))}. \tag {38} \\ \end{array}
$$
Proof. The proof closely follows the proof of Lemma C.3 and Theorem C.4 from [23]. The update rule of SPEG implies for  $k \geq 1$
$$
\begin{array}{l} \left\| x _ {k + 1} - x ^ {*} \right\| ^ {2} = \left\| x _ {k} - x ^ {*} \right\| ^ {2} - 2 \omega \left\langle x _ {k} - x ^ {*}, F _ {v _ {k}} (\hat {x} _ {k}) \right\rangle + \omega^ {2} \left\| F _ {v _ {k}} (\hat {x} _ {k}) \right\| ^ {2} \\ = \| x _ {k} - x ^ {*} \| ^ {2} - 2 \omega \langle \hat {x} _ {k} - x ^ {*}, F _ {v _ {k}} (\hat {x} _ {k}) \rangle - 2 \omega \gamma \langle F _ {v _ {k - 1}} (\hat {x} _ {k - 1}), F _ {v _ {k}} (\hat {x} _ {k}) \rangle \\ + \omega^ {2} \| F _ {v _ {k}} (\hat {x} _ {k}) \| ^ {2} \\ = \| x _ {k} - x ^ {*} \| ^ {2} - 2 \omega \langle \hat {x} _ {k} - x ^ {*}, F _ {v _ {k}} (\hat {x} _ {k}) \rangle - \omega \gamma \| F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \\ - \omega (\gamma - \omega) \| F _ {v _ {k}} (\hat {x} _ {k}) \| ^ {2} + \omega \gamma \| F _ {v _ {k}} (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2}, \\ \end{array}
$$
where in the last step we apply  $2\langle a,b\rangle = \| a\| ^2 +\| b\| ^2 -\| a - b\| ^2$  , which holds for all  $a,b\in \mathbb{R}^d$  Taking the full expectation and using  $\mathbb{E}[\mathbb{E}_{v_k}[\cdot ]] = \mathbb{E}[\cdot ]$  and Weak Minty condition, we derive
$$
\begin{array}{l} \mathbb {E} \left[ \| x _ {k + 1} - x ^ {*} \| ^ {2} \right] \leq \mathbb {E} \left[ \| x _ {k} - x ^ {*} \| ^ {2} \right] - 2 \omega \mathbb {E} \left[ \langle \hat {x} _ {k} - x ^ {*}, F (\hat {x} _ {k}) \rangle \right] - \omega \gamma \mathbb {E} \left[ \| F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \right] \\ - \omega (\gamma - \omega) \mathbb {E} \left[ \| F _ {v _ {k}} (\hat {x} _ {k}) \| ^ {2} \right] + \omega \gamma \mathbb {E} \left[ \| F _ {v _ {k}} (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \right] \\ \end{array}
$$
$$
\stackrel {(4)} {\leq} \mathbb {E} \left[ \| x _ {k} - x ^ {*} \| ^ {2} \right] + 2 \omega \rho \mathbb {E} \left[ \| F (\hat {x} _ {k}) \| ^ {2} \right] - \omega \gamma \mathbb {E} \left[ \| F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \right]
$$
$$
- \omega (\gamma - \omega) \mathbb {E} \left[ \| F _ {v _ {k}} (\hat {x} _ {k}) \| ^ {2} \right] + \omega \gamma \mathbb {E} \left[ \| F _ {v _ {k}} (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \right]
$$
$$
\begin{array}{l} \leq \mathbb {E} \left[ \| x _ {k} - x ^ {*} \| ^ {2} \right] - \omega \gamma \mathbb {E} \left[ \| F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \right] \\ - \omega (\gamma - 2 \rho - \omega) \mathbb {E} \left[ \| F _ {v _ {k}} (\hat {x} _ {k}) \| ^ {2} \right] \\ + \omega \gamma \mathbb {E} \left[ \left\| F _ {v _ {k}} (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \mathbb {E} \left[ \| x _ {k} - x ^ {*} \| ^ {2} \right] - \omega \gamma \mathbb {E} \left[ \| F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \right] \\ + \omega \gamma \mathbb {E} \left[ \| F _ {v _ {k}} (\hat {x} _ {k}) - F _ {v _ {k - 1}} (\hat {x} _ {k - 1}) \| ^ {2} \right], \tag {39} \\ \end{array}
$$
where we apply Jensen's inequality  $\| F(\hat{x}_k)\|^2 = \| \mathbb{E}_{v_k}[F_{v_k}(\hat{x}_k)]\|^2 \leq \mathbb{E}_{v_k}[\| F_{v_k}(\hat{x}_k)\|^2]$  and  $\gamma > 2\rho + \omega$ . For  $k = 0$  we have  $x_1 = x_0 - \omega F_{v_0}(\hat{x}_0) = x_0 - \omega F_{v_0}(x_0)$  and
$$
\begin{array}{l} \mathbb {E} \left[ \| x _ {1} - x ^ {*} \| ^ {2} \right] = \| x _ {0} - x ^ {*} \| ^ {2} - 2 \omega \mathbb {E} \left[ \langle x _ {0} - x ^ {*}, F _ {v _ {0}} (x _ {0}) \rangle \right] + \omega^ {2} \mathbb {E} \left[ \| F _ {v _ {0}} (x _ {0}) \| ^ {2} \right] \\ { = } { \| x _ { 0 } - x ^ { * } \| ^ { 2 } - 2 \omega \langle x _ { 0 } - x ^ { * } , F ( x _ { 0 } ) \rangle + \omega ^ { 2 } \mathbb { E } \left[ \| F _ { v _ { 0 } } ( x _ { 0 } ) \| ^ { 2 } \right] . } \\ \end{array}
$$
Applying Weak Minty condition, we get
$$
\begin{array}{l} \mathbb {E} \left[ \| x _ {1} - x ^ {*} \| ^ {2} \right] = \| x _ {0} - x ^ {*} \| ^ {2} + 2 \omega \rho \| F (x _ {0}) \| ^ {2} + \omega^ {2} \mathbb {E} \left[ \| F _ {v _ {0}} (x _ {0}) \| ^ {2} \right] \\ \leq \left\| x _ {0} - x ^ {*} \right\| ^ {2} + \omega (\omega + 2 \rho) \mathbb {E} \left[ \| F _ {v _ {0}} \left(x _ {0}\right) \| ^ {2} \right]. \tag {40} \\ \end{array}
$$
The next step of our proof is in estimating the last term from (39). Using Young's inequality  $\| a + b \|^2 \leq (1 + \alpha) \| a \|^2 + (1 + \alpha^{-1}) \| b \|^2$ , which holds for any  $a, b \in \mathbb{R}^d$ ,  $\alpha > 0$ , we get for all  $k \geq 2$