SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} 2 \gamma_ {2} \left\langle F (\widetilde {\boldsymbol {x}} ^ {k}), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \\ + 2 \gamma_ {2} \rho \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| \mathbb {1} _ {k} + 2 \gamma_ {2} ^ {2} \rho^ {2} \mathbb {1} _ {k} + \frac {4 \gamma_ {1} \gamma_ {2} \sigma^ {2}}{\left| \mathcal {G} _ {k} \backslash \mathcal {C} _ {k} \right|} \\ + 4 \gamma_ {1} \gamma_ {2} \left\| F (\boldsymbol {x} ^ {k}) - \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} + 4 \gamma_ {1} \gamma_ {2} \left\| \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \widehat {g} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2}. \\ \end{array}
$$
Taking expectation over  $\xi^k$  one obtains that
$$
\begin{array}{l} 2 \gamma_ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\langle F (\widetilde {\boldsymbol {x}} ^ {k}), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] + 2 \gamma_ {2} \rho \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| \mathbb {1} _ {k} + 2 \gamma_ {2} ^ {2} \rho^ {2} \mathbb {1} _ {k} \\ + \gamma_ {1} \gamma_ {2} \left(\frac {4 \sigma^ {2}}{\left| \mathcal {G} _ {k} \backslash \mathcal {C} _ {k} \right|} + 4 \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| F (\boldsymbol {x} ^ {k}) - \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] + 4 \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \widehat {g} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right]\right) \\ \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] + 2 \gamma_ {2} \rho \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| \mathbb {1} _ {k} + 2 \gamma_ {2} ^ {2} \rho^ {2} \mathbb {1} _ {k} \\ + \gamma_ {1} \gamma_ {2} \left(\frac {4 \sigma^ {2}}{\left| \mathcal {G} _ {k} \setminus \mathcal {C} _ {k} \right|} + \frac {4 \sigma^ {2}}{\left| \mathcal {G} _ {k + \frac {1}{2}} \setminus \mathcal {C} _ {k + \frac {1}{2}} \right|} + 4 \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}}\right) \\ \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] + 2 \gamma_ {2} \rho \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| \mathbb {1} _ {k} + 2 \gamma_ {2} ^ {2} \rho^ {2} \mathbb {1} _ {k} \\ + \gamma_ {1} \gamma_ {2} \left(\frac {4 \sigma^ {2}}{n - 2 B - m} + \frac {4 \sigma^ {2}}{n - 2 B - m} + 4 \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}}\right). \\ \end{array}
$$
Finally taking the full expectation gives that
$$
\begin{array}{l} 2 \gamma_ {2} \mathbb {E} \left[ \left\langle F (\widetilde {\boldsymbol {x}} ^ {k}), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ \leq \mathbb {E} \left[ \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] - \mathbb {E} \left[ \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] + 2 \gamma_ {2} \rho \mathbb {E} \left[ \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| \mathbb {1} _ {k} \right] + 2 \gamma_ {2} ^ {2} \rho^ {2} \mathbb {E} [ \mathbb {1} _ {k} ] \\ + \gamma_ {1} \gamma_ {2} \left(\frac {8 \sigma^ {2}}{n - 2 B - m} + 4 \rho^ {2} \mathbb {E} \left[ \mathbb {1} _ {k - \frac {1}{2}} \right]\right). \\ \end{array}
$$
Summing up the results for  $k = 0,1,\ldots ,K - 1$  we derive
$$
\begin{array}{l} \frac {2 \gamma_ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle F \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ \leq \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left(\mathbb {E} \left[ \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| ^ {2} \right] - \mathbb {E} \left[ \| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \| ^ {2} \right]\right) + \frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2}}{n - 2 B - m} \\ + \frac {2 \gamma_ {2} \rho}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| \mathbb {1} _ {k} \right] + \frac {2 \gamma_ {2} ^ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \mathbb {1} _ {k} ] + \frac {4 \gamma_ {1} \gamma_ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \mathbb {1} _ {k - \frac {1}{2}} \right] \\ \leq \frac {\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \| ^ {2} - \mathbb {E} [ \| \boldsymbol {x} ^ {K} - \boldsymbol {x} ^ {*} \| ^ {2} ]}{K} + \frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2}}{n - 2 B - m} \\ + \frac {2 \gamma_ {2} \rho}{K} \sum_ {k = 0} ^ {K - 1} \sqrt {\mathbb {E} \left[ \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| ^ {2} \right] \mathbb {E} \left[ \mathbb {1} _ {k} \right]} + \frac {2 \gamma_ {2} ^ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \mathbb {1} _ {k} \right] + \frac {4 \gamma_ {1} \gamma_ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \mathbb {1} _ {k - \frac {1}{2}} \right]. \\ \end{array}
$$
Assumption 5 implies that  $0 \leq \left\langle F(\pmb{x}^{*}), \widetilde{\pmb{x}}^{k} - \pmb{x}^{*} \right\rangle \leq \left\langle F(\widetilde{\pmb{x}}^{k}), \widetilde{\pmb{x}}^{k} - \pmb{x}^{*} \right\rangle$ . Using this and a the notation  $R_{k} = \| \pmb{x}^{k} - \pmb{x}^{*}\|$ ,  $k > 0$ ,  $R_{0} \geq \| \pmb{x}^{0} - \pmb{x}^{*}\|$  we get
$$
\begin{array}{l} 0 \leq \frac {R _ {0} ^ {2} - \mathbb {E} [ R _ {K} ^ {2} ]}{K} + \frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2}}{n - 2 B - m} \\ + \frac {2 \gamma_ {2} \rho}{K} \sum_ {k = 0} ^ {K - 1} \sqrt {\mathbb {E} \left[ \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \mathbb {E} \left[ \mathbb {1} _ {k} \right]} + \frac {2 \gamma_ {2} ^ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \mathbb {1} _ {k} \right] + \frac {4 \gamma_ {1} \gamma_ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \mathbb {1} _ {k - \frac {1}{2}} \right] (6 2) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ R _ {k} ^ {2} \right] \leq R _ {0} ^ {2} + \frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2} k}{n - 2 B - m} + 2 \gamma_ {2} \rho \sum_ {l = 0} ^ {k - 1} \sqrt {\mathbb {E} \left[ R _ {l} ^ {2} \right] \mathbb {E} \left[ \mathbb {1} _ {l} \right]} (63) \\ + 2 \gamma_ {2} ^ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} \left[ \mathbb {1} _ {l} \right] + 4 \gamma_ {1} \gamma_ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} \left[ \mathbb {1} _ {l - \frac {1}{2}} \right] (64) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ R _ {k} ^ {2} \right] \leq R _ {0} ^ {2} + \frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2} k}{n - 2 B - m} + 2 \gamma_ {2} \rho \sum_ {l = 0} ^ {k - 1} \sqrt {\mathbb {E} \left[ R _ {l} ^ {2} \right] \mathbb {E} \left[ \mathbb {1} _ {l} \right]} (65) \\ + 2 \gamma_ {2} ^ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} [ \mathbb {1} _ {l} ] + 4 \gamma_ {1} \gamma_ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} \left[ \mathbb {1} _ {l - \frac {1}{2}} \right] \leq 2 R _ {0} ^ {2} (66) \\ \end{array}
$$
for all  $k = 0, \ldots, K$ . For  $k = 0$  this inequality trivially holds. Next, assume that it holds for all  $k = 0, 1, \ldots, T - 1, T \leq K - 1$ . Let us show that it holds for  $k = T$  as well. From (64) and (66) we have that  $\mathbb{E}[R_k^2] \leq 2R_0^2$  for all  $k = 0, 1, \ldots, T - 1$ . Therefore,
$$
\begin{array}{l} \mathbb {E} \left[ R _ {T} ^ {2} \right] \leq R _ {0} ^ {2} + \frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2} k}{n - 2 B - m} + 2 \gamma_ {2} \rho \sum_ {l = 0} ^ {k - 1} \sqrt {\mathbb {E} \left[ R _ {l} ^ {2} \right] \mathbb {E} \left[ \mathbb {1} _ {l} \right]} + 2 \gamma_ {2} ^ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} \left[ \mathbb {1} _ {l} \right] \\ + 4 \gamma_ {1} \gamma_ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} \left[ \mathbb {1} _ {l - \frac {1}{2}} \right] \\ \leq R _ {0} ^ {2} + \frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2} k}{n - 2 B - m} + 2 \gamma_ {2} \rho R _ {0} \sum_ {l = 0} ^ {k - 1} \sqrt {\mathbb {E} [ \mathbb {1} _ {l} ]} + 2 \gamma_ {2} ^ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} [ \mathbb {1} _ {l} ] \\ + 4 \gamma_ {1} \gamma_ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} \left[ \mathbb {1} _ {l - \frac {1}{2}} \right]. \\ \end{array}
$$
The latter together with the expected number of at least one peer violations (57) implies
$$
\mathbb {E} [ R _ {T} ^ {2} ] \leq R _ {0} ^ {2} + \frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2} k}{n - 2 B - m} + 2 \gamma_ {2} \rho R _ {0} \frac {n B}{m} + 2 \gamma_ {2} ^ {2} \rho^ {2} \frac {n B}{m} + 4 \gamma_ {1} \gamma_ {2} \rho^ {2} \frac {n B}{m}.
$$
$$
\gamma_ {1} = \min \left\{\frac {1}{2 L}, \sqrt {\frac {(n - 2 B - m) R _ {0} ^ {2}}{1 6 \sigma^ {2} K}}, \sqrt {\frac {m R _ {0} ^ {2}}{8 \rho^ {2} B n}} \right\}
$$
$$
\begin{array}{l} \gamma_ {2} = \min  \left\{\sqrt {\frac {m ^ {2} R _ {0} ^ {2}}{6 4 \rho^ {2} B ^ {2} n ^ {2}}}, \frac {1}{4 L}, \sqrt {\frac {(n - 2 B - m) R _ {0} ^ {2}}{6 4 \sigma^ {2} K}}, \sqrt {\frac {m R _ {0} ^ {2}}{3 2 \rho^ {2} B n}} \right\} \\ = \min \left\{\frac {1}{4 L}, \sqrt {\frac {m ^ {2} R _ {0} ^ {2}}{6 4 \rho^ {2} B ^ {2} n ^ {2}}}, \sqrt {\frac {(n - 2 B - m) R _ {0} ^ {2}}{6 4 \sigma^ {2} K}} \right\} \\ \end{array}
$$
we satisfy conditions of Lemma E.2 and ensure that
$$
\frac {8 \gamma_ {1} \gamma_ {2} \sigma^ {2} k}{n - 2 B - m} + 2 \gamma_ {2} \rho R _ {0} \frac {n B}{m} + 2 \gamma_ {2} ^ {2} \rho^ {2} \frac {n B}{m} + 4 \gamma_ {1} \gamma_ {2} \rho^ {2} \frac {n B}{m} \leq \frac {R _ {0} ^ {2}}{4} + \frac {R _ {0} ^ {2}}{4} + \frac {R _ {0} ^ {2}}{4} + \frac {R _ {0} ^ {2}}{4} = R _ {0} ^ {2},
$$
$$
\mathbb {E} \left[ R _ {T} ^ {2} \right] \leq 2 R _ {0} ^ {2} \equiv 2 R. \tag {67}
$$
Therefore, (66) holds for all  $k = 0,1,\ldots ,K$ . Together with (62) it implies
$$
\sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle F \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \leq \frac {R _ {0} ^ {2}}{\gamma_ {2}}.
$$
E.3.3 Quasi-Strongly Monotone Case
Lemma E.3. Let Assumptions 3, 4 and Corollary 2 hold. If
$$
\gamma_ {1} \leq \frac {1}{2 L} \tag {68}
$$
then  $\overline{\pmb{g}}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k) = \overline{\pmb{g}}_{\pmb{\eta}^k}\left(\pmb{x}^k - \gamma_1\widehat{\pmb{g}}_{\pmb{\xi}^k}(\pmb{x}^k)\right)$  satisfies the following inequality
$$
\gamma_ {1} ^ {2} \mathbb {E} \left[ \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\tilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \mid \boldsymbol {x} ^ {k} \right] \leq 2 \widehat {P} _ {k} + \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}}, \tag {69}
$$
where  $\widehat{P}_k = \gamma_1\mathbb{E}_{\boldsymbol{\xi}^k,\boldsymbol{\eta}^k}\left[\langle \overline{\boldsymbol{g}}_{\boldsymbol{\eta}^k}(\widetilde{\boldsymbol{x}}^k),\boldsymbol{x}^k -\boldsymbol{x}^*\rangle \right]$  and  $\rho^2 = q\sigma^2$  with  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{O}(1)$  by Lemma E.1.
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {x}} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma_ {1} \left\langle \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}, \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\rangle + \gamma_ {1} ^ {2} \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} (70) \\ = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma_ {1} \left\langle \boldsymbol {x} ^ {k} - \gamma \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \boldsymbol {x} ^ {*}, \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\rangle (71) \\ \left. - 2 \gamma_ {1} ^ {2} \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right), \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\rangle + \gamma_ {1} ^ {2} \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2}. \right. (72) \\ \end{array}
$$
Taking the expectation  $\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}[\cdot ] = \mathbb{E}\left[\cdot \mid \pmb{x}^k\right]$  conditioned on  $\pmb{x}^k$  from the above identity, using tower property  $\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}[\cdot ] = \mathbb{E}_{\pmb{\xi}^k}\left[\mathbb{E}_{\pmb{\eta}^k}\left[\cdot \right]\right]$ , and  $\mu$ -quasi strong monotonicity of  $F(x)$ , we derive
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {\boldsymbol {x}} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \\ = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma_ {1} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\langle \boldsymbol {x} ^ {k} - \gamma_ {1} \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \boldsymbol {x} ^ {*}, \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\rangle \right] \\ - 2 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}), \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\rangle \right] + \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} \right] \\ = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} \\ - 2 \gamma_ {1} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\langle \boldsymbol {x} ^ {k} - \gamma_ {1} \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \boldsymbol {x} ^ {*}, F \left(\boldsymbol {x} ^ {k} - \gamma_ {1} \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right)\right) \right\rangle \right] \\ - 2 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right), \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\rangle \right] + \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {\mathrm {(Q S M), (1 4)}} {\leq} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] \\ + \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \right]. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {\boldsymbol {x}} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \quad \stackrel {(1 6)} {\leq} \quad \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| F (\boldsymbol {x} ^ {k}) - F \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \right] \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - F (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - F \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \gamma_ {1} ^ {2} \left(1 - 4 L ^ {2} \gamma_ {1} ^ {2}\right) \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] \tag {Lip,(27),(28)} \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] \\ + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{G} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(1 6), L e m. D. 1} {\leq} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \gamma_ {1} ^ {2} \left(1 - 4 \gamma_ {1} ^ {2} L ^ {2}\right) \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] \\ + \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}} \\ \stackrel {(6 8)} {\leq} \quad \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}}. \\ \end{array}
$$
Finally, we use the above inequality together with (70):
$$
\left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \widehat {P} _ {k} + \gamma_ {1} ^ {2} \mathbb {E} \left[ \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} \mid \boldsymbol {x} ^ {k} \right] \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}},
$$
where  $\widehat{P}_k = \gamma_1\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}\left[\langle \overline{\pmb{g}}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k),\pmb{x}^k -\pmb{x}^*\rangle \right]$ . Rearranging the terms, we obtain (69).
Lemma E.4. Let Assumptions 3, 4 and Corollary 2 hold. If
$$
\gamma_ {1} \leq \frac {1}{2 \mu + 2 L}, \tag {73}
$$
$$
\widehat {P} _ {k} \geq \frac {\mu \gamma_ {1}}{2} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {\gamma_ {1} ^ {2}}{4} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] - \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} - \frac {9 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}}}{2}, \tag {74}
$$
$$
{- \widehat {P} _ {k}} \leq {- \frac {\mu \gamma_ {1}}{2} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}}}
$$
where  $\widehat{P}_k = \gamma_1\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}\left[\langle \overline{\pmb{g}}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k),\pmb{x}^k -\pmb{x}^*\rangle \right]$  where  $\rho^2 = q\sigma^2$  with  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{O}(1)$  by Lemma E.1.
Proof. Since  $\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}[\cdot ] = \mathbb{E}[\cdot \mid \pmb{x}^k ]$  and  $\overline{g}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k) = \overline{g}_{\pmb{\eta}^k}\left(\pmb{x}^k -\gamma_1\widehat{g}_{\pmb{\xi}^k}(\pmb{x}^k)\right)$ , we have
$$
- \widehat {P} _ {k} \quad = - \gamma_ {1} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\langle \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right]
$$
$$
\begin{array}{l} \stackrel {(1 7), (\text {L i p}), L e m. D. 1, C o r. 2} {\leq} \quad - \frac {\mu \gamma_ {1}}{2} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \frac {\gamma_ {1} ^ {2}}{2} (1 - 2 \gamma_ {1} \mu - 4 \gamma_ {1} ^ {2} L ^ {2}) \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] \\ + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{2 g} + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{2 g} + 4 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {I} _ {k - \frac {1}{2}} \\ \leq \quad - \frac {\mu \gamma_ {1}}{2} \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| ^ {2} - \frac {\gamma_ {1} ^ {2}}{2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \| ^ {2} \right] \\ + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}} \\ \end{array}
$$
$$
{- \widehat {P} _ {k}} \leq {- \frac {\mu \gamma_ {1}}{2} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \frac {\gamma_ {1} ^ {2}}{4} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + \frac {9 \gamma_ {1} ^ {2} \rho^ {2} \mathbb {1} _ {k - \frac {1}{2}}}{2}}
$$
Combining Lemmas E.3 and E.4, we get the following result.
Theorem (Theorem 6 duplicate). Let Assumptions 1, 3 and 4 hold. Then after  $T$  iterations SEG-CC (Algorithm 7) with  $\gamma_1 \leq \frac{1}{2\mu + 2L}$  and  $\beta = \gamma_2 / \gamma_1 \leq 1/4$  outputs  $\boldsymbol{x}^T$  such that
where  $q = 2C^2 + 12 + \frac{12}{n - 2B - m}$ ;  $q = \mathcal{O}(1)$  since  $C = \mathcal{O}(1)$ .
Proof of Theorem 6. Since  $\pmb{x}^{k + 1} = \pmb{x}^k -\gamma_2\widehat{\pmb{g}}_{\pmb{\eta}^k}\left(\widetilde{\pmb{x}}^k\right)$ , we have