SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {(\rho - \kappa \theta)}{2 \rho (\gamma - \kappa) \theta} \| \nabla \psi_ {1 / \rho} (z ^ {k}) \| ^ {2} \\ \leq \psi_ {1 / \rho} (z ^ {k}) - \mathbb {E} _ {k} [ \psi_ {1 / \rho} (z ^ {k + 1}) ] + \frac {\rho \beta}{(\gamma - \kappa) \theta^ {2}} \{\psi (x ^ {k}) - \mathbb {E} _ {k} [ \psi (x ^ {k + 1}) ] \} \\ + \frac {\rho (2 \rho \beta^ {2} \theta^ {- 1} + \gamma \beta \theta)}{2 (\gamma - \kappa) \theta^ {2}} \{\| x ^ {k} - x ^ {k - 1} \| ^ {2} - \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \} \\ + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) ^ {2} \theta^ {2}} - \frac {\rho (\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1})}{2 (\gamma - \kappa) \theta^ {2}} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] + \frac {\rho \mathbb {E} _ {k} [ \varepsilon_ {k} ]}{(\gamma - \kappa) \theta^ {2}}, \\ \end{array}
$$
where  $\varepsilon_{k} = (\lambda \theta +\frac{\lambda}{2})\| x^{k + 1} - x^{k - \tau_{k}}\|^{2} + \frac{\lambda(1 - \theta)}{2}\| x^{k} - x^{k - \tau_{k}}\|^{2}$  characterizes the error of delay.
We next bound the stochastic delay using E1 and eliminate the delays.
Lemma 11. Under the same conditions as Lemma 10 as well as E1, given  $\gamma > \max \left\{\rho, 2(\rho + \kappa \beta) \theta^{-1} + 2 \rho \beta^2 \theta^{-3}\right\}$ ,
$$
\mathbb {E} [ \| \nabla \psi_ {1 / \rho} (z ^ {k ^ {*}}) \| ^ {2} ] \leq \frac {2 \rho \theta}{\rho - \kappa \theta} \Big [ \frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{K} + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) \theta^ {2}} + \frac {\rho \lambda (3 \tau^ {2} + 2 \beta)}{2 \theta^ {2} K} \sum_ {k = 1} ^ {K} \mathbb {E} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \Big ],
$$
where  $D = \max \{\psi_{1 / \rho}(z^1) - \psi_{1 / \rho}(z^*)$ ,  $\psi (x^{1}) - \psi (x^{*})\}$ .
Remark 7. Lemma 11 reveals the effect of extrapolation. After simplification  $\theta = 1 - \beta$  appears in the denominator of the error from delay and will enlarge such an error if  $\beta \rightarrow 1$ . This is intuitive since when extrapolating using two iterations generated from delayed information, the error is magnified.
By taking less aggressive steps, we bound the term  $\sum_{k=1}^{K} \mathbb{E}[||x^{k+1} - x^k ||^2]$  and arrive at the final convergence result for DSEPL.
Theorem 8. Under the same conditions as Lemma 11, if we further choose  $\gamma \geq 2\theta^{-1}(\rho + \kappa \beta) + 2\rho\theta^{-3}\beta^2 + 2\lambda\theta^{-2}\beta + 3\lambda\theta^{-1}\tau^2$ , then
$$
\mathbb {E} [ \| \nabla \psi_ {1 / \rho} (z ^ {k ^ {*}}) \| ^ {2} ] \leq \frac {2 \rho \theta}{(\rho - \kappa \theta) \sqrt {K}} \Big (\frac {\lambda (3 \tau^ {2} + 2 \beta)}{\Gamma_ {\beta}} + 1 \Big) \left[ \frac {(\gamma - \kappa + \rho \beta \theta^ {- 2}) D}{\sqrt {K}} + \frac {2 \rho L _ {f} ^ {2} \sqrt {K}}{(\gamma - \kappa) \theta^ {2}} \right],
$$
where  $\Gamma_{\beta} \coloneqq \gamma \theta^{2} - 2\theta (\rho + \kappa \beta) - 2\rho \beta^{2}\theta^{-1} - 3\lambda \tau^{2} - 2\lambda \beta.$
Remark 8. If  $\gamma = \mathcal{O}(\sqrt{K})$ , then  $\frac{(\gamma - \kappa + \rho\beta\theta^{-2})D}{K} + \frac{2\rho L_f^2}{(\gamma - \kappa)\theta^2} = \mathcal{O}\left(\frac{1}{\sqrt{K}}\right), \Gamma_\beta^{-1} = \mathcal{O}\left(\frac{1}{\sqrt{K}}\right)$  and
$$
\mathbb {E} [ \| \nabla \psi_ {1 / \rho} (z ^ {k ^ {*}}) \| ^ {2} ] = \mathcal {O} \Big (\frac {1}{K} + \frac {1}{\sqrt {K}} + \big (\frac {1}{K} + \frac {1}{\sqrt {K}} \big) \frac {\tau^ {2}}{\sqrt {K}} \Big),
$$
which implies that the delay is negligible if  $\tau = o(K^{1 / 4})$
Remark 9. Although our convergence result is based on the extrapolated sequence, we can leverage Lemma 10 to show that  $\mathbb{E}[\| x^{k^* +1} - x^{k^*}\| ^2 ]$ , and subsequently  $\mathbb{E}[\| z^{k^*} - x^{k^*}\| ^2 ]$  is  $\mathcal{O}\left(\frac{1}{K}\right)$ . Using smoothness of the Moreau envelope, we finally have  $\mathbb{E}[\| \nabla \psi_{1 / \rho}(x^{k^*})\| ^2 ]$  is  $\mathcal{O}(\frac{1}{\sqrt{K}})$ .
F.2 Convergence analysis of DSEPL
In this section we present the convergence analysis for DSEPL.
F.2.1 Auxiliary results
To show the convergence of DSEPL, we need to define an auxiliary sequence
$$
z ^ {k} := x ^ {k} + \frac {\beta}{1 - \beta} \left(x ^ {k} - x ^ {k - 1}\right).
$$
Given  $x^k$ , define  $\bar{x} = \beta x^{k} + (1 - \beta)x$  for  $x\in \mathrm{dom}(\omega)$  and  $\theta = 1 - \beta$ . The following identities hold
$$
\bar {x} - x ^ {k} = \theta (x - x ^ {k})
$$
$$
\bar {x} - y ^ {k} = \theta (x - z ^ {k})
$$
$$
\bar {x} - x ^ {k + 1} = \theta (x - z ^ {k + 1}).
$$
and will be used frequently in the analysis.
Proof of Lemma 10
First by the  $(\gamma - \kappa)$ -strong convexity of the proximal sub-problem, we have
$$
\begin{array}{l} f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k + 1}\right) + \frac {\gamma}{2} \| x ^ {k + 1} - y ^ {k} \| ^ {2} \\ \leq f _ {x ^ {k - \tau_ {k}}} (\bar {x}, \xi^ {k - \tau_ {k}}) + \omega (\bar {x}) + \frac {\gamma}{2} \| \bar {x} - y ^ {k} \| ^ {2} - \frac {\gamma - \kappa}{2} \| x ^ {k + 1} - \bar {x} \| ^ {2} \\ = f _ {x ^ {k - \tau_ {k}}} (\bar {x}, \xi^ {k - \tau_ {k}}) + \omega (\bar {x}) + \frac {\gamma \theta^ {2}}{2} \| x - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \| x - z ^ {k + 1} \| ^ {2}. \tag {51} \\ \end{array}
$$
Also, since  $f_{x^{k - \tau_k}}(\cdot ,\xi^{k - \tau_k}) + \omega (\cdot) + \frac{\kappa}{2}\| \cdot -x^k\| ^2$  is convex, we plug  $\bar{x}$  in and apply convexity to get that
$$
\begin{array}{l} f _ {x ^ {k - \tau_ {k}}} (\bar {x}, \xi^ {k - \tau_ {k}}) + \omega (\bar {x}) + \frac {\kappa}{2} \| \bar {x} - x ^ {k} \| ^ {2} \\ \leq (1 - \theta) [ f _ {x ^ {k - \tau_ {k}}} (x ^ {k}, \xi^ {k - \tau_ {k}}) + \omega (x ^ {k}) ] + \theta [ f _ {x ^ {k - \tau_ {k}}} (x, \xi^ {k - \tau_ {k}}) + \omega (x) + \frac {\kappa}{2} \| x - x ^ {k} \| ^ {2} ] \\ \leq (1 - \theta) \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k}\right) \right] + \theta \left[ f (x) + \omega (x) + \frac {\kappa}{2} \| x - x ^ {k} \| ^ {2} + \frac {\lambda}{2} \| x - x ^ {k - \tau_ {k}} \| ^ {2} \right], \\ \end{array}
$$
where the second inequality leverages two-sided approximation since  $\left|f_{x^{k - \tau_k}}(x,\xi^{k - \tau_k}) - f(x)\right|\leq \frac{\lambda}{2}\| x - x^{k - \tau_k}\|^2$ . Then a simple re-arrangement gives
$$
\begin{array}{l} f _ {x ^ {k - \tau_ {k}}} (\bar {x}, \xi^ {k - \tau_ {k}}) + \omega (\bar {x}) \\ \leq (1 - \theta) \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k}\right) \right] \\ + \theta [ f (x) + \omega (x) + \frac {\kappa}{2} \| x - x ^ {k} \| ^ {2} + \frac {\lambda}{2} \| x - x ^ {k - \tau_ {k}} \| ^ {2} ] - \frac {\kappa \theta^ {2}}{2} \| x - x ^ {k} \| ^ {2}. \\ \end{array}
$$
Now combining the above inequality with (51), we get that
$$
\begin{array}{l} f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k + 1}\right) + \frac {\gamma}{2} \| x ^ {k + 1} - y ^ {k} \| ^ {2} \\ \leq (1 - \theta) \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k}\right) \right] + \theta \left[ f (x) + \omega (x) + \frac {\kappa}{2} \| x - x ^ {k} \| ^ {2} + \frac {\lambda}{2} \| x - x ^ {k - \tau_ {k}} \| ^ {2} \right] \\ - \frac {\kappa \theta^ {2}}{2} \| x - x ^ {k} \| ^ {2} + \frac {\gamma \theta^ {2}}{2} \| x - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \| x - z ^ {k + 1} \| ^ {2}. \tag {52} \\ \end{array}
$$
From now on we let  $x = \hat{z}^k$  in (52) and successively deduce that
$$
\begin{array}{l} f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k + 1}\right) + \frac {\gamma}{2} \| x ^ {k + 1} - y ^ {k} \| ^ {2} \\ \leq (1 - \theta) [ f _ {x ^ {k - \tau_ {k}}} (x ^ {k}, \xi^ {k - \tau_ {k}}) + \omega (x ^ {k}) ] + \theta [ f (\hat {z} ^ {k}) + \omega (\hat {z} ^ {k}) + \frac {\kappa}{2} \| \hat {z} ^ {k} - x ^ {k} \| ^ {2} + \frac {\lambda}{2} \| \hat {z} ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} ] \\ - \frac {\kappa \theta^ {2}}{2} \| \hat {z} ^ {k} - x ^ {k} \| ^ {2} + \frac {\gamma \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2}. \\ = (1 - \theta) [ f _ {x ^ {k - \tau_ {k}}} (x ^ {k}, \xi^ {k - \tau_ {k}}) + \omega (x ^ {k}) ] + \theta [ f (\hat {z} ^ {k}) + \omega (\hat {z} ^ {k}) ] + \frac {\kappa \theta - \kappa \theta^ {2}}{2} \| \hat {z} ^ {k} - x ^ {k} \| ^ {2} + \frac {\lambda \theta}{2} \| \hat {z} ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} \\ + \frac {\gamma \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} \\ \leq (1 - \theta) [ f _ {x ^ {k} - \tau_ {k}} (x ^ {k}, \xi^ {k - \tau_ {k}}) + \omega (x ^ {k}) ] + \theta [ f (\hat {z} ^ {k}) + \omega (\hat {z} ^ {k}) ] + \frac {\lambda \theta}{2} \| \hat {z} ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} \\ + \frac {\gamma \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} + \theta \kappa \beta [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} + \| \hat {z} ^ {k} - x ^ {k + 1} \| ^ {2} ] (53) \\ \leq (1 - \theta) \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k}\right) \right] + \theta [ f (\hat {z} ^ {k}) + \omega (\hat {z} ^ {k}) ] + \theta (\kappa \beta + \lambda) \| \hat {z} ^ {k} - x ^ {k + 1} \| ^ {2} \\ + \lambda \theta \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} + \frac {\gamma \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} + \theta \kappa \beta \| x ^ {k + 1} - x ^ {k} \| ^ {2}, (54) \\ \end{array}
$$
where the second inequality (53) applies  $\| \hat{z}^k -x^k\| ^2\leq 2[\| x^{k + 1} - x^k\| ^2 +\| \hat{z}^k -x^{k + 1}\| ^2 ]$  and the last inequality applies  $\| \hat{z}^k -x^{k - \tau_k}\| ^2\leq 2\| \hat{z}^k -x^{k + 1}\| ^2 +2\| x^{k + 1} - x^{k - \tau_k}\| ^2.$
By definition of  $\hat{z}^k$ , for  $\rho > \lambda + \kappa$ , we have  $(\rho - \lambda - \kappa)$ -strong convexity of  $\psi(x) + \frac{\rho}{2} \| \cdot - x \|^2$  and
$$
\begin{array}{l} f (\hat {z} ^ {k}) + \omega (\hat {z} ^ {k}) + \frac {\rho}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} \\ \leq f \left(x ^ {k + 1}\right) + \omega \left(x ^ {k + 1}\right) + \frac {\rho}{2} \| x ^ {k + 1} - z ^ {k} \| ^ {2} - \frac {\rho - \lambda - \kappa}{2} \| x ^ {k + 1} - \hat {z} ^ {k} \| ^ {2}. \tag {55} \\ \end{array}
$$
Multiplying (55) by  $\theta$  and adding it to (54), we have
$$
\begin{array}{l} \frac {\gamma}{2} \left\| x ^ {k + 1} - y ^ {k} \right\| ^ {2} \\ \leq (1 - \theta) \left[ f _ {x ^ {k} - \tau_ {k}} \left(x ^ {k}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k}\right) \right] + \theta f \left(x ^ {k + 1}\right) - f _ {x ^ {k} - \tau_ {k}} \left(x ^ {k + 1}, \xi^ {k - \tau_ {k}}\right) - (1 - \theta) \omega \left(x ^ {k + 1}\right) \\ + \frac {\rho \theta}{2} \| x ^ {k + 1} - z ^ {k} \| ^ {2} - \frac {\theta (\rho - 3 \lambda - 2 \kappa \beta - \kappa)}{2} \| x ^ {k + 1} - \hat {z} ^ {k} \| ^ {2} + \lambda \theta \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} \\ + \frac {\gamma \theta^ {2} - \rho \theta}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} + \theta \kappa \beta \| x ^ {k + 1} - x ^ {k} \| ^ {2}. \tag {56} \\ \end{array}
$$
Then we bound the first line of the right hand side in (56) by
$$
\begin{array}{l} (1 - \theta) \left[ f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k}, \xi^ {k - \tau_ {k}}\right) + \omega \left(x ^ {k}\right) \right] + \theta f \left(x ^ {k + 1}\right) - f _ {x ^ {k - \tau_ {k}}} \left(x ^ {k + 1}, \xi^ {k - \tau_ {k}}\right) - (1 - \theta) \omega \left(x ^ {k + 1}\right) \\ \leq (1 - \theta) [ f (x ^ {k}, \xi^ {k - \tau_ {k}}) + \omega (x ^ {k}) - f (x ^ {k + 1}) - \omega (x ^ {k + 1}) ] + f (x ^ {k + 1}) - \mathbb {E} _ {\xi} [ f _ {x ^ {k - \tau_ {k}}} (x ^ {k + 1}, \xi) ] \\ + \mathbb {E} _ {\xi} \left[ f _ {x ^ {k - \tau_ {k}}} (x ^ {k + 1}, \xi) \right] - f _ {x ^ {k - \tau_ {k}}} (x ^ {k + 1}, \xi^ {k - \tau_ {k}}) + \frac {(1 - \theta) \lambda}{2} \| x ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2} \\ \leq (1 - \theta) [ f (x ^ {k}, \xi^ {k - \tau_ {k}}) + \omega (x ^ {k}) - f (x ^ {k + 1}) - \omega (x ^ {k + 1}) ] + \frac {2 L _ {f} ^ {2}}{\gamma - \kappa} \\ + \frac {\lambda}{2} \left[ \left\| x ^ {k + 1} - x ^ {k - \tau_ {k}} \right\| ^ {2} + (1 - \theta) \left\| x ^ {k} - x ^ {k - \tau_ {k}} \right\| ^ {2} \right], \tag {57} \\ \end{array}
$$
where the inequalities invoke two-sided approximation  $\left|f_{x^{k} - \tau_{k}}(x^{k},\xi^{k - \tau_{k}}) - f(x^{k},\xi^{k - \tau_{k}})\right| \leq \frac{\lambda}{2}\| x^{k} - x^{k - \tau_{k}}\|^{2}$  and bound  $\mathbb{E}_{\xi}\left[f_{x^{k - \tau_{k}}}(x^{k + 1},\xi)\right] - f_{x^{k - \tau_{k}}}(x^{k + 1},\xi^{k - \tau_{k}})$  with Lemma 8. Now plugging (57) back into (56) gives
$$
\begin{array}{l} \frac {\gamma}{2} \left\| x ^ {k + 1} - y ^ {k} \right\| ^ {2} \\ \leq (1 - \theta) [ f (x ^ {k}, \xi^ {k - \tau_ {k}}) + \omega (x ^ {k}) - f (x ^ {k + 1}) - \omega (x ^ {k + 1}) ] + \frac {2 L _ {f} ^ {2}}{\gamma - \kappa} \\ + \frac {\rho \theta}{2} \| x ^ {k + 1} - z ^ {k} \| ^ {2} - \frac {\theta (\rho - 3 \lambda - 2 \kappa \beta - \kappa)}{2} \| x ^ {k + 1} - \hat {z} ^ {k} \| ^ {2} \\ + \frac {\gamma \theta^ {2} - \rho \theta}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} + \theta \kappa \beta \| x ^ {k + 1} - x ^ {k} \| ^ {2}. \\ + \left(\lambda \theta + \frac {\lambda}{2}\right) \| x ^ {k + 1} - x ^ {k - \tau_ {k}} \| ^ {2} + \frac {\lambda (1 - \theta)}{2} \| x ^ {k} - x ^ {k - \tau_ {k}} \| ^ {2}. \tag {58} \\ \end{array}
$$
Isolating the delayed error by  $\varepsilon_{k} = \left(\lambda \theta +\frac{\lambda}{2}\right)\| x^{k + 1} - x^{k - \tau_{k}}\|^{2} + \frac{\lambda(1 - \theta)}{2}\| x^{k} - x^{k - \tau_{k}}\|^{2}$  and we have, by algebraic manipulations of the momentum terms, that
$$
\begin{array}{l} \| x ^ {k + 1} - y ^ {k} \| ^ {2} = \| x ^ {k + 1} - x ^ {k} + x ^ {k} - y ^ {k} \| ^ {2} \\ \geq \left\| x ^ {k + 1} - x ^ {k} \right\| ^ {2} + \beta^ {2} \left\| x ^ {k} - x ^ {k - 1} \right\| ^ {2} - \beta \left\| x ^ {k + 1} - x ^ {k} \right\| ^ {2} - \beta \left\| x ^ {k} - x ^ {k - 1} \right\| ^ {2} \\ = \theta \| x ^ {k + 1} - x ^ {k} \| ^ {2} - \beta \theta \| x ^ {k} - x ^ {k - 1} \| ^ {2} \tag {59} \\ \end{array}
$$
$$
\frac {\rho \theta}{2} \| x ^ {k + 1} - z ^ {k} \| ^ {2} \leq \rho \theta \| x ^ {k + 1} - x ^ {k} \| ^ {2} + \rho \beta^ {2} \theta^ {- 1} \| x ^ {k} - x ^ {k - 1} \| ^ {2}. \tag {60}
$$
Combining (59), (60) with (58) and taking expectation, we have
$$
\begin{array}{l} \frac {\gamma \theta}{2} \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - x ^ {k} \right\| ^ {2} \right] - \frac {\gamma \beta \theta}{2} \| x ^ {k} - x ^ {k - 1} \| ^ {2} \\ \leq (1 - \theta) \{\psi (x ^ {k}) - \mathbb {E} _ {k} [ \psi (x ^ {k + 1}) ] \} + \frac {2 L _ {f} ^ {2}}{\gamma - \kappa} \\ + (\rho + \kappa \beta) \theta \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] + \rho \beta^ {2} \theta^ {- 1} \| x ^ {k} - x ^ {k - 1} \| ^ {2} - \frac {\theta (\rho - 3 \lambda - 2 \kappa \beta - \kappa)}{2} \mathbb {E} _ {k} [ \| x ^ {k + 1} - \hat {z} ^ {k} \| ^ {2} ] \\ + \frac {\gamma \theta^ {2} - \rho \theta}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} - \frac {(\gamma - \kappa) \theta^ {2}}{2} \mathbb {E} _ {k} [ \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} ] + \mathbb {E} _ {k} [ \varepsilon_ {k} ]. \\ \end{array}
$$
After re-arrangement, we arrive at
$$
\begin{array}{l} \frac {(\gamma - \kappa) \theta^ {2}}{2} \mathbb {E} _ {k} [ \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} ] \\ \leq (1 - \theta) \{\psi (x ^ {k}) - \mathbb {E} _ {k} [ \psi (x ^ {k + 1}) ] \} + \frac {\gamma \theta^ {2} - \rho \theta}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} \\ + \left(\rho \beta^ {2} \theta^ {- 1} + \frac {\gamma \beta \theta}{2}\right) \{\| x ^ {k} - x ^ {k - 1} \| ^ {2} - \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \} - \frac {\theta (\rho - 3 \lambda - 2 \kappa \beta - \kappa)}{2} \mathbb {E} _ {k} [ \| x ^ {k + 1} - \hat {z} ^ {k} \| ^ {2} ] \\ + \frac {2 L _ {f} ^ {2}}{\gamma - \kappa} - \frac {\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1}}{2} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] + \mathbb {E} _ {k} [ \varepsilon_ {k} ]. \tag {61} \\ \end{array}
$$
Dividing both sides of (61) by  $\frac{(\gamma - \kappa)\theta^2}{2}$ , we obtain that
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} \right] \\ \leq \frac {2 (1 - \theta)}{(\gamma - \kappa) \theta^ {2}} \left\{\psi \left(x ^ {k}\right) - \mathbb {E} _ {k} \left[ \psi \left(x ^ {k + 1}\right) \right] \right\} + \frac {\gamma \theta - \rho}{(\gamma - \kappa) \theta} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} \\ + \frac {2 \rho \beta^ {2} \theta^ {- 1} + \gamma \beta \theta}{(\gamma - \kappa) \theta^ {2}} \left\{\| x ^ {k} - x ^ {k - 1} \| ^ {2} - \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \right\} - \frac {\rho - 3 \lambda - 2 \kappa \beta - \kappa}{(\gamma - \kappa) \theta} \mathbb {E} _ {k} [ \| x ^ {k + 1} - \hat {z} ^ {k} \| ^ {2} ] \\ + \frac {4 L _ {f} ^ {2}}{(\gamma - \kappa) ^ {2} \theta^ {2}} - \frac {\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1}}{(\gamma - \kappa) \theta^ {2}} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] + \frac {2 \mathbb {E} _ {k} [ \varepsilon_ {k} ]}{(\gamma - \kappa) \theta^ {2}} \\ = \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} - \frac {\rho - \kappa \theta}{(\gamma - \kappa) \theta} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} + \frac {2 \beta}{(\gamma - \kappa) \theta} \left\{\psi \left(x ^ {k}\right) - \mathbb {E} _ {k} [ \psi \left(x ^ {k + 1}\right) ] \right\} \\ + \frac {2 \rho \beta^ {2} \theta^ {- 1} + \gamma \beta \theta}{(\gamma - \kappa) \theta^ {2}} \left\{\| x ^ {k} - x ^ {k - 1} \| ^ {2} - \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - x ^ {k} \| ^ {2} \right] \right\} - \frac {\rho - 3 \lambda - 2 \kappa \beta - \kappa}{(\gamma - \kappa) \theta} \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - \hat {z} ^ {k} \| ^ {2} \right] \\ + \frac {4 L _ {f} ^ {2}}{(\gamma - \kappa) ^ {2} \theta^ {2}} - \frac {\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1}}{(\gamma - \kappa) \theta^ {2}} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] + \frac {2 \mathbb {E} _ {k} [ \varepsilon_ {k} ]}{(\gamma - \kappa) \theta^ {2}}. \tag {62} \\ \end{array}
$$
Next, we consider the Moreau envelope and
$$
\begin{array}{l} \mathbb {E} _ {k} \left[ \psi_ {1 / \rho} (z ^ {k + 1}) \right] \\ = \mathbb {E} _ {k} \left[ \psi (\hat {z} ^ {k + 1}) + \frac {\rho}{2} \| \hat {z} ^ {k + 1} - z ^ {k + 1} \| ^ {2} \right] \\ \leq \mathbb {E} _ {k} \left[ f (\hat {z} ^ {k}) + \omega (\hat {z} ^ {k}) + \frac {\rho}{2} \| \hat {z} ^ {k} - z ^ {k + 1} \| ^ {2} \right] \\ \leq f \left(\hat {z} ^ {k}\right) + \omega \left(\hat {z} ^ {k}\right) + \frac {\rho}{2} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} + \frac {\rho \beta}{(\gamma - \kappa) \theta^ {2}} \left\{\psi \left(x ^ {k}\right) - \mathbb {E} _ {k} \left[ \psi \left(x ^ {k + 1}\right) \right] \right\} \\ - \frac {\rho (\rho - \kappa \theta)}{2 (\gamma - \kappa) \theta} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} + \frac {\rho (2 \rho \beta^ {2} \theta^ {- 1} + \gamma \beta \theta)}{2 (\gamma - \kappa) \theta^ {2}} \left\{\| x ^ {k} - x ^ {k - 1} \| ^ {2} - \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \right\} \\ + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) ^ {2} \theta^ {2}} - \frac {\rho (\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1})}{2 (\gamma - \kappa) \theta^ {2}} \mathbb {E} _ {k} \left[ \left\| x ^ {k + 1} - x ^ {k} \right\| ^ {2} \right] + \frac {\rho \mathbb {E} _ {k} [ \varepsilon_ {k} ]}{(\gamma - \kappa) \theta^ {2}} (63) \\ = \psi_ {1 / \rho} (z ^ {k}) + \frac {\rho \beta}{(\gamma - \kappa) \theta^ {2}} \left\{\psi \left(x ^ {k}\right) - \mathbb {E} _ {k} [ \psi \left(x ^ {k + 1}\right) ] \right\} \\ - \frac {\rho (\rho - \kappa \theta)}{2 (\gamma - \kappa) \theta} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} + \frac {\rho (2 \rho \beta^ {2} \theta^ {- 1} + \gamma \beta \theta)}{2 (\gamma - \kappa) \theta^ {2}} \left\{\| x ^ {k} - x ^ {k - 1} \| ^ {2} - \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \right\} \\ + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) ^ {2} \theta^ {2}} - \frac {\rho (\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1})}{2 (\gamma - \kappa) \theta^ {2}} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] + \frac {\rho \mathbb {E} _ {k} [ \varepsilon_ {k} ]}{(\gamma - \kappa) \theta^ {2}}, (64) \\ \end{array}
$$
where (63) uses the relation from (62). Last we re-arrangement the terms in (64) to get
$$
\begin{array}{l} \frac {\rho (\rho - \kappa \theta)}{2 (\gamma - \kappa) \theta} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} \\ \leq \psi_ {1 / \rho} (z ^ {k}) - \mathbb {E} _ {k} [ \psi_ {1 / \rho} (z ^ {k + 1}) ] + \frac {\rho \beta}{(\gamma - \kappa) \theta^ {2}} \left\{\psi (x ^ {k}) - \mathbb {E} _ {k} [ \psi (x ^ {k + 1}) ] \right\} \\ + \frac {\rho (2 \rho \beta^ {2} \theta^ {- 1} + \gamma \beta \theta)}{2 (\gamma - \kappa) \theta^ {2}} \left\{\| x ^ {k} - x ^ {k - 1} \| ^ {2} - \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] \right\} \\ + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) ^ {2} \theta^ {2}} - \frac {\rho (\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1})}{2 (\gamma - \kappa) \theta^ {2}} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] + \frac {\rho \mathbb {E} _ {k} [ \varepsilon_ {k} ]}{(\gamma - \kappa) \theta^ {2}}. \\ \end{array}
$$
Recalling that  $\| \nabla \psi_{1 / \rho}(\hat{z}^k)\| ^2 = \rho^2\| \hat{z}^k -z^k\| ^2$  , we complete the proof.
Proof of Lemma 11
By Lemma 10 we have that
$$
\begin{array}{l} \frac {\rho - \kappa \theta}{2 \rho (\gamma - \kappa) \theta} \| \hat {z} ^ {k} - z ^ {k} \| ^ {2} \\ \leq \psi_ {1 / \rho} (z ^ {k}) - \mathbb {E} _ {k} [ \psi_ {1 / \rho} (z ^ {k + 1}) ] + \frac {\rho \beta}{(\gamma - \kappa) \theta^ {2}} \{\psi (x ^ {k}) - \mathbb {E} _ {k} [ \psi (x ^ {k + 1}) ] \} \\ + \frac {\rho (2 \rho \beta^ {2} \theta^ {- 1} + \gamma \beta \theta)}{2 (\gamma - \kappa) \theta^ {2}} \left\{\| x ^ {k} - x ^ {k - 1} \| ^ {2} - \mathbb {E} _ {k} \left[ \| x ^ {k + 1} - x ^ {k} \| ^ {2} \right] \right\} \tag {65} \\ + \frac {2 \rho L _ {f} ^ {2}}{(\gamma - \kappa) ^ {2} \theta^ {2}} - \frac {\rho (\gamma \theta^ {2} - 2 \theta (\rho + \kappa \beta) - 2 \rho \beta^ {2} \theta^ {- 1})}{2 (\gamma - \kappa) \theta^ {2}} \mathbb {E} _ {k} [ \| x ^ {k + 1} - x ^ {k} \| ^ {2} ] + \frac {\rho \mathbb {E} _ {k} [ \varepsilon_ {k} ]}{(\gamma - \kappa) \theta^ {2}}. \\ \end{array}
$$
Summing up the inequality (65) from  $k = 1$  to  $K$ , multiplying both sides by  $\gamma + \kappa > 0$  and taking expectation, we have