SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} + \rho \mathbb {E} _ {t} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + \frac {2 p \omega}{n \bar {L}} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2}\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| u ^ {t} - x ^ {*} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - x ^ {*} \| ^ {2} \right]\right) \\ + p \left(\frac {4 \omega L _ {\max}}{n \bar {L}} + \theta_ {t + 1} - 1\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \frac {p \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] - \frac {p \theta_ {t + 1} ^ {2} \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] \\ + \kappa \left(8 p \left(1 + \frac {p}{\beta}\right) L _ {\max } D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) + 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] + \left(1 - \frac {\beta}{2}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t}\right) \right\| ^ {2}\right) \\ + \nu_ {t} \left(\left(1 - \frac {\alpha}{2}\right) \left\| w ^ {t} - u ^ {t} \right\| ^ {2} + \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \right. \\ + \frac {2 \omega}{n} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(z ^ {t}\right) - h _ {i} ^ {t} \right\| ^ {2} + \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{n} + \frac {8 L}{\alpha}\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \rho \left(2 p \mathbb {E} _ {t} \left[ \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \right] + 8 p L D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) + 4 p L ^ {2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] + (1 - p) \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2}\right) \\ + \lambda \left(\left(1 - \frac {\tau}{2}\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} + \frac {2 \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \right. \\ \left. + \left(4 p \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 p \tau^ {2} \omega L _ {\max}}{n}\right) D _ {f} (z ^ {t}, y ^ {t + 1}) + \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right) \mathbb {E} _ {t} \left[ \| x ^ {t + 1} - y ^ {t + 1} \| ^ {2} \right]\right). \\ \end{array}
$$
We regroup the terms and obtain
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f (z ^ {t + 1}) - f (x ^ {*}) \right] + \kappa \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right] + \nu_ {t} \mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right] \\ + \rho \mathbb {E} _ {t} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| u ^ {t} - x ^ {*} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - x ^ {*} \| ^ {2} \right]\right) \\ + p \left(\frac {4 \omega L _ {\max}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\max} + \nu_ {t} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \right. \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\max }}{n}\right) + \theta_ {t + 1} - 1\left. \right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \lambda \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) \mathbb {E} _ {t} \left[ \| x ^ {t + 1} - y ^ {t + 1} \| ^ {2} \right] \\ - \frac {p \theta_ {t + 1} ^ {2} \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] \\ + \nu_ {t} \left(1 - \frac {\alpha}{2}\right) \left\| w ^ {t} - u ^ {t} \right\| ^ {2} \\ \left. + \left(\nu_ {t} \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \rho (1 - p)\right) \| k ^ {t} - \nabla f (z ^ {t}) \| ^ {2} \right. \\ \end{array}
$$
$$
\begin{array}{l} + \left(\rho 2 p + \lambda \left(1 - \frac {\tau}{2}\right)\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + \left(\frac {2 p \omega}{n \bar {L}} + \nu_ {t} \frac {2 \omega}{n} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \lambda \frac {2 \tau^ {2} \omega}{n} + \kappa \left(1 - \frac {\beta}{2}\right)\right) \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2}\right). \\ \end{array}
$$
Using  $x^{t + 1} - y^{t + 1} = \theta_{t + 1}\left(u^{t + 1} - w^t\right)$ , we get (we mark the changes with color)
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f (z ^ {t + 1}) - f (x ^ {*}) \right] + \kappa \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right] + \nu_ {t} \mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right] \\ + \rho \mathbb {E} _ {t} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| u ^ {t} - x ^ {*} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - x ^ {*} \right\| ^ {2} \right]\right) \\ + p \left(\frac {4 \omega L _ {\max}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\max} + \nu_ {t} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \right. \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\max }}{n}\right) + \theta_ {t + 1} - 1\left. \right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \theta_ {t + 1} ^ {2} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \lambda \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - w ^ {t} \| ^ {2} \right] \\ - \frac {p \theta_ {t + 1} ^ {2} \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] \\ + \nu_ {t} \left(1 - \frac {\alpha}{2}\right) \left\| w ^ {t} - u ^ {t} \right\| ^ {2} \\ \left. + \left(\nu_ {t} \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \rho (1 - p)\right) \| k ^ {t} - \nabla f (z ^ {t}) \| ^ {2} \right. \\ + \left(\rho 2 p + \lambda \left(1 - \frac {\tau}{2}\right)\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ \left. \right. + \left(\frac {2 p \omega}{n \bar {L}} + \nu_ {t} \frac {2 \omega}{n} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \lambda \frac {2 \tau^ {2} \omega}{n} + \kappa \left(1 - \frac {\beta}{2}\right)\right)\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2}\right). \\ \end{array}
$$
The inequality (17) implies  $\left\| u^{t + 1} - w^t\right\|^2\leq 2\left\| u^{t + 1} - u^t\right\|^2 +2\left\| u^t -w^t\right\|^2$  and
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f (z ^ {t + 1}) - f (x ^ {*}) \right] + \kappa \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right] + \nu_ {t} \mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right] \\ + \rho \mathbb {E} _ {t} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| u ^ {t} - x ^ {*} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - x ^ {*} \right\| ^ {2} \right]\right) \\ + p \left(\frac {4 \omega L _ {\max}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\max} + \nu_ {t} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \right. \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\max }}{n}\right) + \theta_ {t + 1} - 1\left. \right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + 2 \theta_ {t + 1} ^ {2} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \lambda \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - u ^ {t} \| ^ {2} \right] \\ - \frac {p \theta_ {t + 1} ^ {2} \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \left. + \left(2 \theta_ {t + 1} ^ {2} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \lambda \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) + \nu_ {t} \left(1 - \frac {\alpha}{2}\right)\right) \| w ^ {t} - u ^ {t} \| ^ {2} \right. \\ \left. + \left(\nu_ {t} \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \rho (1 - p)\right) \| k ^ {t} - \nabla f (z ^ {t}) \| ^ {2} \right. \\ + \left(\rho 2 p + \lambda \left(1 - \frac {\tau}{2}\right)\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ \left. \right. + \left(\frac {2 p \omega}{n \bar {L}} + \nu_ {t} \frac {2 \omega}{n} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \lambda \frac {2 \tau^ {2} \omega}{n} + \kappa \left(1 - \frac {\beta}{2}\right)\right)\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2}\right). \\ \end{array}
$$
Now, we want to find appropriate  $\kappa$ ,  $\rho$ ,  $\lambda$ , and  $\nu_{t}$  such that
$$
\begin{array}{l} 2 \theta_ {t + 1} ^ {2} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \lambda \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) + \nu_ {t} \left(1 - \frac {\alpha}{2}\right) \leq \nu_ {t} \left(1 - \frac {\alpha}{4}\right), \\ \nu_ {t} \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \rho (1 - p) \leq \rho \left(1 - \frac {p}{2}\right), \\ \rho 2 p + \lambda \left(1 - \frac {\tau}{2}\right) \leq \lambda \left(1 - \frac {\tau}{4}\right), \\ \frac {2 p \omega}{n \bar {L}} + \nu_ {t} \frac {2 \omega}{n} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} + \lambda \frac {2 \tau^ {2} \omega}{n} + \kappa \left(1 - \frac {\beta}{2}\right) \leq \kappa \left(1 - \frac {\beta}{4}\right). \tag {46} \\ \end{array}
$$
We analyze inequalities (46) in the following lemma:
Lemma E.10 (First Symbolically Computed). Assume that for the parameter  $\bar{L}$ , the inequalities from Sections I and J hold. Then, for all  $t \geq 0$ , exists  $\rho$  in (90),  $\kappa$  in (88),  $\lambda$  in (82), and  $\nu_t$  in (83) such that (46) holds.
We proof lemma separately in Section G. Using the lemma, we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f (z ^ {t + 1}) - f (x ^ {*}) \right] + \kappa \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right] + \nu_ {t} \mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right] \\ + \rho \mathbb {E} _ {t} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f (z ^ {t}) - f \left(x ^ {*}\right)\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| u ^ {t} - x ^ {*} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - x ^ {*} \right\| ^ {2} \right]\right) \\ + p \left(\frac {4 \omega L _ {\max}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\max} + \nu_ {t} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \right. \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\max}}{n}\right) + \theta_ {t + 1} - 1\left. \right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + 2 \theta_ {t + 1} ^ {2} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \lambda \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - u ^ {t} \| ^ {2} \right] \\ - \frac {p \theta_ {t + 1} ^ {2} \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] \\ + \nu_ {t} \left(1 - \frac {\alpha}{4}\right) \left\| w ^ {t} - u ^ {t} \right\| ^ {2} \\ + \rho \left(1 - \frac {p}{2}\right) \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + \lambda \left(1 - \frac {\tau}{4}\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + \kappa \left(1 - \frac {\beta}{4}\right) \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t}\right) \right\| ^ {2}\right). \\ \end{array}
$$
Let us separately analyze the terms w.r.t.  $D_{f}(z^{t},y^{t + 1})$  and  $\mathbb{E}_t\left[\| u^{t + 1} - u^t\| ^2\right]$
Lemma E.11 (Second Symbolically Computed). Consider the parameters  $\rho$ ,  $\kappa$ ,  $\lambda$ , and  $\nu_{t}$  from Lemma E.10. Assume that for the parameter  $\bar{L}$ , the inequalities from Sections  $L$  and  $N$  hold, and the step size  $\theta_{t+1} \leq 1/4$  for all  $t \geq 0$ . Then, for all  $t \geq 0$ , the following inequalities are satisfied:
$$
\begin{array}{l} p \left(\frac {4 \omega L _ {\max}}{n \bar {L}} + \kappa 8 \left(1 + \frac {p}{\beta}\right) L _ {\max} + \nu_ {t} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{p n} + \frac {8 L}{p \alpha}\right) + \rho 8 L + \right. \tag {47} \\ + \lambda \left(4 \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 \tau^ {2} \omega L _ {\max}}{n}\right) + \theta_ {t + 1} - 1\left. \right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \leq 0 \\ \end{array}
$$
$$
\begin{array}{l} 2 \theta_ {t + 1} ^ {2} \left(\frac {p L}{2} + \kappa 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} + \rho 4 p L ^ {2} + \lambda \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right)\right) \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - u ^ {t} \| ^ {2} \right] \\ - \frac {p \theta_ {t + 1} ^ {2} \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] \leq 0. \tag {48} \\ \end{array}
$$
We prove Lemma E.11 in Section H. Using the lemma, we get
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f \left(z ^ {t + 1}\right) - f \left(x ^ {*}\right) \right] + \kappa \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t + 1} - \nabla f _ {i} \left(z ^ {t + 1}\right) \right| \right| ^ {2} \right] + \nu_ {t} \mathbb {E} _ {t} \left[ \left| \left| w ^ {t + 1} - u ^ {t + 1} \right| \right| ^ {2} \right] \\ + \rho \mathbb {E} _ {t} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| u ^ {t} - x ^ {*} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - x ^ {*} \| ^ {2} \right]\right) \\ + \nu_ {t} \left(1 - \frac {\alpha}{4}\right) \left\| w ^ {t} - u ^ {t} \right\| ^ {2} \\ + \rho \left(1 - \frac {p}{2}\right) \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + \lambda \left(1 - \frac {\tau}{4}\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + \kappa \left(1 - \frac {\beta}{4}\right) \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2}\right). \\ \end{array}
$$
Note that  $0 \leq \nu_{t+1} \leq \nu_t$  for all  $t \geq 0$ , since  $\theta_{t+1}$  is a non-increasing sequence (see Lemma E.1 and the definition of  $\nu_t$  in (83)). Using
$$
\theta_ {t + 1} \leq \frac {1}{4} \min  \left\{1, \frac {\alpha}{p}, \frac {\tau}{p}, \frac {\beta}{p} \right\}
$$
for all  $t\geq 0$  , we obtain
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f (z ^ {t + 1}) - f (x ^ {*}) \right] + \kappa \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right] + \nu_ {t + 1} \mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right] \\ + \rho \mathbb {E} _ {t} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f (z ^ {t}) - f (x ^ {*}) + \kappa \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2}\right) + \nu_ {t} \| w ^ {t} - u ^ {t} \| ^ {2} \right. \\ \left. + \rho \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} + \lambda \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2}\right) \\ \end{array}
$$
$$
\left. + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| u ^ {t} - x ^ {*} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - x ^ {*} \| ^ {2} \right]\right)\right).
$$
Let us multiply the inequality by  $\frac{\gamma_{t + 1}}{p\theta_{t + 1}}$
$$
\begin{array}{l} \frac {\gamma_ {t + 1}}{p \theta_ {t + 1}} \left(\mathbb {E} _ {t} [ f (z ^ {t + 1}) - f (x ^ {*}) ] + \kappa \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \| ^ {2} \right] + \nu_ {t + 1} \mathbb {E} _ {t} [ \| w ^ {t + 1} - u ^ {t + 1} \| ^ {2} ] \right. \\ \left. + \rho \mathbb {E} _ {t} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right]\right) \\ \leq \left(\frac {\gamma_ {t + 1}}{p \theta_ {t + 1}} - \gamma_ {t + 1}\right) \left(f (z ^ {t}) - f (x ^ {*}) + \kappa \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \| ^ {2}\right) + \nu_ {t} \| w ^ {t} - u ^ {t} \| ^ {2} \right. \\ \left. + \rho \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} + \lambda \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2}\right) \\ \left. \right. + \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2} \| u ^ {t} - x ^ {*} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2} \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - x ^ {*} \| ^ {2} \right]\right). \\ \end{array}
$$
It is left to use that  $\Gamma_{t + 1} \coloneqq \Gamma_t + \gamma_{t + 1}$  and  $\gamma_{t + 1} = p\theta_{t + 1}\Gamma_{t + 1}$  (see Lemma E.1) and take the full expectation to obtain (45).
In the proof we require that, for the parameter  $\bar{L}$ , the inequalities from Sections I, J, L and N hold. In Section O, we show that these inequalities follow from (44).
E.5 Strongly-convex case
Theorem E.12. Suppose that Assumptions 1.2, 1.1, 1.3, 2.4 hold. Let
$$
\bar {L} = 6 6 0 5 0 8 \times \max  \left\{\frac {L}{\alpha}, \frac {L p}{\alpha \tau}, \frac {\sqrt {L L _ {\operatorname* {m a x}}} p \sqrt {\omega \tau}}{\alpha \beta \sqrt {n}}, \frac {\sqrt {L L _ {\operatorname* {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {\beta} \sqrt {n}}, \frac {L _ {\operatorname* {m a x}} \omega p ^ {2}}{\beta^ {2} n}, \frac {L _ {\operatorname* {m a x}} \omega}{n} \right\}, \tag {49}
$$
$\beta = \frac{1}{\omega + 1}$ ,  $\theta_{\mathrm{min}} = \frac{1}{4}\min \left\{1,\frac{\alpha}{p},\frac{\tau}{p},\frac{\beta}{p}\right\}$ ,  $h_i^0 = \nabla f_i(z^0)$  for all  $i\in [n]$ ,  $w^0 = u^0$ ,  $k^0 = \nabla f(z^0)$ ,  $v^{0} = \nabla f(z^{0})$ , and  $\Gamma_0\geq 1$ . Then Algorithm 1 guarantees that
$$
\mathbb {E} \left[ f \left(z ^ {T}\right) - f \left(x ^ {*}\right) \right] + \frac {\mu}{2} \mathbb {E} \left[ \left\| u ^ {T} - x ^ {*} \right\| ^ {2} \right] \leq 2 \exp \left(- \frac {T}{Q}\right) \left(\left(f \left(z ^ {0}\right) - f \left(x ^ {*}\right)\right) + \left(\frac {\bar {L}}{\Gamma_ {0}} + \mu\right) \left\| u ^ {0} - x ^ {*} \right\| ^ {2}\right), \tag {50}
$$
$$
\begin{array}{l} Q := 2 \times \sqrt {6 6 0 5 0 8} \times \\ \max  \left\{\sqrt {\frac {L}{\alpha p \mu}}, \sqrt {\frac {L}{\alpha \tau \mu}}, \sqrt {\frac {\sqrt {L L _ {\operatorname* {m a x}}} (\omega + 1) \sqrt {\omega \tau}}{\alpha \sqrt {n} \mu}}, \sqrt {\frac {\sqrt {L L _ {\operatorname* {m a x}}} \sqrt {\omega + 1} \sqrt {\omega \tau}}{\alpha \sqrt {p} \sqrt {n} \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega (\omega + 1) ^ {2} p}{n \mu}}, \sqrt {\frac {L _ {\operatorname* {m a x}} \omega}{n p \mu}}, \right. \\ \left. \frac {1}{\alpha}, \frac {1}{\tau}, (\omega + 1), \frac {1}{p} \right\}. \\ \end{array}
$$
Remark E.13. Up to a constant factor of 2, one can see that the optimal  $\Gamma_0$  in (50) from Theorem E.12 equals  $\Gamma_0 = \bar{L} /\mu$ . But the dependence on  $\Gamma_0$  is under the logarithm, so if the dependence on the logarithm is not critical, one can take any  $\Gamma_0\geq 1$ .
Proof. All conditions from Theorem E.9 are satisfied. Let us sum the inequality (45) for  $t = 0$  to  $T - 1$ :
$$
\begin{array}{l} \Gamma_ {T} \left(\mathbb {E} \left[ f (z ^ {T}) - f (x ^ {*}) \right] + \kappa \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| h _ {i} ^ {T} - \nabla f _ {i} (z ^ {T}) \| ^ {2} \right] + \nu_ {T} \mathbb {E} \left[ \| w ^ {T} - u ^ {T} \| ^ {2} \right] \right. \\ \left. + \rho \mathbb {E} \left[ \left\| k ^ {T} - \nabla f (z ^ {T}) \right\| ^ {2} \right] + \lambda \mathbb {E} \left[ \left\| v ^ {T} - \nabla f (z ^ {T}) \right\| ^ {2} \right]\right) \\ \end{array}
$$