SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t + 1} - \nabla f _ {i} \left(z ^ {t + 1}\right) \right| \right| ^ {2} \right] \\ \leq (1 - \beta) p \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} \right] + (1 - \beta) (1 - p) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t}\right) \right\| ^ {2} \\ \stackrel {(1 7)} {\leq} \left(1 + \frac {2 p}{\beta}\right) (1 - \beta) p \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| f _ {i} \left(z ^ {t}\right) - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} \right] + \left(1 + \frac {\beta}{2 p}\right) (1 - \beta) p \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t}\right) \right\| ^ {2} \\ + (1 - \beta) (1 - p) \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t}\right) \right\| ^ {2} \right] \\ = \left(1 + \frac {2 p}{\beta}\right) (1 - \beta) p \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| f _ {i} \left(z ^ {t}\right) - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} \right] + (1 - \beta) \left(1 + \frac {\beta}{2}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t}\right) \right\| ^ {2}. \\ \end{array}
$$
Using  $1 - \beta \leq 1$ , (18) and (20), we get
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t + 1} - \nabla f _ {i} \left(z ^ {t + 1}\right) \right| \right| ^ {2} \right] \\ \leq 4 p \left(1 + \frac {p}{\beta}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| f _ {i} \left(z ^ {t}\right) - \nabla f _ {i} \left(y ^ {t + 1}\right) \right\| ^ {2} + 4 p \left(1 + \frac {p}{\beta}\right) \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(y ^ {t + 1}\right) \right\| ^ {2} \right] \\ + \left(1 - \frac {\beta}{2}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2}. \\ \end{array}
$$
From Assumptions 1.1 and 1.3 and Lemma D.1, we obtain
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t + 1} - \nabla f _ {i} \left(z ^ {t + 1}\right) \right| \right| ^ {2} \right] \\ \leq 8 p \left(1 + \frac {p}{\beta}\right) L _ {\max } \left(f (z ^ {t}) - f (y ^ {t + 1}) - \left\langle \nabla f (y ^ {t + 1}), z ^ {t} - y ^ {t + 1} \right\rangle\right) + 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] \\ + \left(1 - \frac {\beta}{2}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \\ = 8 p \left(1 + \frac {p}{\beta}\right) L _ {\max } D _ {f} (z ^ {t}, y ^ {t + 1}) + 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] + \left(1 - \frac {\beta}{2}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2}. \\ \end{array}
$$
Lemma E.6. Suppose that Assumptions 1.2, 1.1, 1.3 and 2.4 hold. Then, for Algorithm 1, the following inequality holds:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right] \leq \left(1 - \frac {\alpha}{2}\right) \left\| w ^ {t} - u ^ {t} \right\| ^ {2} + \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + \frac {2 \omega}{n} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(z ^ {t}\right) - h _ {i} ^ {t} \right\| ^ {2} + \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max }}{n} + \frac {8 L}{\alpha}\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right). \tag {41} \\ \end{array}
$$
Proof. Using the definition of  $w^{t + 1}$  and Definition 2.2, we get the following inequality
$$
\mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right] = \mathbb {E} _ {t} \left[ \left\| \mathcal {C} ^ {P} (u ^ {t + 1} - q ^ {t + 1}) - (u ^ {t + 1} - q ^ {t + 1}) \right\| ^ {2} \right] \leq (1 - \alpha) \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - q ^ {t + 1} \right\| ^ {2} \right].
$$
We can find the analytical formulas for  $u^{t + 1}$  and  $q^{t + 1}$ , and obtain that
$$
u ^ {t + 1} = \frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu} u ^ {t} + \frac {\mu \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} y ^ {t + 1} - \frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} g ^ {t + 1}
$$
$$
q ^ {t + 1} = \frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu} w ^ {t} + \frac {\mu \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} y ^ {t + 1} - \frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} k ^ {t}.
$$
$$
\mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right]
$$
$$
\leq (1 - \alpha) \mathbb {E} _ {t} \left[ \left\| \frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu} \left(w ^ {t} - u ^ {t}\right) - \frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \left(k ^ {t} - g ^ {t + 1}\right) \right\| ^ {2} \right]
$$
$$
\stackrel {(2 3)} {=} (1 - \alpha) \left\| \frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu} \left(w ^ {t} - u ^ {t}\right) - \frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \left(k ^ {t} - \nabla f \left(y ^ {t + 1}\right)\right) \right\| ^ {2}
$$
$$
+ (1 - \alpha) \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \mathbb {E} _ {t} \left[ \left\| g ^ {t + 1} - \nabla f \left(y ^ {t + 1}\right) \right\| ^ {2} \right]
$$
$$
\begin{array}{l} \stackrel {(1 7)} {\leq} \left(1 - \frac {\alpha}{2}\right) \left(\frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \| w ^ {t} - u ^ {t} \| ^ {2} + \frac {2}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \| k ^ {t} - \nabla f (y ^ {t + 1}) \| ^ {2} \\ + (1 - \alpha) \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \mathbb {E} _ {t} \left[ \left\| g ^ {t + 1} - \nabla f \left(y ^ {t + 1}\right) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(1 8)} {\leq} \left(1 - \frac {\alpha}{2}\right) \left(\frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \| w ^ {t} - u ^ {t} \| ^ {2} \\ + \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \| k ^ {t} - \nabla f (z ^ {t}) \| ^ {2} \\ + \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left\| \nabla f (z ^ {t}) - \nabla f (y ^ {t + 1}) \right\| ^ {2} \\ + (1 - \alpha) \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \mathbb {E} _ {t} \left[ \left\| g ^ {t + 1} - \nabla f \left(y ^ {t + 1}\right) \right\| ^ {2} \right]. \\ \end{array}
$$
One can substitute (35) from Lemma E.3 to the last inequality and get
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| w ^ {t + 1} - u ^ {t + 1} \right| \right| ^ {2} \right] \\ \leq \left(1 - \frac {\alpha}{2}\right) \left(\frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \| w ^ {t} - u ^ {t} \| ^ {2} \\ + \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} + \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left\| \nabla f (z ^ {t}) - \nabla f (y ^ {t + 1}) \right\| ^ {2} \\ + (1 - \alpha) \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {2 \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (z ^ {t}) - h _ {i} ^ {t} \right\| ^ {2} + \frac {4 \omega L _ {\max}}{n} D _ {f} (z ^ {t}, y ^ {t + 1})\right). \\ \end{array}
$$
Using Lemma D.1 and  $1 - \alpha \leq 1$ , we obtain
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| w ^ {t + 1} - u ^ {t + 1} \right| \right| ^ {2} \right] \\ \leq \left(1 - \frac {\alpha}{2}\right) \left(\frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \| w ^ {t} - u ^ {t} \| ^ {2} \\ + \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + \frac {2 \omega}{n} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (z ^ {t}) - h _ {i} ^ {t} \right\| ^ {2} + \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{n} + \frac {8 L}{\alpha}\right) D _ {f} (z ^ {t}, y ^ {t + 1}) \\ \leq \left(1 - \frac {\alpha}{2}\right) \left\| w ^ {t} - u ^ {t} \right\| ^ {2} \\ + \frac {4}{\alpha} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + \frac {2 \omega}{n} \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (z ^ {t}) - h _ {i} ^ {t} \right\| ^ {2} + \left(\frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu}\right) ^ {2} \left(\frac {4 \omega L _ {\max}}{n} + \frac {8 L}{\alpha}\right) D _ {f} (z ^ {t}, y ^ {t + 1}), \\ \end{array}
$$
where we use that  $\Gamma_{t + 1}\geq \Gamma_t$  for all  $t\geq 0$
Lemma E.7. Suppose that Assumptions 1.2 and 1.3 hold. Then, for Algorithm 1, the following inequality holds:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right| \right| ^ {2} \right] \\ \leq 2 p \mathbb {E} _ {t} \left[ \left\| v ^ {t} - \nabla f \left(z ^ {t}\right) \right\| ^ {2} \right] + 8 p L D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) + 4 p L ^ {2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] + (1 - p) \left\| k ^ {t} - \nabla f \left(z ^ {t}\right) \right\| ^ {2}. \tag {42} \\ \end{array}
$$
Proof. Note that  $k^{t + 1}$  and  $z^{t + 1}$  are coupled by the same random variable  $c^t$ . Therefore,
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right| \right| ^ {2} \right] \\ = p \mathbb {E} _ {t} \left[ \left\| v ^ {t} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + (1 - p) \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ \leq 2 p \mathbb {E} _ {t} \left[ \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \right] + 4 p \left\| \nabla f (z ^ {t}) - \nabla f (y ^ {t + 1}) \right\| ^ {2} + 4 p \mathbb {E} _ {t} \left[ \left\| \nabla f (y ^ {t + 1}) - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \\ + (1 - p) \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ \leq 2 p \mathbb {E} _ {t} \left[ \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \right] + 8 p L D _ {f} (z ^ {t}, y ^ {t + 1}) + 4 p L ^ {2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] + (1 - p) \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2}, \\ \end{array}
$$
where we use Assumptions 1.2 and Lemma D.1.
Lemma E.8. Suppose that Assumptions 1.2, 1.1 and 1.3 hold. The momentum  $\tau \in (0,1]$  and the probability  $p \in (0,1]$ . Then, for Algorithm 1, the following inequality holds:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| v ^ {\ell + 1} - \nabla f (z ^ {\ell + 1}) \right| \right| ^ {2} \right] \\ \leq \left(1 - \frac {\tau}{2}\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} + \frac {2 \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \\ + \left(4 p \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 p \tau^ {2} \omega L _ {\max }}{n}\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) + \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right) \mathbb {E} _ {t} \left[ \| x ^ {t + 1} - y ^ {t + 1} \| ^ {2} \right]. \tag {43} \\ \end{array}
$$
Proof. Using the definition of  $v^{t + 1}$ , we get
$$
\mathbb {E} _ {t} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] = \mathbb {E} _ {t} \left[ \left\| (1 - \tau) v ^ {t} + \tau \left(h ^ {t} + \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {C} _ {i} ^ {D, z} (\nabla f _ {i} (z ^ {t + 1}) - h _ {i} ^ {t})\right) - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right].
$$
Assumption 2.4, including the independence and the unbiasedness of the compressors, insures that
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right| \right| ^ {2} \right] \\ \stackrel {(2 3)} {=} (1 - \tau) ^ {2} \mathbb {E} _ {t} \left[ \left\| v ^ {t} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \tau^ {2} \mathbb {E} _ {t} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {C} _ {i} ^ {D, z} (\nabla f _ {i} (z ^ {t + 1}) - h _ {i} ^ {t}) - (\nabla f (z ^ {t + 1}) - h ^ {t}) \right\| ^ {2} \right] \\ = (1 - \tau) ^ {2} \mathbb {E} _ {t} \left[ \left\| v ^ {t} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \frac {\tau^ {2}}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| \mathcal {C} _ {i} ^ {D, z} \left(\nabla f _ {i} (z ^ {t + 1}) - h _ {i} ^ {t}\right) - (\nabla f (z ^ {t + 1}) - h ^ {t}) \right\| ^ {2} \right] \\ \leq (1 - \tau) ^ {2} \mathbb {E} _ {t} \left[ \left\| v ^ {t} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \frac {\tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| \nabla f _ {i} (z ^ {t + 1}) - h _ {i} ^ {t} \right\| ^ {2} \right]. \\ \end{array}
$$
Using the definition of  $z^{t + 1}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right| \right| ^ {2} \right] \\ \leq (1 - \tau) ^ {2} (1 - p) \left| \left| v ^ {t} - \nabla f (z ^ {t}) \right| \right| ^ {2} + (1 - \tau) ^ {2} p \mathbb {E} _ {t} \left[ \left| \left| v ^ {t} - \nabla f (x ^ {t + 1}) \right| \right| ^ {2} \right] \\ + \frac {(1 - p) \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} + \frac {p \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \end{array}
$$
(17),(18)
$$
\begin{array}{l} \leq (1 - \tau) ^ {2} (1 - p) \| v ^ {t} - \nabla f (z ^ {t}) \| ^ {2} \\ + (1 - \tau) ^ {2} \left(1 + \frac {\tau}{2 p}\right) p \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} + (1 - \tau) ^ {2} \left(1 + \frac {2 p}{\tau}\right) p \mathbb {E} _ {t} \left[ \left\| \nabla f (z ^ {t}) - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \\ + \frac {(1 - p) \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \\ + \frac {2 p \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \right] + \frac {2 p \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| \nabla f _ {i} (z ^ {t}) - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ = (1 - \tau) ^ {2} \left(1 + \frac {\tau}{2}\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} + (1 - \tau) ^ {2} \left(1 + \frac {2 p}{\tau}\right) p \mathbb {E} _ {t} \left[ \left\| \nabla f (z ^ {t}) - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \\ + \frac {(1 + p) \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} + \frac {2 p \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| \nabla f _ {i} (z ^ {t}) - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right]. \\ \end{array}
$$
Using  $0 \leq 1 - \tau \leq 1, p \in (0,1]$  and (20), we get
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right| \right| ^ {2} \right] \\ \leq \left(1 - \frac {\tau}{2}\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} + \left(1 + \frac {2 p}{\tau}\right) p \mathbb {E} _ {t} \left[ \left\| \nabla f (z ^ {t}) - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \\ + \frac {2 \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} + \frac {2 p \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| \nabla f _ {i} (z ^ {t}) - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(1 8)} {\leq} \left(1 - \frac {\tau}{2}\right) \| v ^ {t} - \nabla f (z ^ {t}) \| ^ {2} \\ + 2 p \left(1 + \frac {2 p}{\tau}\right) \left\| \nabla f (z ^ {t}) - \nabla f (y ^ {t + 1}) \right\| ^ {2} + 2 p \left(1 + \frac {2 p}{\tau}\right) \mathbb {E} _ {t} \left[ \left\| \nabla f (y ^ {t + 1}) - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} + \frac {2 \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \\ + \frac {4 p \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (z ^ {t}) - \nabla f _ {i} (y ^ {t + 1}) \right\| ^ {2} + \frac {4 p \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (y ^ {t + 1}) \right\| ^ {2} \right]. \\ \end{array}
$$
It is left to use Assumptions 1.2 and 1.1 with Lemma D.1 to obtain
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right| \right| ^ {2} \right] \\ \leq \left(1 - \frac {\tau}{2}\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \\ + 4 p \left(1 + \frac {2 p}{\tau}\right) L D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) + 2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} \mathbb {E} _ {t} \left[ \left\| y ^ {t + 1} - x ^ {t + 1} \right\| ^ {2} \right] \\ + \frac {2 \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \\ + \frac {8 p \tau^ {2} \omega L _ {\max}}{n} D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] \\ = \left(1 - \frac {\tau}{2}\right) \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} + \frac {2 \tau^ {2} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \\ \left. \right. + \left(4 p \left(1 + \frac {2 p}{\tau}\right) L + \frac {8 p \tau^ {2} \omega L _ {\max}}{n}\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) + \left(2 p \left(1 + \frac {2 p}{\tau}\right) L ^ {2} + \frac {4 p \tau^ {2} \omega \widehat {L} ^ {2}}{n}\right) \mathbb {E} _ {t} \left[\left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
E.4 Main theorem
Theorem E.9. Suppose that Assumptions 1.2, 1.1, 1.3, 2.4 hold. Let
$$
\bar {L} = 6 6 0 5 0 8 \times \max \left\{\frac {L}{\alpha}, \frac {L p}{\alpha \tau}, \frac {\sqrt {L L _ {\mathrm {m a x}}} p \sqrt {\omega \tau}}{\alpha \beta \sqrt {n}}, \frac {\sqrt {L L _ {\mathrm {m a x}}} \sqrt {p} \sqrt {\omega \tau}}{\alpha \sqrt {\beta} \sqrt {n}}, \frac {L _ {\mathrm {m a x}} \omega p ^ {2}}{\beta^ {2} n}, \frac {L _ {\mathrm {m a x}} \omega}{n} \right\}, (4 4)
$$
$\beta \leq \frac{1}{\omega + 1}$ , and  $\theta_{\mathrm{min}} = \frac{1}{4} \min \left\{1, \frac{\alpha}{p}, \frac{\tau}{p}, \frac{\beta}{p}\right\}$ . For all  $t \geq 0$ , Algorithm 1 guarantees that
$$
\begin{array}{l} \Gamma_ {t + 1} \left(\mathbb {E} [ f (z ^ {t + 1}) - f (x ^ {*}) ] + \kappa \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \| ^ {2} \right] + \nu_ {t + 1} \mathbb {E} \left[ \| w ^ {t + 1} - u ^ {t + 1} \| ^ {2} \right] \right. \\ + \rho \mathbb {E} \left[ \left\| k ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right] + \lambda \mathbb {E} \left[ \left\| v ^ {t + 1} - \nabla f (z ^ {t + 1}) \right\| ^ {2} \right]) + \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2} \mathbb {E} \left[ \left\| u ^ {t + 1} - x ^ {*} \right\| ^ {2} \right] \\ \leq \Gamma_ {t} \left(\mathbb {E} \left[ f (z ^ {t}) - f (x ^ {*}) \right] + \kappa \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \right] + \nu_ {t} \mathbb {E} \left[ \left\| w ^ {t} - u ^ {t} \right\| ^ {2} \right] \right. \\ \left. + \rho \mathbb {E} \left[ \left\| k ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \right] + \lambda \mathbb {E} \left[ \left\| v ^ {t} - \nabla f (z ^ {t}) \right\| ^ {2} \right]\right) + \frac {\bar {L} + \Gamma_ {t} \mu}{2} \mathbb {E} \left[ \left\| u ^ {t} - x ^ {*} \right\| ^ {2} \right] \tag {45} \\ \end{array}
$$
for some  $\kappa, \rho, \lambda, \nu_t \geq 0$ .
Proof. We fix some constants  $\kappa, \rho, \lambda, \nu_t \geq 0$  for all  $t \geq 0$  that we define later. By combining Lemma E.4 with  $\kappa \times (39)$  from Lemma E.5,  $\nu_t \times (41)$  from Lemma E.6,  $\rho \times (42)$  from Lemma E.7, and  $\lambda \times (43)$  from Lemma E.8, we get the following inequality:
$$
\mathbb {E} _ {t} \left[ f (z ^ {t + 1}) - f (x ^ {*}) \right] + \kappa \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right] + \nu_ {t} \mathbb {E} _ {t} \left[ \left\| w ^ {t + 1} - u ^ {t + 1} \right\| ^ {2} \right]
$$