SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Due to the independence of compressors, we can use Lemma 1 with  $r_i = g_i^t - h_i^t$  and  $s_i =$
$p_{\mathrm{a}}\mathcal{C}_i\left(\frac{1}{p_{\mathrm{a}}} k_i^{t + 1} - \frac{a}{p_{\mathrm{a}}}\left(g_i^t -h_i^t\right)\right) - k_i^{t + 1},$  and obtain
$$
\begin{array}{l} \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left| \left| g ^ {t + 1} - h ^ {t + 1} \right| \right| ^ {2} \right] \right] \\ \leq \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \mathrm {E} _ {\mathcal {C}} \left[ \left\| p _ {\mathrm {a}} \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) - k _ {i} ^ {t + 1} - \mathrm {E} _ {\mathcal {C}} \left[ p _ {\mathrm {a}} \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) - k _ {i} ^ {t + 1} \right] \right\| ^ {2} \right] \\ + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \operatorname {E} _ {\mathcal {C}} \left[ p _ {\mathrm {a}} \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) - k _ {i} ^ {t + 1} \right] \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \\ = \frac {p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathrm {E} _ {\mathcal {C}} \left[ \left\| \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) - \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) \right\| ^ {2} \right] \\ + \frac {a ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2}. \\ \end{array}
$$
From Assumption 7, we have
$$
\begin{array}{l} \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] \right] \\ \leq \frac {\omega p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) \right\| ^ {2} + \frac {a ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \\ = \frac {\omega}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} - a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) \right\| ^ {2} + \frac {a ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \\ \stackrel {(1 5)} {\leq} \frac {2 \omega}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} + \frac {a ^ {2} ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2}. \\ \end{array}
$$
The second inequality can be proved almost in the same way:
$$
\begin{array}{l} \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \right] \\ = \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} - \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right] \right] \right\| ^ {2} \right] \right] + \left\| \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right] \right] \right\| ^ {2} \\ = \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} - g _ {i} ^ {t} + a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) + h _ {i} ^ {t} \right\| ^ {2} \right] \right] + (1 - a) ^ {2} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ = p _ {\mathrm {a}} \mathrm {E} _ {\mathcal {C}} \left[ \left\| \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) - \frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} + a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) \right\| ^ {2} \right] \\ + a ^ {2} \left(1 - p _ {\mathrm {a}}\right) \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ \stackrel {(1 6)} {=} p _ {\mathrm {a}} \mathrm {E} _ {\mathcal {C}} \left[ \left\| \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) - \left(\frac {1}{p _ {\mathrm {a}}} k _ {i} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) \right\| ^ {2} \right] \\ + a ^ {2} \frac {(1 - p _ {\mathrm {a}}) ^ {2}}{p _ {\mathrm {a}}} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ + a ^ {2} \left(1 - p _ {\mathrm {a}}\right) \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ \leq \frac {\omega}{p _ {\mathrm {a}}} \left\| k _ {i} ^ {t + 1} - a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) \right\| ^ {2} \\ + \frac {a ^ {2} (1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ \stackrel {(1 5)} {\leq} \frac {2 \omega}{p _ {\mathrm {a}}} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} + \frac {a ^ {2} (2 \omega + 1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2}. \\ \end{array}
$$
Lemma 6. Suppose that Assumptions 2, 7, and 8 hold and let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ , then
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t + 1} - h ^ {t + 1} \| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \| \nabla f (x ^ {t}) \| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \gamma \| h ^ {t} - \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] + \frac {4 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
Proof. Due to Lemma 2 and the update step from Line 5 in Algorithm 1, we have
$$
\begin{array}{l} \mathbf {E} _ {t + 1} \left[ f (x ^ {t + 1}) \right] \\ \leq \operatorname {E} _ {t + 1} \left[ f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {\gamma}{2} \left\| g ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ = \operatorname {E} _ {t + 1} \left[ f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {\gamma}{2} \left\| g ^ {t} - h ^ {t} + h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ \stackrel {(1 6)} {\leq} \operatorname {E} _ {t + 1} \left[ f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \gamma \left(\left\| g ^ {t} - h ^ {t} \right\| ^ {2} + \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \right). \\ \end{array}
$$
Let us fix some constants  $\kappa, \eta \in [0, \infty)$  that we will define later. Combining the last inequality, bounds (22), (23) and using the law of total expectation, we get
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] \\ + \kappa \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \eta \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ = \operatorname {E} \left[ \operatorname {E} _ {t + 1} \left[ f \left(x ^ {t + 1}\right) \right] \right] \\ + \kappa \mathrm {E} \left[ \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] \right] \right] + \eta \mathrm {E} \left[ \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \right] \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \gamma \left(\left\| g ^ {t} - h ^ {t} \right\| ^ {2} + \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2}\right) \right] \\ + \kappa \mathrm {E} \left[ \frac {2 \omega}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} + \frac {a ^ {2} ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] \\ + \eta \mathrm {E} \left[ \frac {2 \omega}{n p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} + \left(\frac {a ^ {2} (2 \omega + 1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - a) ^ {2}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ = \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \gamma \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\gamma + \kappa (1 - a) ^ {2}\right) \operatorname {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] \\ + \left(\frac {\kappa a ^ {2} ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} + \eta \left(\frac {a ^ {2} (2 \omega + 1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - a) ^ {2}\right)\right) \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \left(\frac {2 \kappa \omega}{n p _ {\mathrm {a}}} + \frac {2 \eta \omega}{p _ {\mathrm {a}}}\right) \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
Now, by taking  $\kappa = \frac{\gamma}{a}$ , we can see that  $\gamma + \kappa (1 - a)^2 \leq \kappa$ , and thus
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] \\ + \frac {\gamma}{a} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \eta \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \gamma \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \frac {\gamma}{a} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] \\ + \left(\frac {\gamma a ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} + \eta \left(\frac {a ^ {2} (2 \omega + 1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - a) ^ {2}\right)\right) \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \left(\frac {2 \gamma \omega}{a n p _ {\mathrm {a}}} + \frac {2 \eta \omega}{p _ {\mathrm {a}}}\right) \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
Next, by taking  $\eta = \frac{\gamma((2\omega + 1)p_{\mathrm{a}} - p_{\mathrm{aa}})}{np_{\mathrm{a}}^2}$  and considering the choice of  $a$ , one can show that  $\left(\frac{\gamma a((2\omega + 1)p_{\mathrm{a}} - p_{\mathrm{aa}})}{np_{\mathrm{a}}^2} + \eta \left(\frac{a^2(2\omega + 1 - p_{\mathrm{a}})}{p_{\mathrm{a}}} + (1 - a)^2\right)\right) \leq \eta$ . Thus
E  $[f(x^{t + 1})]$
$$
\begin{array}{l} + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \| \nabla f (x ^ {t}) \| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \gamma \| h ^ {t} - \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \left(\frac {2 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} + \frac {2 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \omega}{n p _ {\mathrm {a}} ^ {3}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| k _ {i} ^ {t + 1} \| ^ {2} \right]. \\ \end{array}
$$
Considering that  $p_{\mathrm{aa}} \geq 0$ , we can simplify the last term and get
$$
\begin{array}{l} + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ \leq \mathrm {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \| \nabla f (x ^ {t}) \| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \gamma \| h ^ {t} - \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {4 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
E.3 Proof for DASHA-PP
Lemma 7. Suppose that Assumptions 3 and 8 hold. For  $h_i^{t+1}$  and  $k_i^{t+1}$  from Algorithm 1 (DASHA-PP) we have
$$
\begin{array}{l} \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left| \left| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right| \right| ^ {2} \right] \\ \leq \frac {2 (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {2 b ^ {2} (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} + (1 - b) ^ {2} \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2}. \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left| \left| h _ {i} ^ {t + 1} - \nabla f _ {i} \left(x ^ {t + 1}\right) \right| \right| ^ {2} \right] \\ \leq \frac {2 (1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} L _ {i} ^ {2} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \left(\frac {2 b ^ {2} (1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - b) ^ {2}\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2}, \quad \forall i \in [ n ]. \\ \end{array}
$$
$$
\left\| k _ {i} ^ {t + 1} \right\| ^ {2} \leq 2 L _ {i} ^ {2} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + 2 b ^ {2} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2}, \quad \forall i \in [ n ].
$$
Proof. First, let us proof the bound for  $\mathrm{E}_k\left[\mathrm{E}_{p_{\mathrm{a}}}\left[\left\| h^{t + 1} - \nabla f(x^{t + 1})\right\| ^2\right]\right]$ :
$$
\mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right]
$$
$$
= \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \left\| h ^ {t + 1} - \operatorname {E} _ {p _ {\mathrm {a}}} \left[ h ^ {t + 1} \right] \right\| ^ {2} \right] + \left\| \operatorname {E} _ {p _ {\mathrm {a}}} \left[ h ^ {t + 1} \right] - \nabla f (x ^ {t + 1}) \right\| ^ {2}.
$$
$$
\mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i} ^ {t + 1} \right] = h _ {i} ^ {t} + \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - b \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right)
$$
and (16), we have
$$
\begin{array}{l} \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \\ = \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \left\| h ^ {t + 1} - \operatorname {E} _ {p _ {\mathrm {a}}} \left[ h ^ {t + 1} \right] \right\| ^ {2} \right] + (1 - b) ^ {2} \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2}. \\ \end{array}
$$
We can use Lemma 1 with  $r_i = h_i^t$  and  $s_i = k_i^{t + 1}$  to obtain
$$
\begin{array}{l} \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} - k _ {i} ^ {t + 1} \right\| ^ {2} + \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} + (1 - b) ^ {2} \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \\ = \frac {p _ {\mathrm {a}} - p _ {\mathrm {a a}}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) - b \left(h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t})\right) \right\| ^ {2} + (1 - b) ^ {2} \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(1 5)} {\leq} \frac {2 (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} + \frac {2 b ^ {2} (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} + (1 - b) ^ {2} \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \\ \leq \frac {2 (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {2 b ^ {2} (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} + (1 - b) ^ {2} \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2}. \\ \end{array}
$$
In the last in inequality, we used Assumption 3. Now, we prove the second inequality:
$$
\begin{array}{l} \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ = \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i} ^ {t + 1} - \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i} ^ {t + 1} \right] \right\| ^ {2} \right] + \left\| \mathrm {E} _ {p _ {\mathrm {a}}} \left[ h _ {i} ^ {t + 1} \right] - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \\ = \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \left\| h _ {i} ^ {t + 1} - \left(h _ {i} ^ {t} + \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) - b (h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}))\right) \right\| ^ {2} \right] + (1 - b) ^ {2} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ = \frac {\left(1 - p _ {\mathrm {a}}\right) ^ {2}}{p _ {\mathrm {a}}} \left| \left| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) - b \left(h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right| \right| ^ {2} \\ + \left(1 - p _ {\mathrm {a}}\right) \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) - b (h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t})) \right\| ^ {2} + (1 - b) ^ {2} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ = \frac {(1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} \left\| \nabla f _ {i} (x ^ {t + 1}) - \nabla f _ {i} (x ^ {t}) - b (h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t})) \right\| ^ {2} + (1 - b) ^ {2} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \\ \leq \frac {2 \left(1 - p _ {\mathrm {a}}\right)}{p _ {\mathrm {a}}} L _ {i} ^ {2} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \left(\frac {2 b ^ {2} \left(1 - p _ {\mathrm {a}}\right)}{p _ {\mathrm {a}}} + (1 - b) ^ {2}\right) \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2}. \\ \end{array}
$$
Finally, the third inequality of the theorem follows from (15) and Assumption 3.
Theorem 2. Suppose that Assumptions 1, 2, 3, 7 and 8 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b = \frac{p_{\mathrm{a}}}{2 - p_{\mathrm{a}}}$ ,
Proof. Let us fix constants  $\nu, \rho \in [0, \infty)$  that we will define later. Considering Lemma 6, Lemma 7, and the law of total expectation, we obtain
$$
\operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right]
$$