SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Up to Lipschitz constants, DASHA-PP-MVR has the state-of-the-art oracle complexity under PL-condition (see (Li et al., 2021a)). Moreover, DASHA-PP-MVR has the state-of-the-art communication complexity of DASHA for a small enough  $\mu$ .
For simplicity, we omitted  $\frac{d}{\zeta c}$  term from the complexity in the stochastic setting, where  $\zeta_{\mathcal{C}}$  is defined in Definition 12. For instance, for the Rand  $K$  compressor (see Definition 5 and Theorem 6),  $\zeta_{\mathcal{C}} = K$  and  $\frac{d}{\zeta c} = \Theta (\omega)$ .
F.4 Proofs of Theorems
The following proofs almost repeat the proofs from Section E. And one of the main changes is that instead of Lemma 3, we use the following lemma.
F.4.1 Standard Lemma under Polyak-Lojasiewicz Condition
Lemma 11. Suppose that Assumptions 1 and 9 hold and
$$
\operatorname {E} \left[ f (x ^ {t + 1}) \right] + \gamma \Psi^ {t + 1} \leq \operatorname {E} \left[ f (x ^ {t}) \right] - \frac {\gamma}{2} \operatorname {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] + (1 - \gamma \mu) \gamma \Psi^ {t} + \gamma C,
$$
where  $\Psi^t$  is a sequence of numbers,  $\Psi^t \geq 0$  for all  $t \in [T]$ , constant  $C \geq 0$ , constant  $\mu > 0$ , and constant  $\gamma \in (0,1/\mu)$ . Then
$$
\operatorname {E} \left[ f \left(x ^ {T}\right) - f ^ {*} \right] \leq (1 - \gamma \mu) ^ {T} \left(\left(f \left(x ^ {0}\right) - f ^ {*}\right) + \gamma \Psi^ {0}\right) + \frac {C}{\mu}. \tag {33}
$$
Proof. We subtract  $f^{*}$  and use PL-condition (30) to get
$$
\begin{array}{l} \operatorname {E} \left[ f \left(x ^ {t + 1}\right) - f ^ {*} \right] + \gamma \Psi^ {t + 1} \leq \operatorname {E} \left[ f \left(x ^ {t}\right) - f ^ {*} \right] - \frac {\gamma}{2} \operatorname {E} \left[ \left\| \nabla f \left(x ^ {t}\right) \right\| ^ {2} \right] + \gamma \Psi^ {t} + \gamma C \\ \leq (1 - \gamma \mu) \mathrm {E} [ f (x ^ {t}) - f ^ {*} ] + (1 - \gamma \mu) \gamma \Psi^ {t} + \gamma C \\ = (1 - \gamma \mu) (\mathrm {E} [ f (x ^ {t}) - f ^ {*} ] + \gamma \Psi^ {t}) + \gamma C. \\ \end{array}
$$
Unrolling the inequality, we have
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) - f ^ {*} \right] + \gamma \Psi^ {t + 1} \leq (1 - \gamma \mu) ^ {t + 1} \left(\left(f (x ^ {0}) - f ^ {*}\right) + \gamma \Psi^ {0}\right) + \gamma C \sum_ {i = 0} ^ {t} (1 - \gamma \mu) ^ {i} \\ \leq (1 - \gamma \mu) ^ {t + 1} \left(\left(f (x ^ {0}) - f ^ {*}\right) + \gamma \Psi^ {0}\right) + \frac {C}{\mu}. \\ \end{array}
$$
It is left to note that  $\Psi^t\geq 0$  for all  $t\in [T]$
F.4.2 Generic Lemma
We now provide the counterpart of Lemma 6.
Lemma 12. Suppose that Assumptions 2, 7, 8 and 9 hold and let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ , then
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \| \nabla f (x ^ {t}) \| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \gamma \| h ^ {t} - \nabla f (x ^ {t}) \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {1 0 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
Proof. Let us fix some constants  $\kappa, \eta \in [0, \infty)$  that we will define later. Using the same reasoning as in Lemma 6, we can get
$$
\begin{array}{l} \operatorname {E} \left[ f \left(x ^ {t + 1}\right) \right] \\ + \kappa \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \eta \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \gamma \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} + \left(\gamma + \kappa (1 - a) ^ {2}\right) \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] \\ \left. \right. + \left(\frac {\kappa a ^ {2} ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} + \eta \left(\frac {a ^ {2} (2 \omega + 1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - a) ^ {2}\right)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \\ + \left(\frac {2 \kappa \omega}{n p _ {\mathrm {a}}} + \frac {2 \eta \omega}{p _ {\mathrm {a}}}\right) \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
Let us take  $\kappa = \frac{2\gamma}{a}$ . One can show that  $\gamma + \kappa (1 - a)^2 \leq (1 - \frac{a}{2})\kappa$ , and thus
$$
\begin{array}{l} E [ f (x ^ {t + 1}) ] \\ + \frac {2 \gamma}{a} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \eta \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ \leq \mathrm {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \| \nabla f (x ^ {t}) \| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \gamma \| h ^ {t} - \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \left(1 - \frac {a}{2}\right) \frac {2 \gamma}{a} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] \\ \left. \right. + \left(\frac {2 \gamma a ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} + \eta \left(\frac {a ^ {2} (2 \omega + 1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - a) ^ {2}\right)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \left(\frac {4 \gamma \omega}{a n p _ {\mathrm {a}}} + \frac {2 \eta \omega}{p _ {\mathrm {a}}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| k _ {i} ^ {t + 1} \| ^ {2} \right]. \\ \end{array}
$$
Considering the choice of  $a$ , one can show that  $\left(\frac{a^2(2\omega + 1 - p_{\mathrm{a}})}{p_{\mathrm{a}}} + (1 - a)^2\right) \leq 1 - a$ . If we take  $\eta = \frac{4\gamma((2\omega + 1)p_{\mathrm{a}} - p_{\mathrm{aa}})}{np_{\mathrm{a}}^2}$ , then  $\left(\frac{2\gamma a((2\omega + 1)p_{\mathrm{a}} - p_{\mathrm{aa}})}{np_{\mathrm{a}}^2} + \eta \left(\frac{a^2(2\omega + 1 - p_{\mathrm{a}})}{p_{\mathrm{a}}} + (1 - a)^2\right)\right) \leq \left(1 - \frac{a}{2}\right)\eta$  and
$$
\begin{array}{l} E [ f (x ^ {t + 1}) ] \\ + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \| \nabla f (x ^ {t}) \| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \gamma \| h ^ {t} - \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \left(1 - \frac {a}{2}\right) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \left(1 - \frac {a}{2}\right) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \left(\frac {2 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \omega}{n p _ {\mathrm {a}} ^ {3}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \| \nabla f (x ^ {t}) \| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \gamma \| h ^ {t} - \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \left(1 - \frac {a}{2}\right) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \left(1 - \frac {a}{2}\right) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {1 0 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| k _ {i} ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
It it left to consider that  $\gamma \leq \frac{a}{2\mu}$ , and therefore  $1 - \frac{a}{2} \leq 1 - \gamma \mu$ .
F.4.3 Proof for DASHA-PP under PL-condition
$$
\gamma \leq \min \left\{\left(L + \sqrt {\frac {2 0 0 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {4 8}{n p _ {\mathrm {a}} ^ {2}} \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)} \widehat {L}\right) ^ {- 1}, \frac {a}{4 \mu} \right\},
$$
Proof. Let us fix constants  $\nu, \rho \in [0, \infty)$  that we will define later. Considering Lemma 12, Lemma 7, and the law of total expectation, we obtain
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \nu \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \left(\frac {1}{2 \gamma} - \frac {L}{2}\right) \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \gamma \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {1 0 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ 2 \widehat {L} ^ {2} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + 2 b ^ {2} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + \nu \mathrm {E} \left[ \frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {2 b ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} + (1 - b) ^ {2} \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \rho \mathrm {E} \left[ \frac {2 (1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} \widehat {L} ^ {2} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \left(\frac {2 b ^ {2} (1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - b) ^ {2}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t + 1} - h ^ {t + 1} \| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \| ^ {2} \right] \\ + \nu \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {2 0 \gamma \omega (2 \omega + 1) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} - \nu \frac {2 (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} - \rho \frac {2 (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{p _ {\mathrm {a}}}\right) \operatorname {E} \left[ \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \right] \\ + \left(\gamma + \nu (1 - b) ^ {2}\right) \operatorname {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\frac {2 0 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \nu \frac {2 b ^ {2} (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} + \rho \left(\frac {2 b ^ {2} (1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - b) ^ {2}\right)\right) \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
By taking  $\nu = \frac{2\gamma}{b}$ , one can show that  $(\gamma + \nu (1 - b)^2) \leq \left(1 - \frac{b}{2}\right)\nu$ , and
$$
\operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right]
$$
$$
\begin{array}{l} + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \left| \left| \nabla f (x ^ {t}) \right| \right| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {2 0 \gamma \omega (2 \omega + 1) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} - \frac {4 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{b n p _ {\mathrm {a}} ^ {2}} - \rho \frac {2 (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{p _ {\mathrm {a}}}\right) \operatorname {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ \left. \right. + \left(\frac {2 0 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {4 \gamma b \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}} + \rho \left(\frac {2 b ^ {2} (1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - b) ^ {2}\right)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
$$
\begin{array}{l} \left(\frac {2 0 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {4 \gamma b \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}} + \rho \left(\frac {2 b ^ {2} (1 - p _ {\mathrm {a}})}{p _ {\mathrm {a}}} + (1 - b) ^ {2}\right)\right) \\ \leq \left(\frac {2 0 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {4 \gamma b (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} + \rho (1 - b)\right). \\ \end{array}
$$
And if we take  $\rho = \frac{40b\gamma\omega(2\omega + 1)}{np_{\mathrm{a}}^2} +\frac{8\gamma(p_{\mathrm{a}} - p_{\mathrm{aa}})}{np_{\mathrm{a}}^2}$ , then
$$
\left(\frac {2 0 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {4 \gamma b \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}} + \rho (1 - b)\right) \leq \left(1 - \frac {b}{2}\right) \rho ,
$$
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {2 0 \gamma \omega (2 \omega + 1) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} - \frac {4 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{b n p _ {\mathrm {a}} ^ {2}} \right. \\ \left. - \frac {8 0 b \gamma \omega (2 \omega + 1) (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {3}} - \frac {1 6 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {3}}\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ \left. \right. + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[\left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(1 - \frac {b}{2}\right)\left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
Due to  $\frac{p_{\mathrm{a}}}{2} \leq b \leq p_{\mathrm{a}}$ , we have
$$
\begin{array}{l} \operatorname {E} \left[ f \left(x ^ {t + 1}\right) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {1 0 0 \gamma \omega (2 \omega + 1) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}} - \frac {2 4 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {3}}\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ \left. \right. + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[\left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(1 - \frac {b}{2}\right)\left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t + 1} - h ^ {t + 1} \| ^ {2} \right] + \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \left| \left| \nabla f (x ^ {t}) \right| \right| ^ {2} \right] \\ + (1 - \gamma \mu) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t} - h ^ {t} \| ^ {2} \right] + (1 - \gamma \mu) \frac {4 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \\ \left. \right. + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[\left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(1 - \frac {b}{2}\right)\left(\frac {4 0 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
Note that  $\gamma \leq \frac{a}{4\mu} \leq \frac{p_{\mathrm{a}}}{4\mu} \leq \frac{b}{2\mu}$ , thus  $1 - \frac{b}{2} \leq 1 - \gamma \mu$  and