SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} + \frac {\gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {8 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {2 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {2 4 \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {(1 - b) ^ {2} L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) \right. \\ \left. - \frac {6 \gamma}{n p _ {\mathrm {a}} b} \left(\frac {(1 - b) ^ {2} L _ {\sigma} ^ {2}}{B} + \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \frac {\gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(\frac {8 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {2 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\frac {2 4 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {6 \gamma b}{n p _ {\mathrm {a}}}\right) \frac {\sigma^ {2}}{B}. \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {t + 1}) \right] + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \operatorname {E} \left[ \| g ^ {t + 1} - h ^ {t + 1} \| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \| ^ {2} \right] \\ + \frac {\gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \left(\frac {8 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {2 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {\gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(\frac {8 b \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {2 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\frac {2 4 b ^ {2} \gamma \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {6 \gamma b}{n p _ {\mathrm {a}}}\right) \frac {\sigma^ {2}}{B}. \\ \end{array}
$$
$$
\begin{array}{l} \Psi^ {t} = \frac {(2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {\left((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {1}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(\frac {8 b \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {2 (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}}\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ \end{array}
$$
and  $C = \left(\frac{24b^2\omega(2\omega + 1)}{p_{\mathrm{a}}^2} +\frac{6b}{p_{\mathrm{a}}}\right)\frac{\sigma^2}{nB}$  to conclude the proof.
Proof. Using the result from Theorem 4, we have
$$
\begin{array}{l} \operatorname {E} \left[ \left\| \nabla f (\widehat {x} ^ {T}) \right\| ^ {2} \right] \\ \leq \frac {1}{T} \left[ 2 \Delta_ {0} \left(L + \sqrt {\frac {4 8 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {(1 - b) ^ {2} L _ {\sigma} ^ {2}}{B}\right) + \frac {1 2}{n p _ {\mathrm {a}} b} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - b) ^ {2} L _ {\sigma} ^ {2}}{B}\right)}\right) \right. \\ \left. + \frac {2}{b} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \left(\frac {3 2 b \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {4 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {\mathrm {a}}}\right) \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2}\right) \right] \\ + \left(\frac {4 8 b ^ {2} \omega (2 \omega + 1)}{p _ {\mathrm {a}} ^ {2}} + \frac {1 2 b}{p _ {\mathrm {a}}}\right) \frac {\sigma^ {2}}{n B} \\ \end{array}
$$
We choose  $b$  to ensure  $\left(\frac{48b^2\omega(2\omega + 1)}{p_{\mathrm{a}}^2} +\frac{12b}{p_{\mathrm{a}}}\right)\frac{\sigma^2}{nB} = \Theta (\varepsilon)$ . Note that  $\frac{1}{b} = \Theta \left(\max \left\{\frac{\omega}{p_{\mathrm{a}}}\sqrt{\frac{\sigma^2}{n\varepsilon B}},\frac{\sigma^2}{p_{\mathrm{a}}n\varepsilon B}\right\}\right) \leq \Theta \left(\max \left\{\frac{\omega^2}{p_{\mathrm{a}}},\frac{\sigma^2}{p_{\mathrm{a}}n\varepsilon B}\right\}\right)$ , thus
$$
\begin{array}{l} \mathrm {E} \left[ \left\| \nabla f (\widehat {x} ^ {T}) \right\| ^ {2} \right] \\ = \mathcal {O} \left(\frac {1}{T} \left[ \Delta_ {0} \left(L + \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \left(\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right)\right) \right. \right. \\ \left. + \frac {1}{b} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \left(\frac {b \omega^ {2}}{n p _ {\mathrm {a}} ^ {2}} + \frac {1}{n p _ {\mathrm {a}}}\right)\left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2}\right)\right] + \varepsilon\left. \right), \\ \end{array}
$$
where  $\mathbb{1}_{p_{\mathrm{a}}} = \sqrt{1 - \frac{p_{\mathrm{aa}}}{p_{\mathrm{a}}}}$ . It enough to take the following  $T$  to get  $\varepsilon$ -solution.
$$
\begin{array}{l} T = \mathcal {O} \left(\frac {1}{\varepsilon} \left[ \Delta_ {0} \left(L + \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \left(\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right)\right) \right. \right. \\ \left. \left. + \frac {1}{b} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \left(\frac {b \omega^ {2}}{n p _ {\mathrm {a}} ^ {2}} + \frac {1}{n p _ {\mathrm {a}}}\right) \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2}\right) \right]\right). \\ \end{array}
$$
Let us bound the norms:
$$
\begin{array}{l} \operatorname {E} \left[ \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} \right] = \operatorname {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{B _ {\mathrm {i n i t}}} \sum_ {k = 1} ^ {B _ {\mathrm {i n i t}}} \nabla f _ {i} (x ^ {0}; \xi_ {i k} ^ {0}) - \nabla f (x ^ {0}) \right\| ^ {2} \right] \\ = \frac {1}{n ^ {2} B _ {\mathrm {i n i t}} ^ {2}} \sum_ {i = 1} ^ {n} \sum_ {k = 1} ^ {B _ {\mathrm {i n i t}}} \operatorname {E} \left[ \left\| \nabla f _ {i} (x ^ {0}; \xi_ {i k} ^ {0}) - \nabla f _ {i} (x ^ {0}) \right\| ^ {2} \right] \\ \leq \frac {\sigma^ {2}}{n B _ {\mathrm {i n i t}}}. \\ \end{array}
$$
Using the same reasoning, one can get  $\frac{1}{n}\sum_{i=1}^{n}\mathrm{E}\left[\left\|h_i^0 - \nabla f_i(x^0)\right\|^2\right] \leq \frac{\sigma^2}{B_{\mathrm{init}}}$ . Combining all inequalities, we have
$$
\begin{array}{l} T = \mathcal {O} \left(\frac {1}{\varepsilon} \left[ \Delta_ {0} \left(L + \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \left(\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right)\right) \right. \right. \\ \left. \left. + \frac {\sigma^ {2}}{b n B _ {\mathrm {i n i t}}} + \frac {b \omega^ {2} \sigma^ {2}}{n p _ {\mathrm {a}} ^ {2} B _ {\mathrm {i n i t}}} + \frac {\sigma^ {2}}{n p _ {\mathrm {a}} B _ {\mathrm {i n i t}}} \right] \right). \\ \end{array}
$$
Using the choice of  $B_{\mathrm{init}}$  and  $b$ , we obtain
$$
\begin{array}{l} T = \mathcal {O} \left(\frac {1}{\varepsilon} \left[ \Delta_ {0} \left(L + \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \left(\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right)\right) \right. \right. \\ \left. \left. + \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n B} + \frac {b ^ {2} \omega^ {2} \sigma^ {2}}{n p _ {\mathrm {a}} ^ {5 / 2} B} + \frac {b \sigma^ {2}}{p _ {\mathrm {a}} ^ {3 / 2} n B} \right] \right) \\ = \mathcal {O} \left(\frac {1}{\varepsilon} \left[ \Delta_ {0} \left(L + \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \left(\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right)\right) \right. \right. \\ \left. \left. + \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n B} + \frac {\varepsilon}{\sqrt {p _ {\mathrm {a}}}} \right] \right) \\ = \mathcal {O} \left(\frac {\Delta_ {0}}{\varepsilon} \left[ L + \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \left(\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) \right] + \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon B} + \frac {1}{\sqrt {p _ {\mathrm {a}}}}\right). \\ \end{array}
$$
Using  $\frac{\sigma^2}{n\varepsilon B} \geq 1$ , we can conclude the proof of the inequality. The number of stochastic gradients that each node calculates equals  $B_{\mathrm{init}} + 2BT = \mathcal{O}(B_{\mathrm{init}} + BT)$ .
Corollary 4. Suppose that assumptions of Corollary 3 hold, batch size  $B \leq \min \left\{\frac{\sigma}{p_{\mathrm{a}}\sqrt{\varepsilon}n}, \frac{L_{\sigma}^2}{1_{p_{\mathrm{a}}}^2\hat{L}^2}\right\}$ , we take RandK compressors with  $K = \Theta\left(\frac{Bd\sqrt{\varepsilon n}}{\sigma}\right)$ . Then the communication complexity equals  $\mathcal{O}\left(\frac{d\sigma}{\sqrt{p_{\mathrm{a}}\sqrt{n\varepsilon}}} + \frac{L_{\sigma}\Delta_0d}{p_{\mathrm{a}}\sqrt{n\varepsilon}}\right)$ , and the expected number of stochastic gradient calculations per node equals  $\mathcal{O}\left(\frac{\sigma^2}{\sqrt{p_{\mathrm{a}}n\varepsilon}} + \frac{L_{\sigma}\Delta_0\sigma}{p_{\mathrm{a}}\varepsilon^{3/2}n}\right)$ .
$$
\mathcal {O} (d + K T) = \mathcal {O} \left(d + \frac {\Delta_ {0}}{\varepsilon} \left[ K L + K \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + K \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \left(\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) \right] + K \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon B}\right).
$$
Due to  $B \leq \frac{L_{\sigma}^{2}}{\mathbb{1}_{p_{\mathrm{a}}}^{2}\widehat{L}^{2}}$ , we have  $\mathbb{1}_{p_{\mathrm{a}}}\widehat{L} + \frac{L_{\sigma}}{\sqrt{B}} \leq \frac{2L_{\sigma}}{\sqrt{B}}$  and
$$
\mathcal {O} (d + K T) = \mathcal {O} \left(d + \frac {\Delta_ {0}}{\varepsilon} \left[ K L + K \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + K \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \frac {L _ {\sigma}}{\sqrt {B}} \right] + K \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon B}\right).
$$
From Theorem 6, we have  $\omega + 1 = \frac{d}{K}$ . Since  $K = \Theta\left(\frac{Bd\sqrt{\varepsilon n}}{\sigma}\right) = \mathcal{O}\left(\frac{d}{p_{\mathrm{a}}\sqrt{n}}\right)$ , the communication complexity equals
$$
\begin{array}{l} \mathcal {O} (d + K T) = \mathcal {O} \left(d + \frac {\Delta_ {0}}{\varepsilon} \left[ \frac {d}{p _ {\mathrm {a}} \sqrt {n}} L + \frac {d}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {d}{p _ {\mathrm {a}} \sqrt {n}} L _ {\sigma} \right] + \frac {d \sigma}{\sqrt {p _ {\mathrm {a}}} \sqrt {n \varepsilon}}\right) \\ = \mathcal {O} \left(\frac {d \sigma}{\sqrt {p _ {\mathrm {a}}} \sqrt {n \varepsilon}} + \frac {L _ {\sigma} \Delta_ {0} d}{p _ {\mathrm {a}} \sqrt {n} \varepsilon}\right) \\ \end{array}
$$
And the expected number of stochastic gradient calculations per node equals
$$
\begin{array}{l} \mathcal {O} \left(B _ {\text {i n i t}} + B T\right) \\ = \mathcal {O} \left(\frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon} + \frac {B \omega}{\sqrt {p _ {\mathrm {a}}}} \sqrt {\frac {\sigma^ {2}}{n \varepsilon B}} + \frac {\Delta_ {0}}{\varepsilon} \left[ B L + B \frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + B \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \left(\mathbb {1} _ {p _ {\mathrm {a}}} \widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) \right] + B \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon B}\right) \\ = \mathcal {O} \left(\frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon} + \frac {B d}{K \sqrt {p _ {\mathrm {a}}}} \sqrt {\frac {\sigma^ {2}}{n \varepsilon B}} + \frac {\Delta_ {0}}{\varepsilon} \left[ B L + B \frac {d}{K p _ {\mathrm {a}} \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + B \sqrt {\frac {\sigma^ {2}}{p _ {\mathrm {a}} ^ {2} \varepsilon n ^ {2} B}} \frac {L _ {\sigma}}{\sqrt {B}} \right] + \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon}\right) \\ = \mathcal {O} \left(\frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon} + \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon \sqrt {B}} + \frac {\Delta_ {0}}{\varepsilon} \left[ \frac {\sigma}{p _ {\mathrm {a}} \sqrt {\varepsilon} n} L + \frac {\sigma}{p _ {\mathrm {a}} \sqrt {\varepsilon} n} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {\sigma}{p _ {\mathrm {a}} \sqrt {\varepsilon} n} L _ {\sigma} \right]\right) \\ = \mathcal {O} \left(\frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon} + \frac {L _ {\sigma} \Delta_ {0} \sigma}{p _ {\mathrm {a}} \varepsilon^ {3 / 2} n}\right). \\ \end{array}
$$
F Analysis of DASHA-PP under Polyak-Lojasiewicz Condition
In this section, we provide the theoretical convergence rates of DASHA-PP under Polyak-Lojasiewicz Condition.
Assumption 9. The function  $f$  satisfy (Polyak-Lojasiewicz) PL-condition:
$$
\left\| \nabla f (x) \right\| ^ {2} \geq 2 \mu \left(f (x) - f ^ {*}\right), \quad \forall x \in \mathbb {R}, \tag {30}
$$
where  $f^{*} = \inf_{x\in \mathbb{R}^{d}}f(x) > - \infty$ .
Under Polyak-Lojasiewicz condition, a (random) point  $\widehat{x}$  is  $\varepsilon$ -solution, if  $\operatorname{E}[f(\widehat{x})] - f^{*} \leq \varepsilon$ .
We now provide the convergence rates of DASHA-PP under PL-condition.
F.1 Gradient Setting
Theorem 8. Suppose that Assumption 1, 2, 3, 7, 8 and 9 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b = \frac{p_{\mathrm{a}}}{2 - p_{\mathrm{a}}}$ ,
$$
\gamma \leq \min  \left\{\left(L + \sqrt {\frac {2 0 0 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {4 8}{n p _ {\mathrm {a}} ^ {2}} \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)} \widehat {L}\right) ^ {- 1}, \frac {a}{4 \mu} \right\},
$$
and  $h_i^0 = g_i^0 = \nabla f_i(x^0)$  for all  $i \in [n]$  in Algorithm 1 (DASHA-PP), then  $\operatorname{E}\left[f(x^T)\right] - f^* \leq (1 - \gamma \mu)^T \Delta_0$ .
Let us provide bounds up to logarithmic factors and use  $\tilde{\mathcal{O}} (\cdot)$  notation. The provided theorem states that to get  $\varepsilon$ -solution DASHA-PP have to run
$$
\widetilde {\mathcal {O}} \left(\frac {\omega + 1}{p _ {\mathrm {a}}} + \frac {L}{\mu} + \frac {\omega \widehat {L}}{p _ {\mathrm {a}} \mu \sqrt {n}} + \frac {\widehat {L}}{p _ {\mathrm {a}} \mu \sqrt {n}}\right),
$$
communication rounds. The method DASHA from (Tyurin and Richtárik, 2023), have to run
$$
\widetilde {\mathcal {O}} \left(\omega + \frac {L}{\mu} + \frac {\omega \widehat {L}}{\mu \sqrt {n}}\right),
$$
communication rounds to get  $\varepsilon$ -solution. The difference is the same as in the general nonconvex case (see Section 6.1). Up to Lipschitz constants factors, we get the degeneration up to  $1 / p_{\mathrm{a}}$  factor due to the partial participation.
F.2 Finite-Sum Setting
Theorem 9. Suppose that Assumption 1, 2, 3, 7, 4, 8, and 9 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ , probability  $p_{page} = \frac{B}{m + B}$ ,  $b = \frac{p_{page}p_{\mathrm{a}}}{2 - p_{\mathrm{a}}}$
$$
\gamma \leq \min \left\{\left(L + \sqrt {\frac {2 0 0 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {(1 - p _ {p a g e}) L _ {\mathrm {m a x}} ^ {2}}{B}\right) + \frac {4 8}{n p _ {\mathrm {a}} ^ {2} p _ {p a g e}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {(1 - p _ {p a g e}) L _ {\mathrm {m a x}} ^ {2}}{B}\right)\right) ^ {- 1}, \frac {a}{2 \mu}, \frac {b}{2 \mu} \right\},
$$
and  $h_i^0 = g_i^0 = \nabla f_i(x^0)$  for all  $i \in [n]$  in Algorithm 1 (DASHA-PP-PAGE), then  $\operatorname{E}\left[f(x^T)\right] - f^* \leq (1 - \gamma \mu)^T \Delta_0$ .
The provided theorem states that to get  $\varepsilon$ -solution DASHA-PP have to run
$$
\widetilde {\mathcal {O}} \left(\frac {\omega + 1}{p _ {\mathrm {a}}} + \frac {m}{p _ {\mathrm {a}} B} + \frac {L}{\mu} + \frac {\omega}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\operatorname* {m a x}}}{\sqrt {B}}\right) + \frac {\sqrt {m}}{p _ {\mathrm {a}} \mu \sqrt {n B}} \left(\widehat {L} + \frac {L _ {\operatorname* {m a x}}}{\sqrt {B}}\right)\right),
$$
communication rounds. The method DASHA-PAGE from (Tyurin and Richtárik, 2023), have to run
$$
\widetilde {\mathcal {O}} \left(\omega + \frac {m}{B} + \frac {L}{\mu} + \frac {\omega}{\mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\mathrm {m a x}}}{\sqrt {B}}\right) + \frac {\sqrt {m}}{\mu \sqrt {n B}} \left(\frac {L _ {\mathrm {m a x}}}{\sqrt {B}}\right)\right),
$$
communication rounds to get  $\varepsilon$ -solution. We can guarantee the degeneration up to  $1 / p_{\mathrm{a}}$  factor due to the partial participation only if  $B = \mathcal{O}\left(\frac{L_{\max}^2}{\hat{L}^2}\right)$ . The same conclusion we have in Section 6.2.
F.3 Stochastic Setting
Theorem 10. Suppose that Assumption 1, 2, 3, 7, 5, 6, 8 and 9 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b \in \left(0, \frac{p_{\mathrm{a}}}{2 - p_{\mathrm{a}}} \right]$ ,
$$
\gamma \leq \min  \left\{\left(L + \sqrt {\frac {2 0 0 \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {(1 - b) ^ {2} L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) + \frac {4 0}{n p _ {\mathrm {a}} b} \left(\frac {(1 - b) ^ {2} L _ {\sigma} ^ {2}}{B} + \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2}\right)}\right) ^ {- 1}, \frac {a}{2 \mu}, \frac {b}{2 \mu} \right\},
$$
and  $h_i^0 = g_i^0$  for all  $i \in [n]$  in Algorithm 1 (DASHA-PP-MVR), then
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {T}) - f ^ {*} \right] \\ \leq \left(1 - \gamma \mu\right) ^ {T} \left(\Delta_ {0} + \frac {2 \gamma}{b} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \left(\frac {4 0 \gamma b \omega (2 \omega + 1)}{n p _ {\mathrm {a}} ^ {2}} + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2}\right) \\ + \frac {1}{\mu} \left(\frac {1 0 0 b ^ {2} \omega (2 \omega + 1)}{p _ {\mathrm {a}} ^ {2}} + \frac {2 0 b}{p _ {\mathrm {a}}}\right) \frac {\sigma^ {2}}{n B}. \\ \end{array}
$$
The provided theorems states that to get  $\varepsilon$ -solution DASHA-PP have to run
$$
\widetilde {\mathcal {O}} \left(\frac {\omega + 1}{p _ {\mathrm {a}}} + \underbrace {\frac {\omega}{p _ {\mathrm {a}}} \sqrt {\frac {\sigma^ {2}}{\mu n \varepsilon B}}} _ {\mathcal {P} _ {2}} + \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon B} + \frac {L}{\mu} + \frac {\omega}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \underbrace {\frac {\sigma}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon B}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right)} _ {\mathcal {P} _ {1}}\right) \tag {31}
$$
communication rounds. We take  $b = \Theta \left( \min \left\{ \frac{p_{\mathrm{a}}}{\omega} \sqrt{\frac{\mu n \varepsilon B}{\sigma^2}}, \frac{p_{\mathrm{a}} \mu n \varepsilon B}{\sigma^2} \right\} \right) \geq$
$$
\Theta \left(\min  \left\{ \begin{array}{c} p _ {\mathrm {a}} \\ \omega^ {2}, \end{array} \frac {p _ {\mathrm {a}} \mu n \varepsilon B}{\sigma^ {2}} \right\}\right).
$$
The method DASHA-SYNC-MVR from (Tyurin and Richtárik, 2023), have to run
$$
\widetilde {\mathcal {O}} \left(\omega + \frac {\sigma^ {2}}{\mu n \varepsilon B} + \frac {L}{\mu} + \frac {\omega}{\mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {\sigma}{n \mu^ {3 / 2} \sqrt {\varepsilon B}} \left(\frac {L _ {\sigma}}{\sqrt {B}}\right)\right) \tag {32}
$$
communication rounds to get  $\varepsilon$ -solution<sup>9</sup>.
In the stochastic setting, the comparison is a little bit more complicated. As in the finite-sum setting, we have to take  $B = \mathcal{O}\left(\frac{L_{\sigma}^{2}}{\widehat{L}^{2}}\right)$  to guarantee the degeneration up to  $1 / p_{\mathrm{a}}$  of the term  $\mathcal{P}_1$  from (31). However, DASHA-PP-MVR has also suboptimal term  $\mathcal{P}_2$ . This suboptimality is tightly connected with the suboptimality of  $B_{\mathrm{init}}$  in the general nonconvex case, which we discuss in Section 6.3, and it also appears in the analysis of DASHA-MVR (Tyurin and Richtárik, 2023). Let us provide the counterpart of Corollary 4. The corollary reveals that we can escape regimes when  $\mathcal{P}_2$  is the bottleneck by choosing the parameters of the compressors.
Corollary 5. Suppose that assumptions of Theorem 10 hold, batch size  $B \leq \min \left\{\frac{\sigma}{p_{\mathrm{a}}\sqrt{\mu\varepsilon n}}, \frac{L_{\sigma}^{2}}{\hat{L}^{2}}\right\}$ , we take RandK compressors with  $K = \Theta\left(\frac{Bd\sqrt{\mu\varepsilon n}}{\sigma}\right)$ . Then the communication complexity equals
$$
\widetilde {\mathcal {O}} \left(\frac {d \sigma}{p _ {\mathrm {a}} \sqrt {\mu \varepsilon n}} + \frac {d L _ {\sigma}}{p _ {\mathrm {a}} \mu \sqrt {n}}\right),
$$
and the expected number of stochastic gradient calculations per node equals
$$
\widetilde {\mathcal {O}} \left(\frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon} + \frac {\sigma L _ {\sigma}}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon}}\right).
$$