SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left. \right. - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {4 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) - \left(\frac {8 \gamma L _ {\sigma} ^ {2}}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}} ^ {2} B} + \frac {8 \gamma \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2}}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}} ^ {2}}\right)\right) E \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {\gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {2 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + \frac {6 \gamma \sigma^ {2}}{n B ^ {\prime}}. \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t + 1} - h ^ {t + 1} \| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \| ^ {2} \right] \\ + \frac {\gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \frac {2 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ + \frac {\gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \frac {\gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \frac {\gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {2 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + \frac {6 \gamma \sigma^ {2}}{n B ^ {\prime}}. \\ \end{array}
$$
$$
\begin{array}{l} \Psi^ {t} = \frac {(2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \| g ^ {t} - h ^ {t} \| ^ {2} \right] + \frac {\left((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \\ + \frac {1}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {2 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {\mathrm {a}} p _ {\mathrm {m e g a}}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ \end{array}
$$
and  $C = \frac{6\sigma^2}{nB'}$  to conclude the proof.
Corollary 6. Suppose that assumptions from Theorem 11 hold, probability  $p_{\text{mega}} = \min \left\{ \frac{\zeta_{\mathcal{C}}}{d}, \frac{n\varepsilon B}{\sigma^2} \right\}$ , batch size  $B' = \Theta \left( \frac{\sigma^2}{n\varepsilon} \right)$ , and  $h_i^0 = g_i^0 = \frac{1}{B_{\text{init}}} \sum_{k=1}^{B_{\text{init}}} \nabla f_i(x^0; \xi_{ik}^0)$  for all  $i \in [n]$ , initial batch size  $B_{\text{init}} = \Theta \left( \frac{B}{p_{\text{mega}} \sqrt{p_{\text{a}}}} \right) = \Theta \left( \max \left\{ \frac{Bd}{\sqrt{p_{\text{a}}} \zeta_{\mathcal{C}}}, \frac{\sigma^2}{\sqrt{p_{\text{a}}} n\varepsilon} \right\} \right)$ , then DASHA-PP-SYNC-MVR needs
Proof. Due to the choice of  $B'$ , we have
$$
\begin{array}{l} \mathrm {E} \left[ \left\| \nabla f (\widehat {x} ^ {T}) \right\| ^ {2} \right] \leq \frac {1}{T} \left[ 2 \Delta_ {0} \left(L + \sqrt {\frac {8 (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {L _ {\sigma} ^ {2}}{B}\right) + \frac {1 6}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}} ^ {2}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {L _ {\sigma} ^ {2}}{B}\right)}\right) \right. \\ \left. + \frac {4}{p _ {\mathrm {m e g a}} p _ {\mathrm {a}}} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \frac {4 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2} \right] \\ + \frac {2 \varepsilon}{3}. \\ \end{array}
$$
$$
\operatorname {E} \left[ \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} \right] = \operatorname {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{B _ {\mathrm {i n i t}}} \sum_ {k = 1} ^ {B _ {\mathrm {i n i t}}} \nabla f _ {i} (x ^ {0}; \xi_ {i k} ^ {0}) - \nabla f (x ^ {0}) \right\| ^ {2} \right] \leq \frac {\sigma^ {2}}{n B _ {\mathrm {i n i t}}}
$$
$$
\frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathrm {E} \left[ \left\| h _ {i} ^ {0} - \nabla f _ {i} \left(x ^ {0}\right) \right\| ^ {2} \right] = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathrm {E} \left[ \left\| \frac {1}{B _ {\mathrm {i n i t}}} \sum_ {k = 1} ^ {B _ {\mathrm {i n i t}}} \nabla f _ {i} \left(x ^ {0}; \xi_ {i k} ^ {0}\right) - \nabla f _ {i} \left(x ^ {0}\right) \right\| ^ {2} \right] \leq \frac {\sigma^ {2}}{n B _ {\mathrm {i n i t}}},
$$
$$
\begin{array}{l} \mathrm {E} \left[ \left\| \nabla f (\widehat {x} ^ {T}) \right\| ^ {2} \right] \leq \frac {1}{T} \left[ 2 \Delta_ {0} \left(L + \sqrt {\frac {8 (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {L _ {\sigma} ^ {2}}{B}\right) + \frac {1 6}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}} ^ {2}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {L _ {\sigma} ^ {2}}{B}\right)}\right) \right. \\ \left. + \frac {8 \sigma^ {2}}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}} B _ {\mathrm {i n i t}}} \right] \\ + \frac {2 \varepsilon}{3}. \\ \end{array}
$$
Therefore, we can take the following  $T$  to get  $\varepsilon$ -solution.
$$
T = \mathcal {O} \left(\frac {1}{\varepsilon} \left[ \Delta_ {0} \left(L + \sqrt {\frac {\omega^ {2}}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {L _ {\sigma} ^ {2}}{B}\right) + \frac {1}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {L _ {\sigma} ^ {2}}{B}\right)}\right) + \frac {\sigma^ {2}}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}} B _ {\mathrm {i n i t}}} \right]\right)
$$
Considering the choice of  $p_{\mathrm{mega}}$  and  $B_{\mathrm{init}}$ , we obtain
$$
\begin{array}{l} T = \mathcal {O} \left(\frac {1}{\varepsilon} \left[ \Delta_ {0} \left(L + \left(\frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} + \sqrt {\frac {d}{p _ {\mathrm {a}} ^ {2} \zeta_ {\mathcal {C}} n}}\right) \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {\sigma}{p _ {\mathrm {a}} \sqrt {\varepsilon} n} \left(\frac {\widehat {L}}{\sqrt {B}} + \frac {L _ {\sigma}}{B}\right)\right) + \frac {\sigma^ {2}}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}} B _ {\mathrm {i n i t}}} \right]\right) \\ = \mathcal {O} \left(\frac {\Delta_ {0}}{\varepsilon} \left[ L + \left(\frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} + \sqrt {\frac {d}{p _ {\mathrm {a}} ^ {2} \zeta_ {\mathcal {C}} n}}\right) \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {\sigma}{p _ {\mathrm {a}} \sqrt {\varepsilon} n} \left(\frac {\widehat {L}}{\sqrt {B}} + \frac {L _ {\sigma}}{B}\right) \right] + \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon B}\right). \\ \end{array}
$$
The expected communication complexity equals  $\mathcal{O}(d + p_{\mathrm{mega}}d + (1 - p_{\mathrm{mega}})\zeta_{\mathcal{C}}) = \mathcal{O}(d + \zeta_{\mathcal{C}})$  and the expected number of stochastic gradient calculations per node equals  $\mathcal{O}\left(B_{\mathrm{init}} + p_{\mathrm{mega}}B' + (1 - p_{\mathrm{mega}})B\right) = \mathcal{O}\left(B_{\mathrm{init}} + B\right)$ .
$$
\gamma \leq \min \left\{\left(L + \sqrt {\frac {1 6 (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) + \left(\frac {4 8 L _ {\sigma} ^ {2}}{n p _ {m e g a} p _ {\mathrm {a}} ^ {2} B} + \frac {2 4 (1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}) \widehat {L} ^ {2}}{n p _ {m e g a} p _ {\mathrm {a}} ^ {2}}\right)}\right) ^ {- 1}, \frac {a}{2 \mu}, \frac {b}{2 \mu} \right\},
$$
$$
\begin{array}{l} E \left[ f (x ^ {T}) - f ^ {*} \right] \\ \leq (1 - \gamma \mu) ^ {T} \left(\Delta_ {0} + \frac {2 \gamma}{b} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {m e g a}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2}\right) + \frac {2 0 \sigma^ {2}}{\mu n B ^ {\prime}}. \\ \end{array}
$$
Proof. Let us fix constants  $\kappa, \eta, \nu, \rho \in [0, \infty)$  that we will define later. As in the proof of Theorem 11, we can get
$$
\begin{array}{l} \operatorname {E} \left[ f \left(x ^ {t + 1}\right) \right] + \kappa \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \eta \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \nu \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {2 \kappa \omega}{n p _ {\mathrm {a}}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) - \frac {2 \eta \omega}{p _ {\mathrm {a}}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) \right. \\ \left. - \nu \left(\frac {2 L _ {\sigma} ^ {2}}{n p _ {\mathrm {a}} B} + \frac {2 (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}}\right) - \rho \left(\frac {2 L _ {\sigma} ^ {2}}{p _ {\mathrm {a}} B} + \frac {2 (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{p _ {\mathrm {a}}}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(\gamma + \kappa (1 - a) ^ {2}\right) \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] \\ \left. + \left(\kappa \frac {\left((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) a ^ {2}}{n p _ {\mathrm {a}} ^ {2}} + \eta \left(\frac {(2 \omega + 1 - p _ {\mathrm {a}}) a ^ {2}}{p _ {\mathrm {a}}} + (1 - a) ^ {2}\right)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \| g _ {i} ^ {t} - h _ {i} ^ {t} \| ^ {2} \right] \right. \\ + (\gamma + \nu (1 - b)) \operatorname {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\nu \frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) b ^ {2}}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} + \rho (1 - b)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right| \right| ^ {2} \right] \\ + \left(\frac {2 \nu b ^ {2}}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}}} + \frac {2 \rho b ^ {2}}{p _ {\mathrm {a}} p _ {\mathrm {m e g a}}}\right) \frac {\sigma^ {2}}{B ^ {\prime}}. \\ \end{array}
$$
Let us take  $\kappa = \frac{2\gamma}{a}$ , thus  $\gamma + \kappa (1 - a)^2 \leq \left(1 - \frac{a}{2}\right)\kappa$  and
$$
\begin{array}{l} \operatorname {E} \left[ f \left(x ^ {t + 1}\right) \right] + \frac {2 \gamma}{a} \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \eta \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \nu \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} \left[ f (x ^ {t}) \right] - \frac {\gamma}{2} \operatorname {E} \left[ \left| \left| \nabla f (x ^ {t}) \right| \right| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {4 \gamma \omega}{a n p _ {\mathrm {a}}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) - \frac {2 \eta \omega}{p _ {\mathrm {a}}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) \right. \\ \end{array}
$$
$$
\begin{array}{l} \left. - \nu \left(\frac {2 L _ {\sigma} ^ {2}}{n p _ {\mathrm {a}} B} + \frac {2 (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}}\right) - \rho \left(\frac {2 L _ {\sigma} ^ {2}}{p _ {\mathrm {a}} B} + \frac {2 (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{p _ {\mathrm {a}}}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(1 - \frac {a}{2}\right) \frac {2 \gamma}{a} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] \\ \left. + \left(\frac {2 \gamma \left((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) a}{n p _ {\mathrm {a}} ^ {2}} + \eta \left(\frac {(2 \omega + 1 - p _ {\mathrm {a}}) a ^ {2}}{p _ {\mathrm {a}}} + (1 - a) ^ {2}\right)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \right. \\ + (\gamma + \nu (1 - b)) E \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\nu \frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) b ^ {2}}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} + \rho (1 - b)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right| \right| ^ {2} \right] \\ + \left(\frac {2 \nu b ^ {2}}{n p _ {\text {m e g a}} p _ {\mathrm {a}}} + \frac {2 \rho b ^ {2}}{p _ {\mathrm {a}} p _ {\text {m e g a}}}\right) \frac {\sigma^ {2}}{B ^ {\prime}}. \\ \end{array}
$$
Next, since  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ , we have  $\left(\frac{(2\omega + 1 - p_{\mathrm{a}})a^2}{p_{\mathrm{a}}} + (1 - a)^2\right) \leq 1 - a$ . We the choice  $\eta = \frac{2\gamma((2\omega + 1)p_{\mathrm{a}} - p_{\mathrm{aa}})}{np_{\mathrm{a}}^2}$ , we guarantee  $\frac{\gamma((2\omega + 1)p_{\mathrm{a}} - p_{\mathrm{aa}})a}{np_{\mathrm{a}}^2} + \eta \left(\frac{(2\omega + 1 - p_{\mathrm{a}})a^2}{p_{\mathrm{a}}} + (1 - a)^2\right) \leq \left(1 - \frac{a}{2}\right)\eta$  and
$$
\operatorname {E} \left[ f \left(x ^ {t + 1}\right) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {2 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right]
$$
$$
\begin{array}{l} + \nu \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {8 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) \right. \\ \left. - \nu \left(\frac {2 L _ {\sigma} ^ {2}}{n p _ {\mathrm {a}} B} + \frac {2 (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{n p _ {\mathrm {a}} ^ {2}}\right) - \rho \left(\frac {2 L _ {\sigma} ^ {2}}{p _ {\mathrm {a}} B} + \frac {2 (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{p _ {\mathrm {a}}}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(1 - \frac {a}{2}\right) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] \\ + \left(1 - \frac {a}{2}\right) \frac {2 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + (\gamma + \nu (1 - b)) E \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\nu \frac {2 \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right) b ^ {2}}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} + \rho (1 - b)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right| \right| ^ {2} \right] \\ + \left(\frac {2 \nu b ^ {2}}{n p _ {\text {m e g a}} p _ {\mathrm {a}}} + \frac {2 \rho b ^ {2}}{p _ {\mathrm {a}} p _ {\text {m e g a}}}\right) \frac {\sigma^ {2}}{B ^ {\prime}}, \\ \end{array}
$$
where simplified the term using  $p_{\mathrm{aa}} \geq 0$ . Let us take  $\nu = \frac{2\gamma}{b}$  to obtain
$$
\begin{array}{l} \mathrm {E} \left[ f (x ^ {t + 1}) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {2 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \rho \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {8 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) \right. \\ \left. - \left(\frac {4 \gamma L _ {\sigma} ^ {2}}{b n p _ {\mathrm {a}} B} + \frac {4 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{b n p _ {\mathrm {a}} ^ {2}}\right) - \rho \left(\frac {2 L _ {\sigma} ^ {2}}{p _ {\mathrm {a}} B} + \frac {2 (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{p _ {\mathrm {a}}}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(1 - \frac {a}{2}\right) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \left(1 - \frac {a}{2}\right) \frac {2 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\frac {4 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) b}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} + \rho (1 - b)\right) \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right] \\ + \left(\frac {4 \gamma b}{n p _ {\text {m e g a}} p _ {\mathrm {a}}} + \frac {2 \rho b ^ {2}}{p _ {\mathrm {a}} p _ {\text {m e g a}}}\right) \frac {\sigma^ {2}}{B ^ {\prime}}, \\ \end{array}
$$
Next, we take  $\rho = \frac{8\gamma(p_{\mathrm{a}} - p_{\mathrm{aa}})}{np_{\mathrm{a}}^2p_{\mathrm{mega}}}$ , thus
$$
\begin{array}{l} \operatorname {E} \left[ f \left(x ^ {t + 1}\right) \right] + \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \operatorname {E} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] + \frac {2 \gamma ((2 \omega + 1) p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2}} \operatorname {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right] \\ + \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] + \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \\ \leq \operatorname {E} [ f (x ^ {t}) ] - \frac {\gamma}{2} \operatorname {E} \left[ \| \nabla f (x ^ {t}) \| ^ {2} \right] \\ - \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {8 \gamma (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) \right. \\ \left. - \left(\frac {4 \gamma L _ {\sigma} ^ {2}}{b n p _ {\mathrm {a}} B} + \frac {4 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) \widehat {L} ^ {2}}{b n p _ {\mathrm {a}} ^ {2}}\right) - \left(\frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}}\right) \left(\frac {2 L _ {\sigma} ^ {2}}{p _ {\mathrm {a}} B} + \frac {2 (1 - p _ {\mathrm {a}}) \widehat {L} ^ {2}}{p _ {\mathrm {a}}}\right)\right) \mathrm {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right] \\ + \left(1 - \frac {a}{2}\right) \frac {2 \gamma (2 \omega + 1)}{p _ {\mathrm {a}}} \mathrm {E} \left[ \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \right] + \left(1 - \frac {a}{2}\right) \frac {2 \gamma \left(\left(2 \omega + 1\right) p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \right] \\ + \left(1 - \frac {b}{2}\right) \frac {2 \gamma}{b} \mathrm {E} \left[ \left\| h ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \left(1 - \frac {b}{2}\right) \frac {8 \gamma \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n p _ {\mathrm {a}} ^ {2} p _ {\mathrm {m e g a}}} \mathrm {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\frac {4 \gamma b}{n p _ {\mathrm {m e g a}} p _ {\mathrm {a}}} + \frac {1 6 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}}) b ^ {2}}{n p _ {\mathrm {a}} ^ {3} p _ {\mathrm {m e g a}} ^ {2}}\right) \frac {\sigma^ {2}}{B}, \\ \end{array}
$$
Since  $\frac{p_{\mathrm{mega}} p_{\mathrm{a}}}{2} \leq b \leq p_{\mathrm{mega}} p_{\mathrm{a}}$  and  $1 - p_{\mathrm{a}} \leq 1 - \frac{p_{\mathrm{aa}}}{p_{\mathrm{a}}} \leq 1$ , we get