SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \widetilde {\mathcal {O}} \left(K \frac {\omega + 1}{p _ {\mathrm {a}}} + K \frac {\omega}{p _ {\mathrm {a}}} \sqrt {\frac {\sigma^ {2}}{\mu n \varepsilon B}} + K \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon B} + K \frac {L}{\mu} + K \frac {\omega}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + K \frac {\sigma}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon B}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right)\right) \\ = \widetilde {\mathcal {O}} \left(K \frac {\omega + 1}{p _ {\mathrm {a}}} + K \frac {\omega}{p _ {\mathrm {a}}} \sqrt {\frac {\sigma^ {2}}{\mu n \varepsilon B}} + K \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon B} + K \frac {L}{\mu} + K \frac {\omega}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + K \frac {\sigma L _ {\sigma}}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon} B}\right) \\ = \widetilde {\mathcal {O}} \left(\frac {d}{p _ {\mathrm {a}}} + \frac {d}{p _ {\mathrm {a}}} \sqrt {\frac {\sigma^ {2}}{\mu n \varepsilon B}} + \frac {K \sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon B} + \frac {d L}{p _ {\mathrm {a}} \mu \sqrt {n}} + \frac {d}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {K \sigma L _ {\sigma}}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon} B}\right) \\ = \widetilde {\mathcal {O}} \left(\frac {d}{p _ {\mathrm {a}}} + \frac {d \sigma}{p _ {\mathrm {a}} \sqrt {\mu n \varepsilon B}} + \frac {d \sigma}{p _ {\mathrm {a}} \sqrt {\mu \varepsilon n}} + \frac {d L}{p _ {\mathrm {a}} \mu \sqrt {n}} + \frac {d}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {d L _ {\sigma}}{p _ {\mathrm {a}} \mu \sqrt {n}}\right) \\ = \widetilde {\mathcal {O}} \left(\frac {d \sigma}{p _ {\mathrm {a}} \sqrt {\mu \varepsilon n}} + \frac {d L _ {\sigma}}{p _ {\mathrm {a}} \mu \sqrt {n}}\right). \\ \end{array}
$$
The expected number of stochastic gradient calculations per node equals
$$
\widetilde {\mathcal {O}} \left(B \frac {\omega + 1}{p _ {\mathrm {a}}} + B \frac {\omega}{p _ {\mathrm {a}}} \sqrt {\frac {\sigma^ {2}}{\mu n \varepsilon B}} + B \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon B} + B \frac {L}{\mu} + B \frac {\omega}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + B \frac {\sigma}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon B}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right)\right)
$$
$$
\begin{array}{l} = \widetilde {\mathcal {O}} \left(B \frac {\omega + 1}{p _ {\mathrm {a}}} + B \frac {\omega}{p _ {\mathrm {a}}} \sqrt {\frac {\sigma^ {2}}{\mu n \varepsilon B}} + B \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon B} + B \frac {L}{\mu} + B \frac {\omega}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + B \frac {\sigma}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon B}} \left(\frac {L _ {\sigma}}{\sqrt {B}}\right)\right) \\ = \widetilde {\mathcal {O}} \left(\frac {B d}{K p _ {\mathrm {a}}} + \frac {B d}{K p _ {\mathrm {a}}} \sqrt {\frac {\sigma^ {2}}{\mu n \varepsilon B}} + \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon} + B \frac {L}{\mu} + \frac {B d}{K p _ {\mathrm {a}} \mu \sqrt {n}} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {\sigma L _ {\sigma}}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon}}\right) \\ = \widetilde {\mathcal {O}} \left(\frac {\sigma}{p _ {\mathrm {a}} \sqrt {\mu \varepsilon n}} + \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu \varepsilon n \sqrt {B}} + \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon} + \frac {\sigma L}{p _ {\mathrm {a}} \mu^ {3 / 2} \sqrt {\varepsilon} n} + \frac {\sigma}{p _ {\mathrm {a}} \mu^ {3 / 2} \sqrt {\varepsilon} n} \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {\sigma L _ {\sigma}}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon}}\right) \\ = \widetilde {\mathcal {O}} \left(\frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon} + \frac {\sigma L _ {\sigma}}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon}}\right). \\ \end{array}
$$
By analogy to (Tyurin and Richtárik, 2023), we provide a "synchronized" version of the algorithm. With a small probability, participating nodes calculate and send a mega batch without compression. This helps us to resolve the suboptimality of DASHA-PP-MVR w.r.t.  $\omega$ . Note that this suboptimality is not a problem. We show in Corollary 4 that DASHA-PP-MVR can have the optimal oracle complexity and SOTA communication complexity with the particular choices of parameters of the compressors.
Algorithm 8 DASHA-PP-SYNC-MVR
1: Input: starting point  $x^0\in \mathbb{R}^d$  , stepsize  $\gamma >0$  , momentum  $a\in (0,1]$  , momentum  $b\in$    
(0,1], probability  $p_{\mathrm{mega}}\in (0,1]$  , batch size  $B^{\prime}$  and  $B$  , probability  $p_a\in (0,1]$  that a node is   
participating(a), number of iterations  $T\geq 1$    
2: Initialize  $g_i^0,h_i^0$  on the nodes and  $g^0 = \frac{1}{n}\sum_{i = 1}^{n}g_i^0$  on the server   
3: for  $t = 0,1,\ldots ,T - 1$  do   
4:  $x^{t + 1} = x^{t} - \gamma g^{t}$    
5:  $c^{t + 1} = \left\{ \begin{array}{ll}1, & \text{with probability } p_{\mathrm{mega}},\\ 0, & \text{with probability } 1 - p_{\mathrm{mega}} \end{array} \right.$    
6: Broadcast  $x^{t + 1},x^{t}$  to all participating(a) nodes   
7: for  $i = 1,\dots ,n$  in parallel do   
8: if ith node is participating(a) then   
9: if  $c^{t + 1} = 1$  then   
10: Generate i.i.d. samples  $\{\xi_{ik}^{t + 1}\}_{k = 1}^{B'}$  of size  $B^{\prime}$  from  $\mathcal{D}_i$    
11:  $k_{i}^{t + 1} = \frac{1}{B'}\sum_{k = 1}^{B'}\nabla f_{i}(x^{t + 1};\xi_{ik}^{t + 1}) - \frac{1}{B'}\sum_{k = 1}^{B'}\nabla f_{i}(x^{t};\xi_{ik}^{t + 1}) - \frac{b}{p_{\mathrm{mega}}}\left(h_{i}^{t} - \frac{1}{B'}\sum_{k = 1}^{B'}\nabla f_{i}(x^{t};\xi_{ik}^{t + 1})\right)$    
12:  $m_i^{t + 1} = \frac{1}{p_a} k_i^{t + 1} - \frac{a}{p_a} (g_i^t -h_i^t)$    
13: else   
14: Generate i.i.d. samples  $\{\xi_{ij}^{t + 1}\}_{j = 1}^{B}$  of size  $B$  from  $\mathcal{D}_i$    
15:  $k_{i}^{t + 1} = \frac{1}{B}\sum_{j = 1}^{B}\nabla f_{i}(x^{t + 1};\xi_{ij}^{t + 1}) - \frac{1}{B}\sum_{j = 1}^{B}\nabla f_{i}(x^{t};\xi_{ij}^{t + 1})$    
16:  $m_i^{t + 1} = C_i\left(\frac{1}{p_a} k_i^{t + 1} - \frac{a}{p_a} (g_i^t -h_i^t)\right)$    
17: end if   
18:  $h_i^{t + 1} = h_i^t +\frac{1}{p_a} k_i^{t + 1}$    
19:  $g_i^{t + 1} = g_i^t +m_i^{t + 1}$    
20: Send  $m_i^{t + 1}$  to the server   
21: else   
22:  $h_i^{t + 1} = h_i^t$    
23:  $m_i^{t + 1} = 0$    
24:  $g_i^{t + 1} = g_i^t$    
25: end if   
26: end for   
27:  $g^{t + 1} = g^t +\frac{1}{n}\sum_{i = 1}^{n}m_i^{t + 1}$    
28: end for   
29: Output:  $\hat{x}^T$  chosen uniformly at random from  $\{x^t\}_{k = 0}^{T - 1}$  (a): For the formal description see Section 2.2.
In the following theorem, we provide the convergence rate of DASHA-PP-SYNC-MVR.
Theorem 11. Suppose that Assumptions 1, 2, 3, 5, 6, 7 and 8 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b = \frac{p_{\mathrm{mega}} p_{\mathrm{a}}}{2 - p_{\mathrm{a}}}$ , probability  $p_{\mathrm{mega}} \in (0, 1]$ , batch size  $B' \geq B \geq 1$
$$
\gamma \leq \left(L + \sqrt {\frac {8 (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\widehat {L} ^ {2} + \frac {L _ {\sigma} ^ {2}}{B}\right) + \frac {1 6}{n p _ {m e g a} p _ {\mathrm {a}} ^ {2}} \left(\left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2} + \frac {L _ {\sigma} ^ {2}}{B}\right)}\right) ^ {- 1},
$$
G Description of DASHA-PP-SYNC-MVR
and  $h_i^0 = g_i^0$  for all  $i \in [n]$  in Algorithm 8. Then
$$
\begin{array}{l} \operatorname {E} \left[ \left\| \nabla f (\widehat {x} ^ {T}) \right\| ^ {2} \right] \leq \frac {1}{T} \left[ \frac {2 \Delta_ {0}}{\gamma} + \frac {4}{p _ {m e g a} p _ {\mathrm {a}}} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \frac {4 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right)}{n p _ {m e g a} p _ {\mathrm {a}}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2} \right] \\ + \frac {1 2 \sigma^ {2}}{n B ^ {\prime}}. \\ \end{array}
$$
First, we introduce the expected density of compressors (Gorbunov et al., 2021; Tyurin and Richtárik, 2023).
Definition 12. The expected density of the compressor  $\mathcal{C}_i$  is  $\zeta_{\mathcal{C}_i} \coloneqq \sup_{x \in \mathbb{R}^d} \operatorname{E}\left[\|\mathcal{C}_i(x)\|_0\right]$ , where  $\|x\|_0$  is the number of nonzero components of  $x \in \mathbb{R}^d$ . Let  $\zeta_{\mathcal{C}} = \max_{i \in [n]} \zeta_{\mathcal{C}_i}$ .
Note that  $\zeta_{\mathcal{C}}$  is finite and  $\zeta_{\mathcal{C}} \leq d$ .
In the next corollary, we choose particular algorithm parameters to reveal the communication and oracle complexity.
Corollary 6. Suppose that assumptions from Theorem 11 hold, probability  $p_{\text{mega}} = \min \left\{ \frac{\zeta_c}{d}, \frac{n\varepsilon B}{\sigma^2} \right\}$ , batch size  $B' = \Theta \left( \frac{\sigma^2}{n\varepsilon} \right)$ , and  $h_i^0 = g_i^0 = \frac{1}{B_{\text{init}}} \sum_{k=1}^{B_{\text{init}}} \nabla f_i(x^0; \xi_{ik}^0)$  for all  $i \in [n]$ , initial batch size  $B_{\text{init}} = \Theta \left( \frac{B}{p_{\text{mega}} \sqrt{p_a}} \right) = \Theta \left( \max \left\{ \frac{Bd}{\sqrt{p_a} \zeta_c}, \frac{\sigma^2}{\sqrt{p_a} n\varepsilon} \right\} \right)$ , then DASHA-PP-SYNC-MVR needs
$$
T := \mathcal {O} \left(\frac {\Delta_ {0}}{\varepsilon} \left[ L + \left(\frac {\omega}{p _ {\mathrm {a}} \sqrt {n}} + \sqrt {\frac {d}{p _ {\mathrm {a}} ^ {2} \zeta_ {\mathcal {C}} n}}\right) \left(\widehat {L} + \frac {L _ {\sigma}}{\sqrt {B}}\right) + \frac {\sigma}{p _ {\mathrm {a}} \sqrt {\varepsilon} n} \left(\frac {\widehat {L}}{\sqrt {B}} + \frac {L _ {\sigma}}{B}\right) \right] + \frac {\sigma^ {2}}{\sqrt {p _ {\mathrm {a}}} n \varepsilon B}\right).
$$
communication rounds to get an  $\varepsilon$ -solution, the expected communication complexity is equal to  $\mathcal{O}(d + \zeta_{\mathcal{C}}T)$ , and the expected number of stochastic gradient calculations per node equals  $\mathcal{O}(B_{\mathrm{init}} + BT)$ , where  $\zeta_{\mathcal{C}}$  is the expected density from Definition 12.
The main improvement of Corollary 6 over Corollary 3 is the size of the initial batch size  $B_{\mathrm{init}}$ . However, Corollary 4 reveals that we can avoid regimes when DASHA-PP-MVR is suboptimal.
We also provide a theorem under PL-condition (see Assumption 9).
Theorem 13. Suppose that Assumptions 1, 2, 3, 5, 6, 7, 8 and 9 hold. Let us take  $a = \frac{p_{\mathrm{a}}}{2\omega + 1}$ ,  $b = \frac{p_{\mathrm{mega}} p_{\mathrm{a}}}{2 - p_{\mathrm{a}}}$ , probability  $p_{\mathrm{mega}} \in (0, 1]$ , batch size  $B' \geq B \geq 1$
$$
\gamma \leq \min  \left\{\left(L + \sqrt {\frac {1 6 (2 \omega + 1) \omega}{n p _ {\mathrm {a}} ^ {2}} \left(\frac {L _ {\sigma} ^ {2}}{B} + \widehat {L} ^ {2}\right) + \left(\frac {4 8 L _ {\sigma} ^ {2}}{n p _ {m e g a} p _ {\mathrm {a}} ^ {2} B} + \frac {2 4 \left(1 - \frac {p _ {\mathrm {a a}}}{p _ {\mathrm {a}}}\right) \widehat {L} ^ {2}}{n p _ {m e g a} p _ {\mathrm {a}} ^ {2}}\right)}\right) ^ {- 1}, \frac {a}{2 \mu}, \frac {b}{2 \mu} \right\},
$$
$$
\begin{array}{l} \operatorname {E} \left[ f (x ^ {T}) - f ^ {*} \right] \\ \leq \left(1 - \gamma \mu\right) ^ {T} \left(\Delta_ {0} + \frac {2 \gamma}{b} \left\| h ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} + \frac {8 \gamma (p _ {\mathrm {a}} - p _ {\mathrm {a a}})}{n p _ {\mathrm {a}} ^ {2} p _ {m e g a}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2}\right) + \frac {2 0 \sigma^ {2}}{\mu n B ^ {\prime}}. \\ \end{array}
$$
Let us provide bounds up to logarithmic factors and use  $\tilde{\mathcal{O}} (\cdot)$  notation.
Corollary 7. Suppose that assumptions from Theorem 13 hold, probability  $p_{\text{mega}} = \min \left\{\frac{\zeta_C}{d}, \frac{\mu n\varepsilon B}{\sigma^2}\right\}$ , batch size  $B' = \Theta\left(\frac{\sigma^2}{\mu n\varepsilon}\right)$  then DASHA-PP-SYNC-MVR needs
$$
T := \widetilde {\mathcal {O}} \left(\frac {\omega + 1}{p _ {\mathrm {a}}} + \frac {d}{p _ {\mathrm {a}} \zeta_ {\mathcal {C}}} + \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon B} + \frac {L}{\mu} + \frac {\omega}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\frac {L _ {\sigma}}{\sqrt {B}} + \widehat {L}\right) + \left(\frac {\sqrt {d}}{p _ {\mathrm {a}} \mu \sqrt {\zeta_ {\mathcal {C}} n}} + \frac {\sigma}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon B}}\right) \left(\frac {L _ {\sigma}}{\sqrt {B}} + \widehat {L}\right)\right).
$$
communication rounds to get an  $\varepsilon$ -solution, the expected communication complexity is equal to  $\widetilde{\mathcal{O}}(\zeta_{\mathcal{C}}T)$ , and the expected number of stochastic gradient calculations per node equals  $\widetilde{\mathcal{O}}(BT)$  where  $\zeta_{\mathcal{C}}$  is the expected density from Definition 12.
The proof of this corollary almost repeats the proof of Corollary 6. Note that we can skip the initialization procedure and initialize  $h_i^0$  and  $g_i^0$ , for instance, with zeros because the initialization error is under a logarithm.
Let us assume that  $\frac{d}{\zeta c} = \Theta (\omega)$  (holds for the Rand  $K$  compressor), then the convergence rate of DASHA-PP-SYNC-MVR is
$$
\widetilde {\mathcal {O}} \left(\frac {\omega + 1}{p _ {\mathrm {a}}} + \frac {\sigma^ {2}}{p _ {\mathrm {a}} \mu n \varepsilon B} + \frac {L}{\mu} + \frac {\omega}{p _ {\mathrm {a}} \mu \sqrt {n}} \left(\frac {L _ {\sigma}}{\sqrt {B}} + \widehat {L}\right) + \frac {\sigma}{p _ {\mathrm {a}} n \mu^ {3 / 2} \sqrt {\varepsilon B}} \left(\frac {L _ {\sigma}}{\sqrt {B}} + \widehat {L}\right)\right). \tag {34}
$$
Comparing (34) with the rate of DASHA-PP-MVR (31), one can see that DASHA-PP-SYNC-MVR improves the suboptimal term  $\mathcal{P}_2$  from (31). However, Corollary 5 reveals that we can escape these suboptimal regimes by choosing the parameter  $K$  of Rand  $K$  compressors in a particular way.
G.1 Proof for DASHA-PP-SYNC-MVR
In this section, we provide the proof of the convergence rate for DASHA-PP-SYNC-MVR. There are four different sources of randomness in Algorithm 8: the first one from random samples  $\xi^{t+1}$ , the second one from compressors  $\{\mathcal{C}_i\}_{i=1}^n$ , the third one from availability of nodes, and the fourth one from  $c^{t+1}$ . We define  $\mathrm{E}_k[\cdot]$ ,  $\mathrm{E}_{\mathcal{C}}[\cdot]$ ,  $\mathrm{E}_{p_{\mathrm{a}}}[\cdot]$  and  $\mathrm{E}_{p_{\mathrm{mega}}}[\cdot]$  to be conditional expectations w.r.t.  $\xi^{t+1}$ ,  $\{\mathcal{C}_i\}_{i=1}^n$ , availability, and  $c^{t+1}$ , accordingly, conditioned on all previous randomness. Moreover, we define  $\mathrm{E}_{t+1}[\cdot]$  to be a conditional expectation w.r.t. all randomness in iteration  $t+1$  conditioned on all previous randomness.
$$
k _ {i, 1} ^ {t + 1} := \frac {1}{B ^ {\prime}} \sum_ {k = 1} ^ {B ^ {\prime}} \nabla f _ {i} (x ^ {t + 1}; \xi_ {i k} ^ {t + 1}) - \frac {1}{B ^ {\prime}} \sum_ {k = 1} ^ {B ^ {\prime}} \nabla f _ {i} (x ^ {t}; \xi_ {i k} ^ {t + 1}) - \frac {b}{p _ {\mathrm {m e g a}}} \left(h _ {i} ^ {t} - \frac {1}{B ^ {\prime}} \sum_ {k = 1} ^ {B ^ {\prime}} \nabla f _ {i} (x ^ {t}; \xi_ {i k} ^ {t + 1})\right),
$$
$$
k _ {i, 2} ^ {t + 1} := \frac {1}{B} \sum_ {j = 1} ^ {B} \nabla f _ {i} \left(x ^ {t + 1}; \xi_ {i j} ^ {t + 1}\right) - \frac {1}{B} \sum_ {j = 1} ^ {B} \nabla f _ {i} \left(x ^ {t}; \xi_ {i j} ^ {t + 1}\right),
$$
$$
g _ {i, 1} ^ {t + 1} := \left\{ \begin{array}{l l} g _ {i} ^ {t} + \frac {1}{p _ {\mathrm {a}}} k _ {i, 1} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right), & i ^ {\text {t h}} \text {n o d e i s p a r t i c i p a t i n g}, \\ g _ {i} ^ {t}, & \text {o t h e r w i s e}, \end{array} \right.
$$
$$
g _ {i, 2} ^ {t + 1} := \left\{ \begin{array}{l l} g _ {i} ^ {t} + \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i, 2} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right), & i ^ {\text {t h}} \text {n o d e i s p a r t i c i p a t i n g}, \\ g _ {i} ^ {t}, & \text {o t h e r w i s e}, \end{array} \right.
$$
$$
\begin{array}{l} h _ {1} ^ {t + 1} := \frac {1}{n} \sum_ {i = 1} ^ {n} h _ {i, 1} ^ {t + 1}, h _ {2} ^ {t + 1} := \frac {1}{n} \sum_ {i = 1} ^ {n} h _ {i, 2} ^ {t + 1}, g _ {1} ^ {t + 1} := \frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i, 1} ^ {t + 1}, \text {a n d} g _ {2} ^ {t + 1} := \frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i, 2} ^ {t + 1}. \\ \text {N o t e , t h a t} \end{array}
$$
$$
h ^ {t + 1} = \left\{ \begin{array}{l l} h _ {1} ^ {t + 1}, & c ^ {t + 1} = 1, \\ h _ {2} ^ {t + 1}, & c ^ {t + 1} = 0, \end{array} \right.
$$
$$
g ^ {t + 1} = \left\{ \begin{array}{l l} g _ {1} ^ {t + 1}, & c ^ {t + 1} = 1, \\ g _ {2} ^ {t + 1}, & c ^ {t + 1} = 0 \end{array} \right.
$$
First, we will prove two lemmas.
Lemma 13. Suppose that Assumptions 3, 5, 7 and 8 hold and let us consider sequences  $\{g_i^{t+1}\}_{i=1}^n$  and  $\{h_i^{t+1}\}_{i=1}^n$  from Algorithm 8, then
$$
\begin{array}{l} \left. \right. \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {a}} \left[ \operatorname {E} _ {p _ {m e g a}} \left[\left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right]\right]\right] \\ \leq \frac {2 (1 - p _ {\text {m e g a}}) \omega}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \left\| k _ {i, 2} ^ {t + 1} \right\| ^ {2} + \left(\frac {(p _ {\mathrm {a}} - p _ {\mathrm {a a}}) a ^ {2}}{n ^ {2} p _ {\mathrm {a}} ^ {2}} + \frac {2 (1 - p _ {\text {m e g a}}) a ^ {2} \omega}{n ^ {2} p _ {\mathrm {a}}}\right) \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2}, \\ \end{array}
$$
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \mathrm {E} _ {p _ {\text {m e g a}}} \left[\left\| g _ {i} ^ {t + 1} - h _ {i} ^ {t + 1} \right\| ^ {2} \right]\right]\right] \\ \leq \frac {2 (1 - p _ {\text {m e g a}}) \omega}{p _ {\mathrm {a}}} \left\| k _ {i, 2} ^ {t + 1} \right\| ^ {2} + \left(\frac {(1 - p _ {\mathrm {a}}) a ^ {2}}{p _ {\mathrm {a}}} + \frac {2 (1 - p _ {\text {m e g a}}) a ^ {2} \omega}{p _ {\mathrm {a}}}\right) \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ + (1 - a) ^ {2} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2}, \quad \forall i \in [ n ]. \\ \end{array}
$$
Proof. First, we get the bound for  $\mathrm{E}_{t + 1}\left[\left\| g^{t + 1} - h^{t + 1}\right\| ^2\right]$ :
$$
\begin{array}{l} \left. \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \operatorname {E} _ {p _ {\mathrm {m e g a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] \right] \right] \right. \\ = p _ {\text {m e g a}} \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g _ {1} ^ {t + 1} - h _ {1} ^ {t + 1} \right\| ^ {2} \right] + (1 - p _ {\text {m e g a}}) \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g _ {2} ^ {t + 1} - h _ {2} ^ {t + 1} \right\| ^ {2} \right] \right]. \\ \end{array}
$$
$$
\mathrm {E} _ {p _ {\mathrm {a}}} \left[ g _ {i, 1} ^ {t + 1} - h _ {i, 1} ^ {t + 1} \right] = g _ {i} ^ {t} + k _ {i, 1} ^ {t + 1} - a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) - h _ {i} ^ {t} - k _ {i, 1} ^ {t + 1} = (1 - a) \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)
$$
$$
\operatorname {E c} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ g _ {i, 2} ^ {t + 1} - h _ {i, 2} ^ {t + 1} \right] \right] = g _ {i} ^ {t} + k _ {i, 2} ^ {t + 1} - a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) - h _ {i} ^ {t} - k _ {i, 2} ^ {t + 1} = (1 - a) \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right),
$$
$$
\begin{array}{l} \left. \right. \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \mathrm {E} _ {p _ {\mathrm {m e g a}}} \left[\left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right]\right]\right] \\ \stackrel {(1 6)} {=} p _ {\text {m e g a}} \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \left\| g _ {1} ^ {t + 1} - h _ {1} ^ {t + 1} - \mathrm {E} _ {p _ {\mathrm {a}}} \left[ g _ {1} ^ {t + 1} - h _ {1} ^ {t + 1} \right] \right\| ^ {2} \right] \\ + \left(1 - p _ {\text {m e g a}}\right) \operatorname {E} _ {\mathcal {C}} \left[ \operatorname {E} _ {p _ {\mathrm {a}}} \left[ \left\| g _ {2} ^ {t + 1} - h _ {2} ^ {t + 1} - \operatorname {E} _ {p _ {\mathrm {a}}} \left[ g _ {2} ^ {t + 1} - h _ {2} ^ {t + 1} \right] \right\| ^ {2} \right] \right] \\ + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2}. \\ \end{array}
$$
We can use Lemma 1 two times with i)  $r_i = g_i^t - h_i^t$  and  $s_i = -a(g_i^t - h_i^t)$  and ii)  $r_i = g_i^t - h_i^t$  and  $s_i = p_{\mathrm{a}}\mathcal{C}_i\left(\frac{1}{p_{\mathrm{a}}} k_{i,2}^{t + 1} - \frac{a}{p_{\mathrm{a}}} (g_i^t - h_i^t)\right) - k_{i,2}^{t + 1}$ , to obtain
$$
\begin{array}{l} \mathrm {E} _ {\mathcal {C}} \left[ \mathrm {E} _ {p _ {\mathrm {a}}} \left[ \mathrm {E} _ {p _ {\mathrm {m e g a}}} \left[ \left\| g ^ {t + 1} - h ^ {t + 1} \right\| ^ {2} \right] \right] \right] \\ \leq \frac {p _ {\mathrm {m e g a}} a ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ \left. \right. + \left(1 - p _ {\text {m e g a}}\right)\left(\frac {1}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \mathrm {E} _ {\mathcal {C}} \left[\left\| p _ {\mathrm {a}} \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i, 2} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) - \left(k _ {i, 2} ^ {t + 1} - a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right)\right\| ^ {2} \right]\right) \\ + \left(1 - p _ {\text {m e g a}}\right) \left(\frac {a ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2}\right) \\ + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \\ = \frac {a ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \left. \right. + \left(1 - p _ {\text {m e g a}}\right)\left(\frac {p _ {\mathrm {a}}}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathrm {E} _ {\mathcal {C}} \left[\left\| \mathcal {C} _ {i} \left(\frac {1}{p _ {\mathrm {a}}} k _ {i, 2} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right) - \left(\frac {1}{p _ {\mathrm {a}}} k _ {i, 2} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right)\right)\right\| ^ {2} \right]\right) \\ + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \\ \leq \frac {a ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ + \frac {\left(1 - p _ {\mathrm {m e g a}}\right) p _ {\mathrm {a}} \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \frac {1}{p _ {\mathrm {a}}} k _ {i, 2} ^ {t + 1} - \frac {a}{p _ {\mathrm {a}}} \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) \right\| ^ {2} \\ + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2} \\ = \frac {a ^ {2} \left(p _ {\mathrm {a}} - p _ {\mathrm {a a}}\right)}{n ^ {2} p _ {\mathrm {a}} ^ {2}} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - h _ {i} ^ {t} \right\| ^ {2} \\ + \frac {(1 - p _ {\mathrm {m e g a}}) \omega}{n ^ {2} p _ {\mathrm {a}}} \sum_ {i = 1} ^ {n} \left\| k _ {i, 2} ^ {t + 1} - a \left(g _ {i} ^ {t} - h _ {i} ^ {t}\right) \right\| ^ {2} \\ + (1 - a) ^ {2} \left\| g ^ {t} - h ^ {t} \right\| ^ {2}. \\ \end{array}
$$
In the last inequality, we use Assumption 7. Next, using (15), we have