SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| \boldsymbol {Q} _ {j} \| = \left\| \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (j)) N ^ {\prime} \mathbf {H} _ {t ^ {\prime}})\right) \eta \hat {\alpha} (\sigma (j)) N \right\| \\ \leq \eta \hat {\alpha} (\sigma (j)) N \sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} \| (\mathbf {I} - \eta \hat {\alpha} (\sigma (j)) N ^ {\prime} \mathbf {H} _ {t ^ {\prime}}) \| \\ \leq \eta \hat {\alpha} (\sigma (j)) N \sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (1 + \eta \hat {\alpha} (\sigma (j)) N ^ {\prime} L) \\ \leq \eta \hat {\alpha} (\sigma (j)) N K (1 + \eta N L) ^ {K}. \\ \end{array}
$$
The last step is due to we choose  $\eta$  such that  $\eta NL\leq \frac{1}{K}$
The following lemma establishes the bound regarding cumulative update between two epochs, namely,  $\pmb{v}^{r + 1} - \pmb{v}^r$ . In particular, Lemma 11 below shows that: (a) in shuffling Local SGD, our update from  $\pmb{v}^r$  to  $\pmb{v}^{r + 1}$  approximates performing  $NK$  times of gradient descent with  $\hat{\alpha} (j)N\nabla f_{\sigma (j)}(\pmb{v}^r)$ , namely, the bias is controlled, and (b) the update itself is bounded, and can be related to the norm of full gradient.
Lemma 11. During the dynamic of Algorithm A2, the following statements hold true with probability at least  $1 - p$ :
$$
\begin{array}{l} \left\| \sum_ {j = 1} ^ {N} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \eta N K \sum_ {j = 1} ^ {N} \hat {\alpha} (j) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2} \leq 1 0 \eta^ {2} N ^ {2} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} \| \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \| ^ {2} \\ + 1 2 8 \eta^ {2} N ^ {3} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} G ^ {2} \log (1 / p). \\ \end{array}
$$
(b) for any  $N'$  such that  $0 \leq N' < N$
$$
\left\| \sum_ {j = 1} ^ {N ^ {\prime} - 1} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \leq 3 e \eta N K \left(\| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| + G \sqrt {8 N \log (1 / p)}\right),
$$
$$
\boldsymbol {Q} _ {j} := \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (j)) N ^ {\prime} \mathbf {H} _ {t ^ {\prime}})\right) \eta \hat {\alpha} (\sigma (j)) N. \tag {11}
$$
Proof. We start with proving statement (a). Let  $\mathbf{A}_j = \frac{\mathbf{Q}_j}{\hat{\alpha}(\sigma(j))}$  and  $\mathbf{B}_j = \hat{\alpha}(\sigma(j)) \nabla f_{\sigma(j)}(\mathbf{v}^r)$ , applying the identity of summation by parts yields:
$$
\sum_ {j = 1} ^ {N} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) = \frac {\boldsymbol {Q} _ {N - 1}}{\hat {\alpha} (\sigma (N - 1))} \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {n = 1} ^ {N - 1} \left(\frac {\boldsymbol {Q} _ {n + 1}}{\hat {\alpha} (\sigma (n + 1))} - \frac {\boldsymbol {Q} _ {n}}{\hat {\alpha} (\sigma (n))}\right) \sum_ {j = 1} ^ {n} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r})
$$
$$
\begin{array}{l} \left\| \sum_ {j = 1} ^ {N} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v}) - \eta N K \sum_ {j = 1} ^ {N} \hat {\alpha} (j) \nabla f _ {j} (\boldsymbol {v}) \right\| ^ {2} \\ \leq 2 \underbrace {\left\| \left(\frac {\boldsymbol {Q} _ {N - 1}}{\hat {\alpha} (\sigma (N - 1))} - \eta N K \mathbf {I}\right) \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2}} _ {T _ {1}} + 2 \underbrace {\left\| \sum_ {n = 1} ^ {N - 1} \left(\frac {\boldsymbol {Q} _ {n + 1}}{\hat {\alpha} (\sigma (n + 1))} - \frac {\boldsymbol {Q} _ {n}}{\hat {\alpha} (\sigma (n))}\right) \sum_ {j = 1} ^ {n} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2}} _ {T _ {2}}. \\ \end{array}
$$
According to Proposition 2, we have:
$$
\left\| \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (j)) N \mathbf {H} _ {t ^ {\prime}}) - \mathbf {I} \right\| \leq \sum_ {m = 1} ^ {K - 2 - \tau} \left(\frac {e (K - 2 - \tau)}{m} \eta \hat {\alpha} (\sigma (j)) N L\right) ^ {m}.
$$
Since we choose  $\eta \leq \frac{1}{4NKRL}$ , we have:
$$
\left\| \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (j)) N \mathbf {H} _ {t ^ {\prime}}) - \mathbf {I} \right\| \leq \sum_ {m = 1} ^ {K - 2 - \tau} \left(\frac {e}{4 R m}\right) ^ {m} \leq \frac {e}{4 R - e}, \tag {12}
$$
where we use the fact that  $\sum_{m=1}^{K-2-\tau}\left(\frac{e}{4Rm}\right)^m \leq \sum_{m=1}^{K-2-\tau}\left(\frac{e}{4R}\right)^m \leq \frac{e}{4R}\frac{1}{1-e/4R}$ . Hence we know:
$$
\begin{array}{l} T _ {1} \leq \left\| \left(\frac {\boldsymbol {Q} _ {N - 1}}{\hat {\alpha} (\sigma (N - 1))} - \eta N K \mathbf {I}\right) \right\| ^ {2} \left\| \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2} \\ \leq \left\| \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (N - 1)) N \mathbf {H} _ {t ^ {\prime}})\right) \eta N - \eta N K \mathbf {I} \right\| ^ {2} \left\| \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2} \\ \leq \eta^ {2} N ^ {2} K \sum_ {\tau = 0} ^ {K - 1} \left\| \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (N - 1)) N \mathbf {H} _ {t ^ {\prime}}) - \mathbf {I} \right\| ^ {2} \left\| \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2} \\ \leq \eta^ {2} N ^ {2} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} \left\| \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2}. \\ \end{array}
$$
$$
T _ {1} \leq \eta^ {2} N ^ {2} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| ^ {2}.
$$
For  $T_{2}$ , we first examine the bound of  $\frac{Q_{n + 1}}{\hat{\alpha}(\sigma(n + 1))} -\frac{Q_n}{\hat{\alpha}(\sigma(n))}$ :
$$
\begin{array}{l} \left\| \frac {\boldsymbol {Q} _ {n + 1}}{\hat {\alpha} (\sigma (n + 1))} - \frac {\boldsymbol {Q} _ {n}}{\hat {\alpha} (\sigma (n))} \right\| = \left\| \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (n + 1)) N \mathbf {H} _ {t ^ {\prime}})\right) \eta N - \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (n)) N \mathbf {H} _ {t ^ {\prime}})\right) \eta N \right\| \\ = \eta N \left\| \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (n + 1)) N \mathbf {H} _ {t ^ {\prime}})\right) - \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (n)) N \mathbf {H} _ {t ^ {\prime}})\right) \right\| \\ = \eta N \left\| \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (n + 1)) N \mathbf {H} _ {t ^ {\prime}})\right) - \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (n)) N \mathbf {H} _ {t ^ {\prime}})\right) \right\| \\ \leq \eta N \sum_ {\tau = 0} ^ {K - 1} \left(\sum_ {m = 1} ^ {K - 2 - \tau} \left(\frac {e (K - 2 - \tau)}{m} \eta \hat {\alpha} (\sigma (n)) N L\right) ^ {m} + \sum_ {m = 1} ^ {K - 2 - \tau} \left(\frac {e (K - 2 - \tau)}{m} \eta \hat {\alpha} (\sigma (n + 1)) N L\right) ^ {m}\right). \\ \end{array}
$$
where we evoke Proposition 2 at last step. Given that  $\eta \leq \frac{1}{4NKRL}$  we have:
$$
\begin{array}{l} \left\| \frac {\boldsymbol {Q} _ {n + 1}}{\hat {\alpha} (\sigma (n + 1))} - \frac {\boldsymbol {Q} _ {n}}{\hat {\alpha} (\sigma (n))} \right\| \leq \eta N \sum_ {\tau = 0} ^ {K - 1} \left(\sum_ {m = 1} ^ {K - 2 - \tau} \left(\frac {e}{4 R m} \hat {\alpha} (\sigma (n))\right) ^ {m} + \sum_ {m = 1} ^ {K - 2 - \tau} \left(\frac {e}{4 R m} \hat {\alpha} (\sigma (n + 1)) L\right) ^ {m}\right) \\ \leq \eta N K \left(\frac {\hat {\alpha} (\sigma (n)) e}{4 R - e} + \frac {\hat {\alpha} (\sigma (n + 1)) e}{4 R - e}\right). \\ \end{array}
$$
where we use the reasoning in (12). Hence for  $\sqrt{T_2}$
$$
\begin{array}{l} \sqrt {T _ {2}} \leq \eta N K \sum_ {n = 1} ^ {N - 1} \left(\frac {\hat {\alpha} (\sigma (n)) e}{4 R - e} + \frac {\hat {\alpha} (\sigma (n + 1)) e}{4 R - e}\right) \left\| \sum_ {j = 1} ^ {n} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \\ \leq \eta N K \frac {e}{4 R - e} \sum_ {n = 1} ^ {N - 1} \left(\hat {\alpha} (\sigma (n)) + \hat {\alpha} (\sigma (n + 1))\right) \left(G \sqrt {8 n \log (1 / p)} + \frac {n}{N} \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \|\right) \\ \leq \eta N K \frac {2 e}{4 R - e} \left(G \sqrt {8 N \log (1 / p)} + \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \|\right). \\ \end{array}
$$
where at last step we evoke Lemma 10. So we can conclude  $T_{2} \leq 2\eta^{2}N^{2}K^{2}\left(\frac{2e}{4R - e}\right)^{2}\left(G^{2}8N\log (1 / p) + \| \nabla \Phi (\hat{\alpha},\pmb{v}^{r})\|^{2}\right)$ . Putting the bounds of  $T_{1}$  and  $T_{2}$  together will conclude the proof for (a).
Now we switch to proving (b). Once again by the summation of parts identity we have:
$$
\sum_ {j = 1} ^ {N ^ {\prime}} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) = \frac {\boldsymbol {Q} _ {N ^ {\prime}}}{\hat {\alpha} (\sigma (N ^ {\prime}))} \sum_ {j = 1} ^ {N ^ {\prime}} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {n = 1} ^ {N ^ {\prime} - 1} \left(\frac {\boldsymbol {Q} _ {n + 1}}{\hat {\alpha} (\sigma (n + 1))} - \frac {\boldsymbol {Q} _ {n}}{\hat {\alpha} (\sigma (n))}\right) \sum_ {j = 1} ^ {n} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}).
$$
Taking the norm of both side yields:
$$
\begin{array}{l} \left\| \sum_ {j = 1} ^ {N ^ {\prime}} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| = \underbrace {\left\| \frac {\boldsymbol {Q} _ {N ^ {\prime}}}{\hat {\alpha} (\sigma (N ^ {\prime}))} \sum_ {j = 1} ^ {N ^ {\prime}} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\|} _ {B} \\ + \underbrace {\left\| \sum_ {n = 1} ^ {N ^ {\prime} - 1} \left(\frac {\boldsymbol {Q} _ {n + 1}}{\hat {\alpha} (\sigma (n + 1))} - \frac {\boldsymbol {Q} _ {n}}{\hat {\alpha} (\sigma (n))}\right) \sum_ {j = 1} ^ {n} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\|} _ {C}. \\ \end{array}
$$
Plugging our developed bound for  $\| Q_{N'}\|$  and  $\left\| \sum_{n = 1}^{N' - 1}\left(\frac{Q_{n + 1}}{\hat{\alpha}(\sigma(n + 1))} -\frac{Q_n}{\hat{\alpha}(\sigma(n))}\right)\right\|$  yields:
$$
\begin{array}{l} B \leq \left\| \frac {\boldsymbol {Q} _ {N ^ {\prime}}}{\hat {\alpha} (\sigma (N ^ {\prime}))} \right\| \left\| \sum_ {j = 1} ^ {N ^ {\prime}} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \\ \leq \eta N K (1 + \eta N L) ^ {K} \left(G \sqrt {8 N ^ {\prime} \log (1 / p)} + \frac {N ^ {\prime}}{N} \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \|\right). \\ \end{array}
$$
where at last step we evoke Lemma 10. And for C, we use the similar reasoning:
$$
\begin{array}{l} C \leq \sum_ {n = 1} ^ {N ^ {\prime} - 1} \left\| \left(\frac {\boldsymbol {Q} _ {n + 1}}{\hat {\alpha} (\sigma (n + 1))} - \frac {\boldsymbol {Q} _ {n}}{\hat {\alpha} (\sigma (n))}\right) \right\| \left\| \sum_ {j = 1} ^ {n} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \\ \leq \sum_ {n = 1} ^ {N ^ {\prime} - 1} \eta N K \frac {e}{4 R - e} (\hat {\alpha} (\sigma (n + 1)) + \hat {\alpha} (\sigma (n))) \left(G \sqrt {8 n \log (1 / p)} + \frac {n}{N} \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \|\right) \\ \leq 2 \eta N K \frac {e}{4 R - e} \left(G \sqrt {8 N \log (1 / p)} + \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \|\right). \\ \end{array}
$$
Putting these pieces together yields:
$$
\left\| \sum_ {j = 1} ^ {N ^ {\prime}} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \leq 3 e \eta N K \left(\| \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \| + G \sqrt {8 N \log (1 / p)}\right).
$$
Lemma 12. During the dynamic of Algorithm A2, the following statements hold true with probability at least  $1 - p$ :
$$
\begin{array}{l} \left\| \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) \boldsymbol {Q} _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 1} ^ {n} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2} \\ \leq 1 8 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4} \left(\| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| ^ {2} + 8 G ^ {2} N \log (1 / p)\right) \\ \end{array}
$$
Proof. We first apply Cauchy-Schwartz inequality:
$$
\begin{array}{l} \left\| \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - Q _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) Q _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 1} ^ {n} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \\ \leq \sum_ {n = 1} ^ {N - 1} \left\| \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) \right\| \| \boldsymbol {Q} _ {n + 1} \mathbf {H} _ {n + 1} \| \left\| \sum_ {j = 1} ^ {n} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \\ \leq \left(1 + \eta N K + \eta^ {2} N ^ {2} K L\right) ^ {2 N} \eta N L K (1 + \eta N L) ^ {K} L \sum_ {n = 1} ^ {N - 1} \hat {\alpha} (\sigma (n + 1)) \left\| \sum_ {j = 1} ^ {n} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \\ \leq e ^ {2} \eta N K L ^ {2} \sum_ {n = 1} ^ {N - 1} \hat {\alpha} (\sigma (n + 1)) \left\| \sum_ {j = 1} ^ {n} \boldsymbol {Q} _ {J} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\|. \\ \end{array}
$$
We proceed by applying the bound from Lemma 11 (b):
$$
\left\| \sum_ {j = 1} ^ {n} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \leq 3 e \eta N K \left(\| \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \| + G \sqrt {8 N \log (1 / p)}\right).
$$
Therefore, it follows that:
$$
\begin{array}{l} \left\| \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - Q _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) Q _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 1} ^ {n} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \\ \leq e ^ {2} \eta N K L ^ {2} \sum_ {n = 1} ^ {N - 1} \hat {\alpha} (\sigma (n + 1)) \cdot 3 e \eta N K \left(\| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| + G \sqrt {8 N \log (1 / p)}\right) \\ \leq 3 e ^ {3} \eta^ {2} N ^ {2} K ^ {2} L ^ {2} \left(\| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| + G \sqrt {8 N \log (1 / p)}\right) \\ \end{array}
$$
Lemma 13 (Noise bound). During the dynamic of Algorithm A2, the following statement for gradient noises holds true with probability at least  $1 - p$ :
$$
\mathbb {E} \left\| \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N} ^ {j + 1} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right) \boldsymbol {\delta} _ {j} \right\| \leq \eta N K e ^ {2} \delta ,
$$
$$
\boldsymbol {\delta} _ {j} := \sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = t} ^ {\tau + 1} (\mathbf {I} - \hat {\alpha} (\sigma (j)) N \eta \mathbf {H} _ {t ^ {\prime}}) \eta \hat {\alpha} (\sigma (j)) N \boldsymbol {\delta} _ {\sigma (j)} ^ {t}.
$$
Proof. According to triangle and Cauchy-Schwartz inequalities we have:
$$
\begin{array}{l} \left\| \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N} ^ {j + 1} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right) \boldsymbol {\delta} _ {j} \right\| \leq \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N} ^ {j + 1} \| \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right) \| \| \boldsymbol {\delta} _ {j} \| \\ \leq \sum_ {j = 1} ^ {N} \left(1 + (\eta \hat {\alpha} (\sigma (j)) N K (1 + \eta N L) ^ {K}) L\right) ^ {N} \| \boldsymbol {\delta} _ {j} \| \\ \leq \sum_ {j = 1} ^ {N} \underbrace {\left(1 + (\eta \hat {\alpha} (\sigma (j)) N K (1 + \eta N L) ^ {K}\right) L} _ {\leq e} ^ {N} \cdot \eta \hat {\alpha} (\sigma (j)) N K \underbrace {(1 + \eta N L) ^ {K}} _ {\leq e} \delta \\ \leq \eta N K e ^ {2} \delta . \\ \end{array}
$$
B.4 Proof of Theorem 2
Proof. For notational convenience, let us define
$$
\boldsymbol {g} ^ {r} := \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N} ^ {j + 1} (\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}) \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}),
$$
$$
\boldsymbol {\delta} ^ {r} := \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N} ^ {j + 1} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right) \boldsymbol {\delta} _ {j}.
$$
Then we recall the updating rule of  $\pmb{v}$  (Lemma 7):
$$
\boldsymbol {v} ^ {r + 1} = \mathcal {P} _ {\mathcal {W}} \left(\boldsymbol {v} ^ {r} - \boldsymbol {g} ^ {r} - \boldsymbol {\delta} ^ {r}\right)
$$
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} = \mathbb {E} \left\| \mathcal {P} _ {\mathcal {W}} \left(\boldsymbol {v} ^ {r} - \boldsymbol {g} ^ {r} - \boldsymbol {\delta} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}})\right) \right\| ^ {2} \\ \leq \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {g} ^ {r} - \boldsymbol {\delta} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \| ^ {2} \\ \leq \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \| ^ {2} - 2 \mathbb {E} \langle \boldsymbol {g} ^ {r}, \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \rangle + \mathbb {E} \| \boldsymbol {g} ^ {r} \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2} \\ \leq \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \| ^ {2} - 2 \mathbb {E} \langle \eta N K \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}), \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \rangle - 2 \mathbb {E} \langle \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}), \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \rangle \\ + \mathbb {E} \| \boldsymbol {g} ^ {r} \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2}. \\ \end{array}
$$
Now, applying strongly convexity of  $\Phi (\hat{\alpha},\cdot)$  and Cauchy-Schwartz inequality yields:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} \leq (1 - \mu \eta N K) \mathbb {E} \left\| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} - \eta N K \mathbb {E} [ \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) - \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}})) ] \\ + \frac {1}{2} \left(\frac {1}{\mu \eta N K} \mathbb {E} \| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \| ^ {2} + \mu \eta N K \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\alpha}) \| ^ {2}\right) \\ + \mathbb {E} \| \boldsymbol {g} ^ {r} \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2} \\ \leq \left(1 - \frac {1}{2} \mu \eta N K\right) \mathbb {E} \left\| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} - \eta N K \mathbb {E} \left[ \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) - \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}})) \right] \\ + \frac {1}{2 \mu \eta N K} \mathbb {E} \| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| ^ {2} \\ + 2 \mathbb {E} \| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| ^ {2} + 2 \mathbb {E} \| \eta N K \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2}. \\ \end{array}
$$
Since  $\Phi (\hat{\alpha},\cdot)$  is  $L$  smooth, we have:  $\mathbb{E}\left\| \nabla \Phi (\hat{\alpha},\pmb{v}^r)\right\| ^2\leq 2L\mathbb{E}[\Phi (\hat{\alpha},\pmb{v}^r) - \Phi (\hat{\alpha},\pmb{v}^* (\hat{\alpha}))]$ . Therefore, we have:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} \leq (1 - \frac {1}{2} \mu \eta N K) \mathbb {E} \left\| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} - (\eta N K - 4 \eta^ {2} N ^ {2} K ^ {2} L) \mathbb {E} [ \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) - \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}})) ] (13) \\ + \left(\frac {1}{2 \mu \eta N K} + 2\right) \mathbb {E} \| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2}. (14) \\ \end{array}
$$
Now, we examine the term  $\| g^r - \eta NK\nabla \Phi(\hat{\alpha},\pmb{v}^r)\|^2$ . First according to summation by part (Lemma 8) by letting  $\mathbf{A}_j \coloneqq \prod_{j' = N}^{j + 1}(\mathbf{I} - \mathbf{Q}_{j'}\mathbf{H}_{j'})$  and  $\mathbf{B}_j = \mathbf{Q}_j\nabla f_{\sigma(j)}(\pmb{v}^r)$ , we have:
$$
\begin{array}{l} \boldsymbol {g} ^ {r} = \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N} ^ {j + 1} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right) \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \\ = \sum_ {j = 1} ^ {N} \boldsymbol {A} _ {j} \mathbf {B} _ {j} = \sum_ {j = 1} ^ {N} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right) - \prod_ {j ^ {\prime} = N} ^ {n + 1} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) \sum_ {j = 1} ^ {n} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \\ = \sum_ {j = 1} ^ {N} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - Q _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) Q _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 1} ^ {n} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}). \\ \end{array}
$$
$$
\begin{array}{l} \left\| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \right\| ^ {2} \\ = \left\| \eta N K \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \left(\sum_ {j = 1} ^ {N} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) \boldsymbol {Q} _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 1} ^ {n} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r})\right) \right\| ^ {2} \\ \end{array}
$$