SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {R}) - \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {*}) \right\| ^ {2} = \left\| \boldsymbol {\alpha} _ {W} ^ {*} (\boldsymbol {z} ^ {R}) - \boldsymbol {\alpha} _ {W} ^ {*} (\boldsymbol {z} ^ {*}) \right\| ^ {2} \leq \kappa_ {g} ^ {2} \| \boldsymbol {z} ^ {R} - \boldsymbol {z} ^ {*} \| ^ {2} (7) \\ \leq \kappa_ {g} ^ {2} \sum_ {j = 1} ^ {N} \left| \left\| \nabla f _ {i} (\boldsymbol {w} ^ {R}) - \nabla f _ {j} (\boldsymbol {w} ^ {R}) \right\| ^ {2} - \left\| \nabla f _ {i} (\boldsymbol {w} ^ {*}) - \nabla f _ {j} (\boldsymbol {w} ^ {*}) \right\| ^ {2} \right| ^ {2} (8) \\ \leq \kappa_ {g} ^ {2} \sum_ {j = 1} ^ {N} \left| \left(\nabla f _ {i} (\boldsymbol {w} ^ {R}) - \nabla f _ {j} (\boldsymbol {w} ^ {R}) + \nabla f _ {i} (\boldsymbol {w} ^ {*}) - \nabla f _ {j} (\boldsymbol {w} ^ {*})\right) \right. \\ \left. \times \left(\nabla f _ {i} (\boldsymbol {w} ^ {R}) - \nabla f _ {j} (\boldsymbol {w} ^ {R}) - \nabla f _ {i} (\boldsymbol {w} ^ {*}) + \nabla f _ {j} (\boldsymbol {w} ^ {*})\right) \right| ^ {2} \\ \leq \kappa_ {g} ^ {2} \sum_ {j = 1} ^ {N} \left\| \nabla f _ {i} (\boldsymbol {w} ^ {R}) - \nabla f _ {j} (\boldsymbol {w} ^ {R}) + \nabla f _ {i} (\boldsymbol {w} ^ {*}) - \nabla f _ {j} (\boldsymbol {w} ^ {*}) \right\| ^ {2} 4 L ^ {2} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2} \\ \end{array}
$$
Since  $\left\| \nabla f_{i}(\pmb{w}^{R}) - \nabla f_{j}(\pmb{w}^{R})\right\| \leq \left\| \nabla f_{i}(\pmb{w}^{*}) - \nabla f_{j}(\pmb{w}^{*})\right\| + 2L\left\| \pmb{w}^{R} - \pmb{w}^{*}\right\|$ , we can conclude that
$$
\left\| \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {R}) - \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {*}) \right\| \leq \kappa_ {g} ^ {2} \sum_ {j = 1} ^ {N} \left(2 \left\| \nabla f _ {i} (\boldsymbol {w} ^ {*}) - \nabla f _ {j} (\boldsymbol {w} ^ {*}) \right\| ^ {2} + 4 L ^ {2} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2}\right) 4 L \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2}.
$$
With above lemma, to show the convergence of  $\hat{\alpha}$  to  $\alpha^{*}$ , we do the following decomposition
$$
\begin{array}{l} \left\| \hat {\boldsymbol {\alpha}} - \boldsymbol {\alpha} ^ {*} \right\| ^ {2} \leq 2 \left\| \hat {\boldsymbol {\alpha}} - \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {R}) \right\| ^ {2} + 2 \left\| \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {R}) - \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {*}) \right\| ^ {2} \\ \leq 2 \left(1 - \mu \eta_ {\boldsymbol {\alpha}}\right) ^ {K} + 2 \kappa_ {g} ^ {2} \sum_ {j = 1} ^ {N} \left(2 \left\| \nabla f _ {i} (\boldsymbol {w} ^ {*}) - \nabla f _ {j} (\boldsymbol {w} ^ {*}) \right\| ^ {2} + 4 L ^ {2} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2}\right) 4 L \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2}. \\ \end{array}
$$
Now it remains to show the convergence of Local SGD last iterate  $\boldsymbol{w}^R$  to optimal solution  $\boldsymbol{w}^*$ . By convention, we use  $\boldsymbol{w}^t = \frac{1}{N}\sum_{i=1}^{N}\boldsymbol{w}_i^t$  to denote the virtual average iterates.
Lemma 4 (One iteration analysis of Local SGD). Under the condition of Theorem 1, the following statement holds true for any  $t \in [T]$ :
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {w} ^ {r, t + 1} - \boldsymbol {w} ^ {*} \right\| ^ {2} \leq (1 - \mu \gamma) \mathbb {E} \left\| \boldsymbol {w} ^ {r, t} - \boldsymbol {w} ^ {*} \right\| ^ {2} - (2 \gamma - 4 \gamma^ {2} L) \mathbb {E} \left(F (\boldsymbol {w} ^ {*}) - F (\boldsymbol {w} ^ {r, t})\right) \\ + (\gamma L + 2 \gamma^ {2} L ^ {2}) \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {r, t} \right\| ^ {2} + \gamma^ {2} \frac {\delta^ {2}}{N}. \\ \end{array}
$$
Proof. According to updating rule in Algorithm A1, we have the following identity:
$$
\mathbb {E} \left\| \boldsymbol {w} ^ {r, t + 1} - \boldsymbol {w} ^ {*} \right\| ^ {2} = \mathbb {E} \left\| \boldsymbol {w} ^ {r, t} - \boldsymbol {w} ^ {*} \right\| ^ {2} - 2 \gamma \mathbb {E} \left\langle \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} \left(\boldsymbol {w} _ {i} ^ {r, t}; z _ {i} ^ {r, t}\right), \boldsymbol {w} ^ {r, t} - \boldsymbol {w} ^ {*} \right\rangle + \gamma^ {2} \mathbb {E} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} \left(\boldsymbol {w} _ {i} ^ {r, t}\right) \right\| ^ {2}
$$
$$
\begin{array}{l} = \mathbb {E} \left\| \boldsymbol {w} ^ {t} - \boldsymbol {w} ^ {*} \right\| ^ {2} \underbrace {- 2 \gamma \left\langle \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} \left(\boldsymbol {w} _ {i} ^ {r , t}\right) , \boldsymbol {w} ^ {r , t} - \boldsymbol {w} ^ {*} \right\rangle} _ {T _ {1}} \\ + \gamma^ {2} \underbrace {\mathbb {E} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} \left(\boldsymbol {w} _ {i} ^ {r , t} ; z _ {i} ^ {r , t}\right) \right\| ^ {2}} _ {T _ {2}} + \gamma^ {2} \frac {\delta^ {2}}{N}. \tag {10} \\ \end{array}
$$
For  $T_{1}$ , since each  $f_{j}$  is  $L$  smooth and  $\mu$  strongly convex, we have:
$$
\begin{array}{l} - 2 \gamma \left\langle \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} \left(\boldsymbol {w} _ {i} ^ {r, t}\right), \boldsymbol {w} ^ {t} - \boldsymbol {w} ^ {*} \right\rangle = - 2 \gamma \left\langle \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} \left(\boldsymbol {w} _ {i} ^ {t}\right), \boldsymbol {w} ^ {t} - \boldsymbol {w} _ {i} ^ {t} + \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {*} \right\rangle \\ \leq - 2 \gamma \left\langle \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} \left(\boldsymbol {w} _ {i} ^ {t}\right), \boldsymbol {w} ^ {r, t} - \boldsymbol {w} _ {i} ^ {r, t} + \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {*} \right\rangle \\ \leq 2 \gamma \frac {1}{N} \sum_ {i = 1} ^ {N} \left(f _ {i} (\boldsymbol {w} ^ {*}) - f _ {i} (\boldsymbol {w} ^ {r, t}) - \frac {\mu}{2} \left\| \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {*} \right\| ^ {2} + \frac {L}{2} \left\| \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {r, t} \right\| ^ {2}\right). \\ \end{array}
$$
Due to Jensen's inequality we know:  $-\frac{1}{N}\sum_{i = 1}^{N}\frac{\mu}{2}\left\| \pmb {w}_i^t -\pmb {w}^*\right\|^2\leq -\frac{\mu}{2}\left\| \pmb {w}^t -\pmb {w}^*\right\|^2$  . Hence we know:
$$
- 2 \gamma \left\langle \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} (\boldsymbol {w} _ {i} ^ {r, t}), \boldsymbol {w} ^ {r, t} - \boldsymbol {w} ^ {*} \right\rangle \leq 2 \gamma \left(F (\boldsymbol {w} ^ {*}) - F (\boldsymbol {w} ^ {r, t}) - \frac {\mu}{2} \left\| \boldsymbol {w} ^ {r, t} - \boldsymbol {w} ^ {*} \right\| ^ {2} + \frac {L}{2} \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {r, t} \right\| ^ {2}\right).
$$
For  $T_{2}$ , we have:
$$
\begin{array}{l} \mathbb {E} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} (\boldsymbol {w} _ {i} ^ {r, t}) \right\| ^ {2} = 2 \mathbb {E} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla f _ {i} (\boldsymbol {w} _ {i} ^ {r, t}) - \nabla F (\boldsymbol {w} ^ {r, t}) \right\| ^ {2} + 2 \mathbb {E} \left\| \nabla F (\boldsymbol {w} ^ {r, t}) \right\| ^ {2} \\ \leq 2 L ^ {2} \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} \left| \left| \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {r, t} \right| \right| ^ {2} + 4 L \left(F (\boldsymbol {w} ^ {r, t}) - F (\boldsymbol {w} ^ {*})\right). \\ \end{array}
$$
Now, plugging  $T_{1}$  and  $T_{2}$  back to (10) yields:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {w} ^ {r, t + 1} - \boldsymbol {w} ^ {*} \right\| ^ {2} \leq (1 - \mu \gamma) \mathbb {E} \left\| \boldsymbol {w} ^ {r, t} - \boldsymbol {w} ^ {*} \right\| ^ {2} - (2 \gamma - 4 \gamma^ {2} L) \mathbb {E} \left(F (\boldsymbol {w} ^ {*}) - F (\boldsymbol {w} ^ {r, t})\right) \\ + \left(\gamma L + 2 \gamma^ {2} L ^ {2}\right) \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {r, t} \right\| ^ {2} + \gamma^ {2} \frac {\delta^ {2}}{N}. \\ \end{array}
$$
Lemma 5. [50, Lemma 8] For the iterates  $\{\pmb{w}_i^{r,t}\}$  generated in Algorithm A1, the following statement holds true:
$$
\frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {r, t} \right\| ^ {2} \leq 3 K \gamma^ {2} \delta^ {2} + 6 K ^ {2} \gamma^ {2} \zeta^ {2}.
$$
Lemma 6 (Last iterate convergence of Local SGD). Under the conditions of Theorem 1, the following statement holds true for the iterates in Algorithm A1:
$$
\mathbb {E} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2} \leq (1 - \mu \gamma) ^ {R K} \mathbb {E} \left\| \boldsymbol {w} ^ {0} - \boldsymbol {w} ^ {*} \right\| ^ {2} + \frac {1}{\mu \gamma} (\gamma L + 2 \gamma^ {2} L ^ {2}) \left(3 K \gamma^ {2} \delta^ {2} + 6 K ^ {2} \gamma^ {2} \zeta^ {2}\right) + \frac {\gamma \delta^ {2}}{\mu N}
$$
Proof. We first unroll the recursion in Lemma 4 from  $t = K$  to 0, within one communication round:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {w} ^ {r, K} - \boldsymbol {w} ^ {*} \right\| ^ {2} = (1 - \mu \gamma) ^ {K} \mathbb {E} \left\| \boldsymbol {w} ^ {r, 0} - \boldsymbol {w} ^ {*} \right\| ^ {2} - \sum_ {t = 0} ^ {K - 1} (1 - \mu \gamma) ^ {K - t} (2 \gamma - 4 \gamma^ {2} L) \mathbb {E} \left(F (\boldsymbol {w} ^ {*}) - F (\boldsymbol {w} ^ {r, t})\right) \\ + \sum_ {t = 0} ^ {K - 1} (1 - \mu \gamma) ^ {K - t} (\gamma L + 2 \gamma^ {2} L ^ {2}) \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \boldsymbol {w} _ {i} ^ {r, t} - \boldsymbol {w} ^ {r, t} \right\| ^ {2} + \sum_ {t = 0} ^ {K - 1} (1 - \mu \gamma) ^ {K - t} \gamma^ {2} \frac {\delta^ {2}}{N} \\ \end{array}
$$
Since we choose  $\gamma \leq \frac{1}{2L}$ , we know  $\sum_{t=0}^{K-1}(1 - \mu\gamma)^{K-t}(2\gamma - 4\gamma^2 L)\mathbb{E}\left(F(\boldsymbol{w}^*) - F(\boldsymbol{w}^{r,t})\right) \geq 0$ . Plugging in Lemma 5 yields:
$$
\mathbb {E} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2} = (1 - \mu \gamma) ^ {R K} \mathbb {E} \left\| \boldsymbol {w} ^ {0} - \boldsymbol {w} ^ {*} \right\| ^ {2} + \frac {1}{\mu \gamma} (\gamma L + 2 \gamma^ {2} L ^ {2}) \left(3 K \gamma^ {2} \delta^ {2} + 6 K ^ {2} \gamma^ {2} \zeta^ {2}\right) + \frac {\gamma \delta^ {2}}{\mu N},
$$
Algorithm A2: Shuffling Local SGD (One Client)
Input: Clients  $0,\dots ,N - 1$  Number of Local Steps  $K$  ,Number of Epoch  $R$  ,Mixing parameter  $\hat{\alpha}$    
Epoch for  $r = 0,\ldots ,R - 1$  do Server generates permutation  $\sigma_r:[N]\mapsto [N].$  Client sets initial model  $\pmb{v}^{r,0} = \pmb{v}^{r}$  . for  $j = 0,\dots ,N - 1$  do Server sends  $\pmb{v}^{r,j}$  to Client  $\sigma_r(j)$ $\pmb{v}^{r,j + 1} = \mathsf{SGD - Update}(\pmb{v}^{r,j},\eta ,\sigma_r(j),K,\hat{\alpha}).$  Client  $i$  does projection:  $\pmb{v}^{r + 1} = \mathcal{P}_{\mathcal{W}}(\pmb{v}^{r,N})$    
Output:  $\hat{\boldsymbol{v}} = \boldsymbol{v}^R$    
SGD-Update  $(\boldsymbol {v},\eta ,j,K,\boldsymbol {\alpha})$  Initialize  $\pmb{v}^{0} = \pmb{v}$  for  $t = 0,\dots ,K - 1$  do  $\pmb{v}^{t} = \pmb{v}^{t - 1} - \eta \alpha (j)\pmb {N}\nabla f_{j}(\pmb{v}^{t - 1};\xi^{t - 1})$  Output  $\pmb{v}^{K}$
Plugging in  $\gamma = \frac{\log(RK)}{\mu RK}$  gives the convergence rate:
$$
\mathbb {E} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2} \leq \tilde {O} \left(\frac {\mathbb {E} \left\| \boldsymbol {w} ^ {0} - \boldsymbol {w} ^ {*} \right\| ^ {2}}{R K} + \kappa \left(\frac {\delta^ {2}}{\mu^ {2} R ^ {2} K} + \frac {\zeta^ {2}}{\mu^ {2} R ^ {2}}\right) + \frac {\delta^ {2}}{\mu^ {2} N R K}\right),
$$
Equipped with above results, we are now ready to provide the convergence of main theorem.
Proof of Theorem 1. The proof simply follows from Lemma 3:
$$
\begin{array}{l} \left\| \hat {\boldsymbol {\alpha}} - \boldsymbol {\alpha} ^ {*} \right\| ^ {2} \leq 2 \left\| \hat {\boldsymbol {\alpha}} - \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {R}) \right\| ^ {2} + 2 \left\| \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {R}) - \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {*}) \right\| ^ {2} \\ \leq 2 \left(1 - \mu \eta_ {\alpha}\right) ^ {T _ {\alpha}} + 8 L \kappa_ {g} ^ {2} \sum_ {j = 1} ^ {N} \left(2 \left\| \nabla f _ {i} (\boldsymbol {w} ^ {*}) - \nabla f _ {j} (\boldsymbol {w} ^ {*}) \right\| ^ {2} + 4 L ^ {2} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2}\right) \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2} \\ \leq 2 \left(1 - \mu \eta_ {\boldsymbol {\alpha}}\right) ^ {T _ {\boldsymbol {\alpha}}} + 8 L \kappa_ {g} ^ {2} \left(2 \bar {\zeta} _ {i} (\boldsymbol {w} ^ {*}) + 4 N L ^ {2} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2}\right) \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2} \\ \end{array}
$$
Plugging in the convergence of  $\left\| \boldsymbol{w}^R - \boldsymbol{w}^* \right\|^2$  from Lemma 6, and the stepsize  $\eta_{\alpha} = \frac{1}{L_g}$  for  $\alpha$  yields:
$$
\mathbb {E} \| \boldsymbol {\alpha} _ {i} ^ {R} - \boldsymbol {\alpha} _ {i} ^ {*} \| ^ {2} \leq \tilde {O} \left(\exp (- \frac {T _ {\boldsymbol {\alpha}}}{\kappa_ {g}}) + \kappa_ {g} ^ {2} \bar {\zeta} _ {i} (\boldsymbol {w} ^ {*}) L ^ {2} \left(\frac {D ^ {2}}{R K} + \kappa \left(\frac {\delta^ {2}}{\mu^ {2} R ^ {2} K} + \frac {\zeta^ {2}}{\mu^ {2} R ^ {2}}\right) + \frac {\delta^ {2}}{\mu^ {2} N R K}\right)\right).
$$
B.3 Proof of Convergence of Shuffling Local SGD
In this section, we are going to prove the convergence of proposed shuffled variant of Local SGD (Theorem 2). The whole proof framework follows the analysis of vanilla shuffling SGD, but notice that there are two differences. First, in vanilla shuffling SGD, in each epoch, algorithm only updates on each component function  $f_{j}$  once, while here we have to take  $K$  steps of SGD update on each component function. Second, we are considering a weighted sum objective in contrary to averaged objective in [56], which means we need to rescale the objective when we apply without-replacement concentration inequality. Even though our algorithm solves models for  $N$  clients, for the sake of simplicity, throughout the proof we only show the convergence of one client's model. The algorithm from one client point of view is described in Algorithm A2, where we drop the client index for notational convenience.
Proposition 1. Assume a sequence  $\{\pmb{w}^t\}_{t=1}^K$  is obtained by
$$
\boldsymbol {w} ^ {t} = \boldsymbol {w} ^ {t - 1} - \eta \alpha N \nabla f (\boldsymbol {w} ^ {t - 1}; \xi^ {t - 1}), t = 1, \dots , K,
$$
$$
\boldsymbol {w} ^ {t + 1} = \boldsymbol {w} ^ {0} - \left(\sum_ {\tau = 0} ^ {t} \prod_ {t ^ {\prime} = t} ^ {\tau + 1} \left(\mathbf {I} - \alpha N \eta \mathbf {H} _ {t ^ {\prime}}\right)\right) \eta \alpha N \nabla f (\boldsymbol {w} ^ {0}) - \sum_ {\tau = 0} ^ {t} \prod_ {t ^ {\prime} = t} ^ {\tau + 1} \left(\mathbf {I} - \alpha N \eta \mathbf {H} _ {t ^ {\prime}}\right) \eta \alpha N \boldsymbol {\delta} ^ {t}, \quad \forall 0 \leq t \leq K - 1,
$$
where  $\pmb{\delta}^t \coloneqq \nabla f(\pmb{w}^t; \xi^t) - \nabla f(\pmb{w}^t)$ , and by convention, we define  $\prod_{j=a}^{b} \pmb{A}_j = \mathbf{I}$  if  $a < b$ .
Proof. According to updating rule, we have:
$$
\begin{array}{l} \boldsymbol {w} ^ {t + 1} - \boldsymbol {w} ^ {0} = \boldsymbol {w} ^ {t} - \boldsymbol {w} ^ {0} - \eta \alpha N \nabla f (\boldsymbol {w} ^ {t}; \xi^ {t}) \\ = \boldsymbol {w} ^ {t} - \boldsymbol {w} ^ {0} - \eta \alpha N \nabla f (\boldsymbol {w} ^ {t}) - \eta \alpha N \boldsymbol {\delta} ^ {t} \\ = \boldsymbol {w} ^ {t} - \boldsymbol {w} ^ {0} - \eta \alpha N \nabla f (\boldsymbol {w} ^ {0}) - \eta \alpha N (\nabla f (\boldsymbol {w} ^ {t}) - \nabla f (\boldsymbol {w} ^ {0})) - \eta \alpha N \boldsymbol {\delta} ^ {t}. \\ \end{array}
$$
Since  $f$  is  $L$  smooth, and according to Mean Value Theorem, there is a matrix  $\mathbf{H}_t$  satisfying  $\mu \mathbf{I} \preceq \mathbf{H}_t \preceq L\mathbf{I}$ , such that  $\nabla f(\boldsymbol{w}^t) - \nabla f(\boldsymbol{w}^0) = \mathbf{H}_t(\boldsymbol{w}^t - \boldsymbol{w}^0)$ . Hence we have:
$$
\boldsymbol {w} ^ {t + 1} - \boldsymbol {w} ^ {0} = (\mathbf {I} - \eta \alpha N \mathbf {H} _ {t}) (\boldsymbol {w} ^ {t} - \boldsymbol {w} ^ {0}) - \eta \alpha N \nabla f (\boldsymbol {w} ^ {0}) - \eta \alpha N \boldsymbol {\delta} ^ {t}.
$$
Unrolling the recursion from  $t$  to 0 will conclude the proof.
The following lemma establishes the updating rule of models between epochs  $r$  and  $r + 1$ . For notational convenience, whenever there is no confusion, we drop the superscript  $r$  in  $\sigma^r$ .
Lemma 7 (One epoch updating rule). Let  $\pmb{v}^r$  and  $\pmb{v}^{r + 1}$  be two iterates generated by Shuffling Local SGD (Algorithm A2), then the following updating rule holds:
$$
\boldsymbol {v} ^ {r + 1} = \boldsymbol {v} ^ {r} - \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N} ^ {j + 1} (\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}) (\boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \boldsymbol {\delta} _ {j}),
$$
$$
\boldsymbol {Q} _ {j} := \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = K - 1} ^ {\tau + 1} (\mathbf {I} - \eta \hat {\alpha} (\sigma (j)) N \mathbf {H} _ {t ^ {\prime}})\right) \eta \hat {\alpha} (\sigma (j)) N,
$$
$$
\boldsymbol {\delta} _ {j} := \sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = t} ^ {\tau + 1} (\mathbf {I} - \hat {\alpha} (\sigma (j)) N \eta \mathbf {H} _ {t ^ {\prime}}) \eta \hat {\alpha} (\sigma (j)) N \boldsymbol {\delta} _ {\sigma (j)} ^ {t},
$$
by convention, we define  $\prod_{j=a}^{b} A_{j} = \mathbf{I}$  if  $a < b$ .
Proof. According to Proposition 1, we have
$$
\begin{array}{l} \boldsymbol {v} ^ {r, j + 1} = \boldsymbol {v} ^ {r, j} - \left(\sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = t} ^ {\tau + 1} (\mathbf {I} - \hat {\alpha} (\sigma (j)) N \eta \mathbf {H} _ {t ^ {\prime}})\right) \eta \hat {\alpha} (\sigma (j)) N \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r, j}) \\ - \sum_ {\tau = 0} ^ {K - 1} \prod_ {t ^ {\prime} = t} ^ {\tau + 1} (\mathbf {I} - \hat {\alpha} (\sigma (j)) N \eta \mathbf {H} _ {t ^ {\prime}}) \eta \hat {\alpha} (\sigma (j)) N \boldsymbol {\delta} _ {\sigma (j)} ^ {t}. \\ \end{array}
$$
Plugging our definition of  $Q_{j}$  and  $\delta_j$  yields:
$$
\boldsymbol {v} ^ {r, j + 1} - \boldsymbol {v} ^ {r} = \boldsymbol {v} ^ {r, j} - \boldsymbol {v} ^ {r} - Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r, j}) - \boldsymbol {\delta} _ {j}.
$$
Following the same reasoning in the proof of Proposition 1 will conclude the proof.
Lemma 8 (Summation by parts). Let  $\mathbf{A}_j$  and  $\mathbf{B}_j$  be complex valued matrices. Then the following fact holds:
$$
\sum_ {j = 1} ^ {N} \boldsymbol {A} _ {j} \mathbf {B} _ {j} = \boldsymbol {A} _ {N} \sum_ {j = 1} ^ {N} \mathbf {B} _ {j} - \sum_ {n = 1} ^ {N - 1} \left(\boldsymbol {A} _ {n + 1} - \boldsymbol {A} _ {n}\right) \sum_ {j = 1} ^ {n} \mathbf {B} _ {j}.
$$
Proposition 2 (Spectral bound of polynomial expansion). Given a collection of matrices  $\{\mathbf{A}_t\}$  and  $\{\mathbf{B}_t\}$ , such that  $\mathbf{A}_t \preceq L\mathbf{I}$  and  $\mathbf{B}_t \preceq L\mathbf{I}$ , the following bound hold:
$$
\left\| \prod_ {t = l} ^ {h} (\mathbf {I} - a \mathbf {A} _ {t}) - \mathbf {I} \right\| \leq \sum_ {m = 1} ^ {h - l} \left(\frac {e (h - l)}{m}\right) ^ {m} (a L) ^ {m},
$$
$$
\left\| \prod_ {t = l} ^ {h} (\mathbf {I} - a \mathbf {A} _ {t}) - \prod_ {t = l} ^ {h} (\mathbf {I} - b \mathbf {B} _ {t}) \right\| \leq \sum_ {m = 1} ^ {h - l} \left(\frac {e (h - l)}{m}\right) ^ {m} (a L) ^ {m} + \sum_ {m = 1} ^ {h - l} \left(\frac {e (h - l)}{m}\right) ^ {m} (b L) ^ {m}.
$$
Proof. We start with proving the first statement. Expanding the product yields:
$$
\prod_ {t = l} ^ {h} \left(\mathbf {I} - a \mathbf {A} _ {t}\right) = \mathbf {I} + \sum_ {m = 1} ^ {h - l} (- 1) ^ {m} a ^ {m} \sum_ {| S | = m, | S | \subseteq \{l, \dots , h \}} \prod_ {m ^ {\prime} \in S} \mathbf {A} _ {m ^ {\prime}}.
$$
Hence we have:
$$
\left\| \prod_ {t = l} ^ {h} \left(\mathbf {I} - a \mathbf {A} _ {t}\right) - \mathbf {I} \right\| = \left\| \sum_ {m = 1} ^ {h - l} (- 1) ^ {m} a ^ {m} \sum_ {| S | = m, | S | \subseteq \{l, \dots , h \}} \prod_ {m ^ {\prime} \in S} \mathbf {A} _ {m ^ {\prime}} \right\| \leq \sum_ {m = 1} ^ {h - l} \binom {h - l} {m} (a L) ^ {m}
$$
According to the upper bound for binomial coefficients:  $\binom{h-l}{m} \leq \left( \frac{e(h-l)}{m} \right)^m$ , we have:
$$
\sum_ {m = 1} ^ {h - l} \binom {h - l} {m} (a L) ^ {m} \leq \sum_ {m = 1} ^ {h - l} \left(\frac {e (h - l)}{m}\right) ^ {m} (a L) ^ {m}.
$$
Then we switch to the second one. Using the same expanding product yields:
$$
\begin{array}{l} \left\| \prod_ {t = l} ^ {h} \left(\mathbf {I} - a \mathbf {A} _ {t}\right) - \prod_ {t = l} ^ {h} \left(\mathbf {I} - b \mathbf {B} _ {t}\right) \right\| \\ = \left\| \sum_ {m = 1} ^ {h - l} (- 1) ^ {m} a ^ {m} \sum_ {| S | = m, | S | \subseteq \{l, \dots , h \}} \prod_ {m ^ {\prime} \in S} \mathbf {A} _ {m ^ {\prime}} - \sum_ {m = 1} ^ {h - l} (- 1) ^ {m} b ^ {m} \sum_ {| S | = m, | S | \subseteq \{l, \dots , h \}} \prod_ {m ^ {\prime} \in S} \mathbf {B} _ {m ^ {\prime}} \right\| \\ \leq \sum_ {m = 1} ^ {h - l} \binom {h - l} {m} (a L) ^ {m} + \sum_ {m = 1} ^ {h - l} \binom {h - l} {m} (b L) ^ {m} \\ \leq \sum_ {m = 1} ^ {h - l} \left(\frac {e (h - l)}{m}\right) ^ {m} (a L) ^ {m} + \sum_ {m = 1} ^ {h - l} \left(\frac {e (h - l)}{m}\right) ^ {m} (b L) ^ {m}. \\ \end{array}
$$
The following concentration result is the key to bound variance during shuffling updating. The original result holds for the average of gradients, and we will later on generalize it to an arbitrary weighted sum of gradients.
Lemma 9 ([61, Theorem 2]). Suppose  $n \geq 2$ . Let  $\pmb{g}_1, \pmb{g}_2, \dots, \pmb{g}_n \in \mathbb{R}^d$  satisfy  $\| \pmb{g}_j \| \leq G$  for all  $j$ . Let  $\bar{\pmb{g}} = \frac{1}{n} \sum_{j=1}^{n} \pmb{g}_j$ . Let  $\sigma \in S_n$  be a uniform random permutation of  $n$  elements. Then, for  $i \leq n$  with probability at least  $1 - p$ , we have
$$
\left\| \frac {1}{i} \sum_ {j = 1} ^ {i} \boldsymbol {g} _ {\sigma (j)} - \bar {\boldsymbol {g}} \right\| \leq G \sqrt {\frac {8 \left(1 - \frac {i - 1}{n}\right) \log \frac {2}{p}}{i}}.
$$
Lemma 10 (Concentration of partial sum of gradients). Given a uniformly randomly generated permutation  $\sigma$ , and simplex vector  $\alpha$ , if we assume each  $\sup_{\boldsymbol{v} \in \mathcal{W}} \| \nabla f_j(\boldsymbol{v}) \| \leq G$ , then the following statement holds true:
$$
\left\| \sum_ {j = 0} ^ {n} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \leq G \sqrt {8 n \log (1 / p)} + \frac {n}{N} \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \|.
$$
Proof. The proof works by re-writing weighted sum of vectors to average of the these vectors:
$$
\begin{array}{l} \left\| \sum_ {j = 0} ^ {n} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| = \frac {1}{N} \left\| \sum_ {j = 0} ^ {n} \hat {\alpha} (\sigma (j)) N \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| \\ = \frac {1}{N} \left(\left\| \sum_ {j = 0} ^ {n} \hat {\alpha} (\sigma (j)) N \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - n \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \right\| + n \| \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \|\right) \\ \leq G \sqrt {8 n \log (1 / p)} + \frac {n}{N} \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \|. \\ \end{array}
$$
Proposition 3 (Spectral norm bound of  $\mathbf{Q}$ ). Let  $\mathbf{Q}_j$  be defined in (11). Then the following bound for the spectral norm of  $\mathbf{Q}_j$  holds true for all  $j \in [N]$ :
$$
\left\| \boldsymbol {Q} _ {j} \right\| \leq \eta \hat {\alpha} (\sigma (j)) N K (1 + \eta N L) ^ {K}
$$
Proof. The proof can be completed by writing down the definitin of  $Q_{j}$  and applying Cauchy-Schwartz inequality: