SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the following proof, we consider the partial client participation setting. So we assume that  $\pi = \{\pi_1,\pi_2,\dots ,\pi_M\}$  is a permutation of  $\{1,2,\ldots ,M\}$  in a certain training round and only the first  $S$  selected clients  $\{\pi_1,\pi_2,\dots ,\pi_S\}$  will participate in this round. Without otherwise stated, we use  $\mathbb{E}[\cdot ]$  to represent the expectation with respect to both types of randomness (i.e., sampling data samples  $\xi$  and sampling clients  $\pi$ ).
D.1 Strongly convex case
D.1.1 Finding the recursion
Lemma 5. Let Assumptions 1, 2, 3b hold and assume that all the local objectives are  $\mu$ -strongly convex. If the learning rate satisfies  $\eta \leq \frac{1}{6LSK}$ , then it holds that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \leq \left(1 - \frac {\mu S K \eta}{2}\right) \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] + 4 S K \eta^ {2} \sigma^ {2} + 4 S ^ {2} K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} \\ - \frac {2}{3} S K \eta \mathbb {E} \left[ D _ {F} \left(\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}\right) \right] + \frac {8}{3} L \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} ^ {(r)} - \mathbf {x} ^ {(r)} \right\| ^ {2} \right] \tag {27} \\ \end{array}
$$
Proof. According to Algorithm 1, the overall model updates of SFL after one complete training round (with  $S$  clients selected for training) is
$$
\Delta \mathbf {x} = \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {(r)} = - \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} ^ {(r)},
$$
where  $\mathbf{g}_{\pi_m,k}^{(r)} = \nabla f_{\pi_m}(\mathbf{x}_{m,k}^{(r)};\xi)$  is the stochastic gradient of  $F_{\pi_m}$  regarding the vector  $\mathbf{x}_{m,k}^{(r)}$ . Thus,
$$
\mathbb {E} \left[ \Delta \mathbf {x} \right] = - \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right) \right]
$$
In the following, we focus on a single training round, and hence we drop the superscripts  $r$  for a while, e.g., writing  $\mathbf{x}_{m,k}$  to replace  $\mathbf{x}_{m,k}^{(r)}$ . Specially, we would like to use  $\mathbf{x}$  to replace  $\mathbf{x}_{1,0}^{(r)}$ . Without otherwise stated, the expectation is conditioned on  $\mathbf{x}^{(r)}$ .
We start by substituting the overall updates:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} + \Delta \mathbf {x} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \\ = \left\| \mathbf {x} - \mathbf {x} ^ {*} \right\| ^ {2} + 2 \mathbb {E} \left[ \langle \mathbf {x} - \mathbf {x} ^ {*}, \Delta \mathbf {x} \rangle \right] + \mathbb {E} \left[ \left\| \Delta \mathbf {x} \right\| ^ {2} \right] \\ = \left\| \mathbf {x} - \mathbf {x} ^ {*} \right\| ^ {2} - 2 \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \langle \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}), \mathbf {x} - \mathbf {x} ^ {*} \rangle \right] + \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \tag {28} \\ \end{array}
$$
We can apply Lemma 2 with  $\pmb{x} = \mathbf{x}_{m,k}$ ,  $\pmb{y} = \mathbf{x}^*$ ,  $\pmb{z} = \mathbf{x}$  and  $h = F_{\pi_m}$  for the second term on the right hand side in (28):
$$
\begin{array}{l} - 2 \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right), \mathbf {x} - \mathbf {x} ^ {*} \right\rangle \right] \\ \leq - 2 \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ F _ {\pi_ {m}} (\mathbf {x}) - F _ {\pi_ {m}} (\mathbf {x} ^ {*}) + \frac {\mu}{4} \left\| \mathbf {x} - \mathbf {x} ^ {*} \right\| ^ {2} - L \left\| \mathbf {x} _ {m, k} - \mathbf {x} \right\| ^ {2} \right] \\ \leq - 2 S K \eta D _ {F} \left(\mathbf {x}, \mathbf {x} ^ {*}\right) - \frac {1}{2} \mu S K \eta \| \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} + 2 L \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \tag {29} \\ \end{array}
$$
For the third term on the right hand side in (28), using Jensen's inequality, we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \\ \leq 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\mathbf {g} _ {\pi_ {m}, k} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k})\right) \right\| ^ {2} \right] + 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) - \nabla F _ {\pi_ {m}} (\mathbf {x})\right) \right\| ^ {2} \right] \\ + 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x}) - \nabla F _ {\pi_ {m}} \left(\mathbf {x} ^ {*}\right)\right) \right\| ^ {2} \right] + 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \nabla F _ {\pi_ {m}} \left(\mathbf {x} ^ {*}\right) \right\| ^ {2} \right] \tag {30} \\ \end{array}
$$
Seeing the data sample  $\xi_{m,k}$ , the stochastic gradient  $\mathbf{g}_{\pi_m,k}$ , the gradient  $\nabla F_{\pi_m}(\xi_{m,k})$  as  $\xi_i$ ,  $\mathbf{x}_i$ ,  $\mathbf{e}_i$  in Lemma 1 respectively and applying the result of Lemma 1, the first term on the right hand side in (30) can be bounded by  $4SK\sigma^2$ . For the second term on the right hand side in (30), we have
4E  $\left[\left\| \sum_{m = 1}^{S}\sum_{k = 0}^{K - 1}\bigl {(}\nabla F_{\pi_m}(\mathbf{x}_{m,k}) - \nabla F_{\pi_m}(\mathbf{x})\bigr)\right\| ^2\right]\stackrel {(15)}{\leq}4SK\sum_{m = 1}^{S}\sum_{k = 0}^{K - 1}\mathbb{E}\left[\| \nabla F_{\pi_m}(\mathbf{x}_{m,k}) - \nabla F_{\pi_m}(\mathbf{x})\| ^2\right]$
$$
\stackrel {\mathrm {A s m .}} {\leq} ^ {1} 4 L ^ {2} S K \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right]
$$
For the third term on the right hand side in (30), we have
4E  $\left[\left\| \sum_{m = 1}^{S}\sum_{k = 0}^{K - 1}\bigl {(}\nabla F_{\pi_m}(\mathbf{x}) - \nabla F_{\pi_m}(\mathbf{x}^*)\bigr)\right\| ^2\right]\stackrel {(15)}{\leq}4SK\sum_{m = 1}^{S}\sum_{k = 0}^{K - 1}\mathbb{E}\left[\| \nabla F_{\pi_m}(\mathbf{x}) - \nabla F_{\pi_m}(\mathbf{x}^*)\| ^2\right]$
$$
\begin{array}{l} \stackrel {(1 8)} {\leq} 8 L S K \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ D _ {F _ {\pi_ {m}}} \left(\mathbf {x}, \mathbf {x} ^ {*}\right) \right] \\ = 8 L S ^ {2} K ^ {2} D _ {F} \left(\mathbf {x}, \mathbf {x} ^ {*}\right), \tag {31} \\ \end{array}
$$
where the last inequality is because  $\mathbb{E}\left[D_{F_{\pi_m}}(\mathbf{x},\mathbf{x}^*)\right] = D_F(\mathbf{x},\mathbf{x}^*)$ . The forth term on the right hand side in (30) can be bounded by Lemma 3 as follows:
$$
4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \nabla F _ {\pi_ {m}} (\mathbf {x} ^ {*}) \right\| ^ {2} \right] \stackrel {(2 3)} {\leq} 4 S ^ {2} K ^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2}.
$$
With the preceding four inequalities, we can bound the third term on the right hand side in (28):
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \\ \leq 4 S K \sigma^ {2} + 4 L ^ {2} S K \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] + 8 L S ^ {2} K ^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + 4 S ^ {2} K ^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} \tag {32} \\ \end{array}
$$
Then substituting (29) and (32) into (28), we have
$$
\begin{array}{l} \mathbb {E} \left[ \| \mathbf {x} + \Delta \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} \right] \leq \left(1 - \frac {\mu S K \eta}{2}\right) \| \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} + 4 S K \eta^ {2} \sigma^ {2} + 4 S ^ {2} K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} \\ - 2 S K \eta (1 - 4 L S K \eta) D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + 2 L \eta (1 + 2 L S K \eta) \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \\ \leq \left(1 - \frac {\mu S K \eta}{2}\right) \| \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} + 4 S K \eta^ {2} \sigma^ {2} + 4 S ^ {2} K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} \\ - \frac {2}{3} S K \eta D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + \frac {8}{3} L \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right], \\ \end{array}
$$
where we use the condition that  $\eta \leq \frac{1}{6LSK}$  in the last inequality. The claim of this lemma follows after recovering the superscripts and taking unconditional expectation.
D.1.2 Bounding the client drift with Assumption 3b
Similar to the "client drift" in PFL (Karimireddy et al., 2020), we define the client drift in SFL:
$$
E _ {r} := \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} ^ {(r)} - \mathbf {x} ^ {(r)} \right\| ^ {2} \right] \tag {33}
$$
Lemma 6. Let Assumptions 1, 2, 3b hold and assume that all the local objectives are  $\mu$ -strongly convex. If the learning rate satisfies  $\eta \leq \frac{1}{6LSK}$ , then the client drift is bounded:
$$
E _ {r} \leq \frac {9}{4} S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + \frac {9}{4} S ^ {2} K ^ {3} \eta^ {2} \zeta_ {*} ^ {2} + 3 L S ^ {3} K ^ {3} \eta^ {2} \mathbb {E} \left[ D _ {F} \left(\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}\right) \right] \tag {34}
$$
Proof. According to Algorithm 1, the model updates of SFL from  $\mathbf{x}^{(r)}$  to  $\mathbf{x}_{m,k}^{(r)}$  is
$$
\mathbf {x} _ {m, k} ^ {(r)} - \mathbf {x} ^ {(r)} = - \eta \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbf {g} _ {\pi_ {i}, j} ^ {(r)}
$$
with  $b_{m,k}(i) \coloneqq \begin{cases} K - 1, & i \leq m - 1 \\ k - 1, & i = m \end{cases}$ . In the following, we focus on a single training round, and hence we drop the superscripts  $r$  for a while, e.g., writing  $\mathbf{x}_{m,k}$  to replace  $\mathbf{x}_{m,k}^{(r)}$ . Specially, we would like to use  $\mathbf{x}$  to replace  $\mathbf{x}_{1,0}^{(r)}$ . Without otherwise stated, the expectation is conditioned on  $\mathbf{x}^{(r)}$ .
We use Jensen's inequality to bound the term  $\mathbb{E}\left[\| \mathbf{x}_{m,k} - \mathbf{x}\| ^2\right] = \eta^2\mathbb{E}\left[\left\| \sum_{i = 1}^{m}\sum_{j = 0}^{b_{m,k}(i)}\mathbf{g}_{\pi_i,j}\right\| ^2\right]$ :
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} - \mathbf {x} \right\| ^ {2} \right] \\ \leq 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\mathbf {g} _ {\pi_ {i}, j} - \nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j})\right) \right\| ^ {2} \right] + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j}) - \nabla F _ {\pi_ {i}} (\mathbf {x})\right) \right\| ^ {2} \right] \\ + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\nabla F _ {\pi_ {i}} (\mathbf {x}) - \nabla F _ {\pi_ {i}} (\mathbf {x} ^ {*})\right) \right\| ^ {2} \right] + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \nabla F _ {\pi_ {i}} (\mathbf {x} ^ {*}) \right\| ^ {2} \right] \\ \end{array}
$$
Applying Lemma 1 to the first term and Jensen's inequality to the second, third terms on the right hand side in the preceding inequality respectively, we can get
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} - \mathbf {x} \right\| ^ {2} \right] \\ \leq 4 \eta^ {2} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbb {E} \left[ \| \mathbf {g} _ {\pi_ {i}, j} - \nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j}) \| ^ {2} \right] + 4 \eta^ {2} \mathcal {B} _ {m, k} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbb {E} \left[ \| \nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j}) - \nabla F _ {\pi_ {i}} (\mathbf {x}) \| ^ {2} \right] \\ + 4 \eta^ {2} \mathcal {B} _ {m, k} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbb {E} \left[ \| \nabla F _ {\pi_ {i}} (\mathbf {x}) - \nabla F _ {\pi_ {i}} (\mathbf {x} ^ {*}) \| ^ {2} \right] + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \nabla F _ {\pi_ {i}} (\mathbf {x} ^ {*}) \right\| ^ {2} \right] \tag {35} \\ \end{array}
$$
where  $\mathcal{B}_{m,k} := \sum_{i=1}^{m} \sum_{j=0}^{b_{m,k}(i)} 1 = (m-1)K + k$ . The first term on the right hand side in (35) is bounded by  $4\mathcal{B}_{m,k}\eta^2\sigma^2$ . For the second term on the right hand side in (35), we have
$$
\mathbb {E} \left[ \| \nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j}) - \nabla F _ {\pi_ {i}} (\mathbf {x}) \| ^ {2} \right] \overset {\mathrm {A s m . 1}} {\leq} L ^ {2} \mathbb {E} \left[ \| \mathbf {x} _ {i, j} - \mathbf {x} \| ^ {2} \right]
$$
For the third term on the right hand side in (35), we have
$$
\mathbb {E} \left[ \| \nabla F _ {\pi_ {i}} (\mathbf {x}) - \nabla F _ {\pi_ {i}} \left(\mathbf {x} ^ {*}\right) \| ^ {2} \right] \stackrel {(1 8)} {\leq} 2 L \mathbb {E} \left[ D _ {F _ {\pi_ {i}}} \left(\mathbf {x}, \mathbf {x} ^ {*}\right) \right] = 2 L D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) \tag {36}
$$
As a result, we can get
$$
\begin{array}{l} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \leq 4 \mathcal {B} _ {m, k} \eta^ {2} \sigma^ {2} + 4 L ^ {2} \eta^ {2} \mathcal {B} _ {m, k} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b (i)} \mathbb {E} \left[ \| \mathbf {x} _ {i, j} - \mathbf {x} \| ^ {2} \right] + 8 L \eta^ {2} \mathcal {B} _ {m, k} ^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) \\ + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \nabla F _ {\pi_ {i}} (\mathbf {x} ^ {*}) \right\| ^ {2} \right] \\ \end{array}
$$
Then, returning to  $E_r \coloneqq \sum_{m=1}^{S} \sum_{k=0}^{K-1} \mathbb{E}\left[\|\mathbf{x}_{m,k} - \mathbf{x}\|^2\right]$ , we have
$$
\begin{array}{l} E _ {r} \leq 4 \eta^ {2} \sigma^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathcal {B} _ {m, k} + 4 L ^ {2} \eta^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathcal {B} _ {m, k} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbb {E} \left[ \| \mathbf {x} _ {i, j} - \mathbf {x} \| ^ {2} \right] \\ + 8 L \eta^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathcal {B} _ {m, k} ^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + 4 \eta^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \nabla F _ {\pi_ {i}} (\mathbf {x} ^ {*}) \right\| ^ {2} \right] \\ \end{array}
$$
Applying Lemma 4 with  $\mathbf{x}_{\pi_i} = \nabla F_{\pi_i}(\mathbf{x}^*)$  and  $\overline{\pmb{x}} = \nabla F(\mathbf{x}^{*}) = 0$  and the fact that
$$
\begin{array}{l} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathcal {B} _ {m, k} = \frac {1}{2} S K (S K - 1) \leq \frac {1}{2} S ^ {2} K ^ {2}, \\ \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathcal {B} _ {m, k} ^ {2} = \frac {1}{3} (S K - 1) S K (S K - \frac {1}{2}) \leq \frac {1}{3} S ^ {3} K ^ {3}, \\ \end{array}
$$
we can simplify the preceding inequality:
$$
E _ {r} \leq 2 S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + 2 L ^ {2} S ^ {2} K ^ {2} \eta^ {2} E _ {r} + \frac {8}{3} L S ^ {3} K ^ {3} \eta^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + 2 S ^ {2} K ^ {3} \eta^ {2} \zeta_ {*} ^ {2}
$$
After rearranging the preceding inequality, we get
$$
(1 - 2 L ^ {2} S ^ {2} K ^ {2} \eta^ {2}) E _ {r} \leq 2 S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + 2 S ^ {2} K ^ {3} \eta^ {2} \zeta_ {*} ^ {2} + \frac {8}{3} L S ^ {3} K ^ {3} \eta^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*})
$$
Finally, using the condition that  $\eta \leq \frac{1}{6LSK}$ , which implies  $1 - 2L^2 S^2 K^2\eta^2 \geq \frac{8}{9}$ , we have
$$
E _ {r} \leq \frac {9}{4} S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + \frac {9}{4} S ^ {2} K ^ {3} \eta^ {2} \zeta_ {*} ^ {2} + 3 L S ^ {3} K ^ {3} \eta^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}).
$$
The claim follows after recovering the superscripts and taking unconditional expectations.
D.1.3 Tuning the learning rate
Here we make a clear version of Karimireddy et al. (2020)'s Lemma 1 (linear convergence rate) based on Stich (2019b); Stich and Karimireddy (2019)'s works.
Lemma 7 (Karimireddy et al. (2020)). Two non-negative sequences  $\{r_t\}_{t \geq 0}, \{s_t\}_{t \geq 0}$ , which satisfies the relation
$$
r _ {t + 1} \leq \left(1 - a \gamma_ {t}\right) r _ {t} - b \gamma_ {t} s _ {t} + c \gamma_ {t} ^ {2}, \tag {37}
$$
for all  $t \geq 0$  and for parameters  $b > 0$ ,  $a, c \geq 0$  and non-negative learning rates  $\{\gamma_t\}_{t \geq 0}$  with  $\gamma_t \leq \frac{1}{d}$ ,  $\forall t \geq 0$ , for a parameter  $d \geq a$ ,  $d > 0$ .
Selection of weights for average. Then there exists a constant learning rate  $\gamma_t = \gamma \leq \frac{1}{d}$  and the weights  $w_t \coloneqq (1 - a\gamma)^{-(t + 1)}$  and  $W_T \coloneqq \sum_{t=0}^{T} w_t$ , making it hold that:
$$
\Psi_ {T} = \frac {b}{W _ {T}} \sum_ {t = 0} ^ {T} s _ {t} w _ {t} \leq 3 a r _ {0} (1 - a \gamma) ^ {(T + 1)} + c \gamma \leq 3 a r _ {0} \exp [ - a \gamma (T + 1) ] + c \gamma . \tag {38}
$$
Tuning the learning rate carefully. By tuning the learning rate in (38), for  $(T + 1) \geq \frac{1}{2a\gamma}$ , we have
$$
\Psi_ {T} = \tilde {\mathcal {O}} \left(a r _ {0} \exp \left(- \frac {a T}{d}\right) + \frac {c}{a T}\right). \tag {39}
$$
Proof. We start by rearranging (37) and multiplying both sides with  $w_{t}$ :
$$
b s _ {t} w _ {t} \leq \frac {w _ {t} (1 - a \gamma) r _ {t}}{\gamma} - \frac {w _ {t} r _ {t + 1}}{\gamma} + c \gamma w _ {t} = \frac {w _ {t - 1} r _ {t}}{\gamma} - \frac {w _ {t} r _ {t + 1}}{\gamma} + c \gamma w _ {t}.
$$
By summing from  $t = 0$  to  $t = T$ , we obtain a telescoping sum:
$$
\frac {b}{W _ {T}} \sum_ {t = 0} ^ {T} s _ {t} w _ {t} \leq \frac {1}{\gamma W _ {T}} \left(w _ {0} (1 - a \gamma) r _ {0} - w _ {T} r _ {T + 1}\right) + c \gamma ,
$$
$$
\Psi_ {T} = \frac {b}{W _ {T}} \sum_ {t = 0} ^ {T} s _ {t} w _ {t} \leq \frac {b}{W _ {T}} \sum_ {t = 0} ^ {T} s _ {t} w _ {t} + \frac {w _ {T} r _ {T + 1}}{\gamma W _ {T}} \leq \frac {r _ {0}}{\gamma W _ {T}} + c \gamma \tag {40}
$$
Note that the proof of Stich (2019b)'s Lemma 2 used  $W_{T} \geq w_{T} = (1 - a\gamma)^{-(T + 1)}$  to estimate  $W_{T}$ . It is reasonable given that  $w_{T}$  is extremely larger than all the terms  $w_{t}$  ( $t < T$ ) when  $T$  is large. Yet Karimireddy et al. (2020) goes further, showing that  $W_{T}$  can be estimated more precisely:
$$
W _ {T} = \sum_ {0} ^ {T} w _ {t} = (1 - a \gamma) ^ {- (T + 1)} \sum_ {t = 0} ^ {T} (1 - a \gamma) ^ {t} = (1 - a \gamma) ^ {- (T + 1)} \left(\frac {1 - (1 - a \gamma) ^ {T + 1}}{a \gamma}\right)
$$
When  $(T + 1) \geq \frac{1}{2a\gamma}$ ,  $(1 - a\gamma)^{T + 1} \leq \exp(-a\gamma(T + 1)) \leq e^{-\frac{1}{2}} \leq \frac{2}{3}$ , so it follows that
$$
W _ {T} = (1 - a \gamma) ^ {- (T + 1)} \left(\frac {1 - (1 - a \gamma) ^ {T + 1}}{a \gamma}\right) \geq \frac {(1 - a \gamma) ^ {- (T + 1)}}{3 a \gamma}
$$
With the estimates
-  $W_{T} = (1 - a\gamma)^{-(T + 1)}\sum_{t = 0}^{T}(1 - a\gamma)^{t}\leq \frac{w_{T}}{a\gamma}$  (here we leverage  $a\gamma \leq \frac{a}{d}\leq 1$
and  $W_{T}\geq \frac{(1 - a\gamma)^{-(T + 1)}}{3a\gamma},$
we can further simplify (40):
$$
\Psi_ {T} \leq 3 a r _ {0} (1 - a \gamma) ^ {(T + 1)} + c \gamma \leq 3 a r _ {0} \exp \left[ - a \gamma (T + 1) \right] + c \gamma
$$
which is the first result of this lemma.
Now the lemma follows by carefully tuning  $\gamma$  in (38). Consider the two cases:
- If  $\frac{1}{d} > \frac{\ln(\max\{2, a^2 r_0 T / c\})}{aT}$  then we choose  $\gamma = \frac{\ln(\max\{2, a^2 r_0 T / c\})}{aT}$  and get that
$$
\tilde {\mathcal {O}} \left(3 a r _ {0} \exp \left[ - \ln \left(\max  \{2, a ^ {2} r _ {0} T / c \}\right) \right]\right) + \tilde {\mathcal {O}} \left(\frac {c}{a T}\right) = \tilde {\mathcal {O}} \left(\frac {c}{a T}\right),
$$
as in case  $2 \geq a^{2}r_{0}T / c$  it holds  $ar_0 \leq \frac{2c}{aT}$ .
- If otherwise  $\frac{1}{2a(T + 1)}\leq \frac{1}{d}\leq \frac{\ln(\max\{2,a^2r_0T / c\})}{aT}$  (Note  $\frac{1}{2a(T + 1)}\leq \frac{\ln(2)}{aT}\leq \frac{\ln(\max\{2,a^2r_0T / c\})}{aT}$ )
then we pick  $\gamma = \frac{1}{d}$  and get that
$$
3 a r _ {0} \exp \left(- \frac {a T}{d}\right) + \frac {c}{d} \leq 3 a r _ {0} \exp \left(- \frac {a T}{d}\right) + \frac {c \ln (\max  \{2 , a ^ {2} r _ {0} T / c \})}{a T} = \tilde {\mathcal {O}} \left(a r _ {0} \exp \left(- \frac {a T}{d}\right) + \frac {c}{a T}\right)
$$
Combining these two cases, we get
$$
\Psi_ {T} = \tilde {\mathcal {O}} \left(a r _ {0} \exp \left(- \frac {a T}{d}\right) + \frac {c}{a T}\right)
$$
Note that this lemma holds when  $(T + 1) \geq \frac{1}{2a\gamma}$ , so it restricts the value of  $T$ , while there is no such restriction in Stich (2019b)'s Lemma 2.
D.1.4 Proof of strongly convex case of Theorem 1 and Corollary 1
Proof of the strongly convex case of Theorem 1. Substituting (34) into (27) and using  $\eta \leq \frac{1}{6LSK}$ , we can simplify the recursion as,
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \leq \left(1 - \frac {\mu S K \eta}{2}\right) \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] - \frac {1}{3} S K \eta \mathbb {E} \left[ D _ {F} (\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}) \right] \\ + 4 S K \eta^ {2} \sigma^ {2} + 4 S ^ {2} K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} + 6 L S ^ {2} K ^ {2} \eta^ {3} \sigma^ {2} + 6 L S ^ {2} K ^ {3} \eta^ {3} \zeta_ {*} ^ {2} \\ \end{array}
$$
Let  $\tilde{\eta} = MK\eta$  , we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \leq \left(1 - \frac {\mu \tilde {\eta}}{2}\right) \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] - \frac {\tilde {\eta}}{3} \mathbb {E} \left[ D _ {F} (\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}) \right] \\ + \frac {4 \tilde {\eta} ^ {2} \sigma^ {2}}{S K} + \frac {4 \tilde {\eta} ^ {2} (M - S) \zeta_ {*} ^ {2}}{S (M - 1)} + \frac {6 L \tilde {\eta} ^ {3} \sigma^ {2}}{S K} + \frac {6 L \tilde {\eta} ^ {3} \zeta_ {*} ^ {2}}{S} \tag {41} \\ \end{array}
$$
Applying Lemma 7 with  $t = r$  ( $T = R$ ),  $\gamma = \tilde{\eta}$ ,  $r_t = \mathbb{E}\left[\|\mathbf{x}^{(r)} - \mathbf{x}^*\|^2\right]$ ,  $a = \frac{\mu}{2}$ ,  $b = \frac{1}{3}$ ,  $s_t = \mathbb{E}\left[D_F(\mathbf{x}^{(r)}, \mathbf{x}^*)\right]$ ,  $w_t = (1 - \frac{\mu \tilde{\eta}}{2})^{-(r + 1)}$ ,  $c_1 = \frac{4\sigma^2}{SK} + \frac{4(M-S)\zeta_*^2}{S(M-1)}$ ,  $c_2 = \frac{6L\sigma^2}{SK} + \frac{6L\zeta_*^2}{S}$  and  $\frac{1}{d} = \frac{1}{6L}$  ( $\tilde{\eta} = MK\eta \leq \frac{1}{6L}$ ), it follows that
$$
\begin{array}{l} \mathbb {E} \left[ F (\bar {\mathbf {x}} ^ {(R)}) - F (\mathbf {x} ^ {*}) \right] \leq \frac {1}{W _ {R}} \sum_ {r = 0} ^ {R} w _ {r} \mathbb {E} \left[ F (\mathbf {x} ^ {(r)}) - F (\mathbf {x} ^ {*}) \right] \\ \leq \frac {9}{2} \mu \left\| \mathbf {x} ^ {(0)} - \mathbf {x} ^ {*} \right\| ^ {2} \exp \left(- \frac {1}{2} \mu \tilde {\eta} R\right) + \frac {1 2 \tilde {\eta} \sigma^ {2}}{S K} + \frac {1 2 \tilde {\eta} (M - S) \zeta_ {*} ^ {2}}{S (M - 1)} + \frac {1 8 L \tilde {\eta} ^ {2} \sigma^ {2}}{S K} + \frac {1 8 L \tilde {\eta} ^ {2} \zeta_ {*} ^ {2}}{S} \tag {42} \\ \end{array}
$$
where  $\bar{\mathbf{x}}^{(R)} = \frac{1}{W_R}\sum_{r = 0}^{R}w_r\mathbf{x}^{(r)}$  and we use Jensen's inequality ( $F$  is convex) in the first inequality. Note that there are no terms containing  $\gamma^3$  in Lemma 7. As the terms containing  $\gamma^3$  is not the determinant factor for the convergence rate, Lemma 7 can also be applied to this case (Karimireddy et al., 2020; Koloskova et al., 2020). Thus, by tuning the learning rate carefully, we get