SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] = \tilde {\mathcal {O}} \left(\mu D ^ {2} \exp \left(- \frac {\mu R}{1 2 L}\right) + \frac {\sigma^ {2}}{\mu S K R} + \frac {(M - S) \zeta_ {*} ^ {2}}{\mu S R (M - 1)} + \frac {L \sigma^ {2}}{\mu^ {2} S K R ^ {2}} + \frac {L \zeta_ {*} ^ {2}}{\mu^ {2} S R ^ {2}}\right) \tag {43}
$$
where  $D \coloneqq \left\| \mathbf{x}^{(0)} - \mathbf{x}^{*}\right\|$ . Eq. (42) and Eq. (43) are the upper bounds with partial client participation. In particular, when  $S = M$ , we can get the claim of the strongly convex case of Theorem 1 and Corollary 1.
D.2 General convex case
D.2.1 Tuning the learning rate
Lemma 8 (Koloskova et al. (2020)). Two non-negative sequences  $\{r_t\}_{t \geq 0}, \{s_t\}_{t \geq 0}$ , which satisfies the relation
$$
r _ {t + 1} \leq r _ {t} - b \gamma_ {t} s _ {t} + c _ {1} \gamma_ {t} ^ {2} + c _ {2} \gamma_ {t} ^ {3}
$$
for all  $t \geq 0$  and for parameters  $b > 0$ ,  $c_1, c_2 \geq 0$  and non-negative learning rates  $\{\gamma_t\}_{t \geq 0}$  with  $\gamma_t \leq \frac{1}{d}$ ,  $\forall t \geq 0$ , for a parameter  $d > 0$ .
Selection of weights for average. Then there exists a constant learning rate  $\gamma = \gamma_{t} \leq \frac{1}{d}$  and the weights  $w_{t} = 1$  and  $W_{T} = \sum_{t=0}^{T} w_{t}$ , making it hod that:
$$
\Psi_ {T} := \frac {b}{T + 1} \sum_ {t = 0} ^ {T} s _ {t} \leq \frac {r _ {0}}{\gamma (T + 1)} + c _ {1} \gamma + c _ {2} \gamma^ {2} \tag {44}
$$
Tuning the learning rate carefully. By tuning the learning rate carefully in (44), we have
$$
\Psi_ {T} \leq 2 c _ {1} ^ {\frac {1}{2}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {1}{2}} + 2 c _ {2} ^ {\frac {1}{3}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {2}{3}} + \frac {d r _ {0}}{T + 1}. \tag {45}
$$
Proof. For constant learning rates  $\gamma_{t} = \gamma$  we can derive the estimate
$$
\Psi_ {T} = \frac {1}{\gamma (T + 1)} \sum_ {t = 0} ^ {T} \left(r _ {t} - r _ {t + 1}\right) + c _ {1} \gamma + c _ {2} \gamma^ {2} \leq \frac {r _ {0}}{\gamma (T + 1)} + c _ {1} \gamma + c _ {2} \gamma^ {2},
$$
which is the first result (44) of this lemma. Let  $\frac{r_0}{\gamma(T + 1)} = c_1\gamma$  and  $\frac{r_0}{\gamma(T + 1)} = c_2\gamma^2$ , yielding two choices of  $\gamma$ ,  $\gamma = \left(\frac{r_0}{c_1(T + 1)}\right)^{\frac{1}{2}}$  and  $\gamma = \left(\frac{r_0}{c_2(T + 1)}\right)^{\frac{1}{3}}$ . Then choosing  $\gamma = \min \left\{\left(\frac{r_0}{c_1(T + 1)}\right)^{\frac{1}{2}}, \left(\frac{r_0}{c_2(T + 1)}\right)^{\frac{1}{3}}, \frac{1}{d}\right\} \leq \frac{1}{d}$ , there are three cases:
- If  $\gamma = \frac{1}{d}$ , which implies that  $\gamma = \frac{1}{d} \leq \left(\frac{r_0}{c_1(T + 1)}\right)^{\frac{1}{2}}$  and  $\gamma = \frac{1}{d} \leq \left(\frac{r_0}{c_2(T + 1)}\right)^{\frac{1}{3}}$ , then
$$
\Psi_ {T} \leq \frac {d r _ {0}}{T + 1} + \frac {c _ {1}}{d} + \frac {c _ {2}}{d ^ {2}} \leq \frac {d r _ {0}}{T + 1} + c _ {1} ^ {\frac {1}{2}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {1}{2}} + c _ {2} ^ {\frac {1}{3}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {2}{3}}
$$
- If  $\gamma = \left(\frac{r_0}{c_1(T + 1)}\right)^{\frac{1}{2}}$ , which implies that  $\gamma = \left(\frac{r_0}{c_1(T + 1)}\right)^{\frac{1}{2}} \leq \left(\frac{r_0}{c_2(T + 1)}\right)^{\frac{1}{3}}$ , then
$$
\Psi_ {T} \leq 2 c _ {1} \left(\frac {r _ {0}}{c _ {1} (T + 1)}\right) ^ {\frac {1}{2}} + c _ {2} \left(\frac {r _ {0}}{c _ {1} (T + 1)}\right) \leq 2 c _ {1} ^ {\frac {1}{2}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {1}{2}} + c _ {2} ^ {\frac {1}{3}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {2}{3}}
$$
- If  $\gamma = \left(\frac{r_0}{c_2(T + 1)}\right)^{\frac{1}{3}}$ , which implies that  $\gamma = \left(\frac{r_0}{c_2(T + 1)}\right)^{\frac{1}{3}} \leq \left(\frac{r_0}{c_1(T + 1)}\right)^{\frac{1}{2}}$ , then
$$
\Psi_ {T} \leq c _ {1} \left(\frac {r _ {0}}{c _ {2} (T + 1)}\right) ^ {\frac {1}{3}} + 2 c _ {2} ^ {\frac {1}{3}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {2}{3}} \leq c _ {1} ^ {\frac {1}{2}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {1}{2}} + 2 c _ {2} ^ {\frac {1}{3}} \left(\frac {r _ {0}}{T + 1}\right) ^ {\frac {2}{3}}
$$
Combining these three cases, we get the second result of this lemma.
D.2.2 Proof of general convex case of Theorem 1 and Corollary 1
Proof of the general convex case of Theorem 1. Letting  $\mu = 0$  in (41), we get the recursion of the general convex case,
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \leq \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] - \frac {\tilde {\eta}}{3} \mathbb {E} \left[ D _ {F} (\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}) \right] \\ + \frac {4 \tilde {\eta} ^ {2} \sigma^ {2}}{S K} + \frac {4 \tilde {\eta} ^ {2} (M - S) \zeta_ {*} ^ {2}}{S (M - 1)} + \frac {6 L \tilde {\eta} ^ {3} \sigma^ {2}}{S K} + \frac {6 L \tilde {\eta} ^ {3} \zeta_ {*} ^ {2}}{S} \\ \end{array}
$$
Applying Lemma 8 with  $t = r$  ( $T = R$ ),  $\gamma = \tilde{\eta}$ ,  $r_t = \mathbb{E}\left[\|\mathbf{x}^{(r)} - \mathbf{x}^*\|^2\right]$ ,  $b = \frac{1}{3}$ ,  $s_t = \mathbb{E}\left[D_F(\mathbf{x}^{(r)}, \mathbf{x}^*)\right]$ ,  $w_t = 1$ ,  $c_1 = \frac{4\sigma^2}{SK} + \frac{4(M-S)\zeta_*^2}{S(M-1)}$ ,  $c_2 = \frac{6L\sigma^2}{SK} + \frac{6L\zeta_*^2}{S}$  and  $\frac{1}{d} = \frac{1}{6L}$  ( $\tilde{\eta} = MK\eta \leq \frac{1}{6L}$ ), it follows that
$$
\begin{array}{l} \mathbb {E} \left[ F (\bar {\mathbf {x}} ^ {(R)}) - F (\mathbf {x} ^ {*}) \right] \leq \frac {1}{W _ {R}} \sum_ {r = 0} ^ {R} w _ {r} \left(F (\mathbf {x} ^ {(r)}) - F (\mathbf {x} ^ {*})\right) \\ \leq \frac {3 \left\| \mathbf {x} ^ {(0)} - \mathbf {x} ^ {*} \right\| ^ {2}}{\tilde {\eta} R} + \frac {1 2 \tilde {\eta} \sigma^ {2}}{S K} + \frac {1 2 \tilde {\eta} (M - S) \zeta_ {*} ^ {2}}{S (M - 1)} + \frac {1 8 L \tilde {\eta} ^ {2} \sigma^ {2}}{S K} + \frac {1 8 L \tilde {\eta} ^ {2} \zeta_ {*} ^ {2}}{S} \tag {46} \\ \end{array}
$$
where  $\bar{\mathbf{x}}^{(R)} = \frac{1}{W_R}\sum_{r = 0}^{R}w_r\mathbf{x}^{(r)}$  and we use Jensen's inequality ( $F$  is convex) in the first inequality. By tuning the learning rate carefully, we get
$$
F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) = \mathcal {O} \left(\frac {\sigma D}{\sqrt {S K R}} + \sqrt {1 - \frac {S}{M}} \cdot \frac {\zeta_ {*} D}{\sqrt {S R}} + \frac {\left(L \sigma^ {2} D ^ {4}\right) ^ {1 / 3}}{\left(S K\right) ^ {1 / 3} R ^ {2 / 3}} + \frac {\left(L \zeta_ {*} ^ {2} D ^ {4}\right) ^ {1 / 3}}{S ^ {1 / 3} R ^ {2 / 3}} + \frac {L D ^ {2}}{R}\right) \tag {47}
$$
where  $D \coloneqq \left\| \mathbf{x}^{(0)} - \mathbf{x}^* \right\|$ . Eq. (46) and Eq. (47) are the upper bounds with partial client participation. In particular, when  $S = M$ , we can get the claim of the strongly convex case of Theorem 1 and Corollary 1.
D.3 Non-convex case
Lemma 9. Let Assumptions 1, 2, 3b hold. If the learning rate satisfies  $\eta \leq \frac{1}{6LSK}$ , then it holds that
$$
\begin{array}{l} \mathbb {E} \left[ F (\mathbf {x} ^ {(r + 1)}) - F (\mathbf {x} ^ {(r)}) \right] \leq - \frac {S K \eta}{2} \mathbb {E} \left[ \left\| \nabla F (\mathbf {x} ^ {(r)}) \right\| ^ {2} \right] + L S K \eta^ {2} \sigma^ {2} \\ + \frac {L ^ {2} \eta}{2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} ^ {(r)} - \mathbf {x} ^ {(r)} \right\| ^ {2} \right] \tag {48} \\ \end{array}
$$
$$
\mathbb {E} \left[ \Delta \mathbf {x} \right] = - \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \right]
$$
Starting from the smoothness of  $F$  (applying Eq. (16),  $D_F(\pmb{x},\pmb{y}) \leq \frac{L}{2}\|\pmb{x} - \pmb{y}\|^2$  with  $\pmb{x} = \mathbf{x} + \Delta \mathbf{x}$ ,  $\pmb{y} = \mathbf{x}$ ), and substituting the overall updates, we have
$$
\begin{array}{l} \mathbb {E} \left[ F (\mathbf {x} + \Delta \mathbf {x}) - F (\mathbf {x}) \right] \\ \leq \mathbb {E} \left[ \langle \nabla F (\mathbf {x}), \Delta \mathbf {x} \rangle \right] + \frac {L}{2} \mathbb {E} \left[ \| \Delta \mathbf {x} \| ^ {2} \right] \\ \leq - \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle \nabla F (\mathbf {x}), \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right) \right\rangle \right] + \frac {L \eta^ {2}}{2} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \tag {49} \\ \end{array}
$$
For the first term on the right hand side in (49), using the fact that  $2\langle a,b\rangle = \| a\| ^2 +\| b\| ^2 -\| a - b\| ^2$  with  $a = \nabla F(\mathbf{x})$  and  $b = \nabla F_{\pi_m}(\mathbf{x}_{m,k})$ , we have
$$
\begin{array}{l} - \eta \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle \nabla F (\mathbf {x}), \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right) \right\rangle \right] \\ = - \frac {\eta}{2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F (\mathbf {x}) \| ^ {2} + \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \| ^ {2} - \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) - \nabla F (\mathbf {x}) \| ^ {2} \right] \\ \end{array}
$$
$$
\leq \frac {S K \eta}{2} \| \nabla F (\mathbf {x}) \| ^ {2} - \frac {\eta}{2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \| ^ {2} \right] + \frac {L ^ {2} \eta}{2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \tag {50}
$$
For the third term on the right hand side in (49), using Jensen's inequality, we have
$$
\begin{array}{l} \frac {L \eta^ {2}}{2} \mathbb {E} \left[ \left| \left| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right| \right| ^ {2} \right] \\ \leq L \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} - \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \right\| ^ {2} \right] + L \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \right\| ^ {2} \right] \\ \leq L S K \eta^ {2} \sigma^ {2} + L S K \eta^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \| ^ {2} \right], \tag {51} \\ \end{array}
$$
where we apply Lemma 1 by seeing the data sample  $\xi_{m,k}$ , the stochastic gradient  $\mathbf{g}_{\pi_m,k}$ , the gradient  $\nabla F_{\pi_m}(\xi_{m,k})$  as  $\xi_i, \mathbf{x}_i, \mathbf{e}_i$  respectively in Lemma 1 for the first term and Jensen's inequality for the second term in the preceding inequality.
Substituting (50) and (51) into (49), we have
$$
\begin{array}{l} \mathbb {E} \left[ F (\mathbf {x} + \Delta \mathbf {x}) - F (\mathbf {x}) \right] \leq - \frac {S K \eta}{2} \| \nabla F (\mathbf {x}) \| ^ {2} + L S K \eta^ {2} \sigma^ {2} + \frac {L ^ {2} \eta}{2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \\ - \frac {\eta}{2} (1 - 2 L S K \eta) \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \| ^ {2} \right] \\ \end{array}
$$
Since  $\eta \leq \frac{1}{6LSK}$ , the last term on the right hand side in the preceding inequality is negative. Then
$$
\mathbb {E} \left[ F (\mathbf {x} + \Delta \mathbf {x}) - F (\mathbf {x}) \right] \leq - \frac {S K \eta}{2} \| \nabla F (\mathbf {x}) \| ^ {2} + L S K \eta^ {2} \sigma^ {2} + \frac {L ^ {2} \eta}{2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right]
$$
The claim follows after recovering the superscripts and taking unconditional expectation.
D.3.1 Bounding the client drift with Assumption 3a
Since Eq. (18), which holds only for convex functions, is used in the proof of Lemma 6 (i.e., Eq. (36)), we cannot use the result of Lemma 6. Next, we use Assumption 3a to bound the client drift (defined in (33)).
Lemma 10. Let Assumptions 1, 2, 3a hold. If the learning rate satisfies  $\eta \leq \frac{1}{6LSK}$ , then the client drift is bounded:
$$
E _ {r} \leq \frac {9}{4} S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + \frac {9}{4} S ^ {2} K ^ {3} \eta^ {2} \zeta^ {2} + \left(\frac {9}{4} \beta^ {2} S ^ {2} K ^ {3} \eta^ {2} + \frac {3}{2} S ^ {3} K ^ {3} \eta^ {2}\right) \mathbb {E} \left[ \left\| \nabla F (\mathbf {x} ^ {(r)}) \right\| ^ {2} \right] \tag {52}
$$
We use Jensen's inequality to bound the term  $\mathbb{E}\left[\| \mathbf{x}_{m,k} - \mathbf{x}\| ^2\right] = \eta^2\mathbb{E}\left[\left\| \sum_{i = 1}^{m}\sum_{j = 0}^{b_{m,k}(i)}\mathbf{g}_{\pi_i,j}\right\| ^2\right]$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} - \mathbf {x} \right\| ^ {2} \right] \\ \leq 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\mathbf {g} _ {\pi_ {i}, j} - \nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j})\right) \right\| ^ {2} \right] + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j}) - \nabla F _ {\pi_ {i}} (\mathbf {x})\right) \right\| ^ {2} \right] \\ + 4 \eta^ {2} \mathbb {E} \underbrace {\left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m , k} (i)} (\nabla F _ {\pi_ {i}} (\mathbf {x}) - \nabla F (\mathbf {x})) \right\| ^ {2} \right]} _ {T _ {1}} + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \nabla F (\mathbf {x}) \right\| ^ {2} \right] \\ \end{array}
$$
Applying Lemma 1, Jensen's inequality and Jensen's inequality to the first, third and forth terms on the right hand side in the preceding inequality respectively, we can get
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} - \mathbf {x} \right\| ^ {2} \right] \\ \leq 4 \eta^ {2} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbb {E} \left[ \| \mathbf {g} _ {\pi_ {i}, j} - \nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j}) \| ^ {2} \right] + 4 \eta^ {2} \mathcal {B} _ {m, k} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbb {E} \left[ \| \nabla F _ {\pi_ {i}} (\mathbf {x} _ {i, j}) - \nabla F _ {\pi_ {i}} (\mathbf {x}) \| ^ {2} \right] \\ + 4 \eta^ {2} T _ {1} + 4 \mathcal {B} _ {m, k} ^ {2} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2} \tag {53} \\ \end{array}
$$
where  $\mathcal{B}_{m,k} := \sum_{i=1}^{m} \sum_{j=0}^{b_{m,k}(i)} 1 = (m-1)K + k$ . The first term on the right-hand side in (53) is bounded by  $4\mathcal{B}_{m,k}\eta^2\sigma^2$  with Assumption 2. The second term on the right-hand side in (53) can be bounded by  $4L^2\eta^2\mathcal{B}_{m,k}\sum_{i=1}^{m}\sum_{j=0}^{b_{m,k}(i)}\mathbb{E}\left[\|\mathbf{x}_{i,j}-\mathbf{x}\|^2\right]$  with Assumption 1. Then, we have
$$
\mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \leq 4 \mathcal {B} _ {m, k} \eta^ {2} \sigma^ {2} + 4 L ^ {2} \eta^ {2} \mathcal {B} _ {m, k} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbb {E} \left[ \| \mathbf {x} _ {i, j} - \mathbf {x} \| ^ {2} \right] + 4 \eta^ {2} T _ {1} + 4 \mathcal {B} _ {m, k} ^ {2} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2}
$$
$$
\begin{array}{l} E _ {r} \leq 4 \eta^ {2} \sigma^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathcal {B} _ {m, k} + 4 L ^ {2} \eta^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathcal {B} _ {m, k} \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \mathbb {E} \left[ \| \mathbf {x} _ {i, j} - \mathbf {x} \| ^ {2} \right] \\ + 4 \eta^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\nabla F _ {\pi_ {i}} (\mathbf {x}) - \nabla F (\mathbf {x})\right) \right\| ^ {2} \right] + 4 \eta^ {2} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathcal {B} _ {m, k} ^ {2} \| \nabla F (\mathbf {x}) \| ^ {2} \\ \end{array}
$$
Applying Lemma 4 with  $\mathbf{x}_{\pi_i} = \nabla F_{\pi_i}(\mathbf{x})$  and  $\overline{\pmb{x}} = \nabla F(\mathbf{x})$  to the third term and  $\sum_{m=1}^{S}\sum_{k=0}^{K-1}\mathcal{B}_{m,k} \leq \frac{1}{2}S^2K^2$  and  $\sum_{m=1}^{S}\sum_{k=0}^{K-1}\mathcal{B}_{m,k}^2 \leq \frac{1}{3}S^3K^3$  to the other terms on the right hand side in the preceding inequality, we can simplify it:
$$
\begin{array}{l} E _ {r} \leq 2 S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + 2 L ^ {2} S ^ {2} K ^ {2} \eta^ {2} E _ {r} + 2 S ^ {2} K ^ {3} \eta^ {2} \left(\frac {1}{M} \sum_ {i = 1} ^ {M} \| \nabla F _ {i} (\mathbf {x}) - \nabla F (\mathbf {x}) \| ^ {2}\right) + \frac {4}{3} S ^ {3} K ^ {3} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2} \\ \stackrel {\mathrm {A s m .}} {\leq} ^ {3 a} 2 S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + 2 L ^ {2} S ^ {2} K ^ {2} \eta^ {2} E _ {r} + 2 S ^ {2} K ^ {3} \eta^ {2} \zeta^ {2} + 2 \beta^ {2} S ^ {2} K ^ {3} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2} + \frac {4}{3} S ^ {3} K ^ {3} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2} \\ \end{array}
$$
$$
(1 - 2 L ^ {2} S ^ {2} K ^ {2} \eta^ {2}) E _ {r} \leq 2 S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + 2 S ^ {2} K ^ {3} \eta^ {2} \zeta^ {2} + 2 \beta^ {2} S ^ {2} K ^ {3} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2} + \frac {4}{3} S ^ {3} K ^ {3} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2}
$$
$$
E _ {r} \leq \frac {9}{4} S ^ {2} K ^ {2} \eta^ {2} \sigma^ {2} + \frac {9}{4} S ^ {2} K ^ {3} \eta^ {2} \zeta^ {2} + \frac {9}{4} \beta^ {2} S ^ {2} K ^ {3} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2} + \frac {3}{2} S ^ {3} K ^ {3} \eta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2}.
$$
D.3.2 Proof of non-convex case of Theorem 1 and Corollary 1
Proof of non-convex case of Theorem 1. Substituting (52) into (48) and using  $\eta \leq \frac{1}{6LSK}\min \left\{1,\frac{\sqrt{S}}{\beta}\right\}$ , we can simplify the recursion as follows:
$$
\mathbb {E} \left[ F (\mathbf {x} ^ {(r + 1)}) - F (\mathbf {x} ^ {(r)}) \right] \leq - \frac {1}{3} S K \eta \mathbb {E} \left[ \left\| \nabla F (\mathbf {x} ^ {(r)}) \right\| ^ {2} \right] + L S K \eta^ {2} \sigma^ {2} + \frac {9}{8} L ^ {2} S ^ {2} K ^ {2} \eta^ {3} \sigma^ {2} + \frac {9}{8} L ^ {2} S ^ {2} K ^ {3} \eta^ {3} \zeta^ {2}
$$
Letting  $\tilde{\eta} \coloneqq SK\eta$ , subtracting  $F^{*}$  from both sides and then rearranging the terms, we have
$$
\mathbb {E} \left[ F (\mathbf {x} ^ {(r + 1)}) - F ^ {*} \right] \leq \mathbb {E} \left[ F (\mathbf {x} ^ {(r)}) - F ^ {*} \right] - \frac {\tilde {\eta}}{3} \mathbb {E} \left[ \left\| \nabla F (\mathbf {x} ^ {(r)}) \right\| ^ {2} \right] + \frac {L \tilde {\eta} ^ {2} \sigma^ {2}}{S K} + \frac {9 L ^ {2} \tilde {\eta} ^ {3} \sigma^ {2}}{8 S K} + \frac {9 L ^ {2} \tilde {\eta} ^ {3} \zeta^ {2}}{8 S}
$$
Then applying Lemma 8 with  $t = r$  ( $T = R$ ),  $\gamma = \tilde{\eta}$ ,  $r_t = \mathbb{E}\left[F(\mathbf{x}^{(r)}) - F^*\right]$ ,  $b = \frac{1}{3}$ ,  $s_t = \mathbb{E}\left[\|\nabla F(\mathbf{x}^{(r)})\|^2\right]$ ,  $w_t = 1$ ,  $c_1 = \frac{L\sigma^2}{SK}$ ,  $c_2 = \frac{9L^2\sigma^2}{8SK} + \frac{9L^2\zeta^2}{8S}$  and  $\frac{1}{d} = \frac{1}{6L}\min\left\{1, \frac{\sqrt{S}}{\beta}\right\}$  ( $\tilde{\eta} = SK\eta \leq \frac{1}{6L}\min\left\{1, \frac{\sqrt{S}}{\beta}\right\}$ ), we have
$$
\min  _ {0 \leq r \leq R} \mathbb {E} \left[ \left\| \nabla F \left(\mathbf {x} ^ {(r)}\right) \right\| ^ {2} \right] \leq \frac {3 (F \left(\mathbf {x} ^ {0}\right) - F ^ {*})}{\tilde {\eta} R} + \frac {3 L \tilde {\eta} \sigma^ {2}}{S K} + \frac {2 7 L ^ {2} \tilde {\eta} ^ {2} \sigma^ {2}}{8 S K} + \frac {2 7 L ^ {2} \tilde {\eta} ^ {2} \zeta^ {2}}{8 S} \tag {54}
$$
where we use  $\min_{0\leq r\leq R}\mathbb{E}\left[\left\| \nabla F(\mathbf{x}^{(r)})\right\| ^2\right]\leq \frac{1}{R + 1}\sum_{r = 0}^{R}\mathbb{E}\left[\left\| \nabla F(\mathbf{x}^{(r)})\right\| ^2\right]$ . Then, using  $\tilde{\eta}\leq \frac{1}{6L(\beta + 1)}\leq \min \left\{1,\frac{\sqrt{S}}{\beta}\right\}$  and tuning the learning rate carefully, we get
$$
\min  _ {0 \leq r \leq R} \mathbb {E} \left[ \left\| \nabla F \left(\mathbf {x} ^ {(r)}\right) \right\| ^ {2} \right] = \mathcal {O} \left(\frac {\left(L \sigma^ {2} A\right) ^ {1 / 2}}{\sqrt {S K R}} + \frac {\left(L ^ {2} \sigma^ {2} A ^ {2}\right) ^ {1 / 3}}{(S K) ^ {1 / 3} R ^ {2 / 3}} + \frac {\left(L ^ {2} \zeta^ {2} A ^ {2}\right) ^ {1 / 3}}{S ^ {1 / 3} R ^ {2 / 3}} + \frac {L \beta A}{R}\right) \tag {55}
$$
where  $A \coloneqq F(\mathbf{x}^0) - F^*$ . Eq. (54) and Eq. (55) are the upper bounds with partial client participation. In particular, when  $S = M$ , we get the claim of the non-convex case of Theorem 1 and Corollary 1.
E Proofs of Theorem 2
Here we slightly improve the convergence guarantee for the strongly convex case by combining the works of Karimireddy et al. (2020); Koloskova et al. (2020). Moreover, we reproduce the guarantees for the general convex and non-convex cases based on Karimireddy et al. (2020) for completeness. The results are given in Theorem 2. We provide the proof of Theorem 2 for the strongly convex, general convex and non-convex cases in Sections E.1, E.2 and E.3, respectively.
In the following proof, we consider the partial client participation setting, specifically, selecting partial clients without replacement. So we assume that  $\pi = \{\pi_1,\pi_2,\dots ,\pi_M\}$  is a permutation of  $\{1,2,\ldots ,M\}$  in a certain training round and only the first  $S$  selected clients  $\{\pi_1,\pi_2,\dots ,\pi_S\}$  will participate in this round. Without otherwise stated, we use  $\mathbb{E}[\cdot ]$  to represent the expectation with respect to both types of randomness (i.e., sampling data samples  $\xi$  and sampling clients  $\pi$ ).
Theorem 2. For PFL (Algorithm 2), there exist a constant effective learning rate  $\tilde{\eta} \coloneqq K\eta$  and weights  $w_{r}$ , such that the weighted average of the global parameters  $\bar{\mathbf{x}}^{(R)} \coloneqq \frac{1}{W_R} \sum_{r=0}^{R} w_r \mathbf{x}^{(r)}$  ( $W_R = \sum_{r=0}^{R} w_r$ ) satisfies the following upper bounds:
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] \leq \frac {9}{2} \mu D ^ {2} \exp \left(- \frac {\mu \tilde {\eta} R}{2}\right) + \frac {1 2 \tilde {\eta} \sigma^ {2}}{M K} + \frac {1 8 L \tilde {\eta} ^ {2} \sigma^ {2}}{K} + 1 2 L \tilde {\eta} ^ {2} \zeta_ {*} ^ {2} \tag {56}
$$
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] \leq \frac {3 D ^ {2}}{\tilde {\eta} R} + \frac {1 2 \tilde {\eta} \sigma^ {2}}{M K} + \frac {1 8 L \tilde {\eta} ^ {2} \sigma^ {2}}{K} + 1 2 L \tilde {\eta} ^ {2} \zeta_ {*} ^ {2} \tag {57}
$$
$$
\min  _ {0 \leq r \leq R} \mathbb {E} \left[ \| \nabla F \left(\mathbf {x} ^ {(r)}\right) \| ^ {2} \right] \leq \frac {3 A}{\tilde {\eta} R} + \frac {3 L \tilde {\eta} \sigma^ {2}}{S K} + \frac {2 7 L ^ {2} \tilde {\eta} ^ {2} \sigma^ {2}}{8 K} + \frac {9}{4} L ^ {2} \tilde {\eta} ^ {2} \zeta^ {2} \tag {58}
$$
Corollary 2. Applying the results of Theorem 2. By choosing a appropriate learning rate (see the proof of Theorem 2 in Appendix E), we can obtain the convergence rates for PFL as follows:
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] = \tilde {\mathcal {O}} \left(\frac {\sigma^ {2}}{\mu M K R} + \frac {L \sigma^ {2}}{\mu^ {2} K R ^ {2}} + \frac {L \zeta_ {*} ^ {2}}{\mu^ {2} R ^ {2}} + \mu D ^ {2} \exp \left(- \frac {\mu R}{1 2 L}\right)\right) \tag {59}
$$
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] = \mathcal {O} \left(\frac {\sigma D}{\sqrt {M K R}} + \frac {\left(L \sigma^ {2} D ^ {4}\right) ^ {1 / 3}}{K ^ {1 / 3} R ^ {2 / 3}} + \frac {\left(L \zeta_ {*} ^ {2} D ^ {4}\right) ^ {1 / 3}}{R ^ {2 / 3}} + \frac {L D ^ {2}}{R}\right) \tag {60}
$$
$$
\min  _ {0 \leq r \leq R} \mathbb {E} \left[ \| \nabla F (\mathbf {x} ^ {(r)}) \| ^ {2} \right] = \mathcal {O} \left(\frac {\left(L \sigma^ {2} A\right) ^ {1 / 2}}{\sqrt {M K R}} + \frac {\left(L ^ {2} \sigma^ {2} A ^ {2}\right) ^ {1 / 3}}{K ^ {1 / 3} R ^ {2 / 3}} + \frac {\left(L ^ {2} \zeta^ {2} A ^ {2}\right) ^ {1 / 3}}{R ^ {2 / 3}} + \frac {L \beta A}{R}\right) \tag {61}
$$
E.1 Strongly convex case