SlowGuess/NeurIPS_2023_OCR huggingface dataset:
E.1.1 Find the per-round recursion
Lemma 11. Let Assumptions 1, 2, 3b hold and assume that all the local objectives are  $\mu$ -strongly convex. If the learning rate satisfies  $\eta \leq \frac{1}{6LK}$ , then it holds that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \leq \left(1 - \frac {\mu K \eta}{2}\right) \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] + \frac {4 K \eta^ {2} \sigma^ {2}}{S} + 4 K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} \\ - \frac {2}{3} K \eta \mathbb {E} \left[ D _ {F} \left(\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}\right) \right] + \frac {8}{3} L \eta \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} ^ {(r)} - \mathbf {x} ^ {(r)} \right\| ^ {2} \right] \tag {62} \\ \end{array}
$$
Proof. According to Algorithm 2, the overall model updates of PFL after one complete training round (with  $S$  clients selected for training) is
$$
\Delta \mathbf {x} = \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {(r)} = - \frac {\eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} ^ {(r)},
$$
$$
\mathbb {E} \left[ \Delta \mathbf {x} \right] = - \frac {\eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right) \right]
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} + \Delta \mathbf {x} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \\ = \| \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} + 2 \mathbb {E} \left[ \langle \mathbf {x} - \mathbf {x} ^ {*}, \Delta \mathbf {x} \rangle \right] + \mathbb {E} \left[ \| \Delta \mathbf {x} \| ^ {2} \right] \\ = \left\| \mathbf {x} - \mathbf {x} ^ {*} \right\| ^ {2} - \frac {2 \eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right), \mathbf {x} - \mathbf {x} ^ {*} \right\rangle \right] + \frac {\eta^ {2}}{S ^ {2}} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \tag {63} \\ \end{array}
$$
We can apply Lemma 2 with  $\pmb{x} = \mathbf{x}_{m,k}$ ,  $\pmb{y} = \mathbf{x}^*$ ,  $\pmb{z} = \mathbf{x}$  and  $h = F_{\pi_m}$  for the second term on the right hand side in (63):
$$
\begin{array}{l} - \frac {2 \eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \langle \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}), \mathbf {x} - \mathbf {x} ^ {*} \rangle \right] \\ \leq - \frac {2 \eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ F _ {\pi_ {m}} (\mathbf {x}) - F _ {\pi_ {m}} (\mathbf {x} ^ {*}) + \frac {\mu}{4} \left\| \mathbf {x} - \mathbf {x} ^ {*} \right\| ^ {2} - L \left\| \mathbf {x} _ {m, k} - \mathbf {x} \right\| ^ {2} \right] \\ \leq - 2 K \eta D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) - \frac {1}{2} \mu K \eta \| \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} + \frac {2 L \eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \tag {64} \\ \end{array}
$$
For the third term on the right hand side in (63), using Jensen's inequality, we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \\ \leq 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\mathbf {g} _ {\pi_ {m}, k} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k})\right) \right\| ^ {2} \right] + 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) - \nabla F _ {\pi_ {m}} (\mathbf {x})\right) \right\| ^ {2} \right] \\ + 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x}) - \nabla F _ {\pi_ {m}} \left(\mathbf {x} ^ {*}\right)\right) \right\| ^ {2} \right] + 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \nabla F _ {\pi_ {m}} \left(\mathbf {x} ^ {*}\right) \right\| ^ {2} \right] \tag {65} \\ \end{array}
$$
For the first term on the right hand side in (65), we have
$$
\begin{array}{l} 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\mathbf {g} _ {\pi_ {m}, k} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k})\right) \right\| ^ {2} \right] = 4 \sum_ {m = 1} ^ {S} \mathbb {E} \left[ \left\| \sum_ {k = 0} ^ {K - 1} \left(\mathbf {g} _ {\pi_ {m}, k} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k})\right) \right\| ^ {2} \right] \\ \stackrel {\text {L e m .}} {=} ^ {1} 4 \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {g} _ {\pi_ {m}, k} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \| ^ {2} \right] \\ \stackrel {\text {A s m .} 2} {\leq} 4 S K \sigma^ {2}, \tag {66} \\ \end{array}
$$
where we use the fact that clients are independent to each other in the first equality and apply Lemma 1 by seeing the data sample  $\xi_{m,k}$ , the stochastic gradient  $\mathbf{g}_{\pi_m,k}$ , the gradient  $\nabla F_{\pi_m}(\xi_{m,k})$  as  $\xi_i, \mathbf{x}_i, \mathbf{e}_i$  in the second equality. For the second term on the right hand side in (65), we have
$$
\begin{array}{l} 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) - \nabla F _ {\pi_ {m}} (\mathbf {x})\right) \right\| ^ {2} \right] \stackrel {(1 5)} {\leq} 4 S K \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) - \nabla F _ {\pi_ {m}} (\mathbf {x}) \| ^ {2} \right] \\ \stackrel {\text {A s m .}} {\leq} ^ {1} 4 L ^ {2} S K \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \\ \end{array}
$$
For the third term on the right hand side in (65), we have
$$
\begin{array}{l} 4 \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x}) - \nabla F _ {\pi_ {m}} (\mathbf {x} ^ {*})\right) \right\| ^ {2} \right] \stackrel {(1 5)} {\leq} 4 S K \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x}) - \nabla F _ {\pi_ {m}} (\mathbf {x} ^ {*}) \| ^ {2} \right] \\ \stackrel {(1 8)} {\leq} 8 L S K \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ D _ {F _ {\pi_ {m}}} \left(\mathbf {x}, \mathbf {x} ^ {*}\right) \right] \\ = 8 L S ^ {2} K ^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}), \\ \end{array}
$$
where the last inequality is because  $\mathbb{E}\left[D_{F_{\pi_m}}(\mathbf{x},\mathbf{x}^*)\right] = D_F(\mathbf{x},\mathbf{x}^*)$ . The forth term on the right hand side in (65) can be bounded by Lemma 3 as follows:
With the preceding four inequalities, we can bound the third term on the right hand side in (63):
$$
\begin{array}{l} \frac {\eta^ {2}}{S ^ {2}} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \leq \frac {4 K \eta^ {2} \sigma^ {2}}{S} + 4 L ^ {2} K \eta^ {2} \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \\ + 8 L K ^ {2} \eta^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + 4 K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} \tag {67} \\ \end{array}
$$
Then substituting (64) and (67) into (63), we have
$$
\begin{array}{l} \mathbb {E} \left[ \| \mathbf {x} + \Delta \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} \right] \leq \left(1 - \frac {\mu K \eta}{2}\right) \| \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} + \frac {4 K \eta^ {2} \sigma^ {2}}{S} + 4 K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} \\ - 2 K \eta (1 - 4 L K \eta) D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + 2 L \eta (1 + 2 L K \eta) \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \\ \leq \left(1 - \frac {\mu K \eta}{2}\right) \| \mathbf {x} - \mathbf {x} ^ {*} \| ^ {2} + \frac {4 K \eta^ {2} \sigma^ {2}}{S} + 4 K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} \\ - \frac {2}{3} K \eta D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + \frac {8}{3} L \eta \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right], \\ \end{array}
$$
where we use the condition that  $\eta \leq \frac{1}{6LK}$  in the last inequality. The claim of this lemma follows after recovering the superscripts and taking unconditional expectation.
E.1.2 Bounding the client drift with Assumption 3b
The "client drift" (Karimireddy et al., 2020) in PFL is defined as follows: :
$$
\mathcal {E} _ {r} := \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} ^ {(r)} - \mathbf {x} ^ {(r)} \right\| ^ {2} \right] \tag {68}
$$
Lemma 12. Let Assumptions 1, 2, 3b hold and assume that all the local objectives are  $\mu$ -strongly convex. If the learning rate satisfies  $\eta \leq \frac{1}{6LK}$ , then the client drift is bounded:
$$
\mathcal {E} _ {r} \leq \frac {9}{4} K ^ {2} \eta^ {2} \sigma^ {2} + \frac {3}{2} K ^ {3} \eta^ {2} \zeta_ {*} ^ {2} + 3 L K ^ {3} \eta^ {2} \mathbb {E} \left[ D _ {F} \left(\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}\right) \right] \tag {69}
$$
Proof. According to Algorithm 2, the model updates of PFL from  $\mathbf{x}^{(r)}$  to  $\mathbf{x}_{m,k}^{(r)}$  is
$$
\mathbf {x} _ {m, k} ^ {(r)} - \mathbf {x} ^ {(r)} = - \eta \sum_ {j = 0} ^ {k - 1} \mathbf {g} _ {\pi_ {m}, j} ^ {(r)}
$$
We use Jensen's inequality to bound the term  $\mathbb{E}\left[\| \mathbf{x}_{m,k} - \mathbf{x}\| ^2\right] = \eta^2\mathbb{E}\left[\left\| \sum_{j = 0}^{k - 1}\mathbf{g}_{\pi_{m,j}}\right\| ^2\right]$ :
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} - \mathbf {x} \right\| ^ {2} \right] \leq 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {j = 0} ^ {k - 1} \left(\mathbf {g} _ {\pi_ {m}, j} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, j})\right) \right\| ^ {2} \right] + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {j = 0} ^ {k - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, j}) - \nabla F _ {\pi_ {m}} (\mathbf {x})\right) \right\| ^ {2} \right] \\ + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {j = 0} ^ {k - 1} \left(\nabla F _ {\pi_ {m}} (\mathbf {x}) - \nabla F _ {\pi_ {m}} (\mathbf {x} ^ {*})\right) \right\| ^ {2} \right] + 4 \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {j = 0} ^ {k - 1} \nabla F _ {\pi_ {m}} (\mathbf {x} ^ {*}) \right\| ^ {2} \right] \\ \end{array}
$$
Applying Lemma 1 to the first term and Jensen's inequality to the last three terms on the right hand side in the preceding inequality, respectively, we get
$$
\begin{array}{l} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \leq 4 \sum_ {j = 0} ^ {k - 1} \eta^ {2} \mathbb {E} \left[ \| \mathbf {g} _ {\pi_ {m}, j} - \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, j}) \| ^ {2} \right] + 4 k \sum_ {j = 0} ^ {k - 1} \eta^ {2} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, j}) - \nabla F _ {\pi_ {m}} (\mathbf {x}) \| ^ {2} \right] \\ + 4 k ^ {2} \eta^ {2} \mathbb {E} \left[ \left\| \nabla F _ {\pi_ {m}} (\mathbf {x}) - \nabla F _ {\pi_ {m}} (\mathbf {x} ^ {*}) \right\| ^ {2} \right] + 4 k ^ {2} \eta^ {2} \mathbb {E} \left[ \left\| \nabla F _ {\pi_ {m}} (\mathbf {x} ^ {*}) \right\| ^ {2} \right] \\ \end{array}
$$
The first term can be bounded by  $4k\eta^2\sigma^2$  with Assumption 2. The second term can be bounded by  $4L^{2}k\eta^{2}\sum_{j = 0}^{k - 1}\mathbb{E}\left[\| \mathbf{x}_{m,j} - \mathbf{x}\| ^2\right]$  with Assumption 1. The third term can be bounded by  $8Lk^{2}\eta^{2}\mathbb{E}\left[D_{F_{\pi_{m}}}(\mathbf{x},\mathbf{x}^{*})\right]$  with Eq. (18). Thus, we have
$$
\begin{array}{l} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \leq 4 k \eta^ {2} \sigma^ {2} + 4 L ^ {2} k \eta^ {2} \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, j} - \mathbf {x} \| ^ {2} \right] + 8 L k ^ {2} \eta^ {2} \mathbb {E} \left[ D _ {F _ {\pi_ {m}}} (\mathbf {x}, \mathbf {x} ^ {*}) \right] + 4 k ^ {2} \eta^ {2} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} ^ {*}) \| ^ {2} \right] \\ \leq 4 k \eta^ {2} \sigma^ {2} + 4 L ^ {2} k \eta^ {2} \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, j} - \mathbf {x} \| ^ {2} \right] + 8 L k ^ {2} \eta^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + 4 k ^ {2} \eta^ {2} \zeta_ {*} ^ {2} \\ \end{array}
$$
Then turning to  $\mathcal{E}_r\coloneqq \frac{1}{S}\sum_{m = 1}^S\sum_{k = 0}^{K - 1}\mathbb{E}\left[\| \mathbf{x}_{m,k} - \mathbf{x}\| ^2\right]$ , we have
$$
\mathcal {E} _ {r} = 4 \eta^ {2} \sigma^ {2} \sum_ {k = 0} ^ {K - 1} k + 4 L ^ {2} \eta^ {2} \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} k \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, j} - \mathbf {x} \| ^ {2} \right] + 8 L \eta^ {2} \sum_ {k = 0} ^ {K - 1} k ^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + 4 \eta^ {2} \sum_ {k = 0} ^ {K - 1} k ^ {2} \zeta_ {*} ^ {2}
$$
Using the facts that  $\sum_{k=1}^{K-1} k = \frac{(K-1)K}{2} \leq \frac{K^2}{2}$  and  $\sum_{k=1}^{K-1} k^2 = \frac{(K-1)K(2K-1)}{6} \leq \frac{K^3}{3}$ , we can simplify the preceding inequality:
$$
\mathcal {E} _ {r} \leq 2 K ^ {2} \eta^ {2} \sigma^ {2} + 2 L ^ {2} K ^ {2} \eta^ {2} \frac {1}{S} \sum_ {m = 1} ^ {S} \sum_ {j = 0} ^ {k - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, j} - \mathbf {x} \| ^ {2} \right] + \frac {8}{3} L K ^ {3} \eta^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}) + \frac {4}{3} K ^ {3} \eta^ {2} \zeta_ {*} ^ {2}
$$
$$
(1 - 2 L ^ {2} K ^ {2} \eta^ {2}) \mathcal {E} _ {r} \leq 2 K ^ {2} \eta^ {2} \sigma^ {2} + \frac {4}{3} K ^ {3} \eta^ {2} \zeta_ {*} ^ {2} + \frac {8}{3} L K ^ {3} \eta^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*})
$$
Finally, using the condition that  $\eta \leq \frac{1}{6LK}$ , which implies  $1 - 2L^{2}K^{2}\eta^{2} \geq \frac{8}{9}$ , we have
$$
\mathcal {E} _ {r} \leq \frac {9}{4} K ^ {2} \eta^ {2} \sigma^ {2} + \frac {3}{2} K ^ {3} \eta^ {2} \zeta_ {*} ^ {2} + 3 L K ^ {3} \eta^ {2} D _ {F} (\mathbf {x}, \mathbf {x} ^ {*}).
$$
E.1.3 Proof of strongly convex case of Theorem 2
Proof of strongly convex case of Theorem 2. Substituting (69) into (62) and using  $\eta \leq \frac{1}{6LK}$ , we can simplify the recursion as,
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \leq \left(1 - \frac {\mu K \eta}{2}\right) \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] - \frac {1}{3} K \eta \mathbb {E} \left[ D _ {F} (\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}) \right] \\ + \frac {4 K \eta^ {2} \sigma^ {2}}{S} + 4 K ^ {2} \eta^ {2} \frac {M - S}{S (M - 1)} \zeta_ {*} ^ {2} + 6 L K ^ {2} \eta^ {3} \sigma^ {2} + 4 L K ^ {3} \eta^ {3} \zeta_ {*} ^ {2} \\ \end{array}
$$
Let  $\tilde{\eta} = K\eta$  , we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \leq \left(1 - \frac {\mu \tilde {\eta}}{2}\right) \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] - \frac {\tilde {\eta}}{3} \mathbb {E} \left[ D _ {F} (\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}) \right] \\ + \frac {4 \tilde {\eta} ^ {2} \sigma^ {2}}{S K} + \frac {4 \tilde {\eta} ^ {2} (M - S) \zeta_ {*} ^ {2}}{S (M - 1)} + \frac {6 L \tilde {\eta} ^ {3} \sigma^ {2}}{K} + 4 L \tilde {\eta} ^ {3} \zeta_ {*} ^ {2} \tag {70} \\ \end{array}
$$
Applying Lemma 7 with  $t = r$  ( $T = R$ ),  $\gamma = \tilde{\eta}$ ,  $r_t = \mathbb{E}\left[\left\| \mathbf{x}^{(r)} - \mathbf{x}^*\right\|^2\right]$ ,  $a = \frac{\mu}{2}$ ,  $b = \frac{1}{3}$ ,  $s_t = \mathbb{E}\left[D_F(\mathbf{x}^{(r)},\mathbf{x}^*)\right]$ ,  $w_t = (1 - \frac{\mu\tilde{\eta}}{2})^{-(r + 1)}$ ,  $c_1 = \frac{4\sigma^2}{SK} +\frac{4(M - S)\zeta_*^2}{S(M - 1)}$ ,  $c_2 = \frac{6L\sigma^2}{K} +4L\zeta_*^2$  and  $\frac{1}{d} = \frac{1}{6L}$  ( $\tilde{\eta} = K\eta \leq \frac{1}{6L}$ ), it follows that
$$
\begin{array}{l} \mathbb {E} \left[ F (\bar {\mathbf {x}} ^ {(R)}) - F (\mathbf {x} ^ {*}) \right] \leq \frac {1}{W _ {R}} \sum_ {r = 0} ^ {R} w _ {r} \mathbb {E} \left[ F (\mathbf {x} ^ {(r)}) - F (\mathbf {x} ^ {*}) \right] \\ \leq \frac {9}{2} \mu \left\| \mathbf {x} ^ {(0)} - \mathbf {x} ^ {*} \right\| ^ {2} \exp \left(- \frac {1}{2} \mu \tilde {\eta} R\right) + \frac {1 2 \tilde {\eta} \sigma^ {2}}{S K} + \frac {1 2 \tilde {\eta} (M - S) \zeta_ {*} ^ {2}}{S (M - 1)} + \frac {1 8 L \tilde {\eta} ^ {2} \sigma^ {2}}{K} + 1 2 L \tilde {\eta} ^ {2} \zeta_ {*} ^ {2} \tag {71} \\ \end{array}
$$
where  $\bar{\mathbf{x}}^{(R)} = \frac{1}{W_R}\sum_{r = 0}^{R}w_r\mathbf{x}^{(r)}$  and we use Jensen's inequality ( $F$  is convex) in the first inequality. Thus, by tuning the learning rate carefully, we get
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] = \tilde {\mathcal {O}} \left(\mu D ^ {2} \exp \left(- \frac {\mu R}{1 2 L}\right) + \frac {\sigma^ {2}}{\mu S K R} + \frac {(M - S) \zeta_ {*} ^ {2}}{\mu S R (M - 1)} + \frac {L \sigma^ {2}}{\mu^ {2} K R ^ {2}} + \frac {L \zeta_ {*} ^ {2}}{\mu^ {2} R ^ {2}}\right) \tag {72}
$$
where  $D \coloneqq \left\| \mathbf{x}^{(0)} - \mathbf{x}^{*} \right\|$ . Eq. (71) and Eq. (72) are the upper bounds with partial client participation. When  $M$  is large enough, we have  $\frac{(M-S)}{S(M-1)} \approx (1-\frac{S}{M})\frac{1}{S}$ . This is the constant appearing in Karimireddy et al. (2020); Woodworth et al. (2020b). In particular, when  $S=M$ , we can get the claim of the strongly convex case of Theorem 2 and Corollary 2.
E.2 General convex case
E.2.1 Proof of general convex case of Theorem 2 and Corollary 2
Proof of general convex case of Theorem 2. Let  $\mu = 0$  in (70), we get the simplified per-round recursion of general convex case,
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r + 1)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] \leq \mathbb {E} \left[ \left\| \mathbf {x} ^ {(r)} - \mathbf {x} ^ {*} \right\| ^ {2} \right] - \frac {\tilde {\eta}}{3} \mathbb {E} \left[ D _ {F} (\mathbf {x} ^ {(r)}, \mathbf {x} ^ {*}) \right] \\ + \frac {4 \tilde {\eta} ^ {2} \sigma^ {2}}{S K} + \frac {4 \tilde {\eta} ^ {2} (M - S) \zeta_ {*} ^ {2}}{S (M - 1)} + \frac {6 L \tilde {\eta} ^ {3} \sigma^ {2}}{K} + 4 L \tilde {\eta} ^ {3} \zeta_ {*} ^ {2} \\ \end{array}
$$
Applying Lemma 8 with  $t = r$  ( $T = R$ ),  $\gamma = \tilde{\eta}$ ,  $r_t = \mathbb{E}\left[\left\| \mathbf{x}^{(r)} - \mathbf{x}^*\right\|^2\right]$ ,  $b = \frac{1}{3}$ ,  $s_t = \mathbb{E}\left[D_F(\mathbf{x}^{(r)},\mathbf{x}^*)\right]$ ,  $w_t = 1$ ,  $c_1 = \frac{4\sigma^2}{SK} + \frac{4(M - S)\zeta_*^2}{S(M - 1)}$ ,  $c_2 = \frac{6L\sigma^2}{K} + 4L\zeta_*^2$  and  $\frac{1}{d} = \frac{1}{6L}$  ( $\tilde{\eta} = K\eta \leq \frac{1}{6L}$ ), it follows that
$$
\begin{array}{l} \mathbb {E} \left[ F (\bar {\mathbf {x}} ^ {(R)}) - F (\mathbf {x} ^ {*}) \right] \leq \frac {1}{W _ {R}} \sum_ {r = 0} ^ {R} w _ {r} \left(F (\mathbf {x} ^ {(r)}) - F (\mathbf {x} ^ {*})\right) \\ \leq \frac {3 \left\| \mathbf {x} ^ {(0)} - \mathbf {x} ^ {*} \right\| ^ {2}}{\tilde {\eta} R} + \frac {1 2 \tilde {\eta} \sigma^ {2}}{S K} + \frac {1 2 \tilde {\eta} (M - S) \zeta_ {*} ^ {2}}{S (M - 1)} + \frac {1 8 L \tilde {\eta} ^ {2} \sigma^ {2}}{K} + 1 2 L \tilde {\eta} ^ {2} \zeta_ {*} ^ {2} \tag {73} \\ \end{array}
$$
where  $\bar{\mathbf{x}}^{(R)} = \frac{1}{W_R}\sum_{r = 0}^{R}w_r\mathbf{x}^{(r)}$  and we use Jensen's inequality (  $F$  is convex) in the first inequality. By tuning the learning rate carefully, we get
$$
F \left(\bar {\mathbf {x}} ^ {R}\right) - F \left(\mathbf {x} ^ {*}\right) = \mathcal {O} \left(\frac {\sigma D}{\sqrt {S K R}} + \sqrt {1 - \frac {S}{M}} \cdot \frac {\zeta_ {*} D}{\sqrt {S R}} + \frac {\left(L \sigma^ {2} D ^ {4}\right) ^ {1 / 3}}{K ^ {1 / 3} R ^ {2 / 3}} + \frac {\left(L \zeta_ {*} ^ {2} D ^ {4}\right) ^ {1 / 3}}{R ^ {2 / 3}} + \frac {L D ^ {2}}{R}\right) \tag {74}
$$
where  $D \coloneqq \left\| \mathbf{x}^{(0)} - \mathbf{x}^* \right\|$ . Eq. (73) and Eq. (74) are the upper bounds with partial client participation. In particular, when  $S = M$ , we can get the claim of the strongly convex case of Theorem 2 and Corollary 2.
E.3 Non-convex case
Lemma 13. Let Assumptions 1, 2, 3b hold. If the learning rate satisfies  $\eta \leq \frac{1}{6LK}$ , then it holds that
$$
\begin{array}{l} \mathbb {E} \left[ F (\mathbf {x} ^ {(r + 1)}) - F (\mathbf {x} ^ {(r)}) \right] \leq - \frac {K \eta}{2} \mathbb {E} \left[ \left\| \nabla F (\mathbf {x} ^ {(r)}) \right\| ^ {2} \right] + \frac {L K \eta^ {2} \sigma^ {2}}{S} \\ + \frac {L ^ {2} \eta}{2 S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \mathbf {x} _ {m, k} ^ {(r)} - \mathbf {x} ^ {(r)} \right\| ^ {2} \right] \tag {75} \\ \end{array}
$$
$$
\mathbb {E} \left[ \Delta \mathbf {x} \right] = - \frac {\eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \right]
$$
Starting from the smoothness of  $F$  (applying Eq. (16),  $D_F(\pmb{x}, \pmb{y}) \leq \frac{L}{2} \| \pmb{x} - \pmb{y} \|^2$  with  $\pmb{x} = \mathbf{x} + \Delta \mathbf{x}$ ,  $\pmb{y} = \mathbf{x}$ ), and substituting the overall updates, we have
$$
\begin{array}{l} \mathbb {E} \left[ F (\mathbf {x} + \Delta \mathbf {x}) - F (\mathbf {x}) \right] \\ \leq \mathbb {E} \left[ \langle \nabla F (\mathbf {x}), \Delta \mathbf {x} \rangle \right] + \frac {L}{2} \mathbb {E} \left[ \| \Delta \mathbf {x} \| ^ {2} \right] \\ \leq - \frac {\eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle \nabla F (\mathbf {x}), \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right) \right\rangle \right] + \frac {L \eta^ {2}}{2 S ^ {2}} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \tag {76} \\ \end{array}
$$
For the first term on the right hand side in (76), using the fact that  $2\langle a,b\rangle = \| a\| ^2 +\| b\| ^2 -\| a - b\| ^2$  with  $a = \nabla F(\mathbf{x})$  and  $b = \nabla F_{\pi_m}(\mathbf{x}_{m,k})$ , we have
$$
\begin{array}{l} - \frac {\eta}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle \nabla F (\mathbf {x}), \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right) \right\rangle \right] \\ = - \frac {\eta}{2 S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F (\mathbf {x}) \| ^ {2} + \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \| ^ {2} - \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) - \nabla F (\mathbf {x}) \| ^ {2} \right] \\ \end{array}
$$
$$
\leq - \frac {K \eta}{2} \| \nabla F (\mathbf {x}) \| ^ {2} - \frac {\eta}{2 S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}\right) \| ^ {2} \right] + \frac {L ^ {2} \eta}{2 S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \mathbf {x} _ {m, k} - \mathbf {x} \| ^ {2} \right] \tag {77}
$$
For the third term on the right hand side in (76), using Jensen's inequality, we have
$$
\begin{array}{l} \frac {L \eta^ {2}}{2 S ^ {2}} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} \right\| ^ {2} \right] \\ \leq \frac {L \eta^ {2}}{S ^ {2}} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbf {g} _ {\pi_ {m}, k} - \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \right\| ^ {2} \right] + L \eta^ {2} \mathbb {E} \left[ \left\| \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \right\| ^ {2} \right] \\ \leq \frac {L K \eta^ {2} \sigma^ {2}}{S} + \frac {L K \eta^ {2}}{S} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F _ {\pi_ {m}} (\mathbf {x} _ {m, k}) \| ^ {2} \right], \tag {78} \\ \end{array}
$$
where we use independence and Lemma 1 for the first term (see Eq. (66)) and Jensen's inequality for the second term in the preceding inequality.
Substituting (77) and (78) into (76), we have