SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ f \left(w ^ {t + 1}\right) - f \left(w ^ {t}\right) \right] \\ \leq \mathbb {E} \langle \nabla f (w ^ {t}), w ^ {t + 1} - w ^ {t} \rangle + \frac {L}{2} \underbrace {\mathbb {E} \| w ^ {t + 1} - w ^ {t} \| ^ {2}} _ {V _ {2} ^ {t}} \\ = \mathbb {E} \langle \nabla f (w ^ {t}), \frac {1}{N} \sum_ {i \in \mathcal {N}} w _ {i, K} ^ {t} - w ^ {t} \rangle + \frac {L V _ {2} ^ {t}}{2} \\ = \mathbb {E} \langle \nabla f (w ^ {t}), \frac {1}{C} \sum_ {i \in \mathcal {C}} \left[ \left(w _ {i, K} ^ {t} - w _ {i, 0} ^ {t}\right) + \beta \left(w ^ {t} - w _ {i, K} ^ {t - 1}\right) \right] \rangle + \frac {L V _ {2} ^ {t}}{2} \\ = - \eta \mathbb {E} \left\langle \nabla f \left(w ^ {t}\right), \frac {1}{C} \sum_ {i \in C} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} \left(w _ {i, k} ^ {t}\right) - \frac {1}{C} \sum_ {i \in C} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} \left(w ^ {t}\right) + K \nabla f \left(w ^ {t}\right) \right\rangle + \frac {L V _ {2} ^ {t}}{2} \\ = - \eta K \mathbb {E} \| f (w ^ {t}) \| ^ {2} + \mathbb {E} \langle \sqrt {\eta K} \nabla f (w ^ {t}), \sqrt {\frac {\eta}{K}} \frac {1}{C} \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \left(\nabla f _ {i} (w ^ {t}) - \nabla f _ {i} (w _ {i, k} ^ {t})\right) \rangle + \frac {L V _ {2} ^ {t}}{2} \\ \leq - \eta K \mathbb {E} \| f (w ^ {t}) \| ^ {2} + \frac {\eta K}{2} \mathbb {E} \| f (w ^ {t}) \| ^ {2} + \frac {\eta}{2 C} \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \| \nabla f _ {i} (w ^ {t}) - \nabla f _ {i} (w _ {i, k} ^ {t}) \| ^ {2} \\ - \frac {\eta}{2 C ^ {2} K} \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} \left(w _ {i, k} ^ {t}\right) \| ^ {2} + \frac {L V _ {2} ^ {t}}{2} \\ \leq -\frac{\eta K}{2}\mathbb{E}\| f(w^{t})\|^{2} + \frac{\eta L^{2}}{2}\underbrace{\frac{1}{C}\sum_{i\in\mathcal{C}}\sum_{k = 0}^{K - 1}\mathbb{E}\|w^{t} - w_{i,k}^{t}\|^{2}}_{V_{1}^{t}} - \frac{\eta}{2C^{2}K}\mathbb{E}\| \sum_{i\in \mathcal{C}}\sum_{k = 0}^{K - 1}\nabla f_{i}(w_{i,k}^{t})\|^{2} + \frac{LV_{2}^{t}}{2} \\ \leq - \frac {\eta K}{2} \mathbb {E} \| f (w ^ {t}) \| ^ {2} + \frac {\eta L ^ {2} V _ {1} ^ {t}}{2} - \frac {\eta}{2 C ^ {2} K} \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} (w _ {i, k} ^ {t}) \| ^ {2} + \frac {L V _ {2} ^ {t}}{2}. \\ \end{array}
$$
According to Lemma 2 and lemma 3 to bound the  $V_{1}^{t}$  and  $V_{2}^{t}$ , we can get the following recursive formula:
$$
\begin{array}{l} \mathbb {E} \left[ f \left(w ^ {t + 1}\right) - f \left(w ^ {t}\right) \right] \\ \leq - \frac {\eta K}{2} \mathbb {E} \| f (w ^ {t}) \| ^ {2} + \left(\frac {\eta^ {2} L}{2 C N} - \frac {\eta}{2 C ^ {2} K}\right) \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} \left(w _ {i, k} ^ {t}\right) \| ^ {2} \\ + \frac {\eta L ^ {2}}{2} \left[ 4 K \beta^ {2} \Delta^ {t} + 3 K ^ {2} \eta^ {2} \left(\sigma_ {l} ^ {2} + 4 K G ^ {2}\right) + 1 2 K ^ {3} \eta^ {2} B ^ {2} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} \right] \\ + \frac {3 \beta^ {2} L}{2 N} \left(1 + 4 \eta^ {2} K ^ {2} L ^ {2}\right) \Delta^ {t} + \frac {\eta^ {2} K L}{2 N} \left(1 + 9 K ^ {2} \eta^ {2} L ^ {2}\right) \sigma_ {l} ^ {2} + \frac {3 \eta^ {2} K ^ {2} L}{2 N} \left(1 + 1 2 \eta^ {2} K ^ {2} L ^ {2}\right) G ^ {2} \\ + \frac {3 \eta^ {2} K ^ {2} B ^ {2} L}{2 N} \left(1 + 1 2 \eta^ {2} K ^ {2} L ^ {2}\right) \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} \\ \leq \left(\frac {\eta^ {2} L}{2 C N} - \frac {\eta}{2 C ^ {2} K}\right) \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} \left(w _ {i, k} ^ {t}\right) \| ^ {2} + \frac {3 \beta^ {2} L}{2 N} \left[ \frac {4 N}{3} \eta K L + \left(1 + 4 \eta^ {2} K ^ {2} L ^ {2}\right) \right] \Delta^ {t} \\ + \frac {\eta^ {2} K L}{2 N} \left[ 3 N \eta K L + \left(1 + 9 \eta^ {2} K ^ {2} L ^ {2}\right) \right] \sigma_ {l} ^ {2} + \frac {3 \eta^ {2} K ^ {2} L}{2 N} \left[ 4 N \eta K L + \left(1 + 1 2 \eta^ {2} K ^ {2} L ^ {2}\right) \right] G ^ {2} \\ - \frac {\eta K}{2} \left[ 1 - \frac {3 \eta K L B ^ {2}}{N} \left(1 + 1 2 \eta^ {2} K ^ {2} L ^ {2}\right) - 1 2 \eta^ {2} K ^ {2} L ^ {2} B ^ {2} \right] \mathbb {E} \| f (w ^ {t}) \| ^ {2}. \\ \end{array}
$$
Here we make a comprehensive discussion on the selection of  $\eta$  to simplify the above formula. In fact, in lemma 2, there is a constraint on the learning rate as  $\eta \leq \frac{\sqrt{2}}{4(K - 1)L}$  for  $K \geq 2$ . In lemma 3 and lemma 4, there is a constraint on the learning rate as  $\eta \leq \frac{1}{KL}$ . To further minimize the coefficient, we select the  $N\eta KL$  to be constant order. For convenience, we directly select the  $\eta \leq \frac{1}{NKL}$ . Thus, we have:
$$
\begin{array}{l} \mathbb {E} \left[ f \left(w ^ {t + 1}\right) - f \left(w ^ {t}\right) \right] \\ \leq \frac {3 \beta^ {2} L}{2 N} \left(\frac {4}{3} N \eta K L + 5\right) \Delta^ {t} + \frac {\eta^ {2} K L}{2 N} (3 N \eta K L + 1 0) \sigma_ {l} ^ {2} + \frac {3 \eta^ {2} K ^ {2} L}{2 N} (4 N \eta K L + 1 3) G ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} - \frac {\eta K}{2} \left(1 - \frac {3 9 \eta K L B ^ {2}}{N} - 1 2 \eta^ {2} K ^ {2} L ^ {2} B ^ {2}\right) \mathbb {E} \| f (w ^ {t}) \| ^ {2} \\ + \left(\frac {\eta^ {2} L}{2 C N} - \frac {\eta}{2 C ^ {2} K}\right) \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} \left(w _ {i, k} ^ {t}\right) \| ^ {2} \\ <   \frac {1 0 \beta^ {2} L (\Delta^ {t} - \Delta^ {t + 1})}{(1 - 7 2 \beta^ {2}) N} + \frac {3 \eta^ {2} K ^ {2} L}{2 N} \left(\frac {1 3 0 0 \beta^ {2}}{1 - 7 2 \beta^ {2}} + 1 7\right) G ^ {2} + \frac {\eta^ {2} K L}{2 N} \left(\frac {1 0 2 0 \beta^ {2}}{1 - 7 2 \beta^ {2}} + 1 3\right) \sigma_ {l} ^ {2} \\ + \left[ \frac {3 0 \beta^ {2} \eta^ {2} L}{C N ^ {2} (1 - 7 2 \beta^ {2})} + \frac {\eta^ {2} L}{2 C N} - \frac {\eta}{2 C ^ {2} K} \right] \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} (w _ {i, k} ^ {t}) \| ^ {2} \\ - \frac {\eta K}{2} \left[ 1 - \frac {3 9 \eta K L B ^ {2}}{N} - \frac {3 9 0 0 \beta^ {2} \eta K L B ^ {2}}{(1 - 7 2 \beta^ {2}) N} - 1 2 \eta^ {2} K ^ {2} L ^ {2} B ^ {2} \right] \mathbb {E} \| f (w ^ {t}) \| ^ {2}. \\ \end{array}
$$
Firstly, to remove the gradient term, we follow the [19, 48] and let  $\frac{30\beta^2\eta^2L}{CN^2(1 - 72\beta^2)} + \frac{\eta^2L}{2CN} - \frac{\eta}{2C^2K} \leq 0$  then learning rate  $\eta \leq \frac{N}{2CKL}$ . Then, according to the [48], there is a positive constant  $\lambda \in (0,1)$  to satisfy  $1 - \frac{39\eta KL B^2}{N} - \frac{3900\beta^2\eta KL B^2}{(1 - 72\beta^2)N} - 12\eta^2 K^2 L^2 B^2 > \lambda > 0$ . We denote  $\kappa_1 = \frac{1300\beta^2}{1 - 72\beta^2} + 17$  and  $\kappa_2 = \frac{1020\beta^2}{1 - 72\beta^2} + 13$  as two constants in the formula. Therefore, we have:
$$
\begin{array}{l} \frac {\lambda \eta K}{2} \mathbb {E} \| f (w ^ {t}) \| ^ {2} \\ \leq \mathbb {E} [ f (w ^ {t}) - f (w ^ {t + 1}) ] + \frac {1 0 \beta^ {2} L}{(1 - 7 2 \beta^ {2}) N} (\Delta^ {t} - \Delta^ {t + 1}) + \frac {3 \kappa_ {1} \eta^ {2} K ^ {2} L}{2 N} G ^ {2} + \frac {\kappa_ {2} \eta^ {2} K L}{2 N} \sigma_ {l} ^ {2}. \\ \end{array}
$$
A.2.3 Proof of Theorem 1
Theorem 7 Under Assumption  $6\sim 8$  , let participation ratio is  $N / C$  where  $1 < N < C$  , let the learning rate satisfy  $\eta \leq \min \left\{\frac{N}{2CKL},\frac{1}{NKL}\right\}$  where  $K\geq 2$  , let the relaxation coefficient  $\beta \leq \frac{\sqrt{2}}{12}$  and after training  $T$  rounds, the global model  $\omega^t$  generated by FedInit satisfies:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| f \left(w ^ {t}\right) \| ^ {2} \leq \frac {2 \left(f \left(w ^ {0}\right) - f \left(w ^ {\star}\right)\right)}{\lambda \eta K} + \frac {\kappa_ {2} \eta L}{\lambda N} \sigma_ {l} ^ {2} + \frac {3 \kappa_ {1} \eta K L}{\lambda N} G ^ {2}. \tag {27}
$$
where  $\lambda \in (0,1)$ ,  $\kappa_{1} = \frac{1300\beta^{2}}{1 - 72\beta^{2}} + 17$ , and  $\kappa_{2} = \frac{1020\beta^{2}}{1 - 72\beta^{2}} + 13$  are three constants.
Further, by selecting the proper learning rate  $\eta = \mathcal{O}\left(\sqrt{\frac{N}{KT}}\right)$  and let  $D = f(w^0) - f(w^\star)$  as the initialization bias, the global model  $w^t$  satisfies:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| f \left(w ^ {t}\right) \| ^ {2} = \mathcal {O} \left(\frac {D + L \left(\sigma_ {l} ^ {2} + 3 K G ^ {2}\right)}{\sqrt {N K T}}\right). \tag {28}
$$
Proof According to the expansion of the smoothness inequality, we have:
Taking the accumulation from 0 to  $T - 1$ , we have:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| f (w ^ {t}) \| ^ {2} \\ \leq \frac {2 \mathbb {E} [ f (w ^ {0}) - f (w ^ {T}) ]}{\lambda \eta K T} + \frac {2 0 \beta^ {2} L}{(1 - 7 2 \beta^ {2}) \lambda \eta K N T} (\Delta^ {0} - \Delta^ {T}) + \frac {\kappa_ {2} \eta L}{\lambda N} \sigma_ {l} ^ {2} + \frac {3 \kappa_ {1} \eta K L}{\lambda N} G ^ {2} \\ \leq \frac {2 \left(f \left(w ^ {0}\right) - f \left(w ^ {\star}\right)\right)}{\lambda \eta K T} + \frac {\kappa_ {2} \eta L}{\lambda N} \sigma_ {l} ^ {2} + \frac {3 \kappa_ {1} \eta K L}{\lambda N} G ^ {2}. \\ \end{array}
$$
We select the learning rate  $\eta = \mathcal{O}\left(\sqrt{\frac{N}{KT}}\right)$  and let  $D = f(w^{0}) - f(w^{\star})$  as the initialization bias, then we have:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| f (w ^ {t}) \| ^ {2} = \mathcal {O} \left(\frac {D + L (\sigma_ {l} ^ {2} + 3 K G ^ {2})}{\sqrt {N K T}}\right).
$$
A.2.4 Proof of Theorem 2
Theorem 8 Under Assumption  $6\sim 8$  and 10, let participation ratio is  $N / C$  where  $1 < N < C$ , let the learning rate satisfy  $\eta \leq \min \left\{\frac{N}{2CKL},\frac{1}{NKL},\frac{1}{\lambda\mu K}\right\}$  where  $K\geq 2$ , let the relaxation coefficient  $\beta \leq \frac{\sqrt{2}}{12}$ , and after training  $T$  rounds, the global model  $\boldsymbol{\omega}^{t}$  generated by FedInit satisfies:
$$
\mathbb {E} \left[ f \left(w ^ {T}\right) - f \left(w ^ {\star}\right) \right] \leq e ^ {- \lambda \mu \eta K T} \mathbb {E} \left[ f \left(w ^ {0}\right) - f \left(w ^ {\star}\right) \right] + \frac {3 \kappa_ {1} \eta K L}{2 N \lambda \mu} G ^ {2} + \frac {\kappa_ {2} \eta L}{2 N \lambda \mu} \sigma_ {l} ^ {2}. \tag {29}
$$
Further, by selecting the proper learning rate  $\eta = \mathcal{O}\left(\frac{\log(\lambda\mu NKT)}{\lambda\mu KT}\right)$  and let  $D = f(w^0) - f(w^\star)$  as the initialization bias, the global model  $w^t$  satisfies:
$$
\mathbb {E} \left[ f \left(w ^ {T}\right) - f \left(w ^ {\star}\right) \right] = \mathcal {O} \left(\frac {D + L \left(\sigma_ {l} ^ {2} + K G ^ {2}\right)}{N K T}\right). \tag {30}
$$
According to Assumption 10, we have  $2\mu (f(w) - f(w^{\star}))\leq \| \nabla f(w)\|^{2}$ , we have:
$$
\begin{array}{l} \lambda \mu \eta K \mathbb {E} \left[ f \left(w ^ {t}\right) - f \left(w ^ {\star}\right) \right] \leq \frac {\lambda \eta K}{2} \mathbb {E} \| f \left(w ^ {t}\right) \| ^ {2} \\ \leq \mathbb {E} [ f (w ^ {t}) - f (w ^ {t + 1}) ] + \frac {1 0 \beta^ {2} L}{(1 - 7 2 \beta^ {2}) N} (\Delta^ {t} - \Delta^ {t + 1}) + \frac {3 \kappa_ {1} \eta^ {2} K ^ {2} L}{2 N} G ^ {2} + \frac {\kappa_ {2} \eta^ {2} K L}{2 N} \sigma_ {l} ^ {2}. \\ \end{array}
$$
Combining the terms aligned with  $w^t$  and  $w^{t + 1}$ , we have:
$$
\begin{array}{l} \mathbb {E} \left[ f \left(w ^ {t + 1}\right) - f \left(w ^ {\star}\right) \right] \\ \leq (1 - \lambda \mu \eta K) \mathbb {E} [ f (w ^ {t}) - f (w ^ {\star}) ] + \frac {1 0 \beta^ {2} L}{(1 - 7 2 \beta^ {2}) N} (\Delta^ {t} - \Delta^ {t + 1}) + \frac {3 \kappa_ {1} \eta^ {2} K ^ {2} L}{2 N} G ^ {2} + \frac {\kappa_ {2} \eta^ {2} K L}{2 N} \sigma_ {l} ^ {2}. \\ \end{array}
$$
Taking the recursion from  $t = 0$  to  $T - 1$  and let learning rate  $\eta \leq \frac{1}{\lambda \mu K}$ , we have:
$$
\begin{array}{l} \mathbb {E} \left[ f \left(w ^ {T}\right) - f \left(w ^ {\star}\right) \right] \\ \leq \left(1 - \lambda \mu \eta K\right) ^ {T} \mathbb {E} \left[ f \left(w ^ {0}\right) - f \left(w ^ {\star}\right) \right] + \sum_ {t = 0} ^ {T - 1} \left(1 - \lambda \mu \eta K\right) ^ {T - 1 - t} \frac {1 0 \beta^ {2} L}{(1 - 7 2 \beta^ {2}) N} \left(\Delta^ {t} - \Delta^ {t + 1}\right) \\ + \left(\frac {3 \kappa_ {1} \eta^ {2} K ^ {2} L}{2 N} G ^ {2} + \frac {\kappa_ {2} \eta^ {2} K L}{2 N} \sigma_ {l} ^ {2}\right) \sum_ {t = 0} ^ {T - 1} (1 - \lambda \mu \eta K) ^ {T - 1 - t} \\ \leq \left(1 - \lambda \mu \eta K\right) ^ {T} \mathbb {E} \left[ f \left(w ^ {0}\right) - f \left(w ^ {\star}\right) \right] + \frac {1 0 \beta^ {2} L}{\left(1 - 7 2 \beta^ {2}\right) N} \left(\Delta^ {0} - \Delta^ {T}\right) \\ + \left(\frac {3 \kappa_ {1} \eta^ {2} K ^ {2} L}{2 N} G ^ {2} + \frac {\kappa_ {2} \eta^ {2} K L}{2 N} \sigma_ {l} ^ {2}\right) \frac {1 - (1 - \lambda \mu \eta K) ^ {T}}{\lambda \mu \eta K} \\ \leq \left(1 - \lambda \mu \eta K\right) ^ {T} \mathbb {E} \left[ f \left(w ^ {0}\right) - f \left(w ^ {\star}\right) \right] + \frac {3 \kappa_ {1} \eta K L}{2 N \lambda \mu} G ^ {2} + \frac {\kappa_ {2} \eta L}{2 N \lambda \mu} \sigma_ {l} ^ {2} \\ \leq e ^ {- \lambda \mu \eta K T} \mathbb {E} \left[ f \left(w ^ {0}\right) - f \left(w ^ {\star}\right) \right] + \frac {3 \kappa_ {1} \eta K L}{2 N \lambda \mu} G ^ {2} + \frac {\kappa_ {2} \eta L}{2 N \lambda \mu} \sigma_ {l} ^ {2}. \\ \end{array}
$$
We select the learning rate  $\eta = \mathcal{O}\left(\frac{\log(\lambda\mu NKT)}{\lambda\mu KT}\right)$  and let  $D = f(w^{0}) - f(w^{*})$  as the initialization bias, then we have:
$$
\mathbb {E} \left[ f \left(w ^ {T}\right) - f \left(w ^ {\star}\right) \right] = \mathcal {O} \left(\frac {D + L \left(\sigma_ {l} ^ {2} + K G ^ {2}\right)}{N K T}\right).
$$
A.2.5 Proof of Theorem 4
Theorem 9 Under Assumption  $6\sim 8$  we can bound the divergence term as follows. Let the learning rate satisfy  $\eta \leq \min \left\{\frac{N}{2CKL},\frac{1}{NKL},\frac{\sqrt{N}}{\sqrt{CKL}}\right\}$  where  $K\geq 2$  , and after training  $T$  rounds, let  $0 <   \beta <  \frac{\sqrt{6}}{24}$  , the divergence
term  $\Delta^t$  generated by FedInit satisfies:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Delta^ {t} = \mathcal {O} \left(\frac {N \left(\sigma_ {l} ^ {2} + K G ^ {2}\right)}{T} + \frac {\sqrt {N K} B ^ {2} \left[ D + L \left(\sigma_ {l} ^ {2} + K G ^ {2}\right) \right]}{T ^ {\frac {3}{2}}}\right). \tag {31}
$$
Proof According to Lemma 4, we have:
$$
\begin{array}{l} \Delta^ {t + 1} \leq 7 2 \beta^ {2} \Delta^ {t} + 5 1 \eta^ {2} K \sigma_ {l} ^ {2} + 1 9 5 \eta^ {2} K ^ {2} G ^ {2} + 1 9 5 \eta^ {2} K ^ {2} B ^ {2} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} \\ + \frac {3 \eta^ {2}}{C N} \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} \left(w _ {i, k} ^ {t}\right) \| ^ {2}. \\ \end{array}
$$
Here we further bound the gradient term, we have:
$$
\begin{array}{l} \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} (w _ {i, k} ^ {t}) \| ^ {2} = \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \left(\nabla f _ {i} (w _ {i, k} ^ {t}) - \nabla f _ {i} (w ^ {t}) + \nabla f _ {i} (w ^ {t})\right) \| ^ {2} \\ = \mathbb {E} \| \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \left(\nabla f _ {i} \left(w _ {i, k} ^ {t}\right) - \nabla f _ {i} \left(w ^ {t}\right) + \nabla f \left(w ^ {t}\right)\right) \| ^ {2} \\ \leq C K \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \| \nabla f _ {i} \left(w _ {i, k} ^ {t}\right) - \nabla f _ {i} \left(w ^ {t}\right) + \nabla f \left(w ^ {t}\right) \| ^ {2} \\ \leq 2 C K \sum_ {i \in \mathcal {C}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \| \nabla f _ {i} \left(w _ {i, k} ^ {t}\right) - \nabla f _ {i} \left(w ^ {t}\right) \| ^ {2} + 2 C ^ {2} K ^ {2} \mathbb {E} \| \nabla f \left(w ^ {t}\right) \| ^ {2} \\ \leq 2 C ^ {2} K L ^ {2} V _ {1} ^ {t} + 2 C ^ {2} K ^ {2} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2}. \\ \end{array}
$$
Combining this into the recursive formulation, and let the learning rate satisfy  $\eta \leq \frac{\sqrt{N}}{\sqrt{CKL}}$ , we have:
$$
\begin{array}{l} \Delta^ {t + 1} \leq \beta^ {2} \left(7 2 + \frac {2 4 C \eta^ {2} K ^ {2} L ^ {2}}{N}\right) \Delta^ {t} + \eta^ {2} K ^ {2} B ^ {2} \left(1 9 5 + \frac {7 2 C \eta^ {2} K ^ {2} L ^ {2}}{N}\right) \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} \\ + \eta^ {2} K \left(5 1 + \frac {1 8 C \eta^ {2} K ^ {2} L ^ {2}}{N}\right) \sigma_ {l} ^ {2} + \eta^ {2} K ^ {2} \left(1 9 5 + \frac {7 2 C \eta^ {2} K ^ {2} L ^ {2}}{N}\right) G ^ {2} \\ \leq 9 6 \beta^ {2} \Delta^ {t} + 2 6 7 \eta^ {2} K ^ {2} B ^ {2} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} + 6 9 \eta^ {2} K \sigma_ {l} ^ {2} + 2 6 7 \eta^ {2} K ^ {2} G ^ {2}. \\ \end{array}
$$
Let  $96\beta^2 < 1$  as the decayed coefficient where  $\beta < \frac{\sqrt{6}}{24}$ , similar as Lemma 4, we have:
$$
\Delta^ {t} \leq \frac {\Delta^ {t} - \Delta^ {t + 1}}{1 - 9 6 \beta^ {2}} + \frac {2 6 7 \eta^ {2} K ^ {2} B ^ {2}}{1 - 9 6 \beta^ {2}} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} + \frac {6 9 \eta^ {2} K}{1 - 9 6 \beta^ {2}} \sigma_ {l} ^ {2} + \frac {2 6 7 \eta^ {2} K ^ {2}}{1 - 9 6 \beta^ {2}} G ^ {2}.
$$
by taking the accumulation from  $t = 0$  to  $T - 1$ ,
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Delta^ {t} \leq \frac {\Delta^ {0} - \Delta^ {T}}{1 - 9 6 \beta^ {2}} + \frac {6 9 \eta^ {2} K}{1 - 9 6 \beta^ {2}} \sigma_ {l} ^ {2} + \frac {2 6 7 \eta^ {2} K ^ {2}}{1 - 9 6 \beta^ {2}} G ^ {2} + \frac {2 6 7 \eta^ {2} K ^ {2} B ^ {2}}{1 - 9 6 \beta^ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (w ^ {t}) \| ^ {2} \\ \leq \frac {2 6 7 \eta^ {2} K ^ {2} B ^ {2}}{1 - 9 6 \beta^ {2}} \left(\frac {2 (f (w ^ {0}) - f (w ^ {\star}))}{\lambda \eta K T} + \frac {\kappa_ {2} \eta L}{\lambda N} \sigma_ {l} ^ {2} + \frac {3 \kappa_ {1} \eta K L}{\lambda N} G ^ {2}\right) \\ + \frac {6 9 \eta^ {2} K}{1 - 9 6 \beta^ {2}} \sigma_ {l} ^ {2} + \frac {2 6 7 \eta^ {2} K ^ {2}}{1 - 9 6 \beta^ {2}} G ^ {2} \\ \leq \frac {5 3 4 \eta K B ^ {2} (f (w ^ {0}) - f (w ^ {\star}))}{(1 - 9 6 \beta^ {2}) \lambda T} + \frac {2 6 7 \eta^ {3} K ^ {2} B ^ {2} \kappa_ {2} L}{(1 - 9 6 \beta^ {2}) \lambda N} \sigma_ {l} ^ {2} + \frac {8 0 1 \eta^ {3} K ^ {3} B ^ {2} \kappa_ {1} L}{(1 - 9 6 \beta^ {2}) \lambda N} G ^ {2} \\ + \frac {6 9 \eta^ {2} K}{1 - 9 6 \beta^ {2}} \sigma_ {l} ^ {2} + \frac {2 6 7 \eta^ {2} K ^ {2}}{1 - 9 6 \beta^ {2}} G ^ {2}. \\ \end{array}
$$
The same, the learning rate is selected as  $\eta = \mathcal{O}(\sqrt{\frac{N}{KT}})$  and let  $D = f(w^0) - f(w^\star)$  as the initialization bias and let  $96\beta^2 < 1$ , thus we have:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Delta^ {t} = \mathcal {O} \left(\frac {N \left(\sigma_ {l} ^ {2} + K G ^ {2}\right)}{T} + \frac {\sqrt {N K} B ^ {2} \left[ D + L \left(\sigma_ {l} ^ {2} + K G ^ {2}\right) \right]}{T ^ {\frac {3}{2}}}\right).
$$
This completes this proof.
A.2.6 Proof of Theorem 5
Theorem 10 Under Assumption  $6\sim 8$  and 10, we can bound the divergence term as follows. Let the learning rate satisfy  $\eta \leq \min \left\{\frac{N}{2CKL},\frac{1}{NKL},\frac{1}{\lambda\mu K}\right\}$  where  $K\geq 2$  , and after training  $T$  rounds, let  $0 <   \beta <  \frac{\sqrt{6}}{24}$  , the divergence term  $\Delta^T$  generated by FedInit satisfies:
$$
\Delta^ {T} = \mathcal {O} \left(\frac {D + G ^ {2}}{T ^ {2}} + \frac {N \sigma_ {l} ^ {2} + K G ^ {2}}{N K T ^ {2}}\right) + \mathcal {O} \left(\frac {1}{N K T ^ {3}}\right). \tag {32}
$$
Proof According to the Theorem 8, we have:
$$
\Delta^ {t + 1} \leq 9 6 \beta^ {2} \Delta^ {t} + 2 6 7 \eta^ {2} K ^ {2} B ^ {2} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} + 6 9 \eta^ {2} K \sigma_ {l} ^ {2} + 2 6 7 \eta^ {2} K ^ {2} G ^ {2}.
$$
Taking the recursive formulation from  $t = 0$  to  $T - 1$ , we have:
$$
\begin{array}{l} \Delta^ {T} \leq (9 6 \beta^ {2}) ^ {T} \Delta^ {0} + \sum_ {t = 0} ^ {T - 1} (9 6 \beta^ {2}) ^ {t} \left(2 6 7 \eta^ {2} K ^ {2} B ^ {2} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} + 6 9 \eta^ {2} K \sigma_ {l} ^ {2} + 2 6 7 \eta^ {2} K ^ {2} G ^ {2}\right) \\ \leq \frac {6 9 \eta^ {2} K \sigma_ {l} ^ {2}}{1 - 9 6 \beta^ {2}} + \frac {2 6 7 \eta^ {2} K ^ {2} G ^ {2}}{1 - 9 6 \beta^ {2}} + 2 6 7 \eta^ {2} K ^ {2} B ^ {2} \sum_ {t = 0} ^ {T - 1} (9 6 \beta^ {2}) ^ {T - 1 - t} \mathbb {E} \| \nabla f (w ^ {t}) \| ^ {2} \\ \leq \frac {6 9 \eta^ {2} K \sigma_ {l} ^ {2}}{1 - 9 6 \beta^ {2}} + \frac {2 6 7 \eta^ {2} K ^ {2} G ^ {2}}{1 - 9 6 \beta^ {2}} + \frac {2 6 7 \eta^ {2} K ^ {2} B ^ {2}}{1 - 9 6 \beta^ {2}} \left(\frac {\kappa_ {2} \eta L}{\lambda N} \sigma_ {l} ^ {2} + \frac {3 \kappa_ {1} \eta K L}{\lambda N} G ^ {2}\right) \\ + \frac {5 3 4 \eta K B ^ {2}}{\lambda} \sum_ {t = 0} ^ {T - 1} (9 6 \beta^ {2}) ^ {T - 1 - t} \mathbb {E} [ f (w ^ {t}) - f (w ^ {t + 1}) ] + \frac {1 0 \beta^ {2} L}{(1 - 7 2 \beta^ {2}) N} (\Delta^ {0} - \Delta^ {T}) \\ \leq \frac {6 9 \eta^ {2} K \sigma_ {l} ^ {2}}{1 - 9 6 \beta^ {2}} + \frac {2 6 7 \eta^ {2} K ^ {2} G ^ {2}}{1 - 9 6 \beta^ {2}} + \frac {2 6 7 B ^ {2} L}{(1 - 9 6 \beta^ {2}) \lambda} \left(\frac {\kappa_ {2} \eta^ {3} K ^ {2}}{N} \sigma_ {l} ^ {2} + \frac {3 \kappa_ {1} \eta^ {3} K ^ {3}}{N} G ^ {2}\right) \\ + \frac {5 3 4 \eta K B ^ {2}}{\lambda} \sum_ {t = 0} ^ {T - 1} (9 6 \beta^ {2}) ^ {T - 1 - t} \mathbb {E} [ f (w ^ {t}) - f (w ^ {\star}) ]. \\ \end{array}
$$
According to the Theorem 11, we have:
$$
\mathbb {E} [ f (w ^ {t}) - f (w ^ {\star}) ] \leq e ^ {- \lambda \mu \eta K t} \mathbb {E} [ f (w ^ {0}) - f (w ^ {\star}) ] + \frac {3 \kappa_ {1} \eta K L}{2 N \lambda \mu} G ^ {2} + \frac {\kappa_ {2} \eta L}{2 N \lambda \mu} \sigma_ {l} ^ {2}.
$$
Let  $96\beta^{2}\leq e^{-\lambda \mu \eta K}$  thus we have:
$$
\begin{array}{l} \frac {5 3 4 \eta K B ^ {2}}{\lambda} \sum_ {t = 0} ^ {T - 1} (9 6 \beta^ {2}) ^ {T - 1 - t} \mathbb {E} [ f (w ^ {t}) - f (w ^ {\star}) ] \\ \leq \frac {2 6 7 B ^ {2} L}{(1 - 9 6 \beta^ {2}) \lambda^ {2} \mu} \left(\frac {\kappa_ {2} \eta^ {2} K}{N} \sigma_ {l} ^ {2} + \frac {3 \kappa_ {1} \eta^ {2} K ^ {2}}{N} G ^ {2}\right) + \frac {5 3 4 \eta K B ^ {2}}{\lambda} \mathbb {E} [ f (w ^ {0}) - f (w ^ {\star}) ] \sum_ {t = 0} ^ {T - 1} (9 6 \beta^ {2}) ^ {T - 1 - t} e ^ {- \lambda \mu \eta K t} \\ \leq \frac {2 6 7 B ^ {2} L}{(1 - 9 6 \beta^ {2}) \lambda^ {2} \mu} \left(\frac {\kappa_ {2} \eta^ {2} K}{N} \sigma_ {l} ^ {2} + \frac {3 \kappa_ {1} \eta^ {2} K ^ {2}}{N} G ^ {2}\right) \\ + \frac {5 3 4 \eta K B ^ {2}}{\lambda} \mathbb {E} \left[ f \left(w ^ {0}\right) - f \left(w ^ {\star}\right) \right] e ^ {- \lambda \mu \eta K T} \sum_ {t = 0} ^ {T - 1} e ^ {- 2 \lambda \mu \eta K t}. \\ \end{array}
$$
Thus selecting the same learning rate  $\eta = \mathcal{O}\left(\frac{\log(\lambda\mu NKT)}{\lambda\mu KT}\right)$  and let  $D = f(w^0) - f(w^\star)$  as the initialization bias, we have:
$$
\Delta^ {T} = \mathcal {O} \left(\frac {D + G ^ {2}}{T ^ {2}} + \frac {N \sigma_ {l} ^ {2} + K G ^ {2}}{N K T ^ {2}} + \frac {1}{N K T ^ {3}}\right).
$$
A.3 Proofs for the Generalization Error
In this part, we prove the generalization error for our proposed method. We assume the objective function  $f$  is  $L$ -smooth and  $L_{G}$ -Lipschitz as defined in [11, 51]. We follow the uniform stability to upper bound the generalization error in the FL.
We suppose there are  $C$  clients participating in the training process as a set  $\mathcal{C} = \{i\}_{i=1}^{C}$ . Each client has a local dataset  $S_{i} = \{z_{j}\}_{j=1}^{S}$  with total  $S$  data sampled from a specific unknown distribution  $\mathcal{D}_{i}$ . Now we define a re-sampled dataset  $\widetilde{S}_{i}$  which only differs from the dataset  $S_{i}$  on the  $j^{\star}$ -th data. We replace the  $S_{i^{\star}}$  with  $\widetilde{S}_{i^{\star}}$  and keep other  $C-1$  local datasets, which compose a new set  $\widetilde{C}$ . From the perspective of total data,  $\mathcal{C}$  only differs from the  $\widetilde{C}$  at  $j^{\star}$ -th data on the  $i^{\star}$ -th client. Then, based on these two sets, our method could generate two output models,  $w^{t}$  and  $\widetilde{w}^{t}$  respectively, after  $t$  training rounds. We first introduce some notations used in the proof of the generalization error.