SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the generalization error, Theorem 3 indicates that  $\mathcal{E}_G$  term contains two main parts. The first part comes from the stochastic gradients as the vanilla centralized training process [11], which is of the order  $\mathcal{O}((TK)^{\frac{cL}{1 + cL}} / S)$ . The constant  $c$  is of the order  $\mathcal{O}(1 / K)$  as  $c = \frac{\mu_0}{K}$ , thus we have  $\frac{cL}{1 + cL} = \frac{\mu_0L}{K + \mu_0L}$ . If we assume the  $\mu_0L$  is generally small, we always expect to adopt a larger  $K$  in the FL paradigm to reduce generalization error. For instance, if we select  $K\to \infty$ , then  $\mathcal{O}((TK)^{\frac{cL}{1 + cL}} / S)\rightarrow \mathcal{O}(T^{\frac{cL}{1 + cL}} / S)$  which is a very strong upper bound of the generalization error. However, the selection of local interval  $K$  must be restricted from the optimization conditions and we will discuss the details in Section 4.3.4. In addition, the second part in Theorem 3 comes from the divergence term, which is a unique factor in the FL paradigm. As we mentioned above, the divergence term measures the authentic client-drift in the training process. The divergence term is not affected by the number of samples  $S$  and it is only related to the proposed method and the local heterogeneity of the dataset. Proof details are stated in Appendix A.3.
4.3.3 Divergence Term
In the former two parts, we provide the complete theorem to measure optimization error  $\mathcal{E}_O$  and generalization error  $\mathcal{E}_G$ . And we notice that, in the FL paradigm, the divergence term mainly affects the generalization ability of the model instead of the optimization convergence. In this part, we focus on the analysis of the divergence term of our proposed FedInit method. Due to the relaxed initialization at the beginning of each communication round, according to the Algorithm 1, we have  $w_{i,K}^{t} = w^{t} + \beta (w^{t} - w_{i,K}^{t - 1}) - \eta \sum_{k = 0}^{K - 1}g_{i,k}^{t}$ . Thus, we have the following recursive relationship:
$$
\underbrace {w ^ {t + 1} - w _ {i , K} ^ {t}} _ {\text {l o c a l d i v e r g e n c e a t t + 1}} = \beta \underbrace {(w _ {i , K} ^ {t - 1} - w ^ {t})} _ {\text {l o c a l d i v e r g e n c e a t t}} + \underbrace {(w ^ {t + 1} - w ^ {t})} _ {\text {g l o b a l u p d a t e}} + \underbrace {\sum_ {k = 0} ^ {K - 1} \eta g _ {i , k} ^ {t}} _ {\text {l o c a l u p d a t e s}}. \tag {15}
$$
According to the formulation (15), we can bound the divergence  $\Delta^t$  via the following two theorems.
Theorem 4 Under Assumptions  $1 \sim 3$ , we can bound the divergence term as follows. Let the learning rate satisfy  $\eta \leq \min \left\{\frac{N}{2CKL}, \frac{1}{NKL}, \frac{\sqrt{N}}{\sqrt{CKL}}\right\}$  where  $K \geq 2$ , and after training  $T$  rounds, let  $0 < \beta < \frac{\sqrt{6}}{24}$ , the divergence term  $\{\Delta^t\}$  generated by FedInit satisfies:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \Delta^ {t} = \mathcal {O} \left(\frac {N \left(\sigma_ {l} ^ {2} + K G ^ {2}\right)}{T} + \frac {\sqrt {N K} B ^ {2} \left[ D + L \left(\sigma_ {l} ^ {2} + K G ^ {2}\right) \right]}{T ^ {\frac {3}{2}}}\right). \tag {16}
$$
Theorem 4 points out the convergence order of the divergence  $\Delta^t$  generated by FedInit during the training process. This bound matches the conclusion in Theorem 1 with the same learning rate. The
dominant term achieves the  $\mathcal{O}(NK / T)$  rate on the heterogeneity bias  $G$ . It could be seen that the number of selected clients  $N$  will inhibit its convergence and the local consistency linearly increases with  $N$ . Different from the selection in Theorem 1, local interval  $K$  is expected as small enough to maintain the high consistency. Also, the initialization bias  $D$  is no longer dominant in consistency. We omit the constant weight  $\frac{1}{1 - 96\beta^2}$  in this upper bound. Proof details are stated in Appendix A.2.5.
Theorem 5 Under Assumptions  $1 \sim 3$  and 5, we can bound the divergence term as follows. Let the learning rate satisfy  $\eta \leq \min \left\{\frac{N}{2CKL}, \frac{1}{NKL}, \frac{1}{\lambda\mu K}\right\}$  where  $K \geq 2$ , and after training  $T$  rounds, let  $0 < \beta < \frac{\sqrt{6}}{24}$ , the divergence term  $\Delta^T$  generated by FedInit satisfies:
$$
\Delta^ {T} = \tilde {\mathcal {O}} \left(\frac {D + G ^ {2}}{T ^ {2}} + \frac {N \sigma_ {l} ^ {2} + K G ^ {2}}{N K T ^ {2}} + \frac {1}{N K T ^ {3}}\right). \tag {17}
$$
Theorem 5 indicates the convergence of the divergence  $\Delta$  under the  $PL$ -condition which matches the conclusion in Theorem 2 with the same learning rate selection. Assumption 5 establishes a relationship between the gradient norm and the loss difference on the non-convex function  $f$ . Different from the Theorem 4, the initialization bias  $D$  and the heterogeneous bias  $G$  are the dominant terms. Under Assumption 5, the FedInit supports a larger local interval  $K$  in the training process. This conclusion also matches the selection of  $K$  in Theorem 2. When the model converges, FedInit guarantees the local models towards the global optimum under at least  $\mathcal{O}(1 / T^2)$  rate. Similarly, we omit the constant weight  $\frac{1}{1 - 96\beta^2}$  and we will discuss the  $\beta$  in Section 4.3.4. Proof details are stated in Appendix A.2.6.
4.3.4 Excess Risk
In this part, we analyze the excess risk  $\mathcal{E}_E$  of FedInit method. According to the theorems above,
Theorem 6 Under Assumption  $1 \sim 5$ , let the participation ratio is  $N / C$  where  $1 < N < C$ , let the learning rate satisfies  $\eta \leq \min \left\{\frac{N}{2CKL}, \frac{1}{NKL}, \frac{1}{\lambda\mu K}, \right\}$  where  $K \geq 2$ , let the relaxed coefficient  $0 \leq \beta < \frac{\sqrt{6}}{24}$ , and let  $|\mathcal{S}_i| = S$ . By selecting the learning rate  $\eta = \mathcal{O}\left(\frac{\log(\lambda\mu NKT)}{\lambda\mu KT}\right) \leq \frac{c}{t}$ , after training  $T$  communication rounds, the excess risk of the FedInit method achieves:
$$
\mathcal {E} _ {E} \leq \underbrace {\widetilde {\mathcal {O}} \left(\frac {D + L \left(\sigma_ {l} ^ {2} + K G ^ {2}\right)}{N K T}\right)} _ {\text {o p t i m i z a t i o n b i a s}} + \underbrace {\mathcal {O} \left(\frac {1}{S} \left[ \sigma_ {l} (T K) ^ {c L} \right] ^ {\frac {1}{1 + c L}}\right)} _ {\text {s t a b i l i t y b i a s}} + \underbrace {\widetilde {\mathcal {O}} \left(\frac {\sqrt {D + G ^ {2}} K ^ {\frac {c L}{1 + c L}}}{T ^ {\frac {1}{1 + c L}}}\right)} _ {\text {d i v e r g e n c e b i a s}}. \tag {18}
$$
According to the Theorems 2, 3, and 5, we combine their dominant terms to upper bound the excess risk of FedInit method. The first term comes from the optimization error, the second term comes from the stability bias, and the third term comes from the divergence bias. From the perspective of excess risk, the main restriction in the FL paradigm is the divergence term with the bound of  $\widetilde{\mathcal{O}}\left(\frac{1}{T^{1 + cL}}\right)$ . The second term of excess risk matches the conclusion in SGD [11, 51] which relies on the number  $S$ . Our analysis of the excess risk reveals two important corollaries in FL:
- From the perspective of optimization, the FL paradigm is insensitive to local consistency in the training process (Theorems 1&2).
- From the perspective of generalization, the local consistency level significantly affects the performance in the FL paradigm (Theorem 6).
Then we discuss the best selection of the local interval  $K$  and relaxed coefficient  $\beta$ .
Selection of K. In the first term, to minimize the optimization error, the local interval  $K$  is required to be large enough. In the second term, since  $\frac{cL}{1 + cL} \leq 1$ , the upper bound expects a small local interval  $K$ . In the third term, since  $\frac{1}{1 + cL} = \frac{K}{K + \mu_0 L} < 1$ , it expects a large  $K$  to guarantee the order of  $T$  to approach  $\mathcal{O}(1 / T)$ , where the divergence bias could maintain a high-level consistency. Therefore, there is a specific optimal constant selection for  $K > 1$  to minimize the excess risk.
Selection of  $\beta$ . As the dominant term, the coefficient of the divergence bias also plays a key role in the error bound. In Theorem 5, the constant weight we omit for the divergence term  $\Delta^T$  is  $\frac{1}{1 - 96\beta^2}$ . Thus the coefficient of  $\sqrt{\Delta^T}$  is  $\frac{1}{\sqrt{1 - 96\beta^2}}$ . Combined with Theorem 3, we have the coefficient for the
Table 1: Test accuracy (%) on the CIFAR-10/100 dataset. We test two participation ratios on each dataset. Under each setup, we test two Dirichlet splittings, and each result test for 3 times. This table reports results on ResNet-18-GN (upper part) and VGG-11 (lower part) respectively.
Method	CIFAR-10	CIFAR-100
10%-100 clients	5%-200 clients	10%-100 clients	5%-200 clients
Dir-0.6	Dir-0.1	Dir-0.6	Dir-0.1	Dir-0.6	Dir-0.1	Dir-0.6	Dir-0.1
FedAvg	78.77±.11	72.53±.17	74.81±.18	70.65±.21	46.35±.15	42.62±.22	44.70±.22	40.41±.33
FedAdam	76.52±.14	70.44±.22	73.28±.18	68.87±.26	48.35±.17	40.77±.31	44.33±.26	38.04±.25
FedSAM	79.23±.22	72.89±.23	75.45±.19	71.23±.26	47.51±.26	43.43±.12	45.98±.27	40.22±.27
SCAFFOLD	81.37±.17	75.06±.16	78.17±.28	74.24±.22	51.98±.23	44.41±.15	50.70±.29	41.83±.29
FedDyn	82.43±.16	75.08±.19	79.96±.13	74.15±.34	50.82±.19	42.50±.28	47.32±.21	41.74±.21
FedCM	81.67±.17	73.93±.26	79.49±.17	73.12±.18	51.56±.20	43.03±.26	50.93±.19	42.33±.19
FedInit	83.11±.29	75.95±.19	80.58±.20	74.92±.17	52.21±.09	44.22±.21	51.16±.18	43.77±.36
FedAvg	85.28±.12	78.02±.22	81.23±.14	74.89±.25	53.46±.25	50.53±.20	47.55±.13	45.05±.33
FedAdam	86.44±.13	77.55±.28	81.05±.23	74.04±.17	55.56±.29	53.41±.18	51.33±.25	47.26±.21
FedSAM	86.37±.22	79.10±.07	81.76±.26	75.22±.13	54.85±.31	51.88±.27	48.65±.21	46.58±.28
SCAFFOLD	87.73±.17	81.98±.19	84.81±.15	79.04±.16	59.45±.17	56.67±.24	53.73±.32	50.08±.19
FedDyn	87.35±.19	82.70±.24	84.84±.19	80.01±.22	56.13±.18	53.97±.11	51.74±.18	48.16±.17
FedCM	86.80±.33	79.85±.29	83.23±.31	76.42±.36	53.88±.22	50.73±.35	47.83±.19	46.33±.25
FedInit	88.47±.22	83.51±.13	85.36±.19	79.73±.14	58.84±.11	57.22±.21	54.12±.08	50.27±.29
divergence term in formulation (18) is  $\frac{(1 + \beta)\frac{1}{\beta cL}}{\sqrt{1 - 96\beta^2}}$ . Therefore, to minimize this term, there is a specific optimal constant selection for  $0 < \beta < \frac{\sqrt{6}}{24}$ . We validate their selections in Section 5.2.
In this part, we introduce our empirical studies. Due to the page limitations, the details of the dataset, hyperparameters selection, implementation, and some extra ablation studies are stated in Appendix B.
**Benchmarks.** Our selected benchmarks in this paper are stated as follows. FedAvg [26] proposes the general FL paradigm. FedAdam [30] studies the efficiency of adaptive optimizer in FL. SCAFFOLD [19], FedDyn [1], and FedCM [46] learn the "client-drift" problem and adopt the variance reduction technique, ADMM, and client-level momentum respectively in FL to alleviate its negative impact. FedSAM [29] uses the local SAM objective instead of the vanilla empirical risk objective to search for a smooth loss landscape, which focuses on the generalization performance.
Setup. Here we briefly introduce the setups in our experiments. We test our proposed FedInit on the CIFAR-10 / 100 dataset [20]. To generate local heterogeneity, we follow Hsu et al. [14] to split the local clients through the Dirichlet sampling via a coefficient  $D_r$  to control the heterogeneous level and follow Sun et al. [38] to adopt the sampling with replacement to enhance the heterogeneity level. We test on the ResNet-18-GN [12, 13] and VGG-11 [35] to validate its efficiency. Actually, when the heterogeneity is strong, the performance of personalized initialization will be better. To better demonstrate the performance of our proposed method, we add additional noises to the dataset. Specifically, we first introduce the client-based biases. Among clients, we assume that the data samples are obtained differently. Because the local dataset is private and its construction is unknown, i.e., they are collected from different machines or cameras. Therefore, we change the strength of the RGB channels with a random Gaussian noise for different clients. The second noise is the category-based biases. We assume that samples for each category also contain heterogeneity. In our experiments, we add different brightness perturbations to the samples in each category by a random Gaussian noise. Based on these two noises, the heterogeneity of the local dataset is significantly enlarged. In this more realistic dataset, we can clearly observe the performance of each algorithm.
For each benchmark in our experiments, we adopt two coefficients  $D_r = 0.1$  and 0.6 for each dataset to generate different heterogeneity. We generally select the local learning rate  $\eta = 0.1$  and global learning rate  $\eta = 1$  on all setups except for FedAdam we use 0.1. The learning rate decay is set as multiplying 0.998 per round except for FedDyn we use 0.999. We train 500 rounds on CIFAR-10 and 800 rounds on CIFAR-100 to achieve stable test accuracy. The participation ratios are selected as 10% and 5% respectively of total 100 and 200 clients. More details are stated in Appendix B.1.
5.1 Experiment results
In this part, we mainly introduce the experiment results compared with the other benchmarks.
In Table 1, our proposed FedInit method performs well than the other benchmarks with good stability across different experimental setups. On the results of ResNet-18-GN on CIFAR-10, it achieves about  $3.42\%$  improvement than the vanilla FedAvg on the high heterogeneous splitting with  $D_r = 0.1$ . When the participation ratio decreases to  $5\%$ , the accuracy drops only about  $0.1\%$  while FedAvg drops almost  $1.88\%$ . Similar results on CIFAR-100, when the ratio de
Table 2: We incorporate the relaxed initialization (RI) into the benchmarks to test improvements on ResNet-18-GN on CIFAR-10 with the same hyperparameters and specific relaxed coefficient  $\beta$ .
Method	10%-100 clients	5%-200 clients
Dir-0.6	Dir-0.1	Dir-0.6	Dir-0.1
-	+RI	-	+RI	-	+RI	-	+RI
FedAvg	78.77	83.11	72.53	75.95	74.81	80.58	70.65	74.92
FedAdam	76.52	78.33	70.44	72.55	73.28	78.33	68.87	71.34
FedSAM	79.23	83.36	72.89	76.34	75.45	80.66	71.23	75.08
SCAFFOLD	81.37	83.27	75.06	77.30	78.17	81.02	74.24	76.22
FedDyn	82.43	81.91	75.08	75.11	79.96	79.88	74.15	74.34
FedCM	81.67	81.77	73.93	73.71	79.49	79.72	73.12	72.98
creases, FedInit still achieves  $43.77\%$  while the second best method SCAFFOLD drops about  $3.21\%$ . This indicates the proposed FedInit holds good stability on the varies of the participation. In addition, in Table 2, we incorporate the relaxed initialization (RI) into the other benchmarks to test its benefit. " -" means the vanilla benchmarks, and "+" means adopting the relaxed initialization. It shows that the relaxed initialization holds the promising potential to further enhance the performance. Actually, FedInit could be considered as  $(\mathrm{RI} + \mathrm{FedAvg})$ , whose improvement achieves about over  $3\%$  on each setup. Table 1 shows the poor performance of the vanilla FedAvg. Nevertheless, when adopting the RI, FedInit remains above most benchmarks on several setups. When the RI is incorporated into other benchmarks, it helps them to achieve higher performance without additional communication costs.
5.2 Ablation
In this part, we mainly introduce the ablation results of different hyperparameters.
Hyperparameters Sensitivity. The excess risk and test error of FedInit indicate there exists best selections for local interval  $K$  and relaxed coefficient  $\beta$ , respectively. In this part, we test a series of selections to validate our conclusions. To be aligned with previous studies, we denote  $K$  as training epochs. In Figure 1 (a), we can see that the selection range of the beta is very small while it has great potential to improve performance. When it is larger than the threshold, the training process will diverge quickly. As local interval  $K$  increases, test accuracy rises first
(a) Different  $K$
(b) Different  $\beta$
Figure 1: THyperparameters sensitivity studies of local intervals  $K$  and relaxed coefficient  $\beta$  of the FedInit method on CIFAR-10. To fairly compare their efficiency, we fix the total communication rounds  $T = 500$ .
and then decreases. Our analysis provides a clear explanation of the phenomenon. The optimization error decreases as  $K$  increases when it is small. When  $K$  exceeds the threshold, the divergence term in generalization cannot be ignored. Therefore, the test accuracy will be significantly affected.
Consistency. In this part, we test the relationship between the test accuracy and divergence term  $\Delta^T$  under different  $\beta$  selections. As introduced in Algorithm 1 Line.6, negative  $\beta$  means to adopt the relaxed initialization which is close to the latest local model. FedInit degrades to FedAvg when  $\beta = 0$ . Table 3 validates that RI is required to
be far away from the local model (a positive  $\beta$ ). When  $\beta$  is small, the correction is limited. The local divergence term is difficult to be diminished efficiently. While it becomes too large, the local training begins from a bad initialization, which can not receive enough guidance of global information from the global models. Furthermore, as shown in Table 3, if the initialization is too far from the local model, the quality of the initialization state will not be effectively guaranteed.
Table 3: We test different selections of the relaxed coefficient  $\beta$  of the FedInit method on CIFAR-10  $10\% - 100$  Dir-0.1 splitting to validate the relationship between test error and consistency after 500 rounds. We fix other hyperparameters as the same selection above for a fair comparison.
β	-0.2	-0.1	0	0.01	0.02	0.05	0.1	0.15
Accuracy (%)	64.70	67.47	72.53	72.82	73.45	74.65	75.95	44.47
ΔT	0.873	0.815	0.855	0.875	0.850	0.823	0.760	∞
5.3 Discussions of Relaxed Initialization
In this part, we mainly discuss the improvements of the proposed relaxed initialization.
In vanilla classical FedAvg and the most advanced methods, at the beginning of each communication round, we are always caught in a misunderstanding of the high consistency. Because the target of FL is a globally consistent solution, it is always an involuntary aggregation in the algorithm to ensure consistency. We prove that this does contribute to the efficiency of the optimization process, but it is not the best selection for generalization. To better improve the generalization, we prove that a relaxed initialization state will contribute more. We compare their difference in Figure 5.3.
FedAvg
Figure 2: Schematics of the classical FedAvg and our proposed FedInit.
FedInit
- global model w(t)
global model  $w(t + 1)$
$\star$  local optimum 1
$\star$  local optimum 2
$\star$  global optimum
$\bullet$  local relaxed Init. 1
$\bullet$  local relaxed Init. 2
As shown in the above figure, we can clearly see why FedInit contributes more to the consistency. When we move a little in the opposite direction of the last local optimization state, we will move further away from local optimal solutions in the current communication round. The working mode of RI is similar to the idea of "lookahead". Differently, (1) "lookahead" only works at the end of each stage; (2) "lookahead" only works for the global models on the global server. However, RI helps each local client to backtrack a small distance at the beginning of each stage. Therefore, after the local training in the next stage, the trained local models will get closer to each other than before.
In this work, we propose an efficient and novel FL method, dubbed FedInit, which adopts the stage-wise personalized relaxed initialization to enhance the local consistency level. Furthermore, to clearly understand the essential impact of consistency in FL, we introduce the excess risk analysis in FL and study the divergence term. Our proofs indicate that consistency dominates the test error and generalization error bound while optimization error is insensitive to it. Extensive experiments are conducted to validate the efficiency of relaxed initialization. As a practical and light plug-in, it could also be easily incorporated into other FL paradigms to improve their performance.
Limitations & Broader Impact. In this work, we analyze the excess risk for the FedInit method to understand how consistency works in FL. Actually, the relaxed initialization may also work for the personalized FL (pFL) paradigm. It is a future study to explore its properties in the pFL and decentralized FL, which may inspire us to design novel efficient algorithms in the FL community.
Prof. Dacheng Tao is partially supported by Australian Research Council Project FL-170100117.
[41] Han Wang, Siddartha Marella, and James Anderson. Fedadmm: A federated primal-dual algorithm allowing partial participation. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 287-294. IEEE, 2022.
[42] Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication-efficient distributed sgd with slow momentum. arXiv preprint arXiv:1910.00643, 2019.
[43] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7623, 2020.
[44] Jianyu Wang, Zheng Xu, Zachary Garrett, Zachary Charles, Luyang Liu, and Gauri Joshi. Local adaptivity in federated learning: Convergence and consistency. arXiv preprint arXiv:2106.02305, 2021.
[45] Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems, 33:6281-6292, 2020.
[46] Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. Fedcm: Federated learning with client-level momentum. arXiv preprint arXiv:2106.10874, 2021.
[47] Semih Yagli, Alex Dytso, and H Vincent Poor. Information-theoretic bounds on the generalization error and privacy leakage in federated learning. In 2020 IEEE 21st International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), pages 1-5. IEEE, 2020.
[48] Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. arXiv preprint arXiv:2101.11203, 2021.
[49] Xinwei Zhang, Mingyi Hong, Sairaj Dhople, Wotao Yin, and Yang Liu. Fedpd: A federated learning framework with adaptivity to non-iid data. IEEE Transactions on Signal Processing, 69:6055-6070, 2021.
[50] Yikai Zhang, Wenjia Zhang, Sammy Bald, Vamsi Pingali, Chao Chen, and Mayank Goswami. Stability of sgd: Tightness analysis and improved bounds. In Uncertainty in Artificial Intelligence, pages 2364-2373. PMLR, 2022.
[51] Pan Zhou, Hanshu Yan, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Towards understanding why lookahead generalizes better than sgd and beyond. Advances in Neural Information Processing Systems, 34:27290-27304, 2021.
[52] Shenglong Zhou and Geoffrey Ye Li. Federated learning via inexact admm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
In this section, we introduce our proofs of the main theorems in the main context. In the first part, we introduce some assumptions used in our proofs and point out their functions used for which part. In the second part, we prove the convergence rate and optimization error under the general assumptions. In the third part, we prove the uniform stability to measure the generalization error and analyze how each term affects the accuracy.
We suppose there are  $C$  clients participating in the training process and each has a local heterogeneous dataset. In each round  $t$ , we randomly select  $N$  clients to send the global model and they will train  $K$  iterations to get  $N$  local models. The local models will be aggregated on the global server as the next global model. After  $T$  rounds, our method generates a global model as the final state. We denote the total client set as  $C$  and the selected client set as  $\mathcal{N}$ .
In this part, we state assumptions in our proofs and discuss them. We will introduce each assumption and develop their corollaries.