PartialFed: Cross-Domain Personalized Federated Learning via Partial Initialization
Year: 2022

Abstract:
The burst of applications empowered by massive data have aroused unprecedented privacy concerns in AI society. Currently, data confidentiality protection has been one core issue during deep model training. Federated Learning (FL), which enables privacy-preserving training across multiple silos, gained rising popularity for its parameter-only communication. However, previous works have shown that FL revealed a significant performance drop if the data distributions are heterogeneous among different clients, especially when the clients have cross-domain characteristic, such as traffic, aerial and in-door. To address this challenging problem, we propose a novel idea, PartialFed, which loads a subset of the global model’s parameters rather than loading the entire model used in most previous works. We first validate our algorithm with manually decided loading strategies inspired by various expert priors, named PartialFed-Fix. Then we develop PartialFed-Adaptive, which automatically selects personalized loading strategy for each client. The superiority of our algorithm is proved by demonstrating the new state-of-the-art results on cross-domain federated classification and detection. In particular, solely by initializing a small fraction of layers locally, we improve the performance of FedAvg on Office-Home and UODB by 4.88% and 2.65%, respectively. Further studies show that the adaptive strategy performs significantly better on domains with large deviation, e.g. improves AP50 by 4.03% and 4.89% on aerial and medical image detection compared to FedAvg.

heading: 1 Introduction
text: Endless collection of texts, images and videos for training data-hungry models increases potential threats to the safety systems. Any attacks to these data centers could cause billions of information leakage. A safer way is to keep the user data purely local on their devices. But this contradicts with the widely adopted stochastic gradient descent (SGD) training procedure, which usually requires data communication for random batch sampling. Federated Learning [12] develops a paradigm for training large models under such situation. Locally trained models are aggregated using FedAvg [19] and then served as initialization model for the next local iteration. The training data confidentiality is achieved by only allowing transferring the model’s parameters (rather than the data) under different cryptograph algorithms, which include differential privacy [7], homomorphic encryption [37], block chain [22], etc.
However, recent studies [39, 26, 25] have shown that FedAvg does not provide satisfactory results in the presence of data heterogeneity. A major problem is that all models are designed to fit an “average client” [29], which is difficult when local and global distributions are deviated from each other. This phenomenon is not uniquely observed. Researchers in multi-domain learning have also discovered that directly fitting non-identical domains into a single feature extractor is suboptimal [3]. The leading solution to this problem is to reconfigure the network into domain-agnostic and
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
domain-specific layers [23, 33]. Similar conclusion has also been drawn simultaneously in the field of multi-task learning [27]. The performance is boosted by letting each task choosing task-agnostic and task-specific layers on the fly. Based on such generous observations, we propose a Personalized Federated Learning (PFL) method from the idea of client-agnostic and client-specific initialization. Initialization is considered because it plays the essential role of transferring global knowledge in FL.
Figure 1 dipicts the overall architecture. At the heart of our algorithm is a mixed initialization strategy. Instead of fully utilizing the averaged global parameters for initialization, clients will only select a fraction of them, and load the remaining parameters from previous local models. The selection process is decided by a customized loading strategy, which might vary from client to client and time to time. To validate PartialFed thoroughly, we first propose PartialFed-Fix, where the loading strategies are inspired by human priors in functionality and classification of different parts of the network. Then we propose an automatic strategy which is learned jointly with the network parameters by gradient descent. We name this algorithm PartialFed-Adaptive.
In the case of PartialFed-Fix, our algorithm can be viewed as a federated version of multi-path networks in the multi-domain learning [24, 33]. The shared parameters of different clients are jointly learned by FedAvg, while the client-specific parameters are only learned locally. Treating the network as a combination of global and local blocks makes it possible to learn the knowledge from other clients while keeping the local knowledge stored safely. Although PartialFed-Fix can achieve improved performance, we argue that the fixed loading strategy is a suboptimal solution for PFL since different clients may have different dependencies on the global model. For example, a client might prefer the classifier of the global model at the start of training, which often acquires better generalization ability. In the end, the client is likely to train a local classifier, which brings superior personalized performance. We show that our dynamic algorithm PartialFed-Adaptive is able to capture this change of parameter loading behaviors during the training process.
Despite its simplicity, PartialFed gains surprising performance on many non i.i.d. FL experiments. We construct real world FL experiments by introducing cross-domain classification and detection dataset [30, 33]. For example, on Office-Home dataset, our PartialFed-Fix and PartialFed-Adaptive surpass FedAvg by 4.88% and 5.43% on average accuracy, respectively. Similarly, on UODB dataset, PartialFed-Fix and PartialFed-Adaptive outperform FedAvg by 2.65% and 2.68% on AP50, respectively. More interestingly, the adaptive loading strategy greatly reduce the possibility of clients getting inferior performance caused by the distribution deviation.

heading: 2 Related Work
text: Personalized Federated Learning The pioneering work of FedAvg [19] aims to train a global model for all clients. But the goal is hard to achieve when data distributions are non i.i.d.. Wang et al. [31] directly finetune the global model on local dataset for personalized performance. Mansour et al. [18] provide theoretical study of clustering, data interpolation and model interpolation in personalized learning. FedProx [13] introduces a regularization based algorithm. FedCurv [26] and FedCL [35] further consider using Elastic Weight Consolidation (EWC) for parameters importance estimation. T. Dinh et al. [28] theoretically improve the vanilla regularization with Moreau Envelopes. [5, 8] consider using model interpolation to address the PFL problem. Meta learning is also utilized to learn a better initialization for personalized finetuning [6]. The most related works to this paper is parameter decoupling strategies which split networks into global and private layers. Arivazhagan et al. [2] consider global base network with personalized classifiers while FedBN [14] and SiloBN [1] consider private Batch Normalization (BN) layers. The special cases of PartialFed-Fix are equivalent to these methods, and we further examined their combinations and more strategies.
Multi-Domain Learning The problem of training on heterogeneous distributions has been widely studied in multi-domain learning. The difference is that multi-domain learning aggregate data together while FL store data in silos. Our method is related to the recent work [3, 23, 24], which aims to learn a single network across all domains with minimal number of task specific parameters. Bilen and Vedaldi [3] propose to use domain-specific batch and instance normalization while Rebuffi et al. [23, 24] further take series and parallel residual adapters as the domain-specific parts. Wang et al. [33] extend the idea to object detection with SE-ResNet [10] and introduce a multi-domain detection benchmark. Data-driven architectures are also explored in [20, 38, 32] where network pruning or neural architecture search are employed.

heading: 3 Partially-Loaded Federated Learning

heading: 3.1 Preliminary
text: Federated Learning We formally describe the problem setup in this section. Assume there are N ∈ N clients c ∈ C who are willing to participate in the joint training. Each client owns a private dataset Dc = {(xci , yci ) : i ∈ {1, ..., nc}}, where nc is the cardinality of the dataset. We use n = ∑ c∈C nc to denote the total amount of data in all clients. The data distribution P
c(x, y) varies from client to client. To be more specific, we do not put any i.i.d. assumptions on either P c(x) or P c(y|x), which indicates domain gap and label difference are allowed in the framework. At the start of each federated iteration t, a central server will send a global model Wt to all clients as parameter initialization. Clients will then use their private data Dc to train Wt and get an updated Wct . All W c t , c ∈ C are then sent to server and fused by FedAvg:
Wt+1 ← ∑ c∈C nc n Wct (1)
The resulting model is used as initialization for the next federated iteration. When networks are structurally heterogeneous in each client, Wt denotes the maximum set of shareable parameters. The algorithm will stop on its convergence. Note that we do not incorporate client selection as in McMahan et al. [19], which can be easily added if required.
PartialFed Compared to vanilla FL, PartialFed changes the client’s parameter initialization process. After receiving global model Wt, each client c will additionally load the local model Wct−1 from last iteration (See Alg. 1). Instead of purely initializing training with global model Wt, PartialFed uses a mixed initialization, where each parameter is either loaded from global model Wt or local model Wct−1. We hope that by partially loading global parameters, client can benefit from sharing knowledge while avoiding local knowledge forgetting caused by FedAvg. The choice of mixing strategy Act will be discussed in section 3.2 and 3.3. All the models are finetuned locally in the last training iteration.
A major advantage of PartialFed is the minimal changes to the original framework. Without sharing gradients, subset of data and customized attributes, clients can enjoy themselves from a highest parameter privacy protocol. In addition, since the server do not need extra operations like knowledge distillation [16] except for FedAvg, the uploaded models are not required to do any forward computation. This enables homomorphic encryption [37] and protects client from information leakage caused by model attacks [36].
Algorithm 1 PartialFed: Partially-Loaded Federated Learning ClientIteration(c,Wt):
Load Wct−1 from local storage Get partial loading strategy Act . partial initialization Get partially loaded initialization Wct,init by:
Wct,init[i] =
{ Wct−1[i] if A c t [i] = 0
Wt[i] if Act [i] = 1 , i denotes the index of parameter (2)
Set Wct = W c t,init for each local epoch from 1 to E do . regular training for batch b ∈ Dc do
w ← w − ηO`(w; b), for w ∈Wct end for
end for Save Wct on local storage Return Wct
Server: Initialize W0 = [w0,0, w1,0, ..., wk,0] at random for each round t = 1, 2, . . . do
for client c ∈ C in parallel do Wct ← ClientIteration(c,Wt) . device training end for Wt+1 ← ∑ c∈C nc n W c t . execute FedAvg
end for

heading: 3.2 PartialFed-Fix
text: We first discuss the fixed manual strategies in this section. Deep networks are often intuitively divided into different functional parts: feature extractor, classifier, etc. These intuitions can be transformed into corresponding loading strategies. An example strategy is shown in Equ.3, where parameters except fully connected layer (fc) are loaded from global model. We empirically validate the efficiency by transferring commonly used assumptions to our algorithm in this section.
Act [i] = { 1 if layer i is fc 0 else
(3)
"Bottom-up v.s. Top-down" One of the commonly accepted assumptions in deep network is that blocks close to input image are more related to low-level features while top blocks are associated with high-level concepts. Sharing bottom layers assumes that different clients share similar low-level texture features while the opposite assumes high-level visual concepts are composed analogously. We examine whether PFL training benefits more from sharing global "feature extractor" or global "classifier". To be more specific, we use ResNet18 [9] for these experiments, which has four stages (each with two ResBlock) and a fully-connected layer (Figure 2). A detailed description of the used dataset can be found in section 4.1.
The results are listed in Table 1. Firstly, all experiments with partial parameters loading outperform the full parameter loading, which proves the superiority of the proposed PartialFed. Secondly, there are no significant gap with additional parameters loading from global s1 to s1-s4, but loading global fc in this case severely degrades the performance. Thirdly, we find there’s consistent accuracy drop from strategy fc to s2-fc. We hypothesis that sharing too much high-level concepts will hurt the personalized federated learning.
Batch Normalization Batch statistics are considered as a key factor when heterogeneous distributions occur and has been widely studied in domain adaptation [15], multi-domain learning [3], and federated learning [1, 14]. We study how the loading strategies of BN will affect the PFL training. Table 2 displays the results on Office-Home dataset. Loading BN locally (i.e. w/o BN) gains improvements on all four domains. Given that the parameters of BN layers are only a very small fraction of ResNet, the improvement is non-trivial. On the other hand, since the BN layer is such brittle in non i.i.d. settings, we hope that a multi-distribution normalization strategy can be proposed in the future.
Recalling that previous section shows loading fc and s1-s4 simultaneously could harm the performance . We hope this rule can be combined with the BN rule. This leads to the last row of Table 2, where both global BN and global fc are not loaded. Surprisingly, this simple modification significantly improves FedAvg from 75.44% to 80.32%, which is 4.88% improvements.
Skip Loading Except for the "shared feature extractor - private classifier" network partition, another widely adopted routine in multi-domain and multi-task learning is a loop of "share-private-shareprivate-..." sub-networks [33, 27]. This paradigm is found to be helpful when tasks interference happens. This paragraph examines the performance of skip sharing with PartialFed.
For simplicity, we label each ResBlock in ResNet18 with letters as in Figure 2. The same number of blocks (4 blocks) is loaded globally for all experiments. The input convolution and fc layer are loaded as default. Experimental results are revealed in Table 3. Methods that load adjacent layers (e.g. AaBb) get similar mean accuracy except for the CcDd strategy. Since CcDd is a kind of top-down strategies mentioned above, which does not perform well in general, we infer this is the reason for its inferior performance. Interestingly, the two strategies with skip loading, ABCD and abcd, get higher performance than all the other strategies, which justify the idea of skip sharing. We attribute the efficiency of skip loading to the fact that clients have the demand of learning client-specific knowledge at all levels of the network instead of only a part of them.

heading: 3.3 PartialFed-Adaptive
text: The hand-designed strategies in PartialFed-Fix are intuitive but not efficient. The search space of all fixed strategies has a complexity of 2k (k denoted the number of layers). If we further consider
the factor of time t and client number N , it reaches a complexity of N × t × 2k. It’s hard to find an optimal strategy by hand in such large space. Therefore, we propose a learnable loading strategy, PartialFed-Adaptive. The main idea is to learn the initialization strategy through a data-driven approach that adaptively chooses which parameters to be loaded globally given the specific client. The objective of the algorithm is to minimize the following loss function on client c:
min Act min W ct
Ltrain(Dc,W c t |W ct,init = Act(Wt,W ct−1)) (4)
where Act(Wt,W c t−1)) is the initialization given by strategy A c t and t is the federated iteration. Note that here we only consider the personalized loss for client c, but the custom initialization strategy also affects the global training in an implicit way. We leave this problem for future work and focus on the greedy situation in this paper.
The original strategy Act makes discrete decisions, which is not differentiable. In order to optimize strategies and model parameters jointly with gradient descent, we adopt Gumbel-Softmax sampling [17, 11] for modeling strategy Act . More precisely, A c t is parametrized by a distribution vector αct ∈ [0, 1]2k with shape [k, 2]. Each row αct [i] specifies the probability of loading local and global parameter for the ith parameter wi and satisfies αct [i, 0] + α c t [i, 1] = 1. For differentiable training, reparameterization trick is used:
uct [i, j] = exp
( (logαct [i, j] +G(j))/τt )∑ k∈{0,1} exp ( (logαct [i, k] +G(k))/τt
) , j ∈ 0, 1 (5) where G = − log(− logU) is a standard Gumbel distribution with U sampled i.i.d. from Unif(0, 1) and τt is the temperature parameter of softmax at federated iteration t. We also use the hard sample trick described in PyTorch document 1 to obtain discrete sampling. The main idea is to substitute soft sample output ysoft with hard output yhard − fstop_gradient(ysoft) + ysoft.
Algorithm 2 PartialFed-Adaptive ClientIteration-Adaptive(c,Wt):
Load Wct−1 from local storage Initialize αct = α c t−1 for each local epoch from 1 to E do for batch b ∈ Dc do
Sample strategy Ab by Equ. 5 with parameter αct Composite batch parameter by:
Wcb[i] = { Wct−1[i] if Ab[i] = 0 Wt[i] if Ab[i] = 1 , i denotes the index of parameter (6)
Compute loss l with parameter Wcb if batch index % (fm + fs) < fm then
w ← w − ηO`(w,αct ; b), for w ∈Wcb . model parameter update else
αct ← αct − ηO`(w,αct ; b) . strategy parameter update end if
end for end for Sample best parameter Wct according to α c t Return Wct
Training Strategy The overall learning scheme is summarized in Alg. 2. The model parameters and strategy parameters are iteratively updated by EM algorithm. The updating frequency is denoted as fm and fs. At every training step, a discrete batch strategy Ab is sampled by Gumbel-Softmax, which guides the composition of global and local parameters. The composited parameter Wcb is used to compute loss and either itself or the strategy parameter αct is updated according to training step index. The temperature parameter τt in Equ. 5 is initialized as 5.0 and annealed to 0 as in [27]. The sampling strategy approaches the original discrete distribution with τ getting close to zero limit [17].
1https://pytorch.org/docs/master/generated/torch.nn.functional.gumbel_softmax.html

heading: 4 Experiments
text: We evaluate the performance of PartialFed, on real-world non i.i.d. tasks including cross-domain classification and detection.

heading: 4.1 Settings
text: Datasets For federated classification, we use the popular Office-Home [30] dataset, which contains four domains: Art, Clipart, Product and Real World. All domains share the same 65 typical categories in office and home. Each domain has an average of 3k images. Each client use a single domain as its private data, which in total gives 4 clients.
For the detection task, we adopt challenging multi-domain detection dataset UODB [33], which is composed of 11 detection tasks with different domains, categories and data sizes (Table 5). Similar to Office-Home dataset, we take each domain as a client. Except for domain gap, there’s also a big difference in the number of training images, from minimum 0.5K to maximum 35K. Note that FedAvg weights each client model by their sample numbers, the severe data imbalance makes it extremely difficult for federated learning.
Implementation Details We use PyTorch [21] 2 to implement our algorithms. The detection experiments also use Detectron2 [34]3. For Office-Home Dataset, we adopt ResNet-18 as backbone network. SE-ResNet-50 [10] is used as the detection backbone, which is the same setting as the original paper [33]. Both models are pretrained on ILSVRC2012 ImageNet [4]. For sampling the best parameters in Alg. 2, we try both hard and soft sampling. Hard sampling takes the parameter that maximize the logits while the soft sampling uses a weighted sum of global and local parameters with the sample logits. Soft sampling is also followed with finetuning. We find soft sampling is better in most cases, so we report the performance of the soft sampling strategy if not specified. For details, we list the hyper parameters in the Appendix.

heading: 4.2 Cross-Domain Classification
text: The results on Office-Home dataset are displayed in Table 4. We report the Top-1 accuracy for each domain and on average. On average, the All model trained with data from all domains surpasses single domain training, but it does not work well on Product domain. This reflects that naively sharing all parameters is not suitable for all domains even in the non federated training.
Adaptive Granularity We experiment adaptive learning at three levels from coarse to fine: stage, block and layer level for ResNet. For example, for the stage-level adaptive, the choice space has complexity of 25, which decides whether each group in [wstage1, wstage2, wstage3, wstage4, fc] is selected globally or locally. The adaptive strategies outperforms fixed ones on average. Among them, the layer level PartialFed-Adaptive works best and outperforms stage-level and block-level by 0.41%. Theoretically, learning at smaller granularity improves the performance upper bound, but also increases the burden for training algorithms. In practice, clients should choose granularity according to its computational and data resources.
Comparison With State-Of-The-Art We compare our methods with a set of SOTA algorithms. All of them are reimplemented by referring to the open-sourced code published by the author. Hyper parameters are selected as suggested in the original paper, and the best results are reported. The algorithms are also implemented in a personalized version (i.e. finetune the model at end if needed) for fair comparison. While most existing methods improves the vanilla FedAvg, the best FedBN is still inferior to the single domain training due to domain conflicts. This makes FL unacceptable because the extra communication does not introduce positive impacts. On the other hand, our PartialFed, outperforms both single domain and full data training, which makes it practical in more scenarios.
Although being simple, PartialFed-Fix without global BN and fc is already fairly strong on this dataset. It surpasses vanilla FedAvg by 4.88% and FedBN by 1.53% on average accuracy. But it is still affected by domain conflicts and gets only 90.43% on domain Product, which is lower than single domain training. PartialFed-Adaptive overcomes this problem by automatic learning the suitable strategies for each client. It is the only method that beats single domain training on Product domain.
2licensed under https://github.com/pytorch/pytorch/blob/master/LICENSE 3licensed under the Apache License 2.0
This is a thriving result since it improves all clients performance instead of a subset of them. On average, it surpasses vanilla FedAvg by 5.43% and FedBN by 2.09%, respectively.

heading: 4.3 Cross-Domain Detection
text: The results of federated detection is summarized in Table 6. The vanilla FedAvg improves performance of clients with relative small datasets: Watercolor and Clipart. But it performs poorly on datasets with special properties, e.g. WiderFace for face detection, DeepLesion for medical image detection. FedBN works well and is able to relief the decline in face and aerial image detection, but it works even worse than FedAvg on DeepLesion, which is distributionally dissimilar to natural images. FedProx does not perform well in this dataset. Restricting contrasting client models to fit the averaged model with regularization term seems harmful given domain gaps at such degree.
For the fixed loading strategy, we first examine SE module and its combination with batch normalization, which is related to the attention mechanism [10]. We also implemented a skip loading strategy similar to the one in section 3.2. We tested two strategies: loading local first block and loading both local first and third block in each SE-ResNet stage. The first one works better on smaller datasets because it provides more knowledge by sharing more parameters. For the adaptive strategy, The block-level adaptive strategy gets best performance in this case. We hypothesis that searching a SE-ResNet-50 strategy at layer-level is too costing and hard to learn, which results in the lower performance of Adaptive-layer.
In general, PartialFed-Fix and PartialFed-Adaptive beats FedAvg by 2.65% and 2.68% and beats FedBN by 0.95% and 0.98%. And again, our PartialFed-Adaptive gets superior performance on distributionally difficult domains, including DeepLesion, WiderFace and DOTA. This proves the effectiveness of our algorithm on extreme domain shift.

heading: 4.4 Strategy Analysis
text: We visualize how the strategy changes with FL training, as displaced in Figure 3. Interestingly, there’s a clear tendency for all domains to share low-level knowledge than high-level knowledge (i.e. the probability is larger compared to top stages). This coincides with our conclusion in the bottom-up fixed strategies. In the stage1 and stage2, Clipart domain differs from other domain, possibly due to its unique image style. For the high-level fc layer, all domains tends to use private parameters at very first of the training except for Art domain. Loading local fc layer coincides with our findings in the fixed strategies. On the other hand, we find out that client model with Art data learns slower than other domains at the beginning, possibly due to its deviation from the pretrained ImageNet model. Borrowing the knowledge from other domains seems to be a wiser choice in this situation. It’s fascinating to see that the adaptive methods is capable of capturing these kind of changes.
Our experiments have shown that the adaptive learning strategy has advantages in learning clientspecific strategies, which can relief client from negative impacts caused by distribution shift. But it also suffers from the risk of incomplete training, either underfitting or overfitting. For example, we find out that searching a 50 layers ResNet at layer-level is already hard. How to improve the strategy learning efficiency is related to neural architecture search, which considers balance between exploration and exploitation. The data-driven approach also ignores meta-information like datasets cardinality. A possible way to address this is to incorporate human prior into the adaptive algorithm, which is a combination of PartialFed-Fix and PartialFed-Adaptive.

heading: 5 Conclusion
text: We demonstrate a simple yet novel approach for personalized federated learning. The core of our method is a mixed initialization, which only partially utilizing the global parameters given by FedAvg. Inspired by commonly accepted priors, we develop a set of manual mix strategies and validate their reliability. To make it further, we develop a data-aware strategy which is adaptively learned with model parameters. Our experiments show that the proposed two strategies, PartialFed-Fix and PartialFed-Adaptive, outperform a set of state-of-the-art methods on cross-domain FL experiments including object classification and detection. More importantly, PartialFed-Adaptive is found to automatically reduce performance degradation caused by extreme distribution heterogeneity.
Acknowledgments and Disclosure of Funding We thank Rui Li and Xin Huang for discussions. In the future, we wish to use PartialFed on MindSpore4, which is a new deep learning computing framework.

Review summaries:
The paper proposes a new method called ‘PartialFed’ to solve the performance degrading caused by data heterogeneity in the federated learning scenario. The main difference between this work and previous work is that the proposed method partially loads global parameters given by FedAvg, while the previous methods load all of the global parameters for each starting point of the local epoch.   For the sampling strategy for the parameters that load the global parameter, two methods are developed in this paper: “PartialFed-Fixed” and “PartialFed-Adaptive”. For the “PartialFed-Fixed”, the parameters are sampled manually based on the commonly accepted assumptions in deep network e.g. low-level knowledge learned from bottom layers. With the "ParitalFed-Adaptive", the sampling strategy is trained data-driven. Gumbel Softmax sampling is used to enable back-propagation through the sampling process.   To validate the proposed methods, cross-domain federated classification and detection are examined. Two partial loading strategies outperform a set of state-of-art methods. Specifically, the “PartialFed-Adaptive” further reduces performance losses due to the extreme distribution heterogeneity and automatically shows the tendency that reflects the assumptions of deep network used for the design of “PartialFed-Fixed”. 

This paper proposed a novel idea PartialFed, which loads a subset of the global model’s parameters rather than loading the entire model. Specifically, they validate the algorithm with manually decided loading strategies inspired by various expert priors, named PartialFed-Fix. Then they develop PartialFed-Adaptive, which automatically selects a personalized loading strategy for each client.  

This paper explores a simple but novel modification to FL approaches to better deal with settings involving non-homogeneous clients. They find that by allowing clients to mix local and global parameters for initializing local training, the added flexibility can notably improve the performance of the respective local end-models. They perform both an exploration of natural fixed mixing strategies and also propose an adaptive mixing strategy that allows for further improvements on average.  

This paper proposed an adaptive initialization algorithm for FedAvg, motivated by FedPer in a personalized federated learning scenario. The key contribution lies in the observation of the impact on training accuracy of different initialization strategies and the adaptive initialization algorithm. The empirical result of the algorithm shown significant improvement compared with FedAvg in training with heterogeneous data.