PartialFed: Cross-Domain Personalized Federated Learning via Partial Initialization
Authors: Benyuan Sun
Hongxing Huo
Yi Yang
Bo Bai
Venue: NeurIPS 2021 Poster (2021)
Keywords: Federated Learning
Non i.i.d.
Personalization

Abstract: The burst of applications empowered by massive data have aroused unprecedented privacy concerns in AI society. Currently, data confidentiality protection has been one core issue during deep model training. Federated Learning (FL), which enables privacy-preserving training across multiple silos, gained rising popularity for its parameter-only communication. However, previous works have shown that FL revealed a significant performance drop if the data distributions are heterogeneous among different clients, especially when the clients have cross-domain characteristic, such as traffic, aerial and in-door. To address this challenging problem, we propose a novel idea, PartialFed, which loads a subset of the global model’s parameters rather than loading the entire model used in most previous works. We first validate our algorithm with manually decided loading strategies inspired by various expert priors, named PartialFed-Fix. Then we develop PartialFed-Adaptive, which automatically selects personalized loading strategy for each client. The superiority of our algorithm is proved by demonstrating the new state-of-the-art results on cross-domain federated classification and detection. In particular, solely by initializing a small fraction of layers locally, we improve the performance of FedAvg on Office-Home and UODB by 4.88% and 2.65%, respectively. Further studies show that the adaptive strategy performs significantly better on domains with large deviation, e.g. improves AP50 by 4.03% and 4.89% on aerial and medical image detection compared to FedAvg. 

deviated from each PertialFed: Cross-Domain Personalized Federated Learning via Partial Initialization
Benyuan Sun
Hongxing Huo
Yi Yang
Bo Bai
Media Technology Lab, Huawei
Ssunbenyuan,huohongg yangyi16 baibo3)
Abstract
The burst of applications empowered massive data have aroused unprecedented privacy concerns in Al society. Currently, data confidentiality protection has been one core issue during deep model training. Federated Learning (FL), which enables training across multiple silos, gained rising popularity for its parameter-only communication. However, previous works have shown that FL revealed significant performance drop if the data distributions are heterogeneous among different clients, especially when the clients have cross-domain charae- teristic, such as traffic, aerial and in-door. To address this challenging problem, we propose novel idea, PartialFed, which loads a subset of the global model's parameters rather than loading the entire model used in most previous works. We first validate our algorithm with manually decided loading strategies inspired by various expert priors, named PartialFed-Fit Then develop PartialFed-Adaptine which automatically selects personalized loading strategy for each client. The superiority of our algorithm is proved by demonstrating the new state-of-the-an results on cross-domain federated classification and detection. In particular, solely by initializing a small fraction of layers locally, FedAvg on Office-Homm and UODB by 4.88% and 2.65%, respectively. Further studies show that the adaptive strategy performs significantly better on domains with large deviation, e.g. improves AP50 by 4.03% and 4.89% on aerial and medical image detection compared to FedAvg.
improve the performance of
Introduction
Endless collection of texts, images and videos for training data lata-hungry models increases potential threats to the safety systems. Any attacks to these data centers could cause billions of information leakage. A safer way is keep the user data purely local their devices. But this contradicts with the widely adopted stochastic gradient descent (SGD) training procedure, which usually requires data communication for random batch sampling. Federated Learning [12 develops a paradigm for training large models under such situation. Locally trained models are aggregated using FedAvg [19] and then served initialization model for the next local iteration. The training data confidentiality is achieved by only allowing transferring the model's parameters (rather than the data) under different cryptograph algorithms, which include differential privacy 7 homomorphic eneryption [37]. block chain [22), etc.
However, recent studies 39 26 25 have shown that FedAvg does not provide satisfactory results in the presence of data heterogeneity. A major problem is that all models are designed to fit an "average client" 229,, which is difficult when local and global distributions other. This phenomenon is not uniquely observed. Researchers in multi-domain learning have also discovered that directly fitting non-identical domains into single feature extractor is suboptimal The leading solution to this problem is to reconfigure the network into domain-agnostic and
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

domain-specife layers [23 [3 Similar conclusion has also been drawn simultaneously in the field of multi-task learning [27] The performance is boosted by letting each task choosing aask-agn and task-specific layers on the fly. Based on such generous observations, we propose Personalized Federated Learning (PFL.) method from the idea of client-agnostic and client-specific initialization. Initialization is considered because it plays the essential role of transferring global knowledge in FL..
Figure I dipicct the overall architecture. At the heart of our algorithm is a mixed initialization strategy. Instead of fully utilizing the averaged global parameters for initialization, clients will only select a fraction of them, and load the remaining parameters from previous local models. The selection process is decided by a customized loading strategy, which might vary from client to client and time to time. To validate PartialFed thoroughly, we first propose PartialFed- Fixi where the loading strategies are inspired by human priors in functionality and classification of different parts of the network. Then we propose an automatic strategy which is learned jointly with the network parameters by gradient descent. We name this algorithm PerrialFeedAdappiic
In the case of PartialFed-Fix, our algorithm can be viewed as a federated version of multi-path networks in the multi-domain learning [24] 33). The shared parameters of different clients are jointly learned by FedAvg, while the client-specific parameters are only learned locally. Treating the network as combination of global and local blocks makes it possible to learn the knowledge from other clients while keeping the local knowledge stored safely. Although PartialFed-Fit can achieve improved performance, we argue that the fixed loading strategy is a suboptimal solution for PFL since different clients may have different dependencies on the global model. For example, a client might prefer the classifier of the global model at the start of training, which often acquires better generalization ability. In the end, the client is likely to train local classifier, which brings superior personalized performance. We show that our dynamic algorithm PartialFed-Adapiin is able to capture this change of parameter loading behaviors during the training process.
Despite its simplicity. PartialFed gains surprising performance on many non i.i.d. FL experiments. We construct real world FL experiments by introducing cross-domain elassification and detection dataset 30 33 For example, on Office-Hone dataset, our PartialFed- Fix and PartialFed-Adaptivee surpass FedAvg by 4.88% and 5.43% on average accuracy, respectively. Similarly, on UODB dataset, PartialFed-Fix and PartialFed- -ddaptiiv outperform FedAvg by 2.65% and 2.68% on AP50, respectively. More interestingly, the adaptive loading strategy greatly reduce the possibility of clients getting inferior performance caused by the distribution deviation.
2 Related Work
Personalized Federated Learning The pioneering work of FedAvg 19 aims to train global model for all clients. But the goal is hard achieve when data distributions are non i.i.d. Wang et al. directly finetune the global model on local dataset for personalized performance. Mansour et al.
31
18
provide theoretical study of clustering, data interpolation and model interpolation personalized
learning. FedProx [13 introduces a regularization based algorithm. FedCury (26) and FedCL (35 further consider using Elastic Weight Consolidation (EWC) for parameters importance estimation. T. Dinh et al. (28) theoretically improve the vanilla regularization with Moreau Envelopes. [5] g consider using model interpolation to address the PFL problem. Meta learning is also utilized to learn better initialization for personalized finetuning (6). The most related works to this paper is parameter decoupling strategies which split networks into global and private layers. Arivazhagan al. (2) consider global base network with personalized classifiers while FedBN [14] and SiloBN [ consider private Batch Normalization (BN) layers. The special cases of PartialFed- Fix are equivalent to these methods, and we further examined their combinations and more strategies.
Malti-Dommin Learning The problem of training on heterogencous distributions has been widely studied in multi-domain learning. The difference is that multi-domain learning aggregate data together while FL. store data in silos. Our method is related to the recent work 3
which aims to learn single network across all domains with minimal number of task specific parameters. Bilen and Vedaldi [3] propose to use domain-specifer batch and instance normalization while Rebuffi et al.
23 24
23 24 further take series and parallel residual adapters as the domain-specifie parts. Wang et al.
35 extend the idea to object detection with SE-ResNet [I0] and introduce a multi-domain detection
benchmark. Data-driven architectures are also explored in 20 38 32 neural architecture search are employed.
where network pruning or
2

---

Server
doveload
download
Client
upload
upload
Client
download
upload
Partial Erittiliization
Client
Part Initaliaation
Partal Inttialization
Local Tracning
Figure I: The snapshot of PartialFed. In each iteration of federated learning, the central server will send the global model all clients. Each client will then mix the global parameters from server with local parameters from itself by partial initialization, which is used for the next local training.
Partially-Loaded Federated Learning
3.1 Preliminary
Federated Learning We formally describe the problem setup in this section. Assume there are N E N clients E C who are willing to participate in the joint training. Each client owns private dataset D = {(2f, vf) : {1....,m n where ne is the cardinality of the dataset. We use Enec n. to denote the total amount of data in all clients. The data distribution P' y) varies from client to client. To be more specific, we do not put any i.idd. assumptions on either (x) or pe (y)(z), which indicates domain gap and label difference are allowed in the framework.
At the start of each federated iteration f, central server will send global model W, to all clients as parameter initialization. Clients will then use their private data De to train W, and get an updated We. All W E C are then sent server and fused by FedAvg:
Th W
W++1
n
CEC
The resulting model is used initialization for the next federated iteration. When networks structurally heterogeneous in each client, W, denotes the maximum set of shareable parameters. The algorithm will stop on its convergence. Note that we do not incorporate client selection in McMahan et al.
19 which can be easily added if required.
PartialFed Compared vanilla FL, PartialFed changes the client's parameter initialization process. After receiving global model W., each client will additionally load the local model W from last iteration (See Alg. [). Instead of purely initializing training with global model W., PartialFed uses mixed initialization, where each parameter is either loaded from global model W, or local model W We hope that by partially loading global parameters, client can benefit from sharing knowledge while avoiding local knowledge forgetting caused by FedAvg. The choice of mixing strategy A& will be discussed in section 3.2 and 3.3 training iteration.
All the models are finetuned locally in the last
major advantage of Far is the minimal changes the original framework. Without sharing gradients, subset of data and customized attributes, clients can enjoy themselves from a highest parameter privacy protocol.
addition, since the server do not need extra operations like knowledge distillation [16] except for FedAvg. the uploaded models are not required to do any forward computa- tion. This enables homomorphic eneryption (77 and protects client from information leakage caused by model attacks [36].
3

---

Algorithm 1 PartialFed: Partially-Loaded Federated Learinng
Clienttteration(; Wy):
Load W. from local storage Get partial loading strategy Ai Get partially loaded initialization W
partial initialization
Miinst by:
[
if A[
W Cnnc (8)
i denotes the index of parameter
(2)
[W]]i]
if A(
Set W. = - iiisst for cach local epoch from 1 to E do
> regular training
for batch b & D. do
7 ((; b), for w W/
end for end for
Save W; on local storage Return W
Server:
Initialize Wo ( 11,00 ... 0k.ol at random for each round 1, 2.... do
for client e E C' in parallel do
W + Clientteratton(e, W2) end for
D device training
W4+1 EcEc a W.
execute FedAvg
end for
3.2 PartialFed-Fix
We first discuss the fixed manual strategies in this section. Deep networks are often intuitively divided into different functional parts: feature extractor, classifier, etc. These intuitions can be transformed into corresponding loading strategies. An example strategy is shown in Equ where parameters except fully connected layer (fc) are loaded from global model. We empirically validate the efficiency by transferring commonly used assumptions to our algorithm in this section.
[1
if layer fc
Ag
0
else
"Bottom-up v.s. Top-down" One of the commonly accepted assumptions deep network is that blocks close to input image are more related to low-level features while top blocks are associated with high-level concepts. Sharing bottom layers assumes that different clients share similar low-level texture features while the opposite assumes high-level visual concepts are composed analogously. We examine whether PFI. training benefits more from sharing global "feature extractor" or global "classifier". To be more specific, we use ResNet18 99 for these experiments, which has four stages (each with two ResBlock) and a fully-connected layer (Figure 2 detailed description of the used dataset can be found in section 4.1
The results are listed in Table I Firstly, all experiments with partial parameters loading outperform the full parameter loading, which proves the superiority of the proposed PartialFed. Secondly, there are no significant gap with additional parameters loading from global sl to sl-s4, but loading global fc in this case severely degrades the performance. Thirdly, we find there's 's consistent accuracy drop from strategy fe to s2-fc. We hypothesis that sharing too much high-level concepts will hurt the personalized federated learning.
Batch Normalization Batch statistics are considered key factor when heterogencous distri- butions occur and has been widely studied in domain adaptation [15],
learning [3]. and federated learning [ We study how the loading strategies of BN will affect the PFL training. Table[display the results on Office-Home dataset. Loading BN locally (i.e. w/o BN) gains improvements on all four domains. Given that the parameters of BN layers are only a very small fraction of ResNet, the improvement is non-trivial. On the other hand, since the BN layer is such brittle in non i...d. settings, we hope that multi-distribution normalization strategy can proposed in the future.

---

Table Bottom-up sharing v.s. Top-down sharing on Office-Home dataset: The left table shows bottom-up loading strategies, while the right part shows the opposite top-down loading strategies. The column Load describes the partially loaded global layers and s/-st the abbreviation for loading all global parameters start from stagel to stage4 (See Figure for ResNet stages). "P", "A", "C", "R" are abbreviations for the domain names. The last row the result of loading all parameters from the global model (equivalent sl-fc), which is equivalent vanilla FedAvg with finetuning.
Bottom-up
Top-down
Load
mean
Load
A
R
mean
s/
89.46
70.70
73.33
80.53
78.50
88.70
71.48
72.88
81.86
78.73
sl-s2
89.46
70.12
73.44
80.64
78.41
s4-fc
87.39
71.48
73.77
80.42
78.27
s1-s3
90.54
70.70
73.33
80.53
78.50
s3-fc
87.61
70.70
71.09
80.64
77.51
s1-s4
90.22
70.51
71.21
83.52
78.86
s2-fc
88.37
68.75
70.65
82.74
77.63
Full 87.99
68.70
65,85
79.22
75.44
Full
87.99
68.70
65.85
79.22
75.44
Recalling that previous section shows loading fc and s/-s4 simultaneously could harm the performance We hope this rule can be combined with the BN rule. This leads the last row of Table where both global BN and global fc are not loaded. Surprisingly, this simple modification sigmificantly improves FedAvg from 75.44% to 80.32%, which 4.88% improvements.
Table 2: Batch Normalization Strategy Office Home
Load
A
R
mean
Full
87.99
68.70
65.85
79.22
75.44
w/o BN
88,37
72.46
71.99
82.30
78.78
w/o BN&fc
90.43
73.44
74.33
83.08
80.32
Skip Loading Except for the "shared feature extractor private classifier" network partition, another widely adopted routine in multi-domain and multi-task learning is a loop of "share-private-sll private-..." sub-networks [33 27 This paradigm is found to be helpful when tasks interference happens. This paragraph examines the performance of skip sharing with PartialFed.
For simplicity, we label each ResBlock in ResNet18 with letters in Figure 2 The same number of blocks (4 blocks) loaded globally for all experiments. The input convolution and fc layer are loaded as default. Experimental results are revealed in Table Methods that load adjacent layers (e.g. AaBb) get similar mean accuracy except for the CcDd strategy. Since CcDd a kind of top-down strategies mentioned above, which does not perform well in general, its inferior performance. Interestingly, the two strategies with skip loading, ABCD and abcd, get higher performance than all the other strategies, which justify the idea of skip sharing. We attribute the efficiency of skip loading to the fact that clients have the demand of learning client-specific knowledge all levels of the network instead of only part of them.
infer this is the reason for
Load
P
A
C
R
mcan
AaBb
89.57
71.68
72.88
80.53
78.66
AaCe
89.89
72.27
71.88
80.31
78.59
BbCc
89.78
71.09
71.76
80.75
78.35
b
d
fc
BbDd
89.02
70.70
71.65
80.75
78.03
CcDd
87.83
69.34
69.98
81.19
77.08
ABCD (Skip)
90.00
73.05
72.21
81.79
79.31
abcd (Skip)
89.35
73.24
73.10
80.53
79.06
$2
83
S4
Table 3: Skip strategy results on Office-Home
Figure 2: ResNet18 Partition
3.3 PartialFed-Adaptive
The hend-des strategies in PartialFed-Fix are intuitive not efficient. The search space of all fixed strategics has complexity of 2 denoted the number of layers). If we further consider
5

---

the factor of time t and client number N, it reaches a complexity of N x 2 It's hard to find an optimal strategy by hand in such large space. Therefore, we propose learnable loading strategy, PartialFeedAdaptivee. The main idea is to learn the initialization strategy through data-driven approach that adaptively chooses which parameters to be loaded globally given the specific client. The objective of the algorithm is to minimize the following loss function on client c:
min min Lerasn (De, WEl t,init (W,, W AF WT
where Af, W)-1)) the initialization given by strategy Až and the federated iteration. Note that here we only consider the personalized loss for client but the custom initialization strategy also affects the global training in an implicit way. We leave this problem for future work and focus on the greedy situation in this paper.
The original strategy A makes discrete decisions, which is not differentiable. In order to optimize strategies and model parameters jointly with gradient descent, we adopt Gumbel-Softmax sampling 17 for modeling strategy Aÿ- More precisely, A parametrized by a distribution vector of E [0, 1j2k with shape [k; 2]. Each row af[i] specifies the probability of loading local and global 1] 1. For differentiable training, parameter for the tth parameter w; and satisfies af 0] af
reparameterization trick is used:
exp (llo ( G())/m) J E 0,
(5)
Σ
exp (llog afli, k] G(+))/Ts))
KE (0(3))))
where G -looo- log U) standard Gumbel distribution with U sampled i..d. from fni 1) and Te is the temperature parameter of softmax at federated iteration t. We also use the hard sample trick described in Py PoTcch document to obtain discrete sampling. The main idea is to substitute soft sample output Ysaft with hard output yhard Sstop Sstop_gradient (Ysoft) + Ysoft.
gradient
Algorithm PartialFed Adaptive
ponnltterationAAdatiiieee W,): Load W. from local storage Initialize af = af_ for cach local epoch from 1 to E do for batch b DD do
Sample strategy Ag by Equ. witth parameter of Composite batch parameter by:
[
if Ag =
Wild = [.
denotes the index of parameter
(6)
if Au =
Compute loss with parameter W. batch index % (fm + fa) fm then
2 +- w n/V, af; b), for Wi
model parameter update
else
af + -
strategy parameter update
end if end for end for
Sample best parameter Wi according to of Return We
Training Strategy The overall learning scheme is summarized in Alg. 2 The model parameters and strategy parameters are iteratively updated by EM algorithm. The updating frequency is denoted as Im and fa. At every training step, a discrete batch strategy A sampled by Gumbel--offtmxx, which guides the composition of global and local parameters. The composited parameter W. is used to compute loss and either itself or the strategy parameter of is updated according to training step index. The temperature parameter 7; in Equ. 5lis initialized as 5.0 and annealed to 0 as in [27]. The sampling strategy approaches the original discrete distribution with 7 getting close to zero limit I7
http::/ppocccagooooossmmmeeeeeec www.mmmcnnntooaguuume...... wwt.ggguubbessst..ttenee softmax,htm html

---

4
Experiments
We evaluate the performance of PartialFed, on real-world non i.i.d. tasks including cross-domain classification and detection.
4.1 Settings
Datasets For federated classification, we use the popular Office- [0] dataset, which contains four domains: Art, Clipart, Product and Real World. All domains share the same 65 typical categories in office and home. Each domain has an average of 3k images. Each client use a single domain as its private data, which in total gives 4 clients.
For the detection task, we adopt challenging multi-domain detection dataset UODB 33 which is composed of 1 detection tasks with different domains, categories and data sizes (Table 5 Similar to Office-Home dataset, we take each domain as a client. Except for domain gap. there's i also big difference in the number of training images, from minimum 0.5K to maximum 35K. Note that FedAvg weights each client model by their sample numbers, the severe data imbalance makes extremely difficult for federated learning.
Implementation Details We use PyTorch 21 to implement our algorithms. The detection experiments also use Detectron2 34 For Office- Home Dataset, we adopt ResNet- as backbone network. SE-ResNet-50 [10] is used as the detection backbone, which is the same setting as the original paper B3 Both models are pretrained on ILSVRC2012 ImageNet 4 For sampling the we try both hard and soft sampling. Hard sampling takes the parameter that maximize the logits while the soft sampling uses a weighted sum of global and local parameters with the sample logits. Soft sampling is also followed with finetuning. We find soft sampling is better in most cases, so we report the performance of the soft sampling strategy if not specified. For details, we list the hyper parameters in the Appendix.
best parameters in Alg. 2
4.2 Cross-Domain Classification
The results on Office-Home dataset are displayed in Table We report the Top-1 accuracy for each domain and on average. On average, the All model trained with data from all domains surpasses single domain training, but it does not work well on Product domain. This reflects that naively sharing all parameters not suitable for all domains even in the non federated training.
Adaptive Granularity We experiment adaptive learning at three levels from coarse fine: stage, block and layer level for ResNet. For example, for the stage-level adaptive, the choice space has complexity of 25, which decides whether each group in [Wstagel: Wstege2; Wstage3; Wstage4, fc] is selected globally or locally. The adaptive strategies outperforms fixed ones on average. Among them, the layer level PartialFed-Adaae Adaptive works best and outperforms stage-level and block-level by 0.41%. 1.. Theoretically. learning at smaller granularity improves the performance upper bound, but also increases the burden for training algorithms. In practice, clients should choose granularity according to its computational and data resources.
Comparison With State-Of-The-Art We compare our methods with a set of SOTA algorithms. All of them are reimplemented by referring the open-sourced code published by the author. Hyper parameters are selected as suggested in the original paper, and the best results are reported. The algorithms are also implemented in personalized version (i.e. finetune the model at end if needed) for fair comparison. While most existing methods improves the vanilla FedAvg, the best FedBN is still inferior to the single domain training due to domain conflicts. This makes FL unacceptable because the extra communication does not introduce positive impacts. On the other hand, our PartialFed, outperforms both single domain and full data training, which makes practical in more scenarios.
Although being simple, PartialFeed-Fix without global BN and fc is already fairly strong on this dataset. It surpasses vanilla FedAvg by 4.88% and FedBN by 1.53% on average accuracy. But is still affected by domain conflicts and gets only 90.43% on domain Product, which is lower than single domain training. PartialFed-Adee Adaptive overcomes this problem by automatic learning the suitable strategies for each client. It is the only method that beats single domain training on Product domain.
2 licenseed under httphhhhhw.mmyihcchoooooooeoeene
licensed under the Apache License 2.0
7

---

This is a thriving result since it improves all clients performance instead of subset of them. On average, it surpasses vanilla FedAvg by 5.43% and FedBN by 2.09%. respectively.
Table 4: Office-Home Results:
The first row is the single domain results and the second row is directly training one All model with all data. The next block shows vanilla FedAvg and its improvements. For our methods, we first display three representative fixed strategies discussed in section 3.2 Then PartialFed-Adaa Adaptive is listed in the last block. Three different granularities are experimented as in the table. We emphasize the best two results of each domain with red and orange.
Method
mean
Single
91.59
70.22
73.77
80.42
79.00
All
89.29
71.99
76.71
81.63
79.90
FedAvg 19
87.99
68.70
65.85
79.22
75.44
FedProx 13
87.55
67.52
67.19
79.89
75.54
PFedMe 28
86.24
70.28
68.19
79.56
76.07
FedBN[
88.37
72.46
71.99
82.30
78.78
PartialFed-Fin
sl-s4
90.22
70.51
71.21
83.52
78.86
w/o BN&fc
90.43
73.44
74.33
83,08
80.32
ABCD (skip)
90.00
73.05
72.21
81.79
79.31
PartialFed- PartialFed-Adaptive
stage
91.96
73.44
74.89
81.53
block
91.74
73.54
74.28
82.30
layer
92.16
74.02
75.26
82.05
80.87
4.3 Cross-Domain Detection
The results of federated detection is summarized Table The vanilla FedAvg improves perfor- mance of clients with relative small datasets: Watercolor and Clipart. But it performs poorly on datasets with special properties, c.g. WiderFace for face detection, DeepLesion for medical image detection. FedBN works well and is able relief the decline in face and acrial image detection, but it works even worse than FedAvg on DeepLesion, which is distributionally dissimilar to natural images. FedProx does not perform well in this dataset. Restricting contrasting client models to fit the averaged model with regularization term seems harmful given domain gaps at such degree.
For the fixed loading strategy, we first examine SE module and its combination with batch normaliza- tion, which is related the attention mechanism [O]. We also implemented skip loading strategy similar to the one in section 3.2 We tested two strategies: loading local first block and loading both local first and third block each SEE-ReNNet stage. The first one works better on smaller datasets because it provides more knowledge by sharing more parameters. For the adaptive strategy, The block-level adaptive strategy gets best performance in this case. We hypothesis that searching SE-ResNet-50 strategy at layer-level is too costing and hard to learn, which results in the lower performance of Adaptive-layer.
In general, PartialFed-Fi Fix and PartialFed- PartialFed-Aaaptiee Adaptive beats FedAvg by 2.65% and 2.68% and beats FedBN by 0.95% and 0.98%. And again, our Partialeed-Adaptive gets superior performance on distributionally difficult domains, including Deeplesson ession, WiderFace and DOTA, This proves the effectiveness of our algorithm on extreme domain shift.
Table 5: UODB description
Datasets
KITTI
WiderFace PascalVOC
LISA DOTA COCO
Watercolor Clipart
Comic Kitchen
DeepLessoon
Domain
traffic
face
natural
traffic
acrial
natural
watercolor
clipart
comic
indoor
medical
Train
4k
13k
16k
8k
14k
35k
1k
0.5k
1k
5k
23k
Test
4K
5k
2k
10k
1k
0.5k
2k
Classes
20
4
11

---

Table 6: UODB Results:
The first line is the single domain result. The second line depicts vanilla FedAvg results, which is followed by other competing algorithms. We use blue color to highlight domains with large distributional or task deviations, which are challenging for FL algorithms. PartialFed-Fii Fix-Fix is displayed with four strategies by considering the network structure. wo Be& denotes for PartialFed-Fix without loading global SE and BN parameters. fb and ib means the first and third ResBlock in every ResNet stages. Three levels of adaptive methods are also experimented.
We emphasize the best two results of each domain with red and orange.
Method
KITTI
WFace
VOC
LISA
DOTA
COCO
WC
CP
CM
Kit
DL
mean
Single
59.07
54.99
79.10
94.71
68.18
51.35
40.31
36.13
35.07
96.17
62.01
61.55
FedAvgl 19
59.91
50.99
80.54
93.39
65.04
51.74
43.19
42.81
33.46
95.41
58.26
61.34
FedProx
13
61.98
48.32
80.10
92.28
62.52
51.73
42.51
42.32
32.95
94.98
58.11
60.71
[14
62.14
52.58
81.52
94.28
67.47
51.49
49.11
43.69
39.15
94.86
57.11
63.04
PartialFed-Fix
w/o
63.16
51.34
80.97
93,44
66.10
52.45
46.89
44.86
35.96
95.36
58.43
62.63
w/o Se&BN
64.46
52.29
81.72
92.86
67.17
51.89
44.83
46.33
37.07
95.51
58.96
63.01
w/o fb (skip)
62.69
56.04
80,94
94.47
68.33
51.91
49.21
43.15
40.17
95.92
61.09
63.99
w/o fb+tb (skip)
60.75
56.69
80.81
93.39
69.67
51.90
45.03
39.26
37.38
93.96
61.58
62.77
PartialFed AerialFeddAAdapiiee
stage
61.08
55.83
81.18
93.36
68.93
51.75
48.15
44.07
38.24
95.79
63.00
63.76
block
63.44
56.04
81.40
94,25
69.07
51.89
46.74
45.41
36.50
96.34
63.15
64.02
layer
62.05
54.08
80.83
94.35
68.90
51.91
45.53
44.10
38.83
95.70
62.61
63.53
4.4 Strategy Analysis
We visualize how the strategy changes with FL training, as displaced in Figure 3 Interestingly, there's a clear tendency for all domains to share low-level knowledge than high-level knowledge (i.e. the probability is larger compared to top stages). This coincides with our conclusion in the bottom-up fixed strategies. In the stagel and stage2, Clipart domain differs from other domain, possibly due to its unique image style. For the high-level fc layer, all domains tends to use private parameters at very first of the training except for Art domain. Loading local fc layer coincides with our findings in the fixed strategies. On the other hand, we find out that client model with Art data learns slower than other domains at the beginning, possibly due to its deviation from the pretrained ImageNet model. Borrowing the knowledge from other domains seems to be wiser choice in this situation. It's fascinating to see that the adaptive methods is capable of capturing these kind of changes.
stagel
stage2
stage3
stage4
fc
600
Uost
Duei
C6
655
Kod
Uine 43
Me
Rad, Nuld
6300
0.4
6103
6475
0.4
30
29
29
30 26
Figure 3: The change of stage-level strategy across training time. The and axis denotes the federated iteration and the probability of loading global parameters. From left to right exhibits different stages from low-level stagel to high-level fc. Different domains are colored differently. The red dashed box highlight the increase in the tendency of Art domain to share fc layer during early training stages.
Our experiments have shown that the adaptive learning strategy has advantages in learning client- specific strategies. which can relief client from negative impacts caused by distribution shift. But it also suffers from the risk of incomplete training, either underfitting or overfitting. For example, we find out that searching 50 layers ResNet at layer-level is already hard. How to improve the strategy learning efficiency is related to neural architecture search, which considers balance between exploration and exploitation. The data-driven approach also ignores meta-information like datasets cardinality. A possible way to address this to incorporate human prior into the adaptive algorithm, which is combination of PartialFed-FF Fix and PartialFed-Adeen Adaapive..

---

Conclusion
We demonstrate a simple yet novel approach for personalized federated learning. The core of our method is mixed initialization, which only partially utilizing the global parameters given by FedAvg. Inspired by commonly accepted priors, we develop a set of manual mix strategies and validate their reliability. To make it further, develop a data-aware strategy which is adaptively learned with model parameters. Our experiments show that the proposed two strategies, PartialFed-Fix and serrialFed-Adaaiivee outperform set of state-of-the-art methods on cross-domain FL experiments including object classification and detection. More importantly, PartialFed- -Afattviee is found to automatically reduce performance degradation caused by extreme distribution heterogeneity.
Acknowledgments and Disclosure of Funding
We thank Rui Li and Xin Huang for discussions. In the future, we wish to use PartialFed on MindSpore which is new deep learning computing framework.
References
[1] M. Andreux, O. du Terrail, C. Beguier, and E. W. Tramel. Siloed federated learning for multi-centric histopathology datasets. In MICCAI DCL workshop, 2020.
[2] M. G. Arivazhagan, V. Aggarwal, K. Singh, and S. Choudhary. Federated learning with personalization layers. arXiv preprint arXiv: 1912.00818, 2019.
[3] H. Bilen and A. Vedaldi. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint
2017.
[4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, CVPR, 2009.
[5] Y. Deng, M. M. Kamani, and M. Mahdavi. Adaptive personalized federated learning, arXiv preprint arXiv:2003. 13461, 2020.
[6] A. Fallah, A. Mokhtari, and A. Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Advances Neural Information Processing Systems. NeurIPS, 2020.
[7] R. C. Geyer, T. Klein, and M. Nabi. Differentially private federated learning: A client level perspective. In NeurIPS 2017 Workshop, 2017.
[8] Hanzely and Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.055111 2020.
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
[10] J. Hu, S. Li, and S. Gang. Squeeze-and annezze-and-exciaatio networks. In Conference Computer Vision and Pattern Recognition, CVPR, 2018
[11] E. Jang, S. Gu, and B. Poole. Categorical reparameterizztion with gumbel-softmax. In International Conference on Learning Representations, ICLR, 2017.
[12] J. Konečný, H. B. McMahan, D. Ramage, and P. Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02522. 2016.
[13] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems, MLSSs, 2020.
[14] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q Dou. Fed( Federated learning on non- (idd features via local batch normalization. International Conference on Learning Representations, ICLR, 2021.
'naps//wwww.indssoo..adl
10

---

[15] Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint orXiv:1603.04779, 2016.
[16] T. Lin, Kong. S. U. Stich, and M. Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
[17] C. J. Maddison, A. Mnih, and Y. W. Tch. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, ICLR, 2017.
[18] Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. /06/9, 2020.
[19] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication--fficiennn learning of deep networks from decentralized data. In AISTATS, 2017.
[20] P. Morgado and N. Vasconcelos. Nettailor: Tuning the architecture, not just the weights. In Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
[21] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L.. Fang. Bai, and S. Chintala. Pytorch: An imperative style, perforrrfoe deep learning library. In Advances in Neural Information Processing Systems 32. 2019.
[22] R. Pokhrel and Choi. Federated learning with blockchain for autonomous vehicles: Analysis and design challenges. IEEE Transactions on Communications, 2020.
[23] S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, NeurIPS, 2017.
[24] S.-A. Rebuffi, Bilen, and A. Vedaldi. Efficient parametrization of multi-domain deep neural networks. In Conference on Computer Vision and Pattern Recognition, CVPR, 2018.
[25] F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek. Robust and communication-effici federated learning from non-i.idd data, IEEE Transactions on Neural Networks and Learning Systems, 2020.
[26] N. Shoham, T. Avidor, A. Keren, N. Israel, D. Benditkis, L. Mor-Mosef, and I. Zeitak. Over- coming forgetting federated learning on non-iid data. arXiv preprint arXiv: 1910.0779, 2019.
[27] X. Sun, R. Panda, R. Feris, and K. Saenko. Adashare: Learning what to share for efficient deep multi-task learning. In Advances in Neural Information Processing Systems, NeurlPS, 2020,
[28] C. T. Dinh, N. Tran, and J. Nguyen. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, NeurlPS, 2020.
[29] A. Tan, H. Yu, L. Cui, and Q. Yang. Towards personalized federated learning. arXiv preprint arXiv:2 007 10, 2021.
[30] H. Venkateswara, Eusebio. S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
[31] K. Wang, R. Mathews, C. Kiddon, H. Eichner, F. Beaufays, and D. Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv: 10252, 2019,
[32] Q. Wang, J. Ke, J. Greaves, G. Chu, G. Bender, L. Shaiz, A. Go, A. Howard, M.-H. Yang, Gilbert, Milanfar, and F. Yang. Multi path neural networks for on-device multi-domain visual classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2021.
[33] X. Wang, Z. Cai, D. Gao, and N. Vasconcelos. Towards universal object detection by domain attention. In Conference Computer Vision and Pattern Recognition, CVPR, 2019.
11

---

[34] Y. Wu, A. Kirillov, F. Massa, W.-. Lo, and R. Girshick. Detectron2. https: com/ facebookreeaarcc//dtttcccoooo 2019.
[35] X. Yao and L. Sun. Continual local training for better initialization of federated models. arXiv preprint X.iv:2005./26637 2020.
[36] H. Yin, P. Molchanov, J. M. Alvarez, Z. Li, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
[37] C. Zhang, S. Li, J. Xia, W. Wang. F. Yan, and Liu. Batcherypt: Efficient homomorphic encryption for cross-silo federated learning, In 2020 USENIX Anmual Technical Conference (USENIX ATC 20), 2020.
[38] H. Zhao, H. Zeng, X. Qin, Y. Fu, Wang, B. Omar, and Li. What and where: Learn to plug adapters via NAS for multi-domain learning. arXiv preprint arXiv:2007. 12415,, 2020.
[39] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra. Federated learning with non-iid data. arXiv preprint 10X6:1180600082,, 2018.
12


Reviews:
[{"review_id": "XZK3NAawUBT", "reviewer": "Reviewer_37wQ", "summary": "The paper proposes a new method called ‘PartialFed’ to solve the performance degrading caused by data heterogeneity in the federated learning scenario. The main difference between this work and previous work is that the proposed method partially loads global parameters given by FedAvg, while the previous methods load all of the global parameters for each starting point of the local epoch. \n\nFor the sampling strategy for the parameters that load the global parameter, two methods are developed in this paper: “PartialFed-Fixed” and “PartialFed-Adaptive”. For the “PartialFed-Fixed”, the parameters are sampled manually based on the commonly accepted assumptions in deep network e.g. low-level knowledge learned from bottom layers. With the \"ParitalFed-Adaptive\", the sampling strategy is trained data-driven. Gumbel Softmax sampling is used to enable back-propagation through the sampling process. \n\nTo validate the proposed methods, cross-domain federated classification and detection are examined. Two partial loading strategies outperform a set of state-of-art methods. Specifically, the “PartialFed-Adaptive” further reduces performance losses due to the extreme distribution heterogeneity and automatically shows the tendency that reflects the assumptions of deep network used for the design of “PartialFed-Fixed”.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem of non-IID data distribution in federated learning is becoming an active field recently. More precisely, this work can be integrated into the personalized layer approach, in which only parameters of some layers can be transmitted to a server and aggregated by FedAvg. Specifically, FedBN, FedPer, and LG-FEDAVG are a form of the “PartialFed-Fixed” sampling strategy. However, this paper provides more ablation studies, so they show that there are better variations on the personalized layering method. More generally, this paper proposes a method that adaptively learns the sampling strategy from data which is clearly different from previous contributions.\n\nThat being said, there are several weaknesses.\n\n1) The important task of the work is to find a suitable sampling strategy for personalized parameters, but experiments are carried out with only one network structure for each experiment.\n2) In section 3.3, Table 6 shows the experiment result on the cross-domain detection. In this result, the performance of PartialFed-Adaptive is not significantly better than the PartialFed-Fixed method. If we look at each domain respectively, PartialFed-Fixed models are better than ParitalFed-Adaptive models for eight out of 11 domains. Also, for two over three domains with large distributional or task deviations, “w/o fb (skip)” method in PartialFed-Fixed is better than “block-level” PartialFed-Adaptive model. This result can be interpreted in such a way that the proposed adaptive method may not be the optimal choice.\n3) I could not see any theoretical or experimental analysis of the convergence of the proposed method, while related work including FedBN offers both.\n\nSome minor comments are as follows.\n\nHowever, it would be better if the following points were reinforced.\n1) There is no description about the metric and its unit written in Table 1,2,3,5 and 6.\n2) In line 102 and Equation (3), the notation f_c is used without a description. You can guess it as the fully connected layer before you see Figure 2, but it would be better to define it or to replace it with a word.\n3) In line 146 and following lines, the notation k is used without a description. \nIn addition, following points are typos.\n1) In caption of Table 6, “se&BN denotes..” may be “w/o se&BN denotes for..”.\n2) In line 172, Equ.4 should be Equ.5.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is not clear that the partial averaging idea works for more complicated network architecture.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5"}, "extra_text": {}}, {"review_id": "M3iSjtKvKPB", "reviewer": "Reviewer_9V5p", "summary": "This paper explores a simple but novel modification to FL approaches to better deal with settings involving non-homogeneous clients. They find that by allowing clients to mix local and global parameters for initializing local training, the added flexibility can notably improve the performance of the respective local end-models. They perform both an exploration of natural fixed mixing strategies and also propose an adaptive mixing strategy that allows for further improvements on average. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** As mentioned in Sec 4, specific instances of separating global/local parameters in FL have been touched upon in previous works. However, this paper certainly seems to usefully generalize such approaches, performing a more thorough empirical investigation into the space of mixing strategies and also introducing a new adaptive strategy. \n\nMore on the presentational side though, I do think the manuscript could be more up-front about this relationship with these works (i.e., [2, 1, 14] specifically). To the authors credit, they do mention these citations propose special cases of PartialFed-Fix towards the very end of the paper. However, given this close relationship, I do feel this warrants mention much earlier on, such as in the intro and the relevant subsections of Sec 3. \n\n**Quality:** The results seem solid as a whole. Just had a couple comments here\n\n(1) Given the closeness of the some of the numbers, including information about # of trials and error bars in the main (and also adding them for UODB) would be helpful.\n\n(2) Perhaps not necessary but for completeness, would be interesting to see how the current best PartialFed results compare to SOTA centralized MT/Multi-domain learning results.\n\n**Significance:** The problem seems well-motivated, especially judging by the demonstrated failure of more standard FL methods to be effective on the given datasets (performing worse than fully local learning).  Also, the results seem to indicate that PartialFed is indeed a valuable general strategy, leading to improvements on many of the tasks/clients involved. \n\n**Clarity:** There seems to be some room for improved presentation/clarity. Some things I noticed in particular are:\n\n*  The notation in Eq. (4) seems a bit confusing in that the training loss is minimized over both $A_t^c$ and $\\mathcal{W}_t^c$. The latter doesn't really appear in the loss expression?  Also, may be helpful to clarify the $u_t^c$ are the soft samples. \n\n* That final global models are all assumed to be fine-tuned locally could be made more clear, especially in Sec 2. This is mentioned in Sec 3 but in Sec 2, it is only implicitly mentioned in the Table 1 caption.\n\n* Some abbreviations/notation were used without being formally defined such as fc, BN, PFL\n\n* Not central to the paper’s main claims but “Clients can enjoy themselves from a highest privacy protocol” in line 92 does not seem strictly true, given that these approaches may still be liable to downstream inference attacks, i.e. the field has looked at things like differentially private FL to add even more protection.\n\n* Could be nice to move Table 4 closer to the actual results table for UODB for easier cross-referencing\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I do think the authors have discussed the limitations that come to mind. In terms of societal impact, it does not seem to introduce any new negative outcomes relative to traditional FL. ", "review_text": "", "extra_scores": {"ethical_concerns": "None that I have observed.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "H99qJoVXSSx", "reviewer": "Reviewer_u7s7", "summary": "This paper proposed a novel idea PartialFed, which loads a subset of the global model’s parameters rather than loading the entire model. Specifically, they validate the algorithm with manually decided loading strategies inspired by various expert priors, named PartialFed-Fix. Then they develop PartialFed-Adaptive, which automatically selects a personalized loading strategy for each client.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n-- This paper gives a comprehensive introduction of existing partial loading strategies. The experimental section also provides a comprehensive comparison for these methods with the proposed PartialFed-Adaptive algorithm.\n-- The PartialFed-Adaptive algorithm shows certain levels of novelty to learn the initialization strategy through a data-driven approach that adaptively chooses which parameters to be loaded globally given the specific client.\n\nCons:\n --The PartialFed-Adaptive Algorithm is not clearly explained. e.g., what does $u$ in equation (5) stand for, and how $u$ is used in the optimization process?\n-- The algorithm PartialFed-fix is not innovative enough, because there are already some existing works that split networks into global and private layers.\n-- According to the experimental results, the layer-level PartialFed-Adaptive works best and outperforms stage-level and block-level PartialFed-Adaptive, but also increases the burden for training algorithms. Thus, layer-level PartialFed-Adaptive must be much more computationally demanding. I suggest that the authors can compare the training time of  PartialFed-Adaptive with different granularity to help achieve a balance between the performance and computational resources. \n\n===\nUpdates after rebuttal.\n\nThe authors provided a clearer explanation of the details of the proposed algorithm in the response, and also added some experimental results to compare the computation resources of algorithms of different loading levels which address my concerns. I raised my score but  I still reserve my opinion that compared with existing works that split networks into global and private layers, the innovation of this paper is still limited. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "1I8mMIN3k6Y", "reviewer": "Reviewer_iae8", "summary": "This paper proposed an adaptive initialization algorithm for FedAvg, motivated by FedPer in a personalized federated learning scenario. The key contribution lies in the observation of the impact on training accuracy of different initialization strategies and the adaptive initialization algorithm. The empirical result of the algorithm shown significant improvement compared with FedAvg in training with heterogeneous data.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Good. The observation of different initialization strategies during local training is impressive, and the way of modeling the impact of the initialization patterns as a part of the loss function was also novel to the FL scenario. \n\nQuality: Good. The empirical results demonstrate the accuracy improvement with different initialization patterns is clear. The method used for updating the adaptive initialization pattern is promising. It has enough comparison between the proposed algorithm with the existing algorithms.\n\nClarity: Good. The explanation of the dataset and the key observations are easy to understand. The comparisons in the tables are also clear. However, there is some confusion in the algorithm description that needs to be clarified.\n\nSignificance: Good. The observation is important in the sense that it provides insight to understanding the impact of each block in NNs and provides a way to do personalized federated learning.\n\nComment to the paper:\n\n1. In Algorithm 2, step (6) seems to indicate that for E local epochs, the parameter $w$ has not been updated. The algorithm only does $b$ SGD step on $W_b^c$ instead of doing $Eb$ steps. Is it my misunderstanding or the description of the algorithm is incorrect?\n\n2. In other papers, such as FedPD, FedSplit, SCAFFOLD, and FedDyn which optimize local problems to $\\epsilon$ accuracy and adaptively regulate the difference between the local and the global model, initialization seems not to be affected by the initialization. Is there any relation between PartialFed-Adaptive and the dynamic regularization methods?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no potential negative societal impact of this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission. The reviewers agree that the paper provides an interesting result. The authors should follow the reviewers' suggestions to improve their presentation in the paper.
Decision: Accept (Poster)