Hot-pluggable Federated Learning: Bridging General and Personalized FL via Dynamic Selection
Authors: Lei Shen
Zhenheng Tang
Lijun Wu
Yonggang Zhang
Xiaowen Chu
Tao Qin
Bo Han
Venue: ICLR 2025 Poster (2025)
Keywords: Federated Learning
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: This work proposes a selective federated learning approach to integate personalized modules into general federated learning.

Abstract: Personalized federated learning (PFL) achieves high performance by assuming clients only meet test data locally, which does not meet many generic federated learning (GFL) scenarios. In this work, we theoretically show that PMs can be used to enhance GFL with a new learning problem named Selective FL (SFL), which involves optimizing PFL and model selection. However, storing and selecting whole models requires impractical computation and communication costs. To practically solve SFL, inspired by model components that attempt to edit a sub-model for specific purposes, we design an efficient and effective framework named Hot-Pluggable Federated Learning (HPFL). Specifically, clients individually train personalized plug-in modules based on a shared backbone, and upload them with a plug-in marker on the server modular store. In inference stage, an accurate selection algorithm allows clients to identify and retrieve suitable plug-in modules from the modular store to enhance their generalization performance on the target data distribution. Furthermore, we provide differential privacy protection during the selection with theoretical guarantee. Our comprehensive experiments and ablation studies demonstrate that HPFL significantly outperforms state-of-the-art GFL and PFL algorithms. Additionally, we empirically show HPFL's remarkable potential to resolve other practical FL problems such as continual federated learning and discuss its possible applications in one-shot FL, anarchic FL, and FL plug-in market. Our work is the first attempt towards improving GFL performance through a selecting mechanism with personalized plug-ins.

Published as a conference paper at ICLR 2025
HOT-PLUGGABLE FEDERATED LEARNING: BRIDGING GENERAL AND PERSONALIZED FL VIA DYNAMIC SE-
LECTION
Lei Shen
Zhenheng Tang Lijun Wu
Yonggang Zhang*
Xiaowen Chu 2.4 Tao Qin
Bo Han
TMLR Group, Department Computer Science, Hong Kong Baptist University
2
CSE Department, The Hong Kong University of Science and Technology
3 Shanghai AI Laboratory
4 DSA Thrust, The Hong Kong University of Science and Technology (Guangzhou)
5 Microsoft Research A14Science
ABSTRACT
Personalized federated learning (PFL) achieves high performance by assuming clients only meet test data locally, which does not meet many generic federated learning (GFL) scenarios. In this work. we theoretically show that PMs can be used to enhance GFL with a new learning problem named Selective FL (SFL), which involves optimizing PFL and model selection. However, storing and selecting whole models requires impractical computation and communication costs, To practically solve SFL, inspired by model components that attempt edit a sub- model for specific purposes, we design an efficient and effective framework named Hot- Plt-Plug Federated Learning (HPFL). Specifically, clients individually train personalized plug-in in modules based on a shared backbone, and upload them with a plug-in marker on the server modular store. In inference stage, an accurate selection algorithm allows clients to identify and retrieve suitable plug-in modules from the modular store to enhance their generalization performance on the target data distribution. Furthermore, we provide differential privacy protection during the selection with theoretical guarantee. Our comprehensive experiments and ablation studies demonstrate that HPFL significantly outperforms ofte-ooftthe-art GFL and PFL algorithms. Additionally, we empirically show HPFL's remarkable potential to resolve other practical FL problems such continual federated learning and discuss its possible applications in one-shot FL., anarchic FL., and FL plug- market. Our work the first attempt towards improving GFL performance through selecting mechanism with personalized plug-ins. Our code is released at https: / / thub. com tm r- menllr-greee group/ /HPFL.
INTRODUCTION
The performance of generic federated learning (GFL) (Brendan McMahan et al., 2016) suffers from data heterogeneity (Brendan McMahan et al., 2016; Kairouz et al., 2019; Tang et al., 2022b; Chen & Chao, 2021; Tang al., 2024b), where clients have different data distributions. Personalized federated learning (Smith et al., 2017: Collins et al., 2021; Chen & Chao, 2021) (PFL.) assumes that clients only need inference on local test data, which has similar distributions to local training datasets. To this end, PFL prioritize fitting on local datasets while absorbing knowledge from the global training data. Due to this property, PFL gets rid of data heterogeneity, as the training convergence is not severely disturbed by the client drift (Li et al., 2020b; Tang et al., 2024c; Chen & Chao, 2021).
However, in real-world scenarios, FL users may encounter test data different from local training data (Liu et al., 2020; Luo et al., 2019; Tang et al., 2022b; 2024b), but which may appear in other training data. For example, when one traveling abroad, the personal map app might recommend entirely different restaurants from their residence. In such situation, models trained on local restaurant
Equal Contribution. Correspondence Bo Han (bhanml@ccop hww..cc.ooo.pbu..........

---

Published as a conference paper at ICLR 2025
Communication
Plug-in in or task marker
Plug-in module
flow
Noised marker
Freezed backhone
Add Noise &Uplo
Module store the Server
Selection
Download selected plug-in
Final Output
v N (p Om)
Clients
New incoming client
Test data
(h + R u)(
(1) Personalizing Plug-in Module
(2) Inference
Add noise
Figure 1: The framework of HPFL.
and personal data can make better recommendations In GFL, clients encounter test data of others, instead of only their own test data as in PFL (Smith et al., 2017; Collins et al., 2021; Chen & Chao, 2021). In such realistic cases, PFL algorithms lose their general performance, fitting local datasets with the personalized models. As Table shows, advanced PFI. algorithms FedPer ( (Arivazhagan al., 2019b), FedRep (Collins al., 2021) and FedRoD (Chen & Chao, 2021) perform well in PFI., but performance collapses in GFI,, where personalized clients encounter all test data. This performance gap motivates us following fundamental questions:
they prioritize
Whether global model (GM) is compulsorily needed in GFL? Is it possible enhance GFL with personalized models (PMs) trained in PFL?
Considering that PMs are already solutions of PFL., intuitively, we formulate new learning problem called Selective FL (SFL) bridging GFL and PFL (Equation 5), and prove the so- lution of SFL can achieve better performance than solution of GFL. Both GFL and PFL can Accuracy
Table 1:
Test accuracy on GFL and PFL of personalized models, with ResNet-18 and CIFAR-10.
Algorithm
FedAug
FedPer
FedRop
FedRoD
81.5
92.5
74.1
95.8
85.1
95.6
85.3
94.3
be seen special cases of SFL,, whose core idea is to select suitable PMs for inference on clients according to incoming test data. However, the naive solution to SFL leads to privacy concerns, large system overheads and poor scalability. To this end, inspired by model components (Shah al., 2024; Olsson et al., 2022; Lai et al., 2025), we propose a general and effective framework named Hot Pluggalug Federated Learning (HPFL) practically solve SFL.
As shown in Figure 1, HPFL splits the model into two parts: a backbone (also called feature extractor) be trained using any FL algorithm initialized
and a plug-in module. The backbone pre-trained backbone. Clients train plug-ins based on local datasets and upload them with the according plug-in in markers to the server store. During inference, test data passes the backbone, and suitable plug-in is selected to complete the inference. There are two ways to implement retrieving plug-ins in HPFL, ac Clients upload lask markers to the server and select the appropriate module; 9: Clients download all plug-in in markers to select. a is suitable for situations where clients have limited computation ability, as it selects and completes final inference on the server, While reduces the computation burden on the server. To protect the privacy, we provide differential privacy protection on communicated features during the selection with theoretical guarantee. Our contributions are summarized as follows:
We identify a substantial gap between GFL and PFL,, and formulate new problem SFL to bridge them together to address this performance gap (Section 2). As far know, this is the first work
More real-world examples of GFL-PM problems provided in Appendix F.1.
2

---

Published as a conference paper at ICLR 2025
that enhances GFL through learning, sharing and selecting plug-ins, instead of classic paradigm with one single model.
We propose general, efficient and effective framework HPFL, which practically solves SFL (Section 3). And we add noise on communicated markers to provide differential privacy protection with theoretical guarantee (Section 3.4).
We conduct comprehensive experiments and ablation studies four datasets and three neural networks to demonstrate the effectiveness of HPFL (Section 5).
We show the remarkable potential of HPFL in federated continual learning (Section 5.4) and discuss HPFL's possible applications in one-shot FL, anarchic FL and FL plug-in iin market (Section 6).
SELECTIVE FL: IMPLEMENTING GENERIC FL FROM PERSONALIZED FL
2.1 GENERIC FL
The GFL aims to make M clients collaboratively learn a global model parameterized as 8. Each client has its local data distribution Dn Thus. the local objective function 2m (0) client m is also different. The global optimization object of GFL is defined as (Karimireddy et al., 2019; Woodworth al., 2020; Tang et al., 2022b; 2024b):
M
min (o PERd
Σ Pm (0) = > pm PmmBem-ae E(f(8, Em))
(I)
where Em Dm is the data sampled from Dm, f(0, Em) is the prediction, d is the number of model parameters, Pm. >> 0 and E Pm 1. Usually, Pm "N" where n'm denotes the number of client
m's samples and N Em= 7m GM refers to the model obtained from optimizing GFL.
2.2 PERSONALIZED FI.
Different from the object function of GFL, the PFL aims learn multiple personalized models which fit well on different datasets individually: (L.i & Wang, 2019; Chen & Chao, 2021; et al., 2021e):
min Cr(2, 01, 0M) Σ Pmm ((f(Om, (m)) R( 01,..., Oms),
(2)
Q.01
where R. is regularizer (Chen & Chao, 2021) that varies with different algorithms, Q is used to collaborate clients. We call each obtained locally personalized model p. as PM.
2.3 WHEN PM MEETS GFL
In practice, PMs of clients may meet test data from other clients. Therefore, the learned PMs 81. ..., B& need to perform well on all local data Di...., DM- We formulate the corresponding optimization goal with PMs in GFL scenario (GFL-PM) is:
min Gr-G(S2 01, O1...,, 0m) 0,01....9s
Σ Pmn BerBEnD E(f(O,, Em), + R(2, 01, 0M),
(3)
which can be seen as combination of GFL (Eq. 1) and PFL (Eq. 2): each PM is optimized to
minimize the on all D mm E l...., ... When not personalize 0; on D,, Eq. 3 is reduced to GFL. And if each client's PM only needs to perform well on its local data, Eq. 3 turns into PFL.
One may think that there is no need to endow PMs with global generalization performance because one can optimize GFL to obtain GM that generalizes well on all local datasets (Dm, E (1,... M) However, theoretically and empirically, optimization of GM difficult (Karimireddy et al., 2019; Woodworth et al., 2020) under communication cost and data heterogeneity constraints. Additionally, PMs' performance on local test data (PM on PFL) is usually significantly better than that of GM on global test data (GM GFL) (Chen & Chao, 2021; Collins al., 2021).
However, PMs after PFL usually cannot achieve better performance on unseen data distributions than GM in GFL (Chen & Chao, 2021). FedRoD (Chen & Chao, 2021) simultancously optimizes La and Lp, aiming to learn models that perform well both GFL and PFL. This shares a similar spirit of optimizing GFL-PMM problem (Eq. 3). However, PMs obtained from FedRoD remain
3

---

Published as a conference paper at ICLR 2025
trade-off between minimizers of PFL and GFL. It is challenging to obtain model parameters that are both minimizers of GFL and PFL simultaneously. Next, we show that GFL-PM can be naturally transformed into Selective FL (SFL) problem (Eq. 5), which involves optimizing PFL and model selection problem (Eq. 6 in section 2.4). And the solution of SFL could serve as the minimizer of both GFL and PFL..
2.4 SELECTIVE FL
Successful personalization on client m means the following equation (Chen Chao, 2021; Kairouz et al., 2019; Tan et al., 2022a).
Em D.n (0,, Sm) TE (0, Em), Sm), & m,
(4)
which means that for any client m, its PM outperforms than all PMs of other clients (Chen & Chao, 2021). Now, we are ready to state the following theorem (proof in Appendix B.1).
Theorem 2.1. With Equation and the PMs obtained from optimizing Equation 2 as:
QPf Opfl Opfl
arg arg ming.......... LP(S2, 01, 0M), we have
LP-a(S, .r.., 0x1) L .p. 0
Remark 2.2. Theorem 2.1 implies that LP-G G is lower bounded by the minimum of Lp.
Theorem 2.1 shows that the minimum of PFL is the lower bound of the GFL, problem. Intuitively, this inspires us to think about exploit PMs trained PFL to enhance GFL. Note that Equation comes from the selective property of PFL, i.e. optimized PMs have the best performance its local trained datasets. In light of this, the direct solution to select trained PMs according to the test data. Thus, propose the Selective FL (SFL) problem as the following:
min Ls(e, H) Σ Pm Be.Bee-D... ((((), Em (m.) H
(5)
s.t. f(0, H) = f(0p1) ,mm), == S(), Em, H.
(6)
where θ = {SPP1 opn ..... 0200 = arg ming 0.......001 Ore (S2, ..... Oas).
S called selection
function that outputs the model index to select model from the PMs based on the input &m and the auxiliary information H. which is exploited to select plug-in in module, e.g. noisy feature for
calculating distance metrics like Maximum Mean Discrepancy (MMD) will be illustrated in Section 3. Now, we can state the following theorem to illustrate that we can solve problem 3 by SFL (proof in Appendix B.2):
Theorem 2.3. With Equation SPFf 00 H that guarantees gpft 0.(0.cm H). we have
6Pft arg ming CP 01 ..., 0m)) and the
(7) Remark 2.4. Theorem 2.3 shows that if we can accurately select 0p.i out of all PMs when meeting data samples Em D Dm, the solution of SFL is also the lower bound of GFL-PM (Eq. 3). Therefore, solving SFL means that clients achieve performance in GFL as high as in PFL.
Cp-a(R, 0, .,,.., 0M)) > (P( = Ls(),
HPFL: HOT- NOT-PLUGGABLE FEDERATED LEARNING
In this section, will first illustrate that directly selecting PM faces some fatal obstacles, including the large system overheads and privacy concerns in Section 3.1. Then, we introduce the design of HPFL in Section 3.2 with the Algorithm 1. Lastly, the selection method is introduced in Section 3.3.
3.1 PROBLEMS OF DIRECTLY SELECTING PM
With PMs θ {ΩPf 0prl ..... choose PM based on the similarity between its data D, and the input data Em Dm, thus the selection function 6 is implemented as: Se(O, Em, H) arg Min d(D,, (m)) where d(.,.) is any distance measure, then infer as f(@nfl. 5m.) However, accessing data of other clients will cause privacy concerns. Moreover, communicating the whole model parameter On, is impractical due to large system overhead, especially for large language models and many clients.
".......7 M arg ming, i............................. .2: LP( f1....,
.i......d. 0xx), an intuitive idea is to
3.2 DESIGN OF HPFL
4

---

Published as a conference paper at ICLR 2025
Training the complete model 8. First, HPFL obtains a model & that performs well (not as good as PMs in PFL) on all client datasets with any GFL algorithm Thus, the model 0 owns a backbone y that can extract general fea- tures from all client datasets. Due to the lim- ited space, we chose the classic GFL algorithm FedAvg (McMahan et al., 2017) in experi- ments. Future works can explore other advanced GFL algorithms to learn better A.
Algorithm 1 HPFL.
Initialization: server distributes the initial model g
to all clients. 1. Training the complete model 0: for each round r 0, 1, R do of clients S, (1,.. M). server communicates for each client m E S" in parallel do
server samples
clients m S..
c++1 LocalTraining(Pm. 0r) (GFL..
end for
Training personalized plug-in module 8 Usually, after training, early layers of model learn more general features than late lay-
gr+l + ServerUpdaat((mm E S") (GFL..
end for ers (Yosinski et al., 2014; Asano et al., 2020), while late layers are more specific to some par- ticular datasets. Inspired by this, HPFL decom- poses the model as fm p for each client m. As shown in Figure 1, g a feature extrac- tor, and p is model head that outputs the final model prediction.
2. Training personalized plug-in module 0%:: each client m . in parallel do do Clients share and freeze the 0"
Clients design personalized 0%
Training 0% with object function (PFL). Obtaining plug-in in marker H.m (e.g. noised
features explained in Section 3.3 detail.) for plug-in selection.
Clients can design new personal plug-in mod- ule Pm (or say model head) different from the original head p. based on different computa- tion characteristics. Then, with the frozen gen- eral feature extractor g. each client individually trains personalized Pm on local data Pm by optimizing:
Upload 0% and Hm to server.
end for Server stores 0% and H...
HPFL Inference(0", Di.a)): i + SelectPlugIn(Deear 8", 89,77 H.).
min (0 = Eg, EDm (mm o ((mm), 5.)) 5m). 0%
Get output + o Diest)-
(8)
Now, each client obtains PM fm = Pm og. which enhances the generalization performance pm og on Dm, which is usually better than original GM f o due to the personalization. Thus, the 6nft in SFL problem 5 can be constructed by 4 and g inference becomes as f(p/ {m) = Pm o g(Em)
Inference and selecting plug-in module. In HPFL, we define some plug-in marker Hm as auxiliary
information in Equation 5 that will be exploited to select plug-in module. When training 0g Hmm collected by clients and uploaded the server. Note that general framework, HPFI. does not limit the specific form of H.. which depends on the selection method, As the first attempt this paradigm, We introduce a distance-based selection method in Section 3.3.
3.3 SELECTION METHODS
Decomposing the model also avoids accessing the raw data m. D.n- With the shared feature extractor J. we can select the Pm. based on the intermediate features h., = g(mm) rather than & itself to avoid leading raw data. Several studies have exploited the sharing of intermediate features to improve FL (He et al., 2020a; Lin et al., 2020a; Luo et al., 2021; Liang et al., 2020).
Distance based methods. Intuitively, now that each Pm is trained based on local features Atm. we only need to compare the similarity between hm and hestt g(Eeest), where Etest is the data that needs testing. Now, the select problem turns from Equation into:
Saise Reests fei.... Pexr) arg min d(kim Rrest))
mE.M
in which hi and heeest are noised hm and heest which are illustrated in the next section. In this selection method, the plug-in in marker Hm, = hm. In HPFL, utilize Maximum Mean Discrepancy (MMD) distance (Long et al., 2017) to measure the distance between plug-in markers and noised features of test data (task marker). We also exploit other distance measures like SVCCA (Raghu
5

---

Published as a conference paper at ICLR 2025
al., 2017). CKA (Komblith al., 2019) and out-of-distribution confidence based selection methods and provide results in Appendix D.
3.4 PRIVACY PROTECTION
Differential Privaey. In HPFL, plug-in markers (noised features of training data) and task markers (noised features of test data) are shared for selecting. To protect the privacy, following differential privacy (DP) (Abadi et al., 2016; Balle & Wang, 2018; Wu et al., 2022; Yang et al., 2024), we add Gaussian noises as h = (h + * /1 + k) for both hn and hiest where V ~ (N om)), in which hm = (Am + K v)/(1 + k), where v ~ (pm: Fm) is the noise to enhance privacy protection. The Um and on are mean and variance of features hms K is a coefficient controlling the relative magnitude between Gaussian noise and the features. The following theorem shows that the raw data protected by our noise mechanism with (c, dD.P. The detailed proof of Theorem 3.1 is shown in Appendix B.3.
Theorem 3.1. For the procedure of obtaining and sharing markers H(m)) = (g(xm) + K (1 + rt), (c, holds if the c, conforms to any of the two conditions: (1) Vc, E (0, 1)
0
(21(1.25()mmmm))
): 40 < 8 e /V2TE, O ((x
Model Inversion Attack. Besides theoretical analysis of DP protection HPFI, we also empirically verify the safety of sharing noised plug-in markers against the model inversion attack (Zhao et al., 2020). The failed reconstruction (in Appendix E.2) of raw data demonstrate that HPFL can defend model inversion attacks successfully.
4 RELATED WORKS
Generic Federated Learning. To address the data heterogeneity problem, FedProx (Li et al., 2020b)
and MOON (Li et al., 2021b) propose to add regularization terms to mitigate the negative effect caused by data heterogeneity. Some methods explicitly implicitly modify uploaded gradients to alleviate the gradient dissimilarity (Wang et al., 2020; Karimireddy et al., 2019; Tang et al., 2024c). Some works share intermediate features (Jeong et al., 2018; Hao al., 2021: Tang al., 2024c) or extra data (Tang et al., 2022b) to reduce client drift. There are also bunch of work utilizing Knowledge Distillation (KD) (Hinton, 2015) such as FedDF (Lin et al., 2020b) and Fed-ET (Cho et al., 2022). Some one-shot FL. methods propose to only communicate models with one round, thus significantly reducing the communication time (Tang et al., 2024b). Different from these works, we attempt to enhance the GFL. performance with personalized models.
Personalized Federated Learning. PFL exploits personalizing client models to better suit local heterogeneous training data. Meta-learning (Fallah et al., 2020), knowledge distillation (Yu et al., 2020b; Li & Wang, 2019), adaptive regularization and model mixtures (Hanzely & Richtárik, 2020; Dinh et al., 2020; Deng et al.. 2020) are used to enhance personal knowledge learning of models. Some works (Liang et al., 2020; Li et al., 2021a) allow clients to learn different PM structures. KNN-per (Marfoq et al., 2022) constructs PMs by replacing classifiers with non-parametric methods based on local datasets. FedRep (Collins et al., 2021) and FedRoD (Chen & Chao, 2021) propose to learn global feature extractor and personalized classifiers. While FedRoD conducts inference with different classifiers, they are manually switched according to the prior knowledge about the source of test data, which is also impractical in real-world FL. All of these works only consider PMs in PFL settings, which is impractical in real-world FL, because clients might meet various test data. To address this problem, HLFL select suitable PMs according to the test data during the test time,
Test-time adaptation & domain adaptation methods in FL. Some works (Peng et al., 2019; Liu et al., 2021) focus on generalizing a federated model trained on multiple source domains to unseen target domains. FedTHE (Jiang & Lin, 2023) discussed test-time distribution shift of PMs, which similar to but different from generalizing on global test data. These methods enhance federated models by better training schemes, which is orthogonal to our method. FedTHE & FedTHE+ (Jiang & Lin, 2023) discuss test-time distribution shift, which is similar to our problem setting. Recently, some methods propose to select suitable modules based on the inputs in Large Language models (Lai et al., 2025). However, we narrow down the category of distribution shift to apply to the GFL setting and perform much better in our proposed circumstance, while their method mainly aims dealing with

---

Published as a conference paper at ICLR 2025
unknown distribution shift. This is also the differences between work and all methods applying TTA directly on local client training, therefore we only experimentally compare FedTHE and our work, as FedTHE significantly outperforms this type of works. Different from them, HPFL is the first FI. framework that flexibly selects PMs for inference. Due to the limited space, we leave a more detailed discussion of the literature in Appendix A.
EXPERIMENTS
Table 2:
Experiment results. Noisy coefficient k=1. S: we focus more on GFL setting. Numbers ForestGreen highlight highest values GFL setting. *: FedAvg fine-tunes the whole model instead partial model as in HFFL; FodSAM fine-tunes partial model as in HPFL; L; For these two mothods, we only list the best performance in E, 1 & 10 and denote which epochs get the values in subscript. Plug-in selection is implemented with MMD. Ep denotes the epach of fine-tuning. We provide
bars of experiments in Table 11.
Clients
10 (sample 50% each round)
100 (5% each round)
Non-III
Dir(0. I)
Dir(0.05)
Dir(O.1)
Dir(0.05)
Test Set
GFL
PFL
GFL
PFL
GFL
PFL
GFL
PFL
Method/Model
GM PM
PM
GM PM
PM
GM PM
PM
GM PM
PM
CIFAR-10
FedAvg
62.4
73.6
31.66t
93.4cm
FedPer
74.1
40.9
95.8
58.7
27.3
96.4
44.5
20.6
89.7
24.0
14.3
89.9
FedRoD
87.4
FedRep
85.1
51.3
95.6
73.2
30.2
85.3
66.5
27.4
89.3
59.2
20.4
89.1
PorFodMask Ey
23.4
83.1
53.8
15.6
82.1
87.6
FeaTHH
86.2
64.4
93.4
68.7
31.1
85.7
75.0
23.8
90.8
66.9
44.5
90.1
FodSAM Ey
96.0
33.2m
96. 7:00:
50.4
90.3.0
23.3m
91.3mm
HPFT. F.p
81.5
95.4
95.4
62.4
96.0
96.0
73.6
88.6
94.9
47.9
82.2
93.9
HP32 E, 10
81.5
95.7
95.7
73.6
85.7
95.7
47.9
95.3
FedAvg FedPer
97.8.to
98.4kmm
73.5
39.0
87.5
64.1
27.5
99.1
69.0
29.1
95.9
44,8
22.6
96.8
FedRoD
98.5
FedRep
87.0
43.0
97.5
74.7
39.5
98.0
88.2
72.4
97.9
84.4
59.6
98..3
PerFedMask Eg
93.5
FedTHE
87.9
69.4
96.8
70.7
55
98.5
88.5
83.1
97.5
84.7
74.6
97.6
FedSAM
86.3
76.9m
98.55m
(MM)) Fp
86.0
98.3
98.3
76.
99.0
99.
90.2
97.6
97.9
86.
$1.4
98.1
SPFL(MMD) E.
FodAvg FedRoD
69.1
79.5t:
59.7
66.7/10
75.
69.4
32.5
77.2
67.0
23.6
78.5
52.8
11.2
55.4
48.4
7.3
66.3
FedRop
47.9
18.6
56.5
65.3
PerFeeMMusk Ep
47.3
7.0
40.0
49.4
7.0
39.7
41.7
3.8
35.8
42.1
3.6
35.2
FodTHE
FedSAM Eg
& 10*
68.4
57.4m
85.6kmm
64.1
43.0g)
88.8 mc)
41.3
27.3mm
71.10m
34,8
18.4m
77.3km
(MMD)
74.8
81.2
AFF L/MM) Ep
68.6
72.2
85.7
65.3
73.9
S8.8
59.7
55.7
84.1
47.9
70.9
86.4
Liny-ImageNel
FodAvg E, & 10"
56.5
69. St
54,9
75.3
47.2
67.5
42.1
68. (8.9
PedPer
16.3
0.5
0.5
13.4
0.5
2.4
1.8
23.5
1.3
25.1
1.0
FodRoD
$7.5
26.1
68.5
55.3
12.9
$2.9
48.6
49.3
9.6
43.7
5.9
53.7
FedRep
18.6
PaPFedMask Ey FedTHE
26.9
6.6
35.9
23.2
4.2
31.3
29.9
1.9
23.5
18.7
1.6
32.6
FodSAM Ey = & 10"
57.0
48.6m:
75.1mm
55.0
42, 1m
78.2
43.8
30.4:b
69.3
38.0
21.1.0
72.0
APE (MMD)
50.7
71.3
74.7
LPFL(MMD) Ey 10
56.5
50,9
73.7
54.9
58.8
77.0
47.2
48.0
73.2
42.1
43.9
76.5
5.1
EXPERIMENT SETUP
Federated Datasets and Models. We conduct experiments on four commonly used image clas- sification datasets in FI., including CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), Fashion-MNIST (Xiao et al., 2017), and Tiny-Imageti
(Le & Yang, 2015), with Latent Dirichlet Sampling (Dir) partition method (a 0.1, 0.05) to simulate data heterogeneity following (He et al., 2020b; Li et al., 2021b; Luo et al., 2021; Tang et al., 2022b). We also evaluate the scalability of proposed methods with different numbers of clients (M 10. 100). We implement and experiments based on the popular FL framework FedML (He et al., 2020b; Tang et al., 2023a). We train ResNet- 18 (He al., 2016), MobileNet (Howard al., 2017) and simple-CNN all
algorithm
7

---

Published as a conference paper at ICLR 2025
datasets. We run all algorithms for 1000 communication rounds, with 1 local epoch per round. Hyper-parameters and more details are explained in Appendix C.
Baselines and Metrics. We compare HPFL with GFL algorithms FedAvg (McMahan et al., 2017), FedSAM (Qu et al., 2022); advanced PFL algorithms including FedPer (Arivazhagan et al., 20120019 9a), FedRep (Collins et al., 2021), PerFedMask (Setayesh et al., 2023); FedRoD (Chen & Chao, 2021) both for GFL and PFL: and a test-time adaption method FedTHE (Jiang & Lin, 2023). For all algorithms, we validate the learned global model (GM) on the global test dataset (GFL), the personalized models (PM) on the personalized dataset (PFL), and PMs on GFL. More specifically, our new GFL-PM test setting is: for all clients, we randomly assign the local test data encountered by the clients with equal probability, i.e. vi.j E (1,.., M), Pr(D Diest, PPL) 1 1/M, where Dj D test data IID with local training data on client j as local test data in PFL. More details about metrics are stated in Appendix C.
5.2 EXPERIMENT RESULTS
HPFL consistently outperforms baselines in PM GFL while comparable with classic PFL methods in classic personalized setting. As shown in Table 2, in GFL-PM setting, HPFL excels above all methods and most by large margin, even surpasses accuracies in GFL-GM in most cases, while baselines perform poorly due to a lack of adaption to test data. We attribute the significant performance gain to adaptation test data implemented with precise plug-in selection, which we discuss in Section 5.3. It is worth noting that FedTHE also attempts to adapt its model using test data, but only with the ensemble of its locally personalized and global classifier, thus ignores knowledge from other clients and underperforms HPFL. In terms of GFL-GM accuracy, HPFL actually shares the same GM with GFL backbone training method (in our case, i.e. FedAvg), so its GFL-GM accuracy is exactly the same as that of FedAvg and outperforms the classic PFL algorithms focusing PFL performance like FedPer (Arivazhagan et al., 2019a). As for PFL-PM accuracy. our proposed method HPFI. reports comparable results the PFL. baselines.
HPFL maintains fairly excellent robustness against non-IID degree. As shown in Table 2, the accuracy of HPFL is not only highest in GFL-PM, but also increases when the heterogeneity increases from (ir(0.1) to (ir(0.05) in a similar way as in PFL-PM in some cases. From this phenomenon, we infer that HPFL exploits local information from clients ensemble a model in the form of plug-ins. The server holds these local information in the form of plug-ins instead of fusing these local knowledge in a single model. thus prevents the original local information from being corrupted in model aggregation as it occurs in highly heterogeneous data, and maintains a robustness against non-IlI,, which is a common issue in Federated Learning.
HPFL has excellent scalability in terms of per- formance in accuracy. HPFL adopts a one- client-one-plug method to better modify final inference models according to the data distribu- tion of clients" local data. In this way, HPFL has inherent ability to allow more clients to come and go freely in the FL system. From Table 2, we observe that other PFL methods met extreme problems when dealing with the situation that the number of clients was larger (M=1000, with most of the accuracies lower than 30% on CIFAR- 10, 20% on CIFAR-100, However, though with a little decay in accuracy, HPFL is still applicable in the situation where the system included larger number of clients.
Table 3: Results with diffcrent architectures.
Architecture
Mobilenet
Simple-CNi
Method/Model
GM
PM
GM PM
PM
FedAvg
55.7
92.3
64.6
85.4
FedPer
53.7
10.0
10.0
44.1
27.6
85.5
FedRoD
76.3
36.1
92.3
67.1
28.8
83.5
FedRep
74.1
35.8
85.0
54.6
10.0
10.0
PerlFedMusk
13.0
19.0
76.4
31.5
10.0
50.5
FedTHE
76.3
45.4
82.7
67.1
45.1
70.9
HPFI.
55.7
92.8
92.8
64.6
87.8
87.8
generalized framework applicable dif- ferent model architecture. As a general FL framework, HPFL can be seamlessly applied to model architectures where parameter decou- pling is available. We deploy it on three different model architectures (ResNet- 18, MobileNet (Howard et al., 2017), and simple-Chi CNN structure whose architecture is the same as simple-CNY in (Tang et al., 2022b)), and HPFL outperforms baselines use in the main experiment with all of the architectures, showing that HPFL. can be extensively
Table 4: Accuracy different k
10
100
1000
Accuracy
95.4
95.4
95.4
95.4
95.4
8

---

Published as a conference paper at ICLR 2025
employed in different FL systems and improve their performance of GFL and adaptation ability to new clients. Results are in Table 3. Moreover, HPFL can exploit backbones trained with all kinds of GFL algorithms. An ablation study GFL methods used to learn feature extractor of HPFL is demonstrated in Appendix D.6.
A win-win deal: Efforts to protect privacy is not contradictory to the performance of HPFL. In HPFL, clients share plug-in markers with the server, which may raise privacy concern. To protect clients from the risk of data breaches during communication improper storage on the server, we add noise to the plug-in markers. However, we surprisingly found that noise will not damage the performance of HPFL as shown in Table We attribute the robustness toward noise to robust selection method of HPFL, which we study later in Section 5.3. Discussions and experimental results about the privacy risk against HPFL, are shown in Appendix
k= 10
k 1000
Client ID
0.50
0.8
0.75
0.50
0.25
0.25
0.40
Figure 3: Selection score maps with different noise coefficients. Blocks with green anchor mean the corresponding client selects the plug-in and download it. Blocks with green anchor lying in diagonal indicate that clients choose plug-ins of themselves when met their test data, which conforms the aim of selection methods. X-axis represents Plug- in ID.
layer
# layer
layer
layer 13
layer 17
1.00
3.40
0.33
0.30
0.40
Client
0:75
0.25
4.25
0.50
0.25
0.25
M23
0.20
0.20
Figure 4: Selection score maps with different numbers of layers ResNet-18 plug-ins own. X-axis represents Plug-in ID.
5.3 SELECTION ACCURACY
The more flexible the models are, the better? As shown in Figure 2, the accuracy of HPFL continuously decreases with the increasing num- ber of plug-in layers, propose two possible reasons leading to the phenomenon: (I) local clients' samples
95
90
Aecuracy (%)
not sufficient for training big-scale plugs, resulting severe overfitting is-
80
- PFL
sue, and (2) The selection methods may not be
75
--
PFL F 10
suitable for markers from middle layers. How- believe that fine-
MMD
ever, according Table
E,
tuning larger plug-ins does not lead to such performance degradation, because FedAvg fine- tunes on the whole model without significant performance loss. Therefore, natural to give attention to the potential trouble large plug-ins may cause in plug-in selection. In Section 5.3, conduct experiments testify the specula- tion that the performance loss when increasing the plug-in layer mainly due to the degradation of plug-in selection. Due to the page limit, we aim provide intuitive explanation in Appendix D.
ORL
9
13
17
Figure Accuracy with different numbers plug- in layers. X-axis represents the number of layers in ResNet-18 for plug-ins.
Plug-in in selection plays an important role in HPFL, so here we study how it gets affected by the magnitude of noise added on features and the number of plug-in layers. Experiments in this section are carried out with a=0.1, 1=10 on CIFAR-10 dataset, we include the results of additional configurations in Appendix D.3.

---

Published as a conference paper at ICLR 2025
We observed the expected phenomenon conforming to conjecture in Section 5.2 that it is harder for selection methods to correctly select plug-ins with more layers. With the increasing number of plug-in layers, the score map gradually changes. However, until it actually influences the result of selection, the performance of HPFL, gets unaffected.
Observed from Figure 3, despite the slight variation in the heatmaps of MMD score with the noise coefficient, selecting plug-in with the lowest MMD score instead of combining plug-ins with MMD score adds robustness towards noise HPFL. We shows the accuracies under different in Table 4.
5.4 FEDERATED CONTINUAL LEARNING
Federated continual learning (FCL) (Yoon et al., 2021) is a new problem where clients join FL training after initial training. The trained model must retain previous dataset knowledge and per- form well on data from newly arrived clients.
Table 5: Results of FCL
GM
PM
GM
PM
Naive FCL
69.5
58.4
FCL under HPTL
62.2
80.9
HPFL can address the forgetting problem of FCL by preserving previous training knowledge in a personalized plug-in and providing it for client inference shown in Table 5. It is application of HPFL on the temporal scale, where clients collaboratively learn models that generalize well over time. We present more details about the experiment and discussion in Appendix D.5.
APPLICATIONS
One-shot FL. With an average backbone such as pre-trained model, HPFL can train plug-ing in single communication round like the OFL (Tang et al., 2024b), immediately proceeding to inference. This approach also accommodates new clients joining the FL system. Anarchic FL. HPFL supports the dynamic in anarchic FL (Yang al., 2022), where clients join and leave unpredictably. It operates without the need for immediate aggregation, thus allows clients to train and upload plug-ins asynchronously without disturbing server operations or model convergence with stale updates. FL plug-in market. HPFL provides the possibility of constructing more free and transparent model market, and customers can have better confidence knowing the plug-in they are purchasing is able to meet their requirements with fair plug-in selection mechanism. Plug-in providers can obtain commercial benefits from this market.
LIMITATIONS
Scalability of Plug-In Selection. When the number of clients increase thousands more, selecting plug-in module will consume lot of time and computing resources. The proposed plug-in selection methods in this paper need to communicate and select from M sets of plug-in Ø. Future works may consider to cluster and merge Mc plug-in modules together, reducing the number of plug-ins as M. If we set M .,, the HPFL becomes the FedAvg of GFL. The challenge here is to design an appropriate aggregation method, which can make the merged plug-in in performs well on according test datasets. Moreover, we can also think about skipping some plug-ins to reduce the computing overhead. This needs clients to provide more representative attributes of the plug-ins.
Accurate Plug-in Selection. In many experiments, our selection method choose optimal or nearly However, as an initial trial, our proposed plug-in selection methods select sub-optimal plug-ine -in in some circumstances as shown in Figure 4, 12 and 9 etc.. Future works may consider to design more accurate and robust selection methods.
CONCLUSION
In this paper, we explore how to improve the generalization performance when PMs meet test data from other clients. We formalize the SFL to bridge the GFL and PFL together. Then, We propose HPFL to practically solve the SFL. We verify the effectiveness and robustness of HPFL through comprehensive experiments. And we further experimentally verify the remarkable potential of HPFL to resolve other practical FL problems like FCL. Future work can consider to explore new plug-in selection methods, applying HPFL into more FL related problems. Applying HPFL foundation model is also an interesting future direction, especially with the Parameter- Efffciien Fine- -Tnnngg (PEFT) techniques, such as LoRA, adapter prefix tuning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
LS, YGZ and BH were supported by NSFC General Program No. 62376235, RGC Young Collabora- tive Research Grant No. C2005-24Y, Guangdong Basic and Applied Basic Research Foundation Nos. 2022A15 15 1501 1652 and 2024A151501 MSRA StarTrack Scholars Program, HKBU Faculty Niche Research Areas No. RC-INRA-IG/2223/SCINN and HKBU CSD Departmental Incentive Scheme.
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. CoRR, abs/1912.00818. 2019a. URL http: / ///axxiv.rrg/abb..
00818.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. CoRR, abs/1912.00818 2019b. URL http:
///axxxvv.rg//ab/aaa/.... 00818.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis of self-supervision, or what we can learn from a single image. In ICLR, 2020.
Borja Balle and Yu-Xiang Wang- Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning, pp. 394-403, PMLR, 2018
Wenxuan Bao, Tianxin Wei, Haohan Wang, and Jingrui He. Adaptive test-time personalization for federated learning. Advances in Neural Information Processing Systems, 36, 2024.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Eonmunicationn-fffiieet Learning of Deep Networks from Decentralized Data. arXiv e-prints, art. arXiv:1602.05629, February 2016.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. arXiv preprint arXiv:2004. 11791, 2020.
Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heteroge- neous collaborative learning with black- box knowledge transfer. arXiv preprint 112iv:1912.11279, 2019.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. International Conference on Learning Representations, 2021.
Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogencous ensemble knowledge transfer for training large models in federated learning. In Proceedings of the Thirty-Firs International Joint Conference on Artificial Intelligence (IJCAI) Main Track, 2022.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representa- tions for personalized federated learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2089-2099. PMLR, 18-24 Jul 2021.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, 2020.
Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes, 2020.
11

---

Published as a conference paper at ICLR 2025
Xuefeng Du, Zhaoning Wang, Mu Cai, and Sharon Li. Towards unknown-aware learning with virtual outlier synthesis. ICLR, 2022. URL IW7d65uYu5M.
net wtw.:/rrttteturreo/e /tf./ooommm//
Cynthia Dwork. Differential privacy, In International colloquium on automata, languages, and programming, pp. 1-12. Springer, 2006.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4)221-400) 2014.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint 2XXi::2002.07948., 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR, abs/1703.03400, 2017. URL http: 03400.
aaw.aaxiiivoooo http://twww..ggg.......// abs /1703.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gra- dients how easy is it to break privacy in federated learning? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 16937-16947. Curran Associates, Inc., 2020. URL https: /ttp:://prroeeee http//poooeeenn.ee/eeeee neurips wwwroceeedings..eureeeeeeeee
#ile/c4ede56bbbd9e66 aat.aaeaeaaaaee11eeeeeeeeceeeeee
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242-2251, PMLR, 2019.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586 19597, 2020.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. pp. 63-77. Springer-Verlag, 2005.
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models, 2020.
Weituo Hao, Mostafa El-Khamy, Jungwon Lee, Jianyi Zhang, Kevin J Liang, Changyou Chen, and Lawrence Carin Duke. Towards fair federated learning with zero-shot data augmentation. In Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pp-
3310-3319, 2021.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. In Advances in Neural Information Processing Systems 34, 2020a.
Chaoyang He. Songze Li, Jinhyun So. Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang. Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr, Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007. 2020b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint wrXiv:1503.0253. 2015.
Steven CH Hoi. Doyen Sahoo, Jing Lu. and Peilin Zhao. Online learning: A comprehensive survey. Neurocomputing. 459:249-289, 2021.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04661, 2017.
12

---

Published as a conference paper at ICLR 2025
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In Advances in Neural Information Processing Systems, 2021. URL httpa: // ///oppnrevieww.. net/ wwtpeeee//////tee
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyum Kim. coammumccatinn-effiienn on-device machine learning: Federated distillation and augmentation under non-iid private data. NeurlPS, 2018.
Liangze Jiang and Tao Lin. Test-time robust personalization for federated learning. arXiv preprint arXiv:2205. 2022.
Liangze Jiang and Tao Lin. Test-time robust personalization for federated learning. In International Conference on Learning Representations (ICLR), 2023.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint vrXiv:1912.049977 2019.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-210 2021.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, arXiv preprint arXiv:19000063 2019.
Julian Katz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training OOD detectors in their natural habitats. ICML, Proceedings of Machine Learning Research. PMLR, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A.
Rusu, Kicran Milan, John Quan, Tago Ramalho, Agnieszka Grabska Barsska-BBriiskaa, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):351-3352, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference of Machine Learning Research, pp. 3519-3529. PMLR, 09-15 Jun 2019. URL https://
Machine Learning, volume 97 of Proceedings
proceedings mmlr. wrwps/eesseeee. / https//////kknoo /ooooooobiiii hthmmbliihhhh 9a html.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, 2009.
Kunfeng Lai, Zhenheng Tang, Xinglin Pan, Peijie Dong, Xiang Liu, Haolan Chen, Li Shen, Bo Li, and Xiaowen Chu. Mediator: efficiett Ilm merging with less parameter conflicts and uncertainty based routing, 2025. URL https: http::// //axivaxivv///////////aa/////avr org /bb///////////// 2502. 04411.
John Langford, Alexander J. Smola, and Martin Zinkevich. Slow learners fast. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, NIPS'09 pp. 2331-2339 Red Hook, NY, USA, 2009. Curran Associates Inc. ISBN 978161567919..
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015. URL https:
// //api.. semanti washhollcccllllleo org/ :Dtg://oopu::::::emm 16647676666...
Ang Li, Jingwei Sun, Binghui Wang, Lin Duan, Sicheng Li, Yiran Chen, and Hai Li. Lotteryfl: Empower edge intelligence with personalized and communication-effiiien federated learning. In 2021 IEFF/ACM Symposium on Edge Computing (SEC), pp. 68-79, 2021a. doi: 10.1 14514////33142 3492909.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv: 1910.0358/ 2019.
13

---

Published as a conference paper at ICLR 2025
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713- 10722, 2021b.
Tan Li, Linqi Song, and Christina Fragouli. Federated recommendation system via differential privacy. arXiv preprint XXiv:20000066700 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, volume pp. 429-450, 2020b. URL https: https:://poccedddnne
myssys.o .rgg haag//pp//////// Paper.pde
Tian Li, Shengyuan Hu. Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In ICML,, 2021c.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Moreney. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01522 2020.
Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In NeurlPS, 2020a.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems, 33:2351-2365 2020b.
Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space, 2021.
Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tianjian Chen, Han Yu, and Qiang Yang. Fedvision: An online visual object detection platform powered by federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13172-13179, 2020.
Zelei Liu, Yuanyuan Chen, Han Yu, Yang Liu, and Lizhen Cui. Gtg-shapley: Efficient and accurate participant contribution evaluation in federated learning. ACM Trans. Intell. Syst. Technol., 13(4), may 2022. ISSN 2157-6904.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference on computer vision, pp. 2200-2207, 2013.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pp. 2208-2217. PMLR, 2017.
Jiahuan Luo, Xueyang Wu, Yun Luo, Anbu Huang, Yunfeng Huang, Yang Liu, and Qiang Yang. Real-world image datasets for federated learning. arXiv preprint 1rXiv:1910. 1089, 2019.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-500 5984, 2021.
Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni. Personalized federated learning through local memorization. In International Conference on Machine Learning, pp- 15070-15002. PMLR, 2022.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-cffici learning of deep networks from decentralized data. In Artificiaal Intelli- gence and Statistics, pp. 1273- 1282,1282,11 2017.
Kang Loon Ng, Zichen Chen, Zelei Liu, Han Yu, Yang Liu, and Qiang Yang. A multi-playen game for studying federated learning incentive schemes. In IJCAI International Joint Conference on Artificial Intelligence, pp. 5279, 2020.
14

---

Published as a conference paper at ICLR 2025
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint wwxfv:220..118999 2022.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation. In International Conference on Learning Representations, 2019.
Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference Machine Learning, pp. 18250- 18280. PMLR, 2022.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singu- lar vector canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: pttp///poooeeeeeeeeeeeeeeec neurips cc/paper/2017/EEle// 2ee4440055533366666555 9ee67 (11p667cc69--ppppp pdf.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv e-prints, art. urXiv:1606.04p0 04671, June 2016.
Felix Sattler, Klaus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8):3710-3722 2020.
Mehdi Setayesh, Xiaoxiao Li, and Vincent W.S. Wong. Perfedmask: Personalized federated learning with optimized masking vectors. In Proc. of International Conference Learning Representations (ICLR), Kigali, Rwanda, May 2023.
Harshay Shah, Andrew Ilyas, and Aleksander Madry. Decomposing and editing predictions by modeling model computation. arXiv preprint arXiv:2404. 1534, 2024.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In ICML, 2021.
MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning. arXiv preprint arXiv:2006.051 051488, 2020.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. Proceedings of the 22nd ACM SIGSAC conference computer and communications security, pp. 1310-1321, 2015.
Rachael Hwee Ling Sim, Yehong Zhang, Mun Choon Chan, and Bryan Kian Hsiang Low, Col- laborative machine learning with incentive-aware model rewards. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020.
Virginia Smith, Chao-Kar Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances neural information processing systems, 30, 2017.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML, Proceedings of Machine Learning Research. PMLR, 17-23 Jul 2022.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-17, 2022a.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning.
IEEE Transactions on Neural Networks and Learning Systems, 2022b.
Yue Tan, Chen Chen, Weiming Zhuang, Xin Dong, Lingjuan Lyu, and Guodong Long. Is het- erogeneity notorious? taming heterogeneity to handle test-time shift in federated learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
15

---

Published as a conference paper at ICLR 2025
Zhenheng Tang, Shaohuai Shi, Bo Li, and Xiaowen Chu. Gossipfl: A decentralized federated learning framework with sparsified and adaptive communication. IEEE Transactions on Parallel and Distributed Systems, pp. 1-13, 2022a. doi: 10.109/TPDS.20222233333.
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 21111-21132. PMLR, 17-23 Jul 2022b.
Zhenheng Tang, Xiaowen Chu, Ryan Yide Ran, Sunwoo Lee, Shaohuai Shi, Yonggang Zhang, Yuxin Wang, Alex Qiaozhong Liang, Salman Avestimehr, and Chaoyang He. Fedml parrot: A scalable federated learning system via heterogeneity- aware scheduling sequential and hierarchical training. arXiv preprint arXiv:2303.01778 2023a.
Zhenheng Tang, Yuxin Wang, Xin He, Longteng Zhang, Xinglin Pan, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Bingsheng He, et al. Fusionai: Decentralized training and deploying Ilms with massive consumer-level gpus. arXiv preprint wwXiv:2309.0117., 2023b.
Zhenheng Tang, Xueze Kang, Yiming Yin, Xinglin Pan, Yuxin Wang, Xin He. Qiang Wang, Rongfei Zeng. Kaiyong Zhao, Shaohuai Shi, Amelie Chi Zhou, Bo Li, Bingsheng He, and Xiaowen Chu. Fusionllm: A decentralized IIm training system geo-distributed gpus with adaptive compression, / / 2410 12707.
2024a. URL https: aariv. org/ abs/
Zhenheng Tang, Yonggang Zhang, Peijie Dong, Yiu ming Cheung, Amelie Chi Zhou, Bo Han, and Xiaowen Chu. Fusefl: One-shot federated learning through the lens of causality with progressive model fusion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https: http:////peeeeeeeeeee net. forum? i id=E7 [20o 1EK1.
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xinmei Tian, Tongliang Liu, Bo Han, and Xiaowen Chu. Fedimpro: Measuring and improving client update in federated learning. In The Twelfth International Conference on Learning Representations, 2024c.
Stacey Truex, Ling Liu, Ka-Ho Chow, Mehmet Emre Gursoy, and Wenqi Wei. Ldp-fed: Federated learning with local differential privacy. In Proceedings of the third ACM international workshop on edge systems, analytics and networking, pp. 61-66, 2020.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective incon- sisteney problem in heterogeneous federated optimization. In Advances in Neural Information Pro- cessing Systems, volume 33, pp. 761 1-7623, 2020. http:::::: / //proceed/ ings neuri ps. pss. cc/paper/2020 htt//file///////
paf.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE transactions on information forensics and security, 15:3454-3469, 2020.
Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems, 33:6281 16292, 2020.
Q. Wu, K. He, and X. Chen. Personalized federated learning for intelligent iot applications: A cloud-edge based framework. IEEE Open Journal of the Computer Society, 1:35 44, 2020. doi: 10.1 1L.100/0OCS.222222999333
Zhaomin Wu, Qinbin Li, and Bingsheng He. A coupled design of exploiting record similarity for practical vertical federated learning. Advances in Neural Information Processing Systems, 35: 21087-21100, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprini 17Xiv:1708.07747, 2017.
Cong Xie, Sanmi Koyejo, and Indramil Gupta. Asynchronous Federated Optimization. arXiv e-prints, pp. 10XXvv1903.000 March 2019.
16

---

Published as a conference paper at ICLR 2025
Haibo Yang, Xin Zhang, Prashant Khanduri, and Jia Liu. Anarchic federated learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25331-25363. PMLR, 17-23 Jul 2022.
Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. Fedfed: Feature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36, 2024.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018. URL
pettps://openrevaee
.nt/ forumfrrdeeeoroooo wwwerrummiiskTTn fwO-
Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12073-12086. PMLR, 18-24 Jul 2021.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, 2014.
Han Yu, Zelei Liu, Yang Liu, Tianjian Chen, Mingshu Cong, Xi Weng, Dusit Niyato, and Qiang Yang. A sustainable incentive scheme for federated learning. IEEE Intelligent Systems, 35(4): 58 69, 2020a. doi: 10.1 109/MIII..S. 20t..22000.227....
Tao Yu, Fugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.04758, 2020b.
Rongfei Zeng, Chao Zeng, Xingwei Wang, Bo Li, and Xiaowen Chu. Incentive mechanisms in
federated learning and game-theoretical approach. IEEE Network, pp. 1-7, 2022. doi: 10.1 109/ MNET. 1 12.210006..
Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In International Conference on Learning Representations, 2020.
Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie-Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In ICML. JMLR, 2017.
Yibo Zhou. Rethinking reconstruction autoencoder-basee dusto--dist detection. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7379-7387, June 2022.
17

---

Published as a conference paper at ICLR 2025
APPENDIX
A MORE RELATED WORK
A.1 GENERIC FEDERATED LEARNING
The convergence problem of FL. with high non-III data distribution has always been an important problem in improving the performance of models trained with FL.. To resolve this problem, Fed- Prox (Li et al., 2020b) and MOON (Li et al., 2021b) propose to add new model regularization terms mitigate the client drift caused by data heterogeneity. There are also some methods modifying the uploaded gradient to alleviate the dissimilarity of gradients (Wang et al., 2020; Karimireddy et al., 2019; Tang et al., 2022a; 2023b). With a level of privacy protection, some works propose to share intermediate features (Jeong et al., 2018: Hao et al., 2021) or extra data (Tang et al., 2022b; Shin et al., 2020; Lin al., 2020a) to reduce the gradient variance. Some one-shot Fl. methods propose to only communicate models with one round, thus significantly reducing the communication time (Tang et al., 2024b).
A.2 PERSONALIZED FEDERATED LEARNING
Different from the GFL methods that aim to directly reduce the gradient dissimilarity, PFL exploits the heterogeneous data to personalize client models to better suit the local training data.
Recently, several works have proposed to apply Model-Agnostic Meta-Learning (Finn et al., 2017) to Federated Learning for faster adaptation on local training data in clients. The Model-Agnostic Meta- Learnnng (Finn et al., 2017) (MAML) aims to meta-learn a global model, which will be broadcasted to different users to learn a local model adapted to different datasets. Per-FedAvg (Fallah et al., 2020) makes use of MAML to learn personalized models more efficiently. It first finds an initial global shared model with second-ordes order gradient information, and then the global model is fine-tuned by local models with only several iterations suit the local datasets.
Knowledge distillation is also used to promote efficient local adaptation of personalized models (Yu al., 2020b). Specifically, a federated teacher model GT and an adapted student model Gs are defined with the same structure. G3 is initialized with Gr, which has been trained through a common dataset shared across clients. And G's trained by local private datasets, However, in this method, the global model GT won't get optimized as time goes on. It is more like a local fine-tuning technology rather than federated learning. Some works (Hanzely & Richtárik, 2020; Dinh et al., 2020; Deng et al., 2020) utilize some regularization and adaptive model mixture to learn personalized models. FedMD (Li & Wang. 2019) proposes a federated learning framework based on knowledge distillation using a shared dataset, on which clients transfer knowledge through mimicking the outputs of other client models. With knowledge distillation, it allows clients to independently design their own model architectures with their local private datasets.
In addition to the expected performance of personalized models, there are also works aiming at addressing the problems personalized models may meet when applied in reality. Ditto (Li et al., 2021c) adds the regularizer measuring the difference between personalized models and the global model into the objective functions to guarantee both the fairness and robustness of personalized models.
Apart from the usability of personalized models, the accessibility of personalized models is also key consideration when it comes to real-world applications. Considering the situations where clients have heterogeneous environments like datasets, hardware, software, and the Internet, there are too many unpredictable situations in the real world blocking the access of personalized models. To solve these problems, some works (Wu et al., 2020; Li et al., 2021 a) propose to allow clients to learn different personalized model structures. LotteryFL (Li al., 2021a) proposes to let clients individually learn a lottery model, which a subset of the global model. During the communication, these lottery models will be shared between servers and clients. Without the requirement of communicating global model, this method can significantly reduce the communication cost in its training process. PFedHN (Shamsian et al., 2021) also makes clients learn a sub-model based on the global model.
Recently, there have also been many works exploring personalizing parts of models instead of the whole model to improve the performance of personalized models. I.-FeeAvg (Liang et al., 2020)
18

---

Published as a conference paper at ICLR 2025
proposes to share the upper layers (model head) in the DNN and personalize the bottom layers (base model), which will not be averaged during the training. It utilizes personalized base models to output different local features in different clients, on which the global model head will be collaboratively trained through the FedAvg. Conversely, FedRep (Collins et al., 2021) proposes to learn global feature extractor and personalized classifiers. FedRoD (Chen & Chao, 2021) proposes two-predictor framework in which clients train different model heads to switch between GFL and PFL.
Different from their work, Our framework considers a more challenging FL setting, i.e. every client may meet OOD test data from other clients. Moreover, instead of improving the performance of the model itself, we consider more about how clients collaborate to handle the unpredictable test data.
A.3 INCENTIVE MECHANISM
The purpose of FL collaboration among clients is the improvement of model performance on the test data. Therefore, is important to know how much performance gain can be obtained after FL collaboration (Ghorbani & Zou, 2019; Liu et al., 2022; Sim et al., 2020). Furthermore, there should be a well-designed incentive mechanism (Ng et al., 2020; Yu al., 2020a; Zeng et al., 2022) that motivates clients to join FL. Some works consider to implement decentralized training platforms to let geo-distributed collaborators to contribute their computing resources to jointly train a large language model (Tang et al., 2024a; 2023b). Our modular store essentially provides market economy to let clients autonomously choose and download the needed plug module. The higher the generalization performance of the plug-in in module, the more favorable it is. Therefore, the incentive mechanism of the modular store is naturally connected with the practical benefits of the plug-in module.
A.4 FEDERATED CONTINUAL LEARNING
Continual learning (CL) (Kirkpatrick et al., 2017) is to learn different tasks sequentially. Some former tasks are inaccessible after training. Thus, when training subsequent tasks, the machine learning model may forget previous tasks. EWC (Kirkpatrick et al., 2017) finds the model parameters that good for both previous and subsequent tasks using the Fisher Information Matrix. Progressive Neural Network approach (Rusu et al., 2016) is increasingly construct the model during the training. Thus, the newly added parameters can learn the new tasks, while the old parameters can remember the old tasks. DEN (Yoon et al., 2018) dynamically decides the model capacity to learn a compact overlapping knowledge sharing among tasks.
Federated Continual Learning (FCL) (Yoon et al., 2021) a new problem where, after FL training on some clients, there are some other clients that come and join the FL training. The trained model needs to avoid forgetting the previous dataset while performing well on the later dataset with data from newly arrived clients. We use simple example to show that HPFL is naturally suitable to address the forgetting problem of FCL. Our plug-in can not only be seen as a personalized part of the model helping clients do inference on test data but also considered knowledge obtained from training. So it is natural to think we can store the knowledge in the previous dataset and access it whenever we are in need. In fact, it can be seen the temporal scale. Most of the works in FL talk about many clients in single period of time, i.e. Federated Learning in the spatial scale. FCL itself can be seen as problem that happens at Federated Learning within the temporal scale: clients from different times collaboratively learn models that can generalize well on circumstances varied with time. We experimentally verified the potential of HPFL, to address the forgetting problem of FCL in Section 5.4. Details about that experiment and more discussion are presented in Appendix D.5.
container preserving
application of HPFL on
A.5 ASYNCHRONOUS FL
Asynchronous FL (Async-FL) (Xie et al., 2019) means to ease the constraint of the synchronous communication mechanism of classic federated optimization schemes (McMahan et al., 2017). In Async-FL, clients may download the global model from and return gradients to the server at different times. Thus, the server may receive a stale model update, causing unstable convergence. Such a staleness problem has long existed in the distributed machine learning area (Langford et al., 2009; Zheng et al., 2017). Stale updates are usually controlled by some staleness coefficients (Xie et al., 2019) compensated by (Zheng et al., 2017) other newer gradients. Anarchic FI. (Yang al.,
19

---

Published as a conference paper at ICLR 2025
2022) be seen more extreme version of Async-FL. In Anarchic FL, clients decide to download and upload the models at any time, not controlled by the server at all. To this end, HPFL naturally allows this kind of working paradigm since once average backbone, which can be obtained from pre-trained models or summoning several active clients to train, is accessible, any aggregation operation is not demand for HPFL,, the server doesn't t rely timely respond of client and won't be disturbed by stale model update. Once a plug-in is updated by the client, the plug-in can be utilized to do inference on appropriate test data without concern that the parameter of the model will change over time.
A.6 TEST-ADAPTATION & DOMAIN ADAPTATION METHODS IN FL.
There also emerge works that aim adapt or generalize new unseen clients with seen or unseen data distribution. FADA (Peng et al., 2019) utilize domain adaptation to tackle with seen target distribution. However, their method requires target domain data train an adversarial model and thus cannot handle the situation where the target domain is unknown. FedDG (Liu et al., 2021) first proposed novel setting where a federated model trained on multiple distributed source domains is required to generalize on unseen target domains. However, these methods all aim to train unified global model for adaptation or generalization to new clients. As far as we know, HPFL framework to directly exploit PMs to achieve this goal. TTPFL (Bao et al., 2024) proposed using unlabelled test data from new clients to personalize global model by adaptively learning each module in the personalized models on unlabelled test data. However, personalized model, which is done simply with fine-tuning in our experiments, instead we concentrate on how make use of all clients' personalized models collaboratively. Recently, some methods propose to select suitable modules based on the inputs in Large Language models (Lai et al., 2025). Therefore, personalized models learned in (Bao et al., 2024) can also be utilized in HPFL further boost its performance.
first FL
pay little attention to how obtain
B PROOF
B. LOWER BOUND OF PM WITH GFL
Theorem B.1. With Equation 4 and the PMs obtained from optimizing Equation 2 as: ΩPl Opei ..... g arg arg min........... Cr(Ω, 0I.... OMt), we have
LP-o 01, .... 0m) 2 CP(DP opn (
Proof.
Cr-G 01 1i..0..., 0M)
Pm B -D (ff(:,,..((. R(2, 010,001,000 01, Ans)
M
M
Σ m=1
Pin Be DD (e, e(f(Om, Em), + 0,, )M.))
M
1=1
M
(10)
Pm Pm
A (((0 (m) R(22, (200 Omr)
m=1
Dm OmEEmDDe (((( E R 0p5i (pfL)
CPPPf 0pf1 "Pf,,..,0'! (Pfl).
which completes the proof.
B.2 THE EQUIVALENCE BETWEEN SFL AND PFL
Theorem B.2. With Equation 4, SPIll pppt ...., "'.,,0't = arg ming........ Lr(S O,..., Onr) and the
H' that guarantees Opfi = s(O, Em, H), we have
Lp-c(Ω, .1, 0m) Cr( Ls(0.H))
(11)
20

---

Published as a conference paper at ICLR 2025
Proof.
Ls(0,H))
Σ Pm Beg DD ((f(() ;.
Σ Pm R ~D,, ( Sm), 5m
(12)
Cr(PPf 0p .... ( R(SPI OP ...,, 0
S C 0p ...., B
combining with Theorem 2.1, which completes the proof.
B.3 DIFFERENTIAL PRIVACY
In this section, prove that our protection scheme in Section 3.3 can provide (e, 0DD Privacy guarantee for the transmitted markers, and will not leak the information about the raw data. To prove this conclusion, we first introduce the Gaussian mechanism of Differential Privacy (Dwork et al., 2014):
Theorem B.3. V(e,8) E (0, 1), the Gaussian mechanism M(x) f(x) + N(0, o²) provides (c, DP privacy protection with
ô = A2V 2l.(212//12//)
According [5], the traditional Gaussian mechanism can be extended to Theorem 4 support > 1.
Theorem B.4. Ve > and < 5 < 1/2 - e 3r /eee/////00 //2Te, the Gaussian mechanism M(x) =
[(x) ((0, o²) provides (e, DDP privacy protection with
o 2 A2/V2e.
With Theorem B.3, we can estimate the magnitude of Gaussian noise needed to be apply with certain function as 0 < e < 1. Then, found that for our protection scheme g in section 3.3, we have: Theorem B.5. V(c, 8) E 1),
v 22mmm1.25/ * Jm
c=0
8
the procedure is (c,8)-122
Proof. First, we calculate L2 sensitivity A₂ in our protection scheme g in Section 3.3. Assumption B.6. Let f(-) the backbone (also called feature extractor), is local data extracted from local training data D, i.e. D, then f(x) is the raw features, we have
Vz E D. -C < f(x) < C, where C is constant.
Assumption often assumed protecting gradient with DP as in (Abadi et al., 2016; Shokri & Shmatikov, 2015), then
Aaf max (1((x) f(x))122 < 2C2
Recalled from Section 3.3, with pm and of separately denote the mean and variance of the raw features, and denotes the noisy coefficient, our protection scheme is formed as
M(x) (f(x) * * ( = 1//1+==))))0/0))))((0>)))>))) /
pm + +0 x2 xx
21

---

Published as a conference paper at ICLR 2025
The bound of pm can be obtained with the definition of mean:
- pm
(f(z)) <0
=EDm
Denote g(x)) = 1/(1+ +k) f(x) + k/ N)) * pm have
tx E D, - < ( = 1/(1 + x) f(x) + n/(1 + 1)) pm < C
Therefore,
Ag = max - ((())) < 402.
(13)
Derived from Theorem B.3, Lemma B.7 is obtained:
Lemma B.7. V(e, 6) (0, the procedure g is (e, 6) DP if
2ln(1.25/c)
A29
By rearranging variables in Lemma B.7, we have Lemma B.8: Lemma B.8. 6)c 1). for
21n (1.25//)
A2g
8
the procedure g is (c,8)-DP with M(x) f(x) N()
Here Gaussian mechanism's Theorem B.9. V(e, 6) E (0, for
ms therefore we have
21n 21 (1.25/0))
2ln(1.25/8)
A2g
4C
K.
the procedure g (c.
which completes the proof.
With Theorem B.4, can estimate the magnitude of Gaussian noise needed be apply with certain function as e > 1. Then, we found that for our protection scheme g in section 3.3, we have:
Theorem B.10. Ve 0 and 0 < 6 < 1/2 - 3e / if
33e
e 2 O
2( K T
the procedure is (c,8)-DP.
Proof. According to Equation 13, it holds
Aag < 4C2.
Then, According Theorem B.4,
22

---

Published as a conference paper at ICLR 2025
Lemma B.11. Ye and < < 1/2- c- -0/// the procedure is (c, DP if
2 22/ 4C2 /V22.
By rearranging variables in Lemma B.11, we have
Theorem B.12. and á 1/2 - e3e 2T,, for
16C
16C
2
(14)
20
2 om)
the procedure g ()-P)
which completes the proof.
Combining Theorem B.5 and Theorem B.10, we get
Theorem 3.1. For the procedure of obtaining and sharing markers H(rm) (g(Im) K
v)/(1 + 1i), (c,6)-DP holds if the e, conforms to any of the two conditions: (1) Ve, E (0, 1).e
0 V
; < 1 e ////////// //2Te, 2 0 (((xxom)))))
C EXPERIMENT CONFIGURATION
C.I HARDWARE AND SOFTWARE CONFIGURATION
We conduct experiments using NVIDIA A100 40GB GPU, AMD EPYC 7742 64-Core Processor Units. The operating system is Ubuntu 20.04.1 LTS. The pytorch version 1.12.1. The numpy version is 1.23.2. The cuda version is 12.0.
C.2 IMPLEMENT OF SIMPLIHED METRICS AND PROOF
The original metric under the GFL-PM setting in classification tasks should be:
M
M
Accuracy (Ω, 0, 0m)
Pm. (J (m),
(15)
M
m=1
where T(.,)) is the function judging whether the prediction of the model is the same with the real label, specifically
(pprediction, sample) = 1 (predciton = -/samppe)
(16)
where Yeample is the label of sample. f (6:, Em) is the prediction of the model used in final in-
ference on client for the sample ε.. the model is parameterized with 0,. And our way of de- termining personalized model using when inferencing on client is to select from all the plug- ins, i.c. the PMs obtained from optimizing Equation arg ming, ....... Lr(2, .1,... BMa), so we have
local data: Dpft goft ...., gaft
0: = gpst
C.(Em.)): E 1,2, M
(17)
where Ca(Em, i) is the selection made for client based on test data &m and client i. C₁ is the selection algorithm of the client i.
For traditional personalized methods, the clients will only use personalized models trained locally, i.c.
Ci(Em, i)
(18)
substitute Equation 18 into Equation 17, we have
gprl ()(Emi))
(19)
23

---

Published as a conference paper at ICLR 2025
then substitute Equation 19 into Equation 15, we have Accuracy (ft, 01. AM1) =
M
M
Pm D' (0,, Em)
M
m
Pm Eg Dm T (f (00% .5m) .5m)
M
M
Rm
T (s (0p5)) :5:) (Es)
Pm
Tmm
j=1
Tem
(8pt) :5s) ,E)
M
N Tm
M ΣΣΣN
(5 .E)
MN j=1
i=
M ΣΣT( ( (B05) (Es)..s)s
MN
j=1
E BL (f ((p!! EED EDD)
(20)
accuracy of PM in client on global data
Equation 20 represents the averaged accuracy of all personalized models on the global dataset, so we can calculate the averaged accuracy of all personalized models on the global dataset the metrics of simplified metrics instead of original complicated metrics 15; while for our proposed methods HPFL, because clients have same selection method C
Ci(Em, i) argmaz g Sm) = C(5m)
(21)
n
the origin metric turns into
M
1
Accuracy (2, 0
0m)
Pm Be Din T (S (Oar (Oargmax, g(En Em) Em)
M
1=1
Pml Be D'n T (s (pofe Em) :E
M
M
M
Σ T (f
( .E))
Pm nam
M
j=1
nmm
T (p (ootl
(ootem.) Es) .E,
N Tmm
j=1
M
Non ΣΣΣT
M
T (f Es) ,Es)
MN
m==li=
M
(oets (oere :Es)
N
nm
Ppm
(5
(onft
7m
m=1
j=1
Da
Σ
Σ
T (f(o)t)))))
Pm
Tmm
24

---

Published as a conference paper at ICLR 2025
M
Σ Pm B Bm T T( (Barl (s .kg)
m=1
M
Σ 1=1
Pi RE ( )5)
(22)
accuracy selected by elient on its own data
Equation 22 represents the averaged accuracy of clients testing on their own personalized dataset with models equipped with their selected plug-ins based on their own data, weighted with number of samples in data on clients. With these simplification of metrics, we can more efficiently test GFL-PM performance of both the traditional personalized methods (FedPer, FedRoD, FedRep) and HPFL.
C.3 SYPPERPPARMEES
We SGD without momentum the optimizer for all experiments, with batch size of 128 and weight decay of 0.0001. The learning rate is set as 0.1 for both the training of the global model and the fine-tuning on local datasets. The main results shown in Tabel 2 are conducted with 1-layer plug-ins (i.e. only classifier).
Special hyperparameters of some baseline methods
FedRep: Local personalize epoch is set as 1.
ParFedMask: The partition percent of validation is 0.1, personalized fine-tuning epoch E,, after calculating mask is set as as the official implementation did.
FedTHE: We follow the official implementation: the smoothing factor of test history descriptor main- tained by the Exponential Moving Average (EMA) a equals 0.1; the smoothing factor interpolating the test feature and the test history descriptor 3 equals 0.3.
FedSAM: We follow the official implementation: the parameters for SAM minimizers 0.1, 1
C.4 EXTRA EXPLANATION ON EXPERIMENTS
Due to the limited space of the main text, we show more detailed explanation of the experiment in this section.
For the construction of the personalized test dataset, make the training data and test data of client have the same distribution following the settings most PFL methods (Collins et al., 2021), l), count the number of samples Strain (c, m) in each class of training data of client mn and split test data of that clients in that distribution (which means client m have >
Strein(e.m)
2mm1 Eess Strain(a,m) X Em=1 >e= Sceat(c, m) test samples in class c, here C denotes the number of classes in overall dataset, M denotes the number of clients the FL system), Figure and Figure 6 shows that the data partition of training data and test data almost identical as expected in PFL..
To report the best result of all baseline methods, we report the accuracy of their best inference global model on global data during the whole training process. For our method, we also use the best inference model as the backbone of HPFL for fair comparison.
D EXTRA EXPERIMENT RESULTS
Due to the limited space of the main text, we show more experiment results in this section.
D.1 MORE RESULTS ABOUT DIFFERENT NUMBERS OF LAYERS OF PLUG-IN
In this part, we are trying to explain why the selection method degrades when the number of plug-in layers increases in an intuitive way. We propose two possible ways leading the degradation: (1) selection methods can't handle the large dimension of markers that vastly increases when the number of plug-in layers increases, as shown in Table 6. (2) A larger number of plug-in layers means
25

---

Published as a conference paper at ICLR 2025
Class matrix of trainning data (a=0.1)
Class matrix of test data (a 0.1
2947
3866
38.0
116
666
0
341
705
76
288
23
93
0
68
141
328
27661
280
28:17
0
0
66
552
56
563
0
1721
99
18
19
Client ID
39
252
52
484
0
174
1
51
11
97
0
35
Client
5845
B052
6d9
0
1205
0
3962
0
841
792
0
393
674
0
1029
347
410
0
134
0
206
70
82
45499
19
910
21
555
0
1657
5
1385
0
0
0
111
332
1
5.7
Class
Class
Figure 5: Data partitioning on CIFAR- 10 (a=0.1)
Class matrix of trainning data 0a=5. Class matrix of test data (a 0.05)
155
4608
921
0
24
3490
98
3403
0
9
0
699
19
680
0
0
1217
1220
242
Client ID
Client ID
736
0
5342
6779
0
0
136
0
0
0
4153
3536
850
1419
4009
990
11
208
4799
0
0
41
1000
0
74
Class
Class
Figure 6: Data partitioning on CIFAR-10 (a=0.05)
using markers extracted with more shallow layers, and these markers tend to more local, which may not be so helpful for the selection method to assess the similarity of the distributions they are sampled from. For further study, we may conduct experiments to testify these two conjectures. Once testified, we will try to find ways to solve these two problems. However, despite
the conjectures the difficulty of choosing, large plug-ins also multiply the computation time and resources needed in training them, the network bandwidth required to transmit them, and so on. As a result, large plug-ins are generally not good options in HPFL from our perspective.
We also explore selection with plug-ins inn composed of different numbers of layers in different settings, like o=0.05, M=10 on CIFAR-10. Figure and Figure 7 show that with the number of plug-in layers going up, the selection becomes more difficult and unstable as claimed before. Table 6: # marker dimensions versus # plug-ins layers on CIFAR-IO.
plug-in ins layers
3
512x4x4 (8.192)
256x8x8 (16,384)
128x 16x 16 (32,768)
64x32xx32 32
# marker dimensions
512
(131,072)
D.2 MORE RESULTS ABOUT SELECTION METHODS (MMD, SVCCA, CKA, OOD)
MMD, SVCCA and CKA. MMD (Long et al., 2013) is a kernel-based statistical metric used to determine whether given two distributions are the same. SVCCA (Raghu et al., 2017) exploits singular value decomposition and canonical correlation analysis to compare the markers learned by different DNNs. CKA (Korbblth et al., 2019) utilizes the normalized HSIC (Gretton et al., 2005) to measure the similarity. CKA is invariant to the invertible linear transformation. Thus, it can measure meaningful similarities between representations of high dimension (Kornblith al., 2019).
26

---

Published as a conference paper at ICLR 2025
layer=
layer=3
layer=4
layer=
layer=6
0.40
0.50
Client ID
0.75
0.50
0.30
0.40
0.50
0.25
0.20
0.20
0.20
Plug-in
Figure 7: Selection score maps with different numbers of plug-in layers on CIFAR-10 (a 0.05)
Here we summary the formulas of MMD, SVCCA and CKA to better explain our method:
Definition of MMD
Given observations X = ( F and Y = {111 )n
MX,X m(m
m
m (os, + n(n - 1ml
Σ
n 12
(vi,
2
mL
(z). Us).
1=1
where k(-) is the kernel function.
Definition of CKA
Let K and K' be two kernel functions defined over X'x X such that < E [ < too and
0 < E K < +oo. Then, the alignment between K and K' is defined by
E K
p (K,
VE K K
Definition of SVCCA
1. Input: X, Y.
2. Perform: SVD(X), SVD(Y). Output: X = UXXY VY.
3. Perform CCA(X', Y'). Output: X" X Y Wy Y and corrs (p1,..., Pmin (m m2) where my and m2 is the number of samples of X and Y.
OOD detection based methods. Out-of- -attof-diitrrbbttion (OOD) detection aims find whether the test data is OOD or not. Current OOD) detection methods includes the norm of gradients (Huang al., 2021), distance-based methods (Sun al., 2022), reconstruction based methods (Zhou, 2022), classifier based methods (Katz-Samuel, et al., 2022: Du et al., 2022). Intuitively, if the test data OOD one plug-in, can discard this plug-in. Thus, based on this insight, we train an OOD classifier Tm for each plug-in Pm- Each OOD classifier will output an OOD confidence Tm (heest). Lower Fm (heest), less possible that hrest is OOD to the plug-in Pm. Then, the select problem turns from Equation 6 into:
SooD(heasts TI, TM) arg min Ti (bttst))
(23)
iEM
The Tm is trained during the process of optimizing Pm- On each client m,, we generate random noise < (O, 14) as the OOD data, the fim is seen as the in-distribution (ID) data. OOD data has label 1, and ID data has label O. We use a linear classifier as the OOD classifier Ty whose input dimension is same with h. and the output dimension is 1. For OOD data, we hope the T. outputs 1. We use the cross-entropy loss to train the OOD classifier. Different from the distance based methods, OOD detection based method does not need to communication the markers hmm. which significantly increases the privacy security.
We show the results of our previous attempts using SVCCA, CKA, and OOD as the selection methods in this section. The overall training and inference process is the same as mentioned in Section 3. When implementing SVCCA in M=100, we encountered the problem that the number of samples in client was not enough for SVCCA, which required at least. N Ncomponent components. To deal with
27

---

Published as a conference paper at ICLR 2025
Table 7: Experiment results HPFL using SVCCA, CKA and OOD. Noisy coefficient x=1, FedAvg is fine-tuned with the whole model instead of only part of model in HPFL.
Clients
10
100
Non-IID
Dir(O.1)
Dir(0.05)
Dir(O.1)
Test Set
GFL
PFL
GFL
PFL
GFL
PFL
Method/Muulel
GM PM
PM
GM
PM
GM
PM
CIFAR-10
EPFLISVCCA) E,
81.5
62.8
95.4
62.4
32.7
96.0
73.6
61.0
95.0
BPFL(SVCCA) E,
81.5
62.5
95.8
62.4
34.7
96.3
73.6
47.0
95.7
HPFL(Linear-CKA) Eg
56.1
95.4
55.2
96.0
73.6
70.5
95.0
(EenneLaar-CAA Ep
81.5
55.1
95.8
62.4
44.6
96.3
73.6
60.3
95.7
(PFL.RBF-CKA) Ep
81.5
61.0
95.4
62.4
55.2
96.0
73.6
70.7
95.0
HPFL(RBF-CKA) Ep
10
81.5
55.9
95.8
62.4
44.6
96.3
73.6
59.9
95.7
HPFI(Lineor CK 12K'Adeo Adebiasl
Epp
81.5
63.9
95.4
62.4
47.2
96.0
73.6
66.3
95.0
APFL(Linear CK Adebias)
Ep
10
81.5
59.0
95.8
62.4
37.7
96.3
73.6
53.0
95.7
HPFL(RBF CK Adchias)
Ep
81.5
61.0
95.4
62.4
35.0
96.0
73.6
68.4
95.0
HPEL(RBF CK Adebias)
E,
56.7
95.8
40.2
96.3
73.6
55.7
95.7
(PFL(OOD) Eg
66.3
95.4
62.4
54.4
96.0
73.6
64.0
95.0
HPFL(OOD) E,
81.5
16.0
95.8
62.4
3.9
96.3
73.6
27.9
95.7
FMNIST
HPFL/SVCCA) E,
86.0
61.8
98.3
76.1
41.7
99.0
90.2
90.0
97.9
APFL(SVCCA) Ep = 10
86.0
49.7
98.4
76.1
49.3
99.2
90.2
87.7
98.8
HPFL(Linear CKA) Ep
86.0
67.6
98.3
76.1
73.2
99.0
90.2
89.1
97.9
HPF PeFLLLinarr-CK Ep
86.0
62.0
98.4
76.1
65.5
99.2
90.2
88.6
98.8
#PFL(RBF-CKA) Ep
86.0
67.6
98.3
76.1
73.0
99.0
90.2
89.4
97.9
HPFL(RBF-CKA) Ep
86.0
63.3
98.4
76.1
65.5
99.2
90.2
88.7
98.8
APFL(Linear C. Adebsas)
Ep
86.0
66.6
98.3
76.1
44.7
99.0
90.2
89.9
97.9
HPFL(Linear CK Adatiiu)
Ep
98.4
37.7
99.2
88.3
98.8
HPFL(RBF Adebias)
Pip
86.0
61.2
98.3
76.1
50.7
99.0
90.2
89.9
97.9
HPEL(RBF CK Adstiii)
Ep 10
86.0
51.8
98.4
76.1
27.9
99.2
90.2
87.6
98.8
(PFL(OOD) Eg HPFL(OOD) E,
86.0
83.9
98.3
76.1
73.6
99.0
90.2
87.2
97.9
86.0
42.7
98.4
76.1
37.5
99.2
90.2
88.4
98.8
CIFAR 100
HPFL/SVCCA) E.
68.6
68.2
83.3
65.3
68.2
87.4
59.7
51.8
81.2
BFFL(SVCCA) E, 10
68.6
55.2
85.7
65.3
55.2
88.8
59.7
39.9
84.1
HPFL(Lincar-CH) (enear-KKK))))C Ep
68.6
63.4
83.3
65.3
63.4
87.4
59.7
51.0
81.2
HPFI(Linear-CHe CKA)) Ep
68.6
55.0
85.7
65.3
55.0
88.8
59.7
40.0
84.1
(RFE-CB)
68.6
64.1
83.3
65.3
64.1
87.4
59.7
51.4
81.2
HPFL(RBF-CKA) Ep
50.9
85.7
50.9
88.8
59.7
84.1
HPFL(Limeur C. Atthraa)
Ep
62.8
83.3
87.4
59.7
50.7
81.2
HEFerLinear CK Adebias)
Ep
10
68.6
52.6
85.7
65.3
52.6
88.8
59.7
38.8
84.1
HPFL(RBF CK Adchias)
E
68.6
62.9
83.3
65.3
62.9
87.4
59.7
50.8
81.2
HBFL(RBF CK Adebias)
E,
68.6
55.9
85.7
65.3
55.9
88.8
59.7
38.4
84.1
HPFL(OOD) Ep HPFL(OOD) E,
68.6
63.5
83.3
65.3
63.5
87.4
59.7
49.5
81.2
68.6
66.7
85.7
65.3
66.7
88.8
59.7
46.5
84.1
it, we change the number of components used to calculate SVCCA similarity between the plug-in (apppplenuu_samppee...._.__
markers and task markers min(min(num, sample), Neamponent). From Table 7 and Figure is
easy to conclude that SVCCA, CKA (with 4 kinds of kernel), OOD all failed to select correct plug-ins. Take HPFL based on OOD detection an example, OOD classifiers failed give good predictions on whether test data is OOD or not for plug-ins because in federated learning, it is difficult to get OOD data as negative samples when training locally, which will cause OOD classifiers only see in-distribution(I)) data. We tried to solve this problem by generating random images as OOD data to train the OOD classifiers. It is easy to see this method did not work as shown in Figure 8. Therefore, it is difficult to tell whether there is an appropriate way to train OOD classifiers to determine whether OOD detection can be utilized in HPFL. There we propose a possible way to train OOD classifiers: after clients upload the plug-in markers to the server, it is possible utilize these markers to train good OOD classifier and select plug-ins using these classifiers. We take this as a future direction in exploring more selection methods that can be used in HPFI.
28

---

Published as a conference paper at ICLR 2025
D.3 MORE RESULTS ABOUT SELECTION ACCURACY
In this part, we take closer look at the selection accuracy using MMD by visualizing all the selection situation CIFAR-10, FMNIST, CIFAR-I00, with three settings [a=0.1,M=100) {a=0.05, M==0)) (0=0.1,M=1000 as arrange from left to right in Figure 9, 10, 11. The colors of the blocks denote the MMD scores, where red represents relatively high score, and blue represents relatively low score. The green box on the block(i.j) implies that client is choosing plug-in with minimal MMD score for local test data. MMD always helps clients select plug-in trained on the client itself (the green boxes denoting final choice of plug-ins all lie on the diagonal of score heatmap) when a=0.1, M=11 and a=0.05, a=0.05, MM110 CIFAR-IO and FMNIST as shown Figure 9, Figure 10. When it comes to FL system with more clients, like M=100, even though there is some clients choosing inappropriate plug-ins, ins, most of them can still choose plug-im ins trained on their own data. However, When conducted more heterogeneous dataset CIFAR- 100, judging which plug- ins choose becomes more difficult task, which can be easily observed in Figure 11, as the increasing number green boxes not located the diagonal is indicating worse plug-ins selection. Additionally, the score map of CIFAR- 100 is overall much whiter than the score map of CIFAR-10 and FMNIST, denoting that the scores of different plug-ins are close to each other and more challenging choose from when MMD encountered with CIFAR-100. Thus, how to improve selection methods used in HPFL L is crucial problem when met more heterogeneous data together with fewer samples on each client, and will be important for future work.
SVCCA
lincar- -KA
lincar- CKa dcbias
0.06
0.05
0.01
Client ID
0.01
0.04
0.03
0.01
0.02
0.02
0.01
Plug-in ID rbf-CKA
rbf-CKA debias
personalized with OOD
1.00
0.02
0.08
1.00
0.01
0.06
1.00
1.00
0.04
0.00
1.00
0.02
1.00
Figure 8: Selection score maps of SVCCA, CKA, OOD on CIFAR-10 10 (a = 0.1)
a 0.1, M=10
a 0.05, M=10
0=0.1. M-100
1.00
0.80
1.50
Client ID
0.80
0.60
0.60
1.00
0.40
0.50
0.20
Plug-in ID
Figure 9: Selection maps CIFAR-10
D.4 MORE RESULTS ABOUT NOISE
When using the HPFL methods based MMD, it is required that the distribution of local features, or local features are transmitted together with the plug-in trained in the client, so that the other clients able select appropriate plug-ins based on these information. However, transmitting features
29

---

Published as a conference paper at ICLR 2025
0=0.1, M=10
α 0.055 M=10
a=0.1. 0. M=100
4.00
3.00
3.00
Client ID
2.00
2.00
2.00
1.00
1.00
1.00
Plug-in ID
Figure 10: Selection score maps on FMNIST
a 0-1., M-10
a=0.05, M=10
0.1, M=100
0.34
0.60
0.35
0.32
Client ID
0.33
0.30
0.50
0.30
0.28
0.40
0.28
0.26
0.25
0.24
0.30
Plug-in in ID
0.23
Figure 11: Selection score maps on CIFAR-100
faced with the risk of data leakage when met inversion attacks. order to better protect privacy safety, tried to add Gaussian noise generated with the distribution of local features to the origin features, surprisingly found that adding noise according to the distribution of the features not damage the performance, according Table 8. Further study may transmit Gaussian noise generated with the distribution of the local features instead of the noised features. In fact, when A reaches a high value like 1000 in Figure 3, the noised features can be approximately considered degenerate into the Gaussian noise. From Table 8, Figure and Figure 12, we can observe increasing to a large value doesn't hurt much performance of HPFL. Therefore, we will explore using pure Gaussian noise generated with the distribution of the local features to replace the noised features to better protect privacy in the future.
D.5 EXPERIMENT ABOUT FEDERATED CONTINUAL LEARNING
With the increasing real applications of Federated Learning, Federated Continual Learning (FCL) has attracted the attention researchers. In this part, we conduct an experiment to display the potential of HPFL to solve catastrophic forgetting met in FCL. We first displayed the catastrophic forgetting issue in naive FCL. Then we utilized HPFL to solve this problem. Suppose we had clients the FL. system. We first trained 500 epochs on client 0-4 with FedAvg, and then we trained another 500 epochs on client 5-9. We trained the backbone of HPFL and the global model naive FCL in Nvidia V100 GPU and the rest of the experiment on Nvidia A100. For naive FCL, We had to adjust the learning rate to 0.05 when training on client 5-9 during 500- 10000000-o0 epoch in case of training divergence. For FCL under HPFL, froze the backbone of the model after training 500 epochs training 5 plug-ins on client 0-4 for epoch, respectively. Then we kept training on client 5-9 with the invariant backbone, after another 500 epochs, we trained 5 plug-ins on client 5-9, respectively. From Table 9, we observe that the accuracy of naive FCL significantly drops from 78.6 to 52.8, showing that training on clients 5-9 during 500-1000 rounds makes the global model severely forget the knowledge about clients 0-4. We show promising way of using HPFL to mitigate this problem. When met new task, HPFL allows clients to quickly adapt to their local data by fine-tuning only a few epochs and uploading the plug-in to the server, like what happened at the 500 round in our experiment. After training in some new tasks, it is about time to conduct inference on all clients, we train plug-ins every client. In that case, we are able to select and download the plug-ins ins better suited for test data with similar distribution, instead of having knowledge of previous tasks. As is shown Table 9, our experiment shows HPF can significantly outperform naive FCL in GFL and mitigate the catastrophic forgetting issue in FCL.
client 0-4 and
tasks, we do clients 5-9 in our experiment, and select plug-ins for
choice but
global model having forgotten the
30

---

Published as a conference paper at ICLR 2025
Table 8: Accuracy Different noise coefficient on CIFAR-10.
Noise coefficient
10
100
1000
Fine-tune epoch Etaoe
10
10
10
10
10
0.1, .M
10
Accuracy
95.4
95.7
95.4
95.7
95.4
95.7
95.4
95.7
95.4
95.7
0.05, M
10
Accuracy
96.0
96.3
96.0
96.3
73.7
74.0
71.1
70.5
71.1
70.5
0.1, M
100
Accuracy
95.4
92.0
95.4
95.7
95.4
95.7
95.4
95.7
95.4
95.7
10
100
x-1000
20
3.00
Client ID
0.75
0.75
1.00
1.00
.000
0.00
0.25
0.25
0.60
0.40
0.50
Plug-in ID
Figure 12: Selection score maps with different noise coefficient on CIFAR-10 (a=0.05, M=10)
D.6 MORE RESULTS ABOUT BACKBONE TRAINING METHODS
As long as the used GFL methods are able to train a strong general feature extractor, HPFL is able to utilize the feature extractor to train the personalized plug-ins and extract features. We conduct experiments using FedRoD to lestify HPFL's compatibility with other GFL methods. The results are given in Table 10. Number of clients equal M 10, local fine-tuning epoch E, 10, local datasets are partitioned in (0r(0.1). Other settings remain the same as the main experiments in Table 2.
From the overall performance of HPFL(FedRoD), we can see that HPFL using FedRoD its backbone training method is comparable to that using FedAvg, which confirms HPFL is compatible with the GFL methods other than FedAvg. We also observe an interesting fact that even if FedRoD shows excellent performance in GFL-GM (surpasses FedAvg g in many datasets and settings), fine- tuning the backbone trained with it is not advantageous as shown PFL-PM (only comparable with fine-tuning the backbone trained with FedAvg). From this phenomenon, we presume the advantage of FedRoD in GFL-GM should mainly be attributed its global head trained with a Mass-balanced loss instead of its backbone.
D.7 ERROR BARS
Due to time and resource limits, we can only provide error bars of part of our numerous experiments, the results are shown in Table 11. All experiments are conducted with Cifar- 10 dataset, partition with (ir(0.1), involved 10 clients. Other settings follows that of Table 2. We calculate the error bars with the standard error of the mean.
From the table above, we can observe that HPFL's 's error significantly lower than FedSAM, which shows the robustness of our method HPFL when encountering different random simulations.
DISCUSSION ON PRIVACY PROBLEM
As HPFL requires local clients to share auxiliary information local data and plug-ins help inference, may raise concern about data privacy of HPFL. We attempt to analyze the risk of privacy leakage in HPFI. respectively from sharing auxiliary information and plug-ins.
31

---

Published as a conference paper at ICLR 2025
K-10
k=100
k=1000
Client ID
1,50
1.50
2.00
2.00
2,00
1,00
1.30
150
0.50
.00
Plug-in in ID
Figure 13: Selection score maps with different noise coefficient on CIFAR-10 (o=0.1, M=100)
Table 9: catastrophic forgetting issue in Naive FCL.
Algorithm
Naive FCL (500R)
Naive FCL (1000R)
Test data
data from Client 0-4
Method/Model
GM
Accuracy
78.6
52.8 (5
E.1 PRIVACY RISKS OF SHARING PLUG-INS
In HPFL, we ask local clients to upload part of their personalized models to the server, which means every personalized model is possibly accessible to all clients. This potential sharing with other clients will raise concerns about the risk of privacy leakage. However, in classic Federated Learning algorithms like FedAvg, there also exists similar behavior of sharing global model, and it is difficult to recover training samples from the final model shared over the whole FL system. Instead, research shows that it is possible recover training data of clients from gradients transmitted the server (Geiping et al., 2020), which will not happen in HPFL except for the training period of the backbone model, which is able to be solved with regular privacy protect techniques like differential privacy (DP) which is widely used to protect potential privacy risks of GFL. algorithms, and not special problem of HPFI. Even extreme concern on potential privacy risk of storing plug-ins in the server can be solved by only requesting plug-ins ins after selection as described in Appendix F.2, clients providing the plug-ins can ask the server to delete the plug-ins after sending the plug-ins to the clients in need.
E.2 PRIVACY RISKS OF SHARING AUXILIARY INFORMATION
Since HPFL asks clients to share auxiliary information with the server, once data breach happens in the communication period between clients and the server or the information is not properly kept in the server, the leaked information may lead to attacks, such as feature inversion attacks. Here we resorted image reconstruction by feature inversion method in (Zhao et al., 2020) to check whether the raw image can be reconstructed by inverting the representation through the pretrained global backbone model parameters, exploring whether data privacy will be threatened if both auxiliary information and backbone model is leaked. Experiments are conducted on CIFAR-10 10 with a ResNet- 18 trained on CIFAR-10 under the Federated Learning setting used in our main experiments. As we can see in Figure 14, if we use raw features as the auxiliary information, real pictures can be easily recovered by feature inversion methods. To handle this risk of privacy leakage, here we propose three ways to prevent the problem: (1) add noise to the features; (2) use the averaged feature to select the plug-ins; (3) use model-based selection methods like OOD.
Adding noise to transmitted information is often practiced in the Federated Learning called Differential Privacy(DP). which is utilized protect gradient against Differential attacks. Inspired by DP, we attempt to add noise to the transmitted auxiliary information, and below we use the same recovery method to recover the original image from the markers. More specifically, we add Gaussian noise to protect the features from privacy risk, which is commonly-adopter method for (e, -DP. Plenty of previous works (Luo et al., 2021; Li & Wang, 2019; Hao et al., 2021; Chang et al., 2019) transmit noised features to exchange auxiliary information without privacy leakage. As DP used in Federated Learning is mainly for protecting FL system from differential attacks (also called membership inference attacks) (Dwork, 2006; Wei al., 2020; Truex et al., 2020), which attempt to get information on membership based on differences between models in different rounds. HPFL. doesn't 't involve multiple rounds communication except for traditional GFL backbone training
32

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study backbone training methods.
Clients
10 (sample 50% each round)
Non-IID
Dir(0.1)
Test Setting
GFL-PM
Method
(FFI(FedAvg)
HPFL(FedRo))
CIFAR-10
GFL-GM
81.5
85.3
CIFAR-10
GFL-PM
95.7
96.0 (1.03)
PFL-PM
95.7
96.0 (1.0.))
FMNIST
GFL-GM
86.0
87.9 (1:
FMNIST
GFL-PM
98.4
98.4 (1.0)
PFL-PM
98.4
98.4
CIFAR- 100
GFL-GM
68.6
69.9 (1.1.)
CIFAR-100
GFL-PM
72.2
68.5 (4.3.7)
PFL-PM
85.7
85.5 (4.0.2)
Tiny-ImageNet-2000
GFL-GM
56.5
57.4 (+0:9)
Tiny-ImageNet-e
GFL-PM
50.9
56.0 (1:5.)
PFL-PM
73.7
74.7 (1.100
Table 11: Error bars of partial experiments.
Fine-ture -uning Iteration
Seed=0
Seod=1
Seed=2
Error Bar
FedSAM
47.8
47.4
51.4
+ 2.20
10
42.3
42.3
45.2
+ 1.67
HPFL
1
95.4
95.3
95.4
0.06
10
95.77
95.6
95.8
+ 0.07
Raw image A
Raw image B
Reconstructed image A
Reconstructed image B
Raw image A
Raw image
Reconstructed image
Reconstructed image
Figure 14: Image reconstructed from raw features
33

---

Published as a conference paper at ICLR 2025
phase. Therefore, differential attack raise additional privacy risks to protect from in HFFL, and there is not need for HPFL to protect privacy with differential privacy. To this end, we leave out detailed discussion of differential privacy in paper. Instead, attack and show recovery results with the markers in Figure 15. As noise coefficient x increases, the reconstructed image is less similar the raw images. It is worth noting that when main experiment, the reconstructed images are already hard to tell any information about the raw images. If there is still concern in this situation, the clients can increase the noise coefficient &. to higher level with the risk of hurting performance.
focus more model inversion
as in
Raw image
k=0
k=
x=10
100
x=1000
Figure 15: Image reconstructed from the markers, k is noise coefficient denoting the scale of noise added on raw features, the bigger x is, the larger noise is added on the markers transmitted to select plug-ins.
Using the averaged feature to select the plug-ins a practical way of protecting privacy as practiced in (Luo et al., 2021), inspired by their work, attempted to select plug-ins with the average of all features on local clients. However, we assumed that simply averaging all features leads to the lack of information select plug-ins properly, thus degrades the performance of HPFL. Therefore, we tried to divide the features into groups and take the average in every group. With enough samples in every group, we can prevent privacy leakage as presented Figure 16 and maintain good performance as shown in Table 12.
Raw images
Reconstructed image
Figure 16: Image reconstructed from the averaged feature, every group is composed by raw features of 10 raw images
Table 12: Accuracy of Different average group CIFAR- 10.
# of raw features in every group
10
1
10
1
10
a 0.1, M = 10
Accuracy
81.5
80.1
76.8
79.1
0.05, M
10
Accuracy
96.0
96.1
87.6
86.1
0.1. M
100
Accuracy
81.4
83.8
76.8
75.3
Utilizing model-based selection methods like OOD to select the plug-ins, due these methods avoid sharing direct information about raw data or features, they are exposed to less risk of data leakage. It is difficult for the attacker attack the clients with the model parameters than with the data information due to less information contained it, which can be proved by the data processing inequality (McMahan et al., 2017).
34

---

Published as a conference paper at ICLR 2025
F REAL-WORLD APPLICATION
F.1 REAL-WORLD GM-PFL
To better illustrate the GFL-PM setting we propose and demonstrate its importance, we give some examples exhibiting the significance of our proposed set-up below:
Some clients may have insufficient computing resources or local training data to fine-tune a deep learning model in cross-device setting. In these situations, training distribution can be regarded as an empty set 0. In this way, the client cannot get a personalized model by locally fine-tuning the global model. In traditional GFL and PFL setting, the client has no choice but to adopt the global model and endure the lack of personalization. This problem is caused by the mismatch of training data distribution and test data distribution, assumed in our proposed set-up, and is solvable with our proposed method HPFL by exploiting personalized plug-ins from other clients.
Case I:
A car with a personalized automated driving system (ADS) has driven out of the previous city it used to be. It requires to personalize on geometric data from the present city it is now in for improving the performance of the ADS in this new city. Classic GFL and PFL in this situation leave the ADS no option but to collect the geometric data and personalize on it after the collection completes, and accept the temporary performance loss using the previous personalized model before finishing the new personalization, since the distribution of test data has greatly changed. It's another example where the discrepancy between training data (geometric data from the previous city) and test data (geometric data from the present city) threatens the availability of FL systems. While with our proposed method designed to solve the problem, the ADS can attempt to access the plug-ins from owners living in the present city.
Case 2:
Case 3: Imagine a person is traveling from a high latitude area to an equatorial region, and the recommender system their phone is supported by federated learning. If the recommender system uses the personalized model trained when in the high latitude area, it will continue to prompt thick down jackets for the person, which is clearly an unexpected and unreasonable recommendation. With our method, one can get the same recommendation as the local people with plug-ins on their phones without time to fine-tune the model again.
F.2 SCALABILITY ISSUES OF HPFL AND THEIR SOLUTIONS
F.2.1 POTENTIAL SCALABILITY PROBLEM OF SHARING PLUG-INS
For plug-ins: In fact, HPFL can be applied both cross-device and cross-silo setting, with a slight modification in cross-device setting where the number of clients is overwhelmingly large. We introduce two methods to enhance the scalability problem of HPFL as follows: 1) To handle the massive plug-ins needed to be stored in the server, the server can cluster the plug-ins with client- cluster methods in a similar way as done in IFCA (Ghosh et al., 2020), CFL (Sattler et al., 2020), FL+HC (Briggs et al., 2020), and so Then the server aggregates the plug-ins in the same clusters to keep a controllable number of plug-ins, like in O(1) or (logMM), where M denotes the number of clients. 2) Actually simpler solution can naturally originate from selection method: use selection score to measure the similarity between two plug-ins and abandon some similar plug-ins to reduce the storage space taken up by plug-ins. We carried out an experiment to validate such initial solution and found even with a such naive method, we can significantly reduce the number of plug-ins and maintain higher performance than best baseline GFL methods. As the results shown in Figure 17: (1) HPFL L are able to maintain 1/3 number of plug-ins ins and still surpass the best GFL algorithm Fed under the experiment setting (see threshold = 0.2): (2) HPFL are able achieve similar accuracy with a half number of plug-ins as that in original HPFL (see threshold = 0.1). The server can significantly reduce the number of plug-ins in both ways, thus increase the scalability of our method without much performance loss.
can
simpler method is enough for solving the issue of the massive plug-ins in clients: as our selection method doesn't 't require the presence of plug-ins, clients may not upload the plug-ins after training. Instead, the server can request the appropriate plug-in from the corresponding client after calculating the selection scores. Considering the common issue in cross-client setting, FL systems may encounter client dropout (Li et al., 2020a: Kairouz et al., 2021; Tan et al., 2022b). In a situation where the client with the most appropriate plug-in is out of connection, the server may attempt to request plug-ins one
35

---

Published as a conference paper at ICLR 2025
Plug-in Accuracy (%)
100
Accuracy (%) # Plug--n after reduction
80
FedTHE
60
40
20
0
origin
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Figure 17: Accuracy and Plug-in after picking plug-ins with selection scores. X-axis represents the threshold to eliminate plug-ins. If the normalized MMD score between new-coming plug-ins and present plug-ins lower than the threshold, the new-coming plug-ins will be abandon. Experiments are done with a = 0.1, M = 100, E, = in CIFAR-10 dataset.
Table 13: Experiment results of sharing task markers. $: focus more GFL setting. Numbers in ForestGreen highlight highest values in GFL setting. Ep denotes the epoch of fine-tuning. Other hyper- paramters follows the experiments Table
Clients
(sample 50% each round)
(5% each round)
Non-IID
Dir(0.1)
Dir(0.05)
Dir(O.1)
Dir(0.05)
Test Sct
5FL
PFL
GFL
PFL
GFL
PFL
GFL
PFL
Method/Model
GM PM
PM
GM PM
PM
GM PM
PM
GM PM
PM
CIFAR-10 10
HPF L(heaa) Ep HPF ((Feett Ep 10
81.5
95.4
95.4
62.4
96.0
96.0
73.6
91.7
94.9
47.9
85.2
93.9
81.5
95.7
95.7
62.4
96.3
96.3
73.6
90.3
95.7
47.9
85.2
95.3
FMNIST
HFFT(hies) Ep HPF H2FL 1PFL() E, 10
98.3
98.3
76.1
99.1
99.1
90.2
97.9
97.9
86.1
95.3
98.1
86.0
98.4
98.4
76.1
99.2
99.2
90.2
98.6
98.8
86.1
94.0
98.7
CIFAR- 100
HPFT((ees)) Ep
68.6
75.7
83.3
65.3
78.8
87.4
59.7
67.5
81.2
47.9
72.7
84.1
HPFL.(uue)) Eg 10
68.6
69.5
85.7
65.3
78.0
88.9
59.7
63.8
84.1
75.5
86.4
Tiny 1anyyImmaeNee-2200
HPF L(heet) Ep IPFII(heast) Eg 10
56.5
51.8
70.8
54.9
55.5
74.7
47.2
58.6
71.3
42.1
59.2
74.7
47.4
73.7
54.9
50.3
77.0
47.2
57.7
73.2
42.1
57.8
76.5
by one with the selection score. To avoid downloading all plug-ins and plug-in markers to clients, if the number of clients grows to large number, clients choose to add noise to their local test features and send the task markers to the server to select the plug-ins. In this way, each client can get the exact plug-in they need without the need download all the plug-ins and the plug-in markers,
36

---

Published as a conference paper at ICLR 2025
which will cause great communication cost with a great number of clients in the FL. We conduct experiments to test the feasibility of this method against the communication and storage burden of HPFL in FL systems with plenty of clients. The result are shown in Table 13.
F.2.2 ONLINE INFERENCE OF HPFL
Our method is designed to infer in batches, online test-time adaptation where test samples arrive one by one is not our main application scenario (Hoi et al., 2021; Jiang & Lin, 2022; Tan al., 2023). Selecting plug-ins for every sample may incur expensive computation costs. However, when met a similar situation and the computation costs are unavoidable, clients can give up downloading plug-ins. Similar to method stated in Appendix F.2. following instructions free clients from the storage burden brought by downloading plug-ins and corresponding plug-in ing markers for every sample: simply send the task marker the server; let the server select the appropriate plug-in; then infer the server side; and finally return the inference result back to the client. In this way, communication and latency issues with online inference can be solved with slight modifications in our proposed method.
37


Reviews:
[{"review_id": "YdVxjq9xn9", "reviewer": "Reviewer_VDPF", "summary": "This article proposes a new federated learning framework called Hot-Pluggable Federated Learning (HPFL), which aims to solve the performance gap problem between general federated learning (GFL) and personalized federated learning (PFL). Traditional GFL cannot cope with the diversity of data distribution, while PFL is only suitable for scenarios where local data distribution is similar. When the client encounters test data that is different from local data, PFL's personalized model has difficulty maintaining efficient generalization performance. To this end, this paper proposes a new problem framework of Selective Federated Learning (SFL), which enhances the effect of GFL by selecting an appropriate personalized model for each client in the inference stage. The HPFL framework divides the model into shared backbone and personalized plug-in modules. The client trains and uploads plug-ins based on local data. During inference, it can select appropriate plug-ins to adapt to different data distributions, while protecting data privacy through differential privacy.", "questions": "·How to balance the computational and communication overheads brought by the storage and selection of personalized plug-in modules in the HPFL framework to improve model efficiency?\n\n·Is it possible that differential privacy protection during model selection in this article, or when using other privacy protection methods, may significantly affect model performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Originality：The HPFL framework proposed in this paper innovatively introduces a plug-in selection mechanism into federated learning, realizes the bridge between general models and personalized models, and solves the performance balance problem of traditional GFL and PFL. This is a new attempt in federated learning.\n\nQuality：The experimental part is relatively comprehensive, covering a variety of data sets and model verification, and enhancing security through differential privacy. The overall design is rigorous, and the results demonstrate the advantages of HPFL in performance and adaptability.\n\nClarity：The paper is well-structured, with clear background and problem descriptions, and clear algorithm design, framework details, and experimental procedures, making it easy for readers to understand its core contributions.\n\nSignificance：This study proposed a new solution to the adaptability problem of federated learning under heterogeneous data distribution, which has practical application potential and provides a new direction for the future development of federated learning.", "weaknesses": "·Insufficient details of the selection mechanism (page 5, Section 3.3)：HPFL uses multiple distance metrics such as MMD, SVCCA, and CKA to select plug-ins, but the specific algorithm steps and implementation details are rarely described. The article can add mathematical expressions or pseudocodes for some of the selection methods to increase readability and make it easier for readers to understand the robustness of the selection process.\n\n·Selection of comparison methods (page 7&8, Table 2 and Table 3)：The paper compares a variety of GFL and PFL algorithms, but lacks a comparison of newer solutions that focus on heterogeneous data distribution problems. It is recommended to add more to further highlight the advantages of HPFL in performance and adaptability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vsNfd3a0iB", "reviewer": "Reviewer_FWNo", "summary": "This paper presents a novel framework called Hot-Pluggable Federated Learning (HPFL) that aims to bridge the gap between generic federated learning (GFL) and personalized federated learning (PFL). The authors propose a new learning paradigm, Selective Federated Learning (SFL), which combines model optimization with model selection. HPFL addresses the challenges of storing and selecting whole models by designing an efficient framework that allows clients to train personalized plug-in modules and upload them to a server. During inference, a selection algorithm identifies suitable plug-in modules to enhance performance on target data distributions. The paper also incorporates differential privacy protection during the selection process. Comprehensive experiments demonstrate HPFL's effectiveness in improving GFL performance and its potential in addressing other FL challenges like continual learning and one-shot FL.", "questions": "As shown in the Weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors identify a substantial gap between GFL and PFL, and formulate a new problem SFL to bridge them together to address this performance gap. Both optimization function of GFL and PFL are the special cases of it.\n\n2. The authors propose a general, efficient and effective framework HPFL, which practically solves SFL.\n\n3. Comprehensive experiments and ablation studies on four datasets and three neural networks demonstrate the effectiveness of HPFL.", "weaknesses": "1. It’s adorable that the authors define a paradigm to bridge the gap between GFL and PFL, but the selective method are not novelty enough. For example, [1] allowed each client to choose the appropriate scale model to train. And the training process in HPFL are identical to the split federated learning, the authors only add another select process in inference process. \n2. The authors claim that they theoretically show PMs can be used to enhance GFL with a new learning problem named Selective FL (SFL), which involves optimizing PFL and model selection. But only the statement of the loss cannot totally evaluate the effectivess of the SFL. And in the Eq.4, the greater-than sign \\geq should be a less-than sign \\leq? The better one should gain the less loss?\n3. What’s the originality in the analysis of the privacy protection? It seems that add Gaussian noise to the partial model or full model is identical. Thus I think it’s only an existing result.\n4. The notations needs to be improved. For example, in section 2.4 the definition of Selective FL (SFL) problem, the introducing of auxiliary information is confused. And in Theorem2.3, the function s(\\dot) lacks description.\n5. The presentation in experimental section should be improved. For example, the Table 2 is hard to read. It’s confused that the authors not only give the best result in grey, but also some second best result. And why the proposed HPFL does not gain the best result especially compared to the GFL FedSAM?\n\n[1] Cho, Yae Jee, et al. \"Heterogeneous ensemble knowledge transfer for training large models in federated learning.\" arXiv preprint arXiv:2204.12703 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "geKGokmR6Q", "reviewer": "Reviewer_8vJb", "summary": "The paper introduces Hot-Pluggable Federated Learning (HPFL), a framework aimed at bridging the gap between Generic Federated Learning (GFL) and Personalized Federated Learning (PFL) by proposing Selective Federated Learning (SFL). SFL optimizes PFL while allowing for the selection of personalized models (PMs) to enhance generalization performance across diverse test data. In HPFL, clients train personalized plug-in modules based on a shared backbone model, which are then uploaded to a server for selection during inference. This process also incorporates differential privacy to protect user data during the selection phase. Experimental results demonstrate that HPFL significantly outperforms traditional GFL and PFL methods, suggesting its applicability in various federated learning scenarios, including continual learning.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of SFL effectively addresses the limitations of existing PFL methods, allowing for better adaptation to real-world scenarios where test data may differ significantly from local training data.\n\n2. The HPFL framework’s modular approach enables efficient communication and computation, making it suitable for practical applications in federated learning, including scenarios with resource-constrained clients.", "weaknesses": "1. The dependency on a common backbone model may lead to reduced performance if the backbone fails to generalize well across heterogeneous client data distributions.\n\n2. The plug-in selection process during inference could introduce additional computational delays, particularly for clients with limited resources, potentially hindering real-time performance.\n\n3. While the framework claims differential privacy protection, the effectiveness of this mechanism in preventing information leakage during plug-in selection remains to be empirically validated in diverse operational contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FcYElXewAW", "reviewer": "Reviewer_kxCM", "summary": "After authors responses: The rating has been updated considering the authors' inputs and clarifications. I appreciate their efforts in providing those responses.\n\nIn order to solve the selective FL (SFL) problem, this paper leverages the model components that attempt to edit a submodel for specific purposes to design a  framework referred to as Hot-Pluggable Federated Learning (HPFL). In HPFL, clients individually train\npersonalized plug-in modules based on a shared backbone, and upload them with a plug-in marker on the server modular store. During the inference stage, a selection algorithm allows clients to identify and retrieve suitable plug-in modules from the modular store to enhance their generalization performance on the target data distribution. This paper also provides differential privacy protection during\nthe selection with theoretical guarantee. Key contributions can be summarized as:\n-\tIdentifying a major gap between Generic FL (GFL) and Personalized FL (PFL), and formulate a new problem SFL to bridge this performance \n-\tDeveloping a general, efficient and effective framework HPFL, which practically solves SFL and adding noise on communicated markers to provide differential privacy protection with theoretical guarantee.\n-\tExperiments on four datasets and three neural networks to demonstrate the effectiveness of HPFL", "questions": "Q1. What is the main contribution of the HPFL that makes it outperform existing PFL models? This could be described by adding a table with core update rule of existing PFLs [including more recent studies] and the proposed method;\nQ2. What is the difference of plug-in module and vanilla personalized model? Intuitively they seem to be the same and help considering the local model to personalize the local model of each client.;\nQ3. What is the novelty of integrating DP in the algorithm? How does it lead to advancing the proposed HPFL solution? Is it HPFL+DP or the integration has some challenges? If so, what are the challenges and how does this paper tackle them? Why can’t we use other privacy preservation mechanisms? ;\nQ4. Detailed comparison with two types of studies: i. existing PFL algorithms and comparing with HPFL, ii. Existing privacy preserving algorithms and comparison with DP;\n5. How can the plug-in marker contribute to bridging the gap between GFL and PFL?\nQ6. Can you please clarify which type(s) of differential privacy (local, central, or both) are used, and to provide a more detailed explanation of how the plug-ins interact with the DP mechanisms and what novel contributions are made in this area?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written. It fairly cites prior works that it built on and shows how it leverages those solutions. It clarifies what is the problem and what are research questions to answer.\nAfter carefully reviewing authors' clarifying points and responses to my concerns and other reviewers, I increased two of my scores.", "weaknesses": "The originality of this paper is not clear. For instance, the following are two major claimed contributions according to the paper “Identifying a major gap between Generic FL (GFL) [e.g.,  (Karimireddy et al., 2019; Woodworth\net al., 2020; Tang et al., 2022b)'s works] and Personalized FL (PFL) [e.g.,  (Li & Wang, 2019; Chen & Chao, 2021; Li et al., 2021c):'s works], and formulate a new problem SFL to bridge this performance; and Developing a general, efficient and effective framework HPFL, which practically solves SFL and adding noise on communicated markers to provide differential privacy protection with theoretical guarantee.”\nHowever, the methodology seems to be a combination of existing works on PFL while leveraging several existing work with minimal advances. It is not clear how the mentioned theorems add value to the literature, for example, this statement is vague and is not adequately explained: “solving SFL means that clients achieve performance in GFL as high as in PFL.”. It is not clear how plug-in marker can contribute to bridging the gap between GFL and PFL? It would be very helpful to explain elaborately. Specifically, please provide a more detailed explanation and concrete example of how the plug-in markers specifically help bridge the gap between GFL and PFL performance.\nWhile using differential privacy can add value to the method, it is not clear whether it can benefit from local DP, central DP, or both?  Please clarify which type(s) of differential privacy (local, central, or both) are used, and to provide a more detailed explanation of how the plug-ins interact with the DP mechanisms and what novel contributions are made in this area. Specifically it could be explained how plug-ins affect DP and what is the contribution in this part of the paper.\nFigure 2 should also include results of HPFL. The comparison should be expanded to cover more advanced PFL studies to showcase how HPFL performs compared to those methods. Currently is mainly focuses on basic PFL algorithms for comparison purposes. For instance, some key papers in this domain can help authors to provide a more compelling comparison among proposed method and existing PFL solutions, such as FedAlt/FedSim of Krishna et al, 2022@ICML and for the specific case of PFL with differential privacy, Hu et al, 2020 @ IEEE IoT Journal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)