Hot-pluggable Federated Learning: Bridging General and Personalized FL via Dynamic Selection

Authors: Lei Shen, Zhenheng Tang, Lijun Wu, Yonggang Zhang, Xiaowen Chu, Tao Qin, Bo Han
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: Federated Learning

Abstract:
Personalized federated learning (PFL) achieves high performance by assuming clients only meet test data locally, which does not meet many generic federated learning (GFL) scenarios. In this work, we theoretically show that PMs can be used to enhance GFL with a new learning problem named Selective FL (SFL), which involves optimizing PFL and model selection. However, storing and selecting whole models requires impractical computation and communication costs. To practically solve SFL, inspired by model components that attempt to edit a sub-model for specific purposes, we design an efficient and effective framework named Hot-Pluggable Federated Learning (HPFL). Specifically, clients individually train personalized plug-in modules based on a shared backbone, and upload them with a plug-in marker on the server modular store. In inference stage, an accurate selection algorithm allows clients to identify and retrieve suitable plug-in modules from the modular store to enhance their generalization performance on the target data distribution. Furthermore, we provide differential privacy protection during the selection with theoretical guarantee. Our comprehensive experiments and ablation studies demonstrate that HPFL significantly outperforms state-of-the-art GFL and PFL algorithms. Additionally, we empirically show HPFL's remarkable potential to resolve other practical FL problems such as continual federated learning and discuss its possible applications in one-shot FL, anarchic FL, and FL plug-in market. Our work is the first attempt towards improving GFL performance through a selecting mechanism with personalized plug-ins.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729472051225

thread_type:
official_review

<thread_object>
creator: reviewer_kxcm

<content>
summary:
After authors responses: The rating has been updated considering the authors' inputs and clarifications. I appreciate their efforts in providing those responses.

In order to solve the selective FL (SFL) problem, this paper leverages the model components that attempt to edit a submodel for specific purposes to design a  framework referred to as Hot-Pluggable Federated Learning (HPFL). In HPFL, clients individually train
personalized plug-in modules based on a shared backbone, and upload them with a plug-in marker on the server modular store. During the inference stage, a selection algorithm allows clients to identify and retrieve suitable plug-in modules from the modular store to enhance their generalization performance on the target data distribution. This paper also provides differential privacy protection during
the selection with theoretical guarantee. Key contributions can be summarized as:
-	Identifying a major gap between Generic FL (GFL) and Personalized FL (PFL), and formulate a new problem SFL to bridge this performance 
-	Developing a general, efficient and effective framework HPFL, which practically solves SFL and adding noise on communicated markers to provide differential privacy protection with theoretical guarantee.
-	Experiments on four datasets and three neural networks to demonstrate the effectiveness of HPFL

strengths:
The paper is well-written. It fairly cites prior works that it built on and shows how it leverages those solutions. It clarifies what is the problem and what are research questions to answer.
After carefully reviewing authors' clarifying points and responses to my concerns and other reviewers, I increased two of my scores.

weaknesses:
The originality of this paper is not clear. For instance, the following are two major claimed contributions according to the paper “Identifying a major gap between Generic FL (GFL) [e.g.,  (Karimireddy et al., 2019; Woodworth
et al., 2020; Tang et al., 2022b)'s works] and Personalized FL (PFL) [e.g.,  (Li & Wang, 2019; Chen & Chao, 2021; Li et al., 2021c):'s works], and formulate a new problem SFL to bridge this performance; and Developing a general, efficient and effective framework HPFL, which practically solves SFL and adding noise on communicated markers to provide differential privacy protection with theoretical guarantee.”
However, the methodology seems to be a combination of existing works on PFL while leveraging several existing work with minimal advances. It is not clear how the mentioned theorems add value to the literature, for example, this statement is vague and is not adequately explained: “solving SFL means that clients achieve performance in GFL as high as in PFL.”. It is not clear how plug-in marker can contribute to bridging the gap between GFL and PFL? It would be very helpful to explain elaborately. Specifically, please provide a more detailed explanation and concrete example of how the plug-in markers specifically help bridge the gap between GFL and PFL performance.
While using differential privacy can add value to the method, it is not clear whether it can benefit from local DP, central DP, or both?  Please clarify which type(s) of differential privacy (local, central, or both) are used, and to provide a more detailed explanation of how the plug-ins interact with the DP mechanisms and what novel contributions are made in this area. Specifically it could be explained how plug-ins affect DP and what is the contribution in this part of the paper.
Figure 2 should also include results of HPFL. The comparison should be expanded to cover more advanced PFL studies to showcase how HPFL performs compared to those methods. Currently is mainly focuses on basic PFL algorithms for comparison purposes. For instance, some key papers in this domain can help authors to provide a more compelling comparison among proposed method and existing PFL solutions, such as FedAlt/FedSim of Krishna et al, 2022@ICML and for the specific case of PFL with differential privacy, Hu et al, 2020 @ IEEE IoT Journal.

questions:
Q1. What is the main contribution of the HPFL that makes it outperform existing PFL models? This could be described by adding a table with core update rule of existing PFLs [including more recent studies] and the proposed method;
Q2. What is the difference of plug-in module and vanilla personalized model? Intuitively they seem to be the same and help considering the local model to personalize the local model of each client.;
Q3. What is the novelty of integrating DP in the algorithm? How does it lead to advancing the proposed HPFL solution? Is it HPFL+DP or the integration has some challenges? If so, what are the challenges and how does this paper tackle them? Why can’t we use other privacy preservation mechanisms? ;
Q4. Detailed comparison with two types of studies: i. existing PFL algorithms and comparing with HPFL, ii. Existing privacy preserving algorithms and comparison with DP;
5. How can the plug-in marker contribute to bridging the gap between GFL and PFL?
Q6. Can you please clarify which type(s) of differential privacy (local, central, or both) are used, and to provide a more detailed explanation of how the plug-ins interact with the DP mechanisms and what novel contributions are made in this area?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730341642975

thread_type:
official_review

<thread_object>
creator: reviewer_8vjb

<content>
summary:
The paper introduces Hot-Pluggable Federated Learning (HPFL), a framework aimed at bridging the gap between Generic Federated Learning (GFL) and Personalized Federated Learning (PFL) by proposing Selective Federated Learning (SFL). SFL optimizes PFL while allowing for the selection of personalized models (PMs) to enhance generalization performance across diverse test data. In HPFL, clients train personalized plug-in modules based on a shared backbone model, which are then uploaded to a server for selection during inference. This process also incorporates differential privacy to protect user data during the selection phase. Experimental results demonstrate that HPFL significantly outperforms traditional GFL and PFL methods, suggesting its applicability in various federated learning scenarios, including continual learning.

strengths:
1. The introduction of SFL effectively addresses the limitations of existing PFL methods, allowing for better adaptation to real-world scenarios where test data may differ significantly from local training data.

2. The HPFL framework’s modular approach enables efficient communication and computation, making it suitable for practical applications in federated learning, including scenarios with resource-constrained clients.

weaknesses:
1. The dependency on a common backbone model may lead to reduced performance if the backbone fails to generalize well across heterogeneous client data distributions.

2. The plug-in selection process during inference could introduce additional computational delays, particularly for clients with limited resources, potentially hindering real-time performance.

3. While the framework claims differential privacy protection, the effectiveness of this mechanism in preventing information leakage during plug-in selection remains to be empirically validated in diverse operational contexts.

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681937284

thread_type:
official_review

<thread_object>
creator: reviewer_fwno

<content>
summary:
This paper presents a novel framework called Hot-Pluggable Federated Learning (HPFL) that aims to bridge the gap between generic federated learning (GFL) and personalized federated learning (PFL). The authors propose a new learning paradigm, Selective Federated Learning (SFL), which combines model optimization with model selection. HPFL addresses the challenges of storing and selecting whole models by designing an efficient framework that allows clients to train personalized plug-in modules and upload them to a server. During inference, a selection algorithm identifies suitable plug-in modules to enhance performance on target data distributions. The paper also incorporates differential privacy protection during the selection process. Comprehensive experiments demonstrate HPFL's effectiveness in improving GFL performance and its potential in addressing other FL challenges like continual learning and one-shot FL.

strengths:
1. The authors identify a substantial gap between GFL and PFL, and formulate a new problem SFL to bridge them together to address this performance gap. Both optimization function of GFL and PFL are the special cases of it.

2. The authors propose a general, efficient and effective framework HPFL, which practically solves SFL.

3. Comprehensive experiments and ablation studies on four datasets and three neural networks demonstrate the effectiveness of HPFL.

weaknesses:
1. It’s adorable that the authors define a paradigm to bridge the gap between GFL and PFL, but the selective method are not novelty enough. For example, [1] allowed each client to choose the appropriate scale model to train. And the training process in HPFL are identical to the split federated learning, the authors only add another select process in inference process. 
2. The authors claim that they theoretically show PMs can be used to enhance GFL with a new learning problem named Selective FL (SFL), which involves optimizing PFL and model selection. But only the statement of the loss cannot totally evaluate the effectivess of the SFL. And in the Eq.4, the greater-than sign \geq should be a less-than sign \leq? The better one should gain the less loss?
3. What’s the originality in the analysis of the privacy protection? It seems that add Gaussian noise to the partial model or full model is identical. Thus I think it’s only an existing result.
4. The notations needs to be improved. For example, in section 2.4 the definition of Selective FL (SFL) problem, the introducing of auxiliary information is confused. And in Theorem2.3, the function s(\dot) lacks description.
5. The presentation in experimental section should be improved. For example, the Table 2 is hard to read. It’s confused that the authors not only give the best result in grey, but also some second best result. And why the proposed HPFL does not gain the best result especially compared to the GFL FedSAM?

[1] Cho, Yae Jee, et al. "Heterogeneous ensemble knowledge transfer for training large models in federated learning." arXiv preprint arXiv:2204.12703 (2022).

questions:
As shown in the Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694843106

thread_type:
official_review

<thread_object>
creator: reviewer_vdpf

<content>
summary:
This article proposes a new federated learning framework called Hot-Pluggable Federated Learning (HPFL), which aims to solve the performance gap problem between general federated learning (GFL) and personalized federated learning (PFL). Traditional GFL cannot cope with the diversity of data distribution, while PFL is only suitable for scenarios where local data distribution is similar. When the client encounters test data that is different from local data, PFL's personalized model has difficulty maintaining efficient generalization performance. To this end, this paper proposes a new problem framework of Selective Federated Learning (SFL), which enhances the effect of GFL by selecting an appropriate personalized model for each client in the inference stage. The HPFL framework divides the model into shared backbone and personalized plug-in modules. The client trains and uploads plug-ins based on local data. During inference, it can select appropriate plug-ins to adapt to different data distributions, while protecting data privacy through differential privacy.

strengths:
Originality：The HPFL framework proposed in this paper innovatively introduces a plug-in selection mechanism into federated learning, realizes the bridge between general models and personalized models, and solves the performance balance problem of traditional GFL and PFL. This is a new attempt in federated learning.

Quality：The experimental part is relatively comprehensive, covering a variety of data sets and model verification, and enhancing security through differential privacy. The overall design is rigorous, and the results demonstrate the advantages of HPFL in performance and adaptability.

Clarity：The paper is well-structured, with clear background and problem descriptions, and clear algorithm design, framework details, and experimental procedures, making it easy for readers to understand its core contributions.

Significance：This study proposed a new solution to the adaptability problem of federated learning under heterogeneous data distribution, which has practical application potential and provides a new direction for the future development of federated learning.

weaknesses:
·Insufficient details of the selection mechanism (page 5, Section 3.3)：HPFL uses multiple distance metrics such as MMD, SVCCA, and CKA to select plug-ins, but the specific algorithm steps and implementation details are rarely described. The article can add mathematical expressions or pseudocodes for some of the selection methods to increase readability and make it easier for readers to understand the robustness of the selection process.

·Selection of comparison methods (page 7&8, Table 2 and Table 3)：The paper compares a variety of GFL and PFL algorithms, but lacks a comparison of newer solutions that focus on heterogeneous data distribution problems. It is recommended to add more to further highlight the advantages of HPFL in performance and adaptability.

questions:
·How to balance the computational and communication overheads brought by the storage and selection of personalized plug-in modules in the HPFL framework to improve model efficiency?

·Is it possible that differential privacy protection during model selection in this article, or when using other privacy protection methods, may significantly affect model performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732803874910

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all reviewers for taking the time to review our work. We appreciate you find **our framework novel, innovative, general, efficient and effective** (Reviewer VDPF, FWNo and 8vJb), **our paper well-written** (Reviewer VDPF and kxCM) and **easy to follow** (Reviewer VDPF), **detailed and fair discussion of related works** (Reviewer kxCM), we **identify a substantial gap between GFL and PFL, and formulate a new problem SFL to bridge them together** (Reviewer VDPF, FWNo), **extensive experiments and ablation studies showing superior performance** (Reviewer VDPF, FWNo and 8vJb), **our method promsing and potential in applications** (Reviewer VDPF  and 8vJb), we **provides a new direction for the future development of FL** (Reviewer VDPF). 

Here, we provide a summary of our responses to frequent questions for convenient reading.

**Q1: Clarification on contributions and advantages:** (Reviewer VDPF, FWNo and kxCM)

***Ans for Q1):*** 

We identify our advantages compared with previous works as:

 - **Superb performance on all datasets instead of local datasets.**
 - **Lower communication resource required.**
 - **Free from the requirement of a public dataset, which is diffult to get in many cases like medical information.**
 - **Implementing a possible market mechanism of plug-in sharing between different users.**
 - **Potential application for asynchrnous and continuous learning scenarios.**

, which are achieved by 

 - **We first formulate the performance gap between GFL and PFL, and though this fomulation we find a theoretical framework named Selective FL (SFL) that can utilize the excellent PFL performance of local models to boost the GFL performance.**
 - **We instantiate SFL with an efficient and practical framework named HPFL, which incorporates the plug-in mechanism to improve GFL performance with PFL modules.**
 - **We use markers help to identify the training data distribution of the plug-in modules and adapt test model accordingly.**


**Q2: Selection method:** (Reviewer VDPF, FWNo)

***Ans for Q2):*** 

**Clarification**: we add the specific implementations and formulas of MMD, SVCCA, and CKA in Appendix D.2 to provide better elaboration on the selection process. 

**Issues on formulation and presentation**: We have revised the notation and text presentation to better clarify auxiliary information and selection function ***in Section 2.4 in our revision***. 


**Q3: Differential Privacy:** (Reviewer VDPF, FWNo, 8vJb and kxCM)

***Ans for Q3):*** 

**Novelty:** Compared with applying DP to models, applying DP to other shared information like intermediate features is significantly less explored. 

**Experimental Verification:**
- **In Table 13**, we experimentally shows under the protection of Differential Privacy, HPFL still achieve better GFL performance over all baselines.
- Moreover, **results in Table 8** also show that the model performance are not significantly affected by the Differental Privacy protection.
- In **Appendix E.2 and Figure 15, 16**, we observe that after our protection against the markers, no raw image can be successfully reconstructed with model inversion attack. 


**Q4: Discussion on extra system overheads** (Reviewer VDPF and 8vJb)

***Ans for Q4):***

We provide three real-world deployment schemes of HPFL inference, each targeted at a specific scenario to reduce the according bottleneck of system overheads: 
- (1) In light-weight applications requiring low update frequency (like backend activities), the bottleneck is the storage and communication costs for frequent update, **Plug-in Updating** only update the local plug-in stored locally at a client-defined frequency, which greatly reduce inference costs; 
- (2) In latency-sensitive applications, the bottleneck is communication time to fetch plug-ins, **Cache All** request all plug-ins and markers from the server to omit the need for communication at inference time; 
- (3) When one client meets a limited number of distributions, the bottleneck comes from the frequent download of the same plug-in, we adopt **Cache Selected**, which stores the plug-ins that clients have chosen before.

**Plug-in Updating** and **Cache All** are the **$\alpha$** and **$\beta$** implementation in the introdution. 

Below we briefly summarize the factors reducing the system overheads of HPFL. 
 - **Low update frequency and fewer transmitted parameters**, detailed explanation can be found in responses to Reviewers VDPF, 8vJb. 
 - **Controllable Number of Plug-ins and Storage**: In Appendix F.2.1, we provide some naive ideas on reducing the number of plug-ins, and show HPFL outperform the best GFL-PM baseline FedTHE with only 1/3 number of plug-ins by simply eliminating plug-ins with the calculated selection score.

We have integrated the above mentioned modifications into ***our revision***.
</content>
</thread_object>

</thread>


Metareview:
metareview:
a) Summary

The paper introduces Selective Federated Learning (SFL), a framework to bridge the gap between Generic Federated Learning (GFL) and Personalized Federated Learning (PFL) by enabling clients to share and selectively integrate lightweight plug-in modules trained on local data. The core insight is that personalized modules can enhance generalization across diverse test distributions when combined with a shared backbone and selected using markers. The main result demonstrates that the proposed Hot-Pluggable Federated Learning (HPFL) framework significantly outperforms state-of-the-art GFL and PFL methods while addressing privacy concerns through differential privacy and reducing system overhead with efficient plug-in sharing mechanisms

b) Strengths

- The Hot-Pluggable Federated Learning (HPFL) framework, which sets up a "module store" and picks the most relevant module for the test distribution. This framework is realistic and potentially impactful. This could be interesting to extend to a *marketplace* setting as well.
- Introduces privacy protection for shared plug-in markers using differential privacy, with theoretical guarantees and experimental validation.
- Extensive evaluations across multiple datasets and architectures demonstrate significant improvements over traditional GFL and PFL methods.

c) Weakness

- The core methodology—using modular plug-ins, markers, and differential privacy—largely combines existing techniques, with minimal theoretical innovation beyond problem formulation. Potential extensions could try analyze more interesting data and model acquisition techniques (see e.g. Lu et al. 2023: https://arxiv.org/abs/2403.13893). It is also clearly related to the routing modules in mixture of experts (MoEs): see Yadav et al. 2024: https://arxiv.org/abs/2408.07057.
- Performance depends heavily on the backbone model's ability to generalize across heterogeneous client data

d) Decision to recommend **accept**

The authors introduce a system that allows clients to share and use small, adaptable components (plug-ins), which improves performance and reduces resource requirements. The experiments show that this approach works better than existing methods, and it also includes privacy protection to keep shared information secure. While some aspects build on existing ideas, the practical usefulness and strong results lead all the reviewers to recommend accepting this work.