Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning

Authors: Jingyuan Zhang, Yiyang Duan, Shuaicheng Niu, YANG CAO, Wei Yang Bryan Lim
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Federated Learning; Federated Domain Adaptation; Federated Fine-Tuning

Abstract:
Federated Domain Adaptation (FDA) is a Federated Learning (FL) scenario where models are trained across multiple clients with unique data domains but a shared category space, without transmitting private data. The primary challenge in FDA is data heterogeneity, which causes significant divergences in gradient updates when using conventional averaging-based aggregation methods, reducing the efficacy of the global model. This further undermines both in-domain and out-of-domain performance (within the same federated system but outside the local client), which is critical in certain business applications. To address this, we propose a novel framework called \textbf{M}ulti-domain \textbf{P}rototype-based \textbf{F}ederated Fine-\textbf{T}uning (MPFT). MPFT fine-tunes a pre-trained model using multi-domain prototypes, i.e., several pretrained representations enriched with domain-specific information from category-specific local data. This enables supervised learning on the server to create a globally optimized adapter that is subsequently distributed to local clients, without the intrusion of data privacy. Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation in FDA. Notably, MPFT achieves convergence within a single communication round, greatly reducing computation and communication costs. To ensure privacy, MPFT applies differential privacy to protect the prototypes. Additionally, we develop a prototype-based feature space hijacking attack to evaluate robustness, confirming that raw data samples remain unrecoverable even after extensive training epochs. The complete implementation of MPFL is available at \url{https://anonymous.4open.science/r/DomainFL/}.

Decision: accept (poster)
Reviewer scores: 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730078015066

thread_type:
official_review

<thread_object>
creator: reviewer_ckus

<content>
summary:
This paper introduces Multi-domain Prototype-based Federated Fine-Tuning (MPFT), a framework designed to address data heterogeneity and data privacy in federated learning (FL) by using prototype-based training rather than traditional averaging methods. In MPFT, each client generates a set of representative data embeddings (prototypes) that capture essential domain-specific characteristics without transferring raw data. These prototypes are then aggregated at the server, allowing for a simulated centralized learning approach and enabling fine-tuning of a global adapter. This method aims to achieve performance on par with centralized learning while solving the challenges incurred by aggregating client models.
The efficiency of MPFT lies in that it requires only a single round of global communication, significantly reducing computational and communication costs compared to multi-round FL methods. Furthermore, by selectively sampling prototypes, the framework limits data transfer volumes. To ensure privacy, MPFT integrates differential privacy mechanisms, mitigating risks of data exposure and rendering prototype-based data reconstruction ineffective—even when the prototype encoder is known to potential attackers.

strengths:
+ They propose an interesting idea -- MPFT as a one-round federated fine-tuning approach for multi-domain environments, demonstrating its performance improved over previous methods. 
+ A new metric is introduced to evaluate model adaptability, assessing both out-of-domain and in-domain accuracy to balance knowledge retention and domain adaptation.
+ The writing and logic are clear and easy to follow. The problem formulation is clear and friendly to readers.

weaknesses:
- The proof of convergence in Section B lacks the differential-privacy-related analysis. A supplemental analysis of "MPFT with DP" should be added for theoretical completeness. Intuitively, the addition of differential privacy introduces a bounded randomness during the convergence process.

- The selected experimental counterparts for comparison are mostly in 2017-2022, which are relatively not new. In Section 2, the authors think the "averaging-based aggregation results in poor out-of-domain adaptation performance." It would be better to show the performance of out-of-domain adaptation from the recent advances atop model/parameter aggregation. Some recent works (with or without aggregation) relevant to data heterogeneity in federated learning are expected to compare, such as,

Fedcp: Separating feature information for personalized federated learning via conditional policy. In KDD 2023.
FedFed: Feature Distillation against Data Heterogeneity in Federated Learning. In NeurIPS 2023.
Fedgh: Heterogeneous federated learning with generalized global header. In MM 2023.

questions:
1. Would the authors explain how you chose the parameter $s$ in Section 5.6? 

2. Why is the class $k$ missing in the proof in Appendix B (while formulated before proof)?

3. Can authors make more comparisons with recent (>=2023) works? For example,
Fedcp: Separating feature information for personalized federated learning via conditional policy. In KDD 2023.
FedFed: Feature Distillation against Data Heterogeneity in Federated Learning. In NeurIPS 2023.
Fedgh: Heterogeneous federated learning with generalized global header. In MM 2023.

It would be better to compare aggregation-based improvements in recent works and the new mechanism proposed in this paper. Since the authors stressed the drawbacks of these works, it would be better to give detailed support or explanation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730392873281

thread_type:
official_review

<thread_object>
creator: reviewer_xvys

<content>
summary:
The paper proposes Multi-domain Prototype-based Federated Fine-Tuning (MPFT), a framework designed for Federated Domain Adaptation (FDA) by utilizing domain-specific prototypes. Instead of relying on traditional model aggregation techniques, which often falter due to data heterogeneity, MPFT transfers prototypes (compressed domain representations) to the server. This approach allows the server to learn a global adapter that improves both in-domain and out-of-domain performance. MPFT also incorporates differential privacy to protect prototypes from potential data leakage.

strengths:
The main strength of the paper lies in its comprehensive experimentation, which provides a detailed view of MPFT’s performance across various scenarios, showing superior computational and communication efficiency.

weaknesses:
The Multi-domain Prototype-based Federated Fine-Tuning (MPFT) method proposed in this paper is overly simplistic, primarily relying on basic prototype sampling strategies—such as mean, cluster, and random sampling—to generate client prototypes. Additionally, because the model does not undergo training on each client’s data, it fundamentally contradicts the original design principles of federated learning (FL). The performance gains observed in the experiments likely stem from the strong pre-trained feature extractor, which does not require training on all client data but only a few representative prototypes to achieve reasonable performance. If the pre-trained feature extractor were removed and the model was trained solely on prototypes, the results would likely be poor. If only prototype fine-tuning is needed, then why even employ a federated learning framework? Thus, despite achieving certain results, this approach introduces no complex mechanisms or innovative architectures to enhance the FL system, lacking fresh or original design. It merely follows a standard process: feature extraction with a general pre-trained model, prototype selection, server training, and client fine-tuning, without presenting any real novelty.

The paper does not provide a thorough theoretical analysis of the different prototype sampling methods, nor does it uncover the fundamental differences between these sampling strategies in handling heterogeneous data distributions. Although the paper presents experimental results comparing mean, cluster, and random sampling methods, it lacks detailed explanations on the core differences among these strategies in terms of learning mechanisms, communication efficiency, and personalization effects. Consequently, the paper falls short in theoretical depth, failing to provide any deep insights into the impacts of these sampling strategies.

The theoretical analysis in this paper primarily focuses on convergence. However, training the adapter on aggregated prototype data is no different from centralized data training, so convergence is naturally expected. To provide valuable theoretical insights, the paper would need to show that the adapter’s performance on aggregated prototypes is comparable to that on aggregated raw client data or offer theoretical performance bounds. Furthermore, transmitting prototypes instead of models introduces a higher risk of privacy leakage, so the analysis should prioritize differential privacy (DP) guarantees rather than focusing solely on convergence. The current paper fails to address theoretical guarantees for privacy protection, which is particularly critical in privacy-sensitive federated learning contexts.

Although the experimental section is comprehensive, covering multiple datasets, various sampling strategies, and different DP configurations, it does not clearly reveal the functional differences and applicability of each prototype sampling approach across different scenarios. While the experiments test different prototype configurations, they lack an in-depth discussion on how these configurations perform under different data distribution patterns and sample sparsity conditions. This results in a set of experiments that, while extensive, fails to provide a robust summary of the unique characteristics of the prototype methods, thus missing an opportunity to elevate the research value.

questions:
I suggest that the authors remove the overly powerful pre-trained feature extractor and instead use a simpler, more foundational model for feature extraction, such as ResNet or ConvNet. This would more accurately showcase the contribution of the prototype and methodology itself.

The authors should provide a validity proof to demonstrate that training the adapter on aggregated prototypes achieves comparable performance to training on aggregated raw client data or features—or, alternatively, establish an upper bound on the performance gap between these approaches.

Although differential privacy (DP) experiments were conducted, a theoretical analysis is also necessary. Specifically, the authors should clarify the noise variance conditions required for their method to satisfy DP. In particular, the statement “Furthermore, we observe that specific noise configurations can reduce bias across heterogeneous datasets, enhancing the robustness of prototype data” requires a more robust theoretical explanation.

I would encourage the authors to further explain, both theoretically and experimentally, the differences, advantages, and potential complementarities among various prototype selection mechanisms.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685152769

thread_type:
official_review

<thread_object>
creator: reviewer_zwk6

<content>
summary:
This work studies domain adaptation in federated learning scenarios, employing prototype-based fine-tuning to leverage knowledge from other clients. The fine-tuning procedure requires only one round of communication between clients and the server, making it communication-computationally efficient.

strengths:
1, Experiments on two datasets indicate a significant performance boost from the proposal, increasing accuracy by 2%.

2. The approach is intuitive, sharing prototypes using three sampling strategies.

3. The proposal is communication-computationally efficient, as it involves a low number of interaction rounds and requires only prototypes for communication.

weaknesses:
The motivating scenario may be impractical, as the proposal assumes that clients already possess a well-pretrained model. This assumption can be particularly challenging in sensitive areas where federated learning is essential, such as healthcare or finance, where data privacy concerns limit access to robust pre-trained models. 

Can the authors discuss specific applications or scenarios within these sensitive domains where obtaining a pre-trained model might be more feasible? Additionally, it would be valuable to explore potential solutions or adaptations of their method for situations where a pre-trained model is not available.

questions:
1. The experimental settings are somewhat vague, particularly regarding whether competitive approaches, such as FedAvg and FedProx, are trained from scratch or from the same pretrained backbone model as the MPFT. Could the authors specify this information in the paper, ideally in the experimental setup or implementation details section?

2. The privacy protection measures appear informal. While Gaussian noise is added to prototypes (i.e., embeddings) as noted in Section 5.6, it is important to assess whether the noise level provides adequate privacy protection for sensitive attributes or labels. To strengthen this section, could the authors provide a more rigorous analysis of the privacy guarantees associated with their approach? For example, quantifying the level of privacy protection using established metrics or frameworks would be beneficial.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732457943180

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you for taking the time to review our manuscript and provide your valuable feedback. We have carefully addressed all the comments and incorporated changes to improve the manuscript accordingly. Below, we summarized responses to the most frequently raised concerns:

1. **Applicability of Pretrained Models (Reviewer ZWk6 - Weakness, Reviewer xVyS - W1, Q1):**  
   - We clarified the feasibility of using pretrained models in sensitive domains like healthcare (e.g., BioBERT) and finance (e.g., FinBERT).  
   - Additional experiments with weaker pretrained models (e.g., ResNet-50) validated that our method's effectiveness does not solely rely on strong feature extractors. The results, included in Section 5.1 and Appendix C.3, showed that MPFT outperforms baselines regardless of the feature extractor's strength.

2. **Experimental Details and Fair Comparisons (Reviewer ZWk6 - Q1, Reviewer xVyS - Q1):**  
   - Clarified that all baseline methods, including FedAvg and FedProx, were trained with the same pretrained backbone as MPFT.  
   - We explicitly outlined experimental setups and parameter configurations in Appendix C.2 of the revised manuscript.

3. **More Detailed Differential Privacy (DP) Analysis (Reviewer ZWk6 - Q2, Reviewer xVyS - W3, Q3, Reviewer ckUs - W1):**  
   - We conducted a detailed theoretical analysis linking Gaussian noise to privacy budgets and empirically validated privacy guarantees.  
   - Results in Appendix H confirm that our DP mechanism effectively protects prototypes while maintaining performance, even with increased noise levels.

4. **Novelty of MPFT Framework (Reviewer xVyS - W1):**  
     - MPFT transmits feature prototypes instead of raw data or parameters, reducing communication costs and preserving privacy. MPFT uses class-level prototypes, avoiding the aggregation of features from different classes into a single prototype (Appendix A).  
     - MPFT eliminates reliance on traditional averaging-based methods, making it more suitable for scenarios with high domain heterogeneity.  
     - MPFT balances global generalization through aggregated prototypes and local adaptation via Algorithm 3, ensuring high in-domain accuracy with minimal communication overhead.  

5. **More Detailed Comparison of Sampling Methods (Reviewer xVyS - W2, Q4):**  
   - We analyzed the differences among average, random, and cluster sampling.  
   - Cluster sampling performed best in mixed-domain settings, while random sampling excelled with high domain heterogeneity. These insights, supported by Table 3, are added to Section 5.3.

6. **Comparisons with More Recent Works (Reviewer ckUs - W2, Q3):**  
   - We implemented FedCP and FedFed, showing that MPFT outperforms these methods on DomainNet subset50 (see Table 5).  
   - FedGH was excluded due to unavailable code and different research focus (model heterogeneity). Details are clarified in Appendix G.
   
We deeply appreciate your efforts and would be happy to further clarify or discuss any remaining concerns. Please feel free to let us know if additional modifications or explanations are required.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes to enhance federated domain adaptation and, in particular, to mitigate the challenge of data heterogeneity. The authors propose a framework known as Multi-domain Prototype-based Federated Fine-Tuning (MPFT). It finetunes a pre-trained models using multidomain prototypes. MPFT enables supervised learning on the server to derive a globally optimized adapter that is subsequently distributed to local clients, without the intrusion of data privacy. It is shown empirically that MPFT improves in- and out-of-domain accuracy over conventional federated domain adaptation methods. Privacy is also considered. 

Generally, the reviewers felt positive about the paper, which uses some logical and natural approaches to solve the problem. While nothing is overly surprising, the theoretical and experimental results are sound. MPFT is low cost, efficient and robust. I believe that MPFT may be useful in future federated domain adaptation tasks.