Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning

## Abstract

Abstract Federated Domain Adaptation (FDA) is a Federated Learning (FL) scenario where models are trained across multiple clients with unique data domains but a shared category space, without transmitting private data. The primary challenge in FDA is data heterogeneity, which causes significant divergences in gradient updates when using conventional averaging-based aggregation methods, reducing the efficacy of the global model. This further undermines both in-domain and out-of-domain performance (within the same federated system but outside the local client). To address this, we propose a novel framework called M ulti-domain P rototype-based F ederated Fine- T uning (MPFT). MPFT fine-tunes a pre-trained model using multi-domain prototypes, i.e., pretrained representations enriched with domain-specific information from category-specific local data. This enables supervised learning on the server to derive a globally optimized adapter that is subsequently distributed to local clients, without the intrusion of data privacy. Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation in FDA. Notably, MPFT achieves convergence within a single communication round, greatly reducing computation and communication costs. To ensure privacy, MPFT applies differential privacy to protect the prototypes. Additionally, we develop a prototype-based feature space hijacking attack to evaluate robustness, confirming that raw data samples remain unrecoverable even after extensive training epochs. The complete implementation of MPFL is available at https://anonymous.4open.science/r/DomainFL/ .

## 1 Introduction

Federated Learning (FL) is a privacy-preserving distributed machine learning paradigm designed to protect the data of participating clients (McMahan et al., ). In FL, only models trained on local data are shared between clients and servers, rather than the raw data itself, mitigating the risk of data leaks. Mainstream FL research primarily focuses on optimizing each client’s performance within its local data domain (in-domain performance). However, in Federated Domain Adaptation (FDA) scenarios, clients need to perform well on the collective data domains shared by all participants to meet certain business requirements. For instance, consider a consortium of banks training a model to detect fraudulent transactions. Each bank has distinct customer bases and transaction patterns, leading to variations in their data (domains). A good out-of-domain performance is essential for anticipating new risks that a bank may not be exposed to yet. The challenge is to ensure the global model performs well across all banks, achieving good in-domain accuracy (within each bank’s typical use cases) and out-of-domain accuracy (generalizing across the federated financial system).

Studies have shown that FL achieves results similar to centralized training only when the datasets among clients are independently and identically distributed (i.i.d) and share similar domain characteristics  (Liu et al., ; Seol & Kim, ; Shaheen et al., ). In typical scenarios where all clients’ models are trained on local data from the same domains, update directions are aligned, making it feasible to use averaging-based aggregation algorithms such as FedAvg (McMahan et al., ). However, in FDA, in which clients have unique domains but a shared category space, the significant dissimilarities of feature space among local model updates mean that averaging of weights may not yield an optimal global model (Su et al., ; Sun et al., ).

To address this issue, we propose a novel framework called Multi-domain Prototype-based Federated Fine-Tuning (MPFT). MPFT tackles this issue by having each client generate a specific proportion of data embeddings (i.e., prototypes) to be transmitted to the server to create a prototype training dataset. This allows us to simulate a centralized learning approach without transferring raw data, as prototypes encapsulate sufficient domain-specific features to represent the entire data domain. The server then fine-tunes a global adapter using this comprehensive set of prototypes, with the goal of approximating the performance of centralized learning without relying on conventional averaging approaches to derive a global model. Figure 1 illustrates how our aggregation method compares to centralized learning and traditional averaging-based FL approaches.

Since only a specific proportion of local data embeddings are sampled as prototypes, and MPFT requires only a single round of global communication to converge, our framework incurs significantly lower computation and communication costs compared to other multi-round FL frameworks. MPFT also incorporates a differential privacy mechanism to mitigate the risk that the original data of specific prototypes are exposed during the prototype transmission process. Furthermore, simulations of feature space hijacking attacks on MPFT demonstrate that attackers cannot reconstruct the original data from the uploaded prototypes, even when the pretrained prototype encoder is known.

Our contributions are as follows: a) We propose MPFT, a one-round federated fine-tuning framework with convergence guarantees that outperforms previous methods on multi-domain environments. b) We introduce a novel metric to evaluate the performance of the FDA, specifically assessing out-of-domain and in-domain accuracy to consider the trade-off between knowledge preservation and adaptation. c) We demonstrate empirically that MPFT incurs lower computational and communication overheads as compared to other FL methods while ensuring privacy through differential privacy protection of the prototypes and maintaining robustness against feature space hijacking attacks.

Figure: Figure 1: Comparison of MPFT to centralized learning and previous averaging-based FL approaches.
Refer to caption: x1.png

## 2 Related work

Efforts to enhance performance in FL across heterogeneous datasets (FDA) have been extensive. Regularization methods during training, such as FedProx (Li et al., ) and FedDyn (Jin et al., ), or knowledge distillation techniques like FedGen (Venkateswaran et al., ) and FedNTD (Lee et al., ), aim to align local models more closely with the global model. Such approaches help mitigate the divergence among local models but rely on the assumption that the averaged global model is well-suited for the entire data distribution which often does not hold in FDA.

Updating only parts of the model during aggregation is another strategy, exemplified by personalized FL (PFL)  (Tan et al., ), which aims to customize local models to enhance in-domain performance. Some methods involve using a personalized aggregation base to select specific portions of global information for model aggregation, or updating only parts of the model during the aggregation phase, e.g., APFL (Deng et al., ), FedFomo (Zhang et al., ), FedAMP (Huang et al., ), FedPHP (Li et al., ), APPLE (Luo & Wu, ), and FedALA (Zhang et al., ). Additionally, some methods split model layers to segregate global and local components, such as in FedPer (Arivazhagan et al., ), LG-FedAvg (Liang et al., ), FedRep (Husnoo et al., ), FedRoD (Chen & Chao, ), FedBABU (Oh et al., ), FedCP (Zhang et al., ), FedGH (Yi et al., ), and DBE (Zhang et al., ). However, a key drawback is these methods still rely on averaging-based aggregation, which results in poor out-of-domain adaptation performance, despite achieving some in-domain improvements.

Some FL methods incorporate prototype learning. For instance, FedProto (Tan et al., ) utilizes averaged local prototypes to train as the global prototype on local clients. FPL (Huang et al., ) transmit all the prototypes (data embeddings) to the server during the initial phase for clustering and averaging, which is then sent back to the clients for further optimization. This method introduces significant communication overhead and potential privacy leakage during the first phase. FedNH (Dai et al., ) leverages class prototype transmission to address class imbalance across clients, rather than tackling the more complex issue of domain heterogeneity.

Unlike the above methods, we do not assume that the average aggregation of different client models or prototypes can represent the global model or global prototype in scenarios involving heterogeneous client data. MPFT uses prototypes from different clients as basic units that replace data distribution for centralized training on the server, thus obtaining an approximate global model capable of fitting all heterogeneous client domains distribution, as demonstrated in Appendix A.

## 3 Problem statement

#### Scenario assumption.

Consider a scenario involving $N$ clients, where each client possesses a private training dataset ${\mathbb{D}}_{1},\dots,{\mathbb{D}}_{N}$, each drawn from a unique data domain and shared category space(^1^11These domains may be largely isolated with minimal overlap, but MPFT also generalizes to scenarios where each client may have data from multiple domains, as demonstrated in Section 5.). Our goal is to train a model that balances domain knowledge preservation and domain knowledge adaptation. Domain knowledge preservation refers to the ability of the model to retain each client’s unique domain insights. Domain knowledge adaptation is defined as the model’s ability to indirectly extract and transfer knowledge from each participating client’s domain to others, even if a client $i$ cannot directly train on another client $j$’s data.

#### Optimization goal.

Building on this, we define the Domain Knowledge Preservation loss $\mathcal{L}^{P}$ and the Domain Knowledge Adaptation loss $\mathcal{L}^{A}$ as follows:

$$ $\mathcal{L}^{P}=\sum_{i=1}^{N}\mathcal{L}_{i}(\Theta^{\mathcal{P}}_{i};{ \mathbb{D}}_{i};\Theta^{\mathcal{G}}),$ $$

$$ $\mathcal{L}^{A}=\sum_{i=1}^{N}\sum_{j=1,j\neq i}^{N}\mathcal{L}_{i}(\Theta^{ \mathcal{P}}_{i};{\mathbb{D}}_{j};\Theta^{\mathcal{G}}),$ (1) $$

where $\mathcal{L}(\cdot)$ denotes the loss function, $\Theta^{\mathcal{P}}_{i}$ denotes the local (personalized) model parameters for client $i$, and $\Theta^{\mathcal{G}}$ denotes the global model parameters. In conventional FL, the local models $\Theta^{\mathcal{P}}_{1},\dots,\Theta^{\mathcal{P}}_{N}$ are periodically synchronized with the global model $\Theta^{\mathcal{G}}$.

The optimization goal is to decrease both the $\mathcal{L}^{P}$ and $\mathcal{L}^{A}$. Thus, we formalize the FDA optimization goal as follows:

$$ $\{\Theta^{\mathcal{P}}_{1},\dots,\Theta^{\mathcal{P}}_{N};\Theta^{\mathcal{G}} \}=\text{arg\,min}(\alpha^{1}_{i}\mathcal{L}^{P}+\alpha^{2}_{i}\mathcal{L}^{A}),$ (2) $$

where $\alpha^{1}_{i}$ and $\alpha^{2}_{i}$ are client-defined weight parameters which balance the trade-off between domain knowledge preservation and domain knowledge adaptation, in line with the “no free lunch” theorem.

#### Evaluation metrics.

To quantify the effectiveness of the optimization, we propose two metrics: in-domain accuracy (ind acc) and out-of-domain accuracy (ood acc). Denoting $\text{ACC}_{i}^{(j)}$ to be the accuracy for a client $i$ when tested on domain $j$, ind acc and ood acc are defined as follows:

$$ $\text{ind acc}=\frac{\sum_{i=1}^{N}\text{ACC}_{i}^{(i)}n_{i}}{\sum_{i=1}^{N}n_ {i}},$ $$

$$ $\text{ood acc}=\frac{\sum_{i=1}^{N}\sum_{j\neq i}\text{ACC}_{i}^{(j)}n_{j}}{ \sum_{i=1}^{N}\sum_{j\neq i}n_{j}},$ (3) $$

where $n_{i}$ is the number of test samples for client $i$. The ind acc measures each client’s performance on its own domain data, while the ood acc evaluates adaptation performance when tested on data from other domains.

## 4 Methodology

#### Overview.

We introduce Multi-domain Prototype-based Federated Fine-Tuning (MPFT), which consists of three main components: Prototype Generation, Global Adapter Initialization, and Few-shot Local Adaptation, as depicted in Figure 2. During the Prototype Generation phase, we generate domain-specific prototypes for each client based on specific sampling methods and ratios. The clients subsequently transmit their local prototypes to the server. In the Global Adapter Initialization phase, we utilize these prototypes to train a global adapter designed to handle the multi-domain distribution of all clients, thereby improving the ood performance of the global adapter across clients. The global adapter is then sent back to the clients for local inference. While the global adapter performs well in ood accuracy, some clients may require better ind performance. In such cases, they can proceed to the Few-shot Local Adaptation phase, where a few-shot dataset is sampled locally to further fine-tune the local adapter. Knowledge distillation is employed to mitigate catastrophic forgetting of global knowledge during this phase.

Figure: Figure 2: An overview of MPFL.
Refer to caption: x2.png

#### Prototype Generation.

A prototype is a compact representation of a specific class feature that is unique to each domain. To synchronize the consistency of the prototype’s embedding space across domains, each client utilizes the same pretrained image encoder to generate the prototype set $\{{{\mathbb{P}}^{(1)},\dots,{\mathbb{P}}^{(\mathcal{K})}}\}$, where $\mathcal{K}$ represents the total number of classes. Although all clients share the same label space, each label manifests uniquely within its respective feature domain.

To generate these prototypes, different sampling methods can be applied, including Mean Sampling, Cluster Sampling, and Random Sampling, as described in Algorithm 1. The choice of sampling method depends on the desired trade-off between computational efficiency and prototype representational robustness. In Mean Sampling, each client $i$ generates a prototype for class $k$ by calculating the mean of the pretrained embeddings for that class. In Cluster Sampling, clustering (e.g., k-means) is performed on pretrained embeddings of each class, and a certain number of cluster centers are then selected based on a predefined sampling rate to form the prototype set. In Random Sampling, a fixed number of pretrained embeddings of each class are randomly selected according to the sampling rate, and these selected embeddings constitute the prototype set.

Once each client has generated their prototypes, they transmit these to the server. Consequently, the server accumulates $N$ domain-specific representations subset (prototypes subset) for each class $k$, which are collectively represented as ${\mathbb{D}}^{\mathcal{P}}=\left\{\bigcup_{i=1}^{N}\left\{{\mathbb{P}}_{i}^{(1
)},\dots,{\mathbb{P}}_{i}^{\mathcal{(K)}}\right\}\right\}$.

Figure: Algorithm 1 Different Sampling Methods

#### Global adapter initialization.

Algorithm 2 outlines the global adapter initialization process. Note that we avoid averaging-based aggregation within each class across different clients, as this would distort the global distribution. Utilizing ${\mathbb{D}}^{\mathcal{P}}$, we train the global adapter $A^{\mathcal{G}}$ to adapt to the entire system’s data distribution with cross-entropy loss $\mathcal{L}$:

$$ $\{\Theta^{\mathcal{G}},A^{\mathcal{G}}\}=\text{arg\,min }\mathcal{L}({\mathbb{ D}}^{\mathcal{P}};\Theta^{\mathcal{G}};A^{\mathcal{G}}).$ (4) $$

Upon successful training, the global adapter $A^{\mathcal{G}}$ is sent to the clients, replacing their local adapters.

Figure: Algorithm 2 Federated Learning with Global Adapter Initialization

#### Few-shot local adaptation.

While global adapter $A^{G}$ performs well in ood accuracy, it may not be sufficient for ind accuracy. To address this, clients can use their local few-shot data ${\mathbb{D}}_{i}^{\mathcal{F}}$ to further fine-tune $A^{G}$, adapting it to their local domain and improving ind accuracy, as shown in Algorithm 3.

To avoid catastrophic forgetting (French, ) of global knowledge during the local adaptation process, knowledge distillation (KD) method is employed to regularize the locally trained adapter $A^{\mathcal{L}}$, ensuring it does not deviate too much from the global adapter $A^{\mathcal{G}}$. The loss function for local adaptation is defined as:

$$ $\mathcal{L}=\mathcal{L}_{\text{CE}}+\beta\mathcal{L}_{\text{KD}},$ (5) $$

where $\mathcal{L}_{\text{CE}}$ represents the cross-entropy loss, and $\mathcal{L}_{\text{KD}}$ denotes the KD loss. The hyperparameter $\beta$ balances the influence of the KD loss in the overall objective. In Section 5.4, we compare the effects of different KD weights $\beta$ on balancing ind accuracy and ood accuracy.

Figure: Algorithm 3 Few-shot Local Adaptation

#### Convergence.

We analyze the convergence of MPFT, with detailed proofs provided in Appendix B.

###### Theorem 1 (Convergence of fine-tuning with prototypes)

For a smooth, non-convex loss function $\mathcal{L}$ with a Lipschitz continuous gradient with constant $L$, the global fine-tuning using prototypes ${\mathbb{D}}^{\mathcal{P}}$ converges. The sequence of updates for the global adapter $A^{\mathcal{G}}$ achieves a monotonic decrease in the loss function $L({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}})$. Specifically, choosing a learning rate $\eta$ such that $0<\eta<\frac{2}{L}$ ensures:

$$ $\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t +1})\leq\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{ \mathcal{G}}_{t})-c\|\nabla_{A^{\mathcal{G}}}\mathcal{L}({\mathbb{D}}^{ \mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t})\|^{2}+\frac{\Delta}{N},$ $$

where $c=\eta-\frac{L\eta^{2}}{2}$ is a positive constant, ensuring that the step size is appropriately bounded for convergence. Here, $\Delta$ is the maximum prototype divergence across clients.

###### Corollary 1.1 (Convergence to stationary point and rate)

As $T$ increases, the average gradient norm decreases, indicating convergence to a stationary point:

$$ $\frac{1}{T}\sum_{t=1}^{T}\|\nabla_{A^{\mathcal{G}}}\mathcal{L}({\mathbb{D}}^{ \mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t})\|^{2}\leq\frac{\mathcal {L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{1})- \mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{T })+\frac{T\Delta}{N}}{cT}.$ $$

This indicates that as $T$ increases, the right-hand side approaches zero, confirming that the gradient norm diminishes and the updates converge to a stationary point.

###### Theorem 1 (Convergence of fine-tuning with prototypes)

###### Corollary 1.1 (Convergence to stationary point and rate)

## 5 Experiment

#### Datasets and Models.

To simulate a FDA environment, We use the DomainNet (Peng et al., ) and PACS (Li et al., ) datasets which are widely used in multi-domain data adaptation. For these datasets, we employ pretrained CLIP models from OpenCLIP (Cherti et al., ; Radford et al., ; Schuhmann et al., ). The image encoder of CLIP for DomainNet is a ViT-B-32 pretrained on the LAION-2B dataset, while for PACS, we use a ConvNeXT-Base pretrained on the LAION-400M dataset as the image encoder.

#### Implementation details.

We implement various representative FL algorithms as baselines, including FedAvg (McMahan et al., ), FedProx (Li et al., ), Ditto (Li et al., ), MOON (Li et al., ), FedProto (Tan et al., ), and DBE (Zhang et al., ), using the PyTorch library and based on the integrated FL library PFLlib (Zhang et al., ). To simulate the common FL scenario where data resides only on clients, we split the data for each client into a training set (70%), a test set (20%), and a validation set (10%). We evaluate in-domain (ind) and out-of-domain (ood) accuracy following Equation 3. To demonstrate the scalability of our method, we partition the DomainNet dataset, which includes 345 categories, into subsets containing 50, 100, and 150 classes, respectively. To more conveniently compute the convergence time for each FL method and compare the computational and communication costs of them, we introduce an early stopping strategy during training. More details about our experimental setup and baselines can be found in Appendix C.1 and Appendix C.2, with details on the early stopping strategy provided in Appendix C.3.

#### Diverse FDA Scenarios.

We conducted experiments and analyses from different perspectives on various potential FDA scenarios. Section 5.1 presents the performance of different FL methods in a basic scenario, where each client is assigned a unique data domain, with minimal overlap between domains. Section 5.2 provides a detailed analysis of the global model’s performance on each client, exploring the fairness of different methods in FDA. In Section 5.3, we design a more realistic scenario where each client may hold data from multiple domains, reducing the domain heterogeneity between clients. Appendix F sets up another realistic scenario where multiple clients belong to the same data domain, significantly increasing the number of clients compared to the original setup. In Section 5.4, we investigate the role of local adaptation. Section 5.5 compares the computational and communication costs of different methods. Finally, in Section 5.6 and Appendix G, we explore the differential privacy mechanism in MPFT and its robustness against feature space hijacking attacks.

### 5.1 Performance on multi-domain

We evaluate our method alongside other FL approaches in Table 1, including local training (i.e., each client fine-tunes the pretrained model separately). Empirical results show that local training excels in ind accuracy but performs poorly in ood accuracy. A reason is that local fine-tuning results in catastrophic forgetting (Luo et al., ). Personalized FL methods such as FedProto and DBE, which generally maintain a personalized local model for each client, have higher ind accuracy but compromise ood accuracy. In contrast, methods like FedAvg, MOON, and Ditto demonstrate more balanced improvements in both ind and ood accuracies. FedProx, which introduces a regularization term between the global and local models, improves ood accuracy at the expense of ind accuracy. In comparison, our method consistently achieves superior performance in both ood and ind accuracy across all DomainNet subsets. As the subset size of DomainNet increases, we observe variable convergence stability across methods such as FedAvg, FedProx, and Ditto, while DBE demonstrates accelerated convergence. In contrast, our method requires only one global communication round, which significantly reduces both computational and communication costs. This benefit is further elaborated in section 5.5.

Table: Table 1: Test accuracy and communication rounds for different FL methods on DomainNet subsets and PACS. The communication rounds are determined using an early stopping strategy, where fewer rounds indicate faster convergence. Additionally, we compare the sensitivity of MPFT to different global convergence thresholds to ensure the robustness of the results across various hyperparameters, as refer to Appendix D.

### 5.2 Details about results on each domain

To further explain why MPFT achieves better ind and ood accuracy compared to the baselines, we visualize the performance of each domain in Figure 3. Each axis of the radar chart represents a specific data domain (e.g., Real or Painting), with the shape and coverage area of the curves illustrating the global model’s performance across these domains. Empirically, the roundness of the curve could reflect the fairness of the model across different clients (domains)—the rounder the curve, the more fair the method is in the global distribution, leading to better ood accuracy.

Compared to other FL baselines across different DomainNet subset sizes, MPFT with average sampling method performs exceptionally well in the Quickdraw domain, with a more balanced curve shape. Additionally, MPFT maintains strong performance across other domains relative to the baselines, thereby achieving better overall fairness. For more details about the effects of random and cluster sampling compared
to average sampling on each domain, please refer to Appendix E.

Figure: Figure 3: Comparison of different FL methods across various DomainNet subset sizes.
Refer to caption: x3.png

### 5.3 Impact of Multi-Domain Differences on Performance

In real-world scenarios, a client may contain data from multiple domains rather than a single specific domain(^2^22Additionally, it is common for multiple clients to share the same data domain, as refer to Appendix F.)
. We simulate a situation where each client contains $1-mr$ percent of data from its original domain, mixed with $mr$ percent of data from another domain. Here, $mr$ represents the mixed ratio, indicating the level of domain diversity on the client side. We evaluate our method alongside other FL approaches under this scenario, as shown in Table 2, where DomainNet subset-50 is used. We observe a reduction in the performance advantage of our method compared to others, as the mixed ratio increases. This decline is due to the reduced heterogeneity within the FL system, which diminishes the strengths of our approach. However, it is still evident that our method outperforms most FL algorithms, particularly when using Random and Cluster sampling strategies.

Table: Table 2: Results of mixed multi-domain situation on DomainNet Subset-50.

### 5.4 Performance with local adaptation

We compare the few-shot performance of local adaptation with different knowledge distillation (KD) weights in Figure 4. As the KD weight increases, there is less out-of-domain knowledge forgetting but worse in-domain knowledge alignment. With an increase in the number of few-shot samples, the ood and ind accuracy show a similar trend. We provide more details about the experiment implementation of KD in local adaptation in Appendix C.4.

Figure: Figure 4: Few-shot performance comparison of local adaptation with different KD weights.
Refer to caption: x4.png

### 5.5 Computation cost and communication cost

We further analyze the computation cost and communication cost in Table 3. Computation cost is the total training time of FL, which is related to the number of communication rounds and the computational complexity within each round. Among the methods compared, Ditto is the most time-consuming due to the additional local training epoch it requires. In contrast, the MPFT, which converges in just one global round, significantly reduces training time, particularly when the sampling method is set to average. For other sampling methods, such as Random and Cluster, our approach trades a modest increase in training time for substantial improvements in both ind and ood accuracy. Communication cost is the number of parameters transmitted, which is theoretically influenced by the number of communication rounds $\mathcal{R}$, the model (adapter) parameters $\sum$, and the prototypes $\prod$. The number of communication rounds $\mathcal{R}$ directly contributes to a linear increase in the overall communication cost. Furthermore, the model adapter parameters $\sum$ and the size of the prototypes $\prod$ determine the communication cost per round in these specific algorithms. In our empirical results, MPFT with average sampling achieved the lowest communication cost across all experiments. However, while random or cluster sampling slightly increases communication overhead, it also significantly improves MPFT’s performance (see Table 1).

Table: Table 3: Computation cost and communication cost of FL methods. $\mathcal{R}$ represents the convergence round, $\sum$ represents the number of model (adapter) parameters, $\prod$ represents the size of prototypes, $N$ represents the number of clients. The subsets are derived from the DomainNet dataset.

### 5.6 Privacy preservation analysis

Following DBE (Zhang et al., ), we add Gaussian noise $\mathcal{N}$ to client prototypes $p_{1},\dots,p_{N}$ with a perturbation coefficient $q$ for the noise and a scale parameter $s$ for the noise distribution, the perturbed prototype $\tilde{p}_{i}$ of client $i$ can be defined as $\tilde{p}_{i}=p_{i}+q\cdot\mathcal{N}(0,s^{2})$, where $p_{i}$ is the original prototype of client $i$.

Table 4 shows the results of applying this differential privacy method on the DomainNet subset-50, with the sampling method set to average under various noise parameter combinations. This approach effectively mitigates attackers from inferring individual data points even when they possess the pretrained model and most of prototypes. Furthermore, we observe that specific noise configurations can reduce bias across heterogeneous datasets, enhancing the robustness of prototype data. In some cases, this even leads to improved performance compared to models without noise. For instance, the combinations of $(q=0.5,s=0.1)$, $(q=0.1,s=0.05)$, and $(q=0.5,s=0.05)$ exhibit such effects. According to DBE, setting $q=0.2$ and $s=0.05$ is sufficient to ensure privacy protection. However, excessively large noise can degrade model performance.

Table: Table 4: Performance of differential privacy with varying noise parameters configuration

To further evaluate the robustness of MPFT against adversarial attacks, we simulate a feature space hijacking attack (Vepakomma et al., ) on MPFT, please refer to Appendix G.

## 6 Conclusion and Future Work

We propose an adaptive and lightweight FDA framework, MPFT, designed to align a global model with heterogeneous domains by fitting prototypes from different domains. Extensive experiments demonstrate the effectiveness, low cost, and robustness of MPFT. This study may inspire further research in FDA that focuses on generalizing across heterogeneous domain prototypes, rather than relying on model parameter averaging for aggregation.

While MPFT achieves strong performance, it has some limitations. First, the quality of the prototypes is highly dependent on the pretrained model’s ability to extract meaningful features. Second, although attackers cannot reconstruct specific raw data from the prototypes, they may still be able to perform membership inference attacks (Shokri et al., ) or attribution inference attacks (Fredrikson et al., ) by exploiting statistical information contained within the prototypes. Addressing the aforementioned limitations could further enhance the viability and effectiveness of this approach in practical FL applications, making this method a viable alternative of averaging-based FL methods.

## Appendix A Visualization of prototype

In order to allow readers to intuitively understand the relationship between prototypes and original data, we visualize the data embeddings (all prototypes) and averaged prototypes in two-dimensional coordinates using the t-SNE (Van der Maaten & Hinton, ) algorithm, as shown in Figure 5. Different colors represent different domains: for example, blue indicates the ”Painting” domain, while orange signifies the ”Real” domain. Different markers represent various categories of data samples. The darker markers located within each sample cluster represent the prototypes of the ”domain-class”. It is evident that each prototype effectively reflects the distribution of its specific domain-class information. However, the mean prototype, represented by the large red star, is distant from each individual prototype. This observation underscores why we do not average the prototypes, as it fails to accurately reflect the overall data distribution, particularly in FDA scenarios.

Figure: Figure 5: t-SNE visualization of multiple datasets with their corresponding prototypes.
Refer to caption: x5.png

The Universal Approximation Theorem (Hornik et al., ) suggests that neural networks act as ”universal” data distribution fitters, effectively fitting the distribution of given data samples. However, this also leads to parameter space deviations between different client models in heterogeneous FL. To address these issues, we introduce a more reasonable aggregation method rather than averaging-based aggregation. We treat the same category of data in different domains with significant differences as independent data prototypes. We then use the collection of these data prototypes, which form a more general data distribution that covers the entire FL system, to train a global model that better fits the global distribution.

## Appendix B Convergence analysis

#### Setup:

The global loss function for fine-tuning using prototypes is:

$$ $\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}})= \frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{i}({\mathbb{D}}^{\mathcal{P}};\Theta^{ \mathcal{G}},A^{\mathcal{G}}),$ $$

where ${\mathbb{D}}^{\mathcal{P}}$ is the set of prototypes aggregated from all clients.

#### Assumptions:

The standard assumptions follow those of Li et al. (); Tan et al. ():
1) The loss function $\mathcal{L}$ is non-convex but smooth,
2) The gradient of the loss function is Lipschitz continuous with constant $L$. 3) The divergence between the prototypes of different clients for a given class is bounded. Let $\Delta_{i}^{(k)}$ be the divergence between client $i$’s prototype for class $k$ and the average prototype across all clients:

$$ $\Delta_{i}^{(k)}=\|p_{i}^{(k)}-\bar{p}^{(k)}\|,\quad\bar{p}^{(k)}=\frac{1}{N} \sum_{i=1}^{N}p_{i}^{(k)},$ $$

and assume that $\Delta_{i}^{(k)}\leq\Delta$ for all $i$ and $k$.
Since the prototypes are derived from pretrained image embeddings, this assumption likely holds due to the consistency provided by the pretrained encoder in synchronizing the embedding space across domains.

#### Proof:

Given the update rule $A^{\mathcal{G}}_{t+1}=A^{\mathcal{G}}_{t}-\eta\nabla_{A^{\mathcal{G}}}\mathcal
{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t})$, for a smooth, non-convex loss function $\mathcal{L}$, the Lipschitz continuity implies:

$$ $\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t +1})\leq\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{ \mathcal{G}}_{t})-\eta\|\nabla_{A^{\mathcal{G}}}\mathcal{L}({\mathbb{D}}^{ \mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t})\|^{2}+\frac{L\eta^{2}}{ 2}\|\nabla_{A^{\mathcal{G}}}\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{ \mathcal{G}},A^{\mathcal{G}}_{t})\|^{2}+\frac{\Delta}{N}.$ $$

Rearranging the terms, we obtain:

$$ $\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t +1})\leq\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{ \mathcal{G}}_{t})-\left(\eta-\frac{L\eta^{2}}{2}\right)\|\nabla_{A^{\mathcal{G }}}\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}} _{t})\|^{2}+\frac{\Delta}{N}.$ $$

Choosing $\eta$ such that $0<\eta<\frac{2}{L}$, we have:

$$ $\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t +1})\leq\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{ \mathcal{G}}_{t})-c\|\nabla_{A^{\mathcal{G}}}\mathcal{L}({\mathbb{D}}^{ \mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t})\|^{2}+\frac{\Delta}{N},$ $$

where $c=\eta-\frac{L\eta^{2}}{2}$ is a positive constant. This ensures a decrease in the loss function at each step, leading to convergence, with an additional term accounting for prototype divergence.

### B.1 Corollary 1: On Convergence to Stationary Point and Convergence Rate

To analyze the convergence rate, we sum both sides of the inequality from $t=1$ to $T$:

$$ $\sum_{t=1}^{T}\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{ \mathcal{G}}_{t+1})\leq\sum_{t=1}^{T}\mathcal{L}({\mathbb{D}}^{\mathcal{P}}; \Theta^{\mathcal{G}},A^{\mathcal{G}}_{t})-c\sum_{t=1}^{T}\|\nabla_{A^{\mathcal {G}}}\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G }}_{t})\|^{2}+\frac{T\Delta}{N}.$ $$

This simplifies to:

$$ $\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{1 })-\mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}} _{T})\geq c\sum_{t=1}^{T}\|\nabla_{A^{\mathcal{G}}}\mathcal{L}({\mathbb{D}}^{ \mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t})\|^{2}-\frac{T\Delta}{N}.$ $$

Rearranging the terms and dividing by $cT$, we obtain the convergence rate:

$$ $\frac{1}{T}\sum_{t=1}^{T}\|\nabla_{A^{\mathcal{G}}}\mathcal{L}({\mathbb{D}}^{ \mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{t})\|^{2}\leq\frac{\mathcal {L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{1})- \mathcal{L}({\mathbb{D}}^{\mathcal{P}};\Theta^{\mathcal{G}},A^{\mathcal{G}}_{T })+\frac{T\Delta}{N}}{cT}.$ $$

As $T$ increases, the average gradient norm decreases, indicating convergence to a stationary point, adjusted for prototype divergence. Note that the size of the divergence $\Delta$ affects the rate of convergence. A smaller $\Delta$ implies that the prototypes across clients are more similar, leading to faster convergence. Conversely, a larger $\Delta$ suggests greater variability between client data, which may slow down the convergence rate.

### B.2 Remark: On Error Bounds

By incorporating the average prototype divergence, the error bound for client $i$ is:

$$ $\epsilon_{i}^{\text{ind}}\leq\epsilon_{i}^{\text{local}}+\alpha\Delta_{i}^{ \text{avg}},$ $$

where $\epsilon_{i}^{\text{local}}$ is the local model error without federation, $\alpha$ is a constant that measures the sensitivity of the error to the prototype divergence, and $\Delta_{i}^{\text{avg}}=\frac{1}{K}\sum_{k=1}^{K}\Delta_{i}^{(k)}$ is the average prototype divergence for client $i$. Thus, the in-domain error bound can be written as:

$$ $\epsilon^{\text{ind}}\leq\frac{\sum_{i=1}^{N}\left(\epsilon_{i}^{\text{local}} +\alpha\Delta_{i}^{\text{avg}}\right)n_{i}}{\sum_{i=1}^{N}n_{i}}.$ $$

For out-of-domain accuracy (ood acc), we are interested in how well the global model generalizes across different client domains. The maximum divergence of prototypes captures the worst-case divergence between any two domains.

Given the metric:

$$ $\text{ood acc}=\frac{\sum_{i=1}^{N}\sum_{j\neq i}\text{ACC}_{i}^{(j)}n_{j}}{ \sum_{i=1}^{N}\sum_{j\neq i}n_{j}},$ $$

By incorporating the maximum prototype divergence, the error bound for client $i$ on domain $j$ is:

$$ $\epsilon_{ij}^{\text{ood}}\leq\epsilon_{i}^{\text{local}}+\beta\Delta_{ij}^{ \text{max}},$ $$

where $\epsilon_{i}^{\text{local}}$ is the local model error without federation, $\beta$ is a constant that measures the sensitivity of the error to the prototype divergence, and $\Delta_{ij}^{\text{max}}=\max_{k}\|p_{i}^{(k)}-p_{j}^{(k)}\|$.

Thus, the overall out-of-domain error bound is:

$$ $\epsilon^{\text{ood}}\leq\frac{\sum_{i=1}^{N}\sum_{j\neq i}\left(\epsilon_{ij} ^{\text{local}}+\beta\Delta_{ij}^{\text{max}}\right)n_{j}}{\sum_{i=1}^{N}\sum_ {j\neq i}n_{j}}.$ $$

We note that the in- and out-of-domain error bounds are directly influenced by average prototype divergence and maximum prototype divergence (worst-case scenario), respectively.

- •
Impact of higher sampling rate: A higher sampling rate generally leads to better accuracy because more prototypes are generated per class, providing a richer representation of the feature distribution across domains. This helps the global model better generalize to out-of-domain data by capturing a diverse set of variations.
- •
Sampling methods: Cluster sampling is particularly effective for OOD accuracy because it captures the underlying structure of the data distribution within each class by selecting multiple representative prototypes (e.g., cluster centers). Mean sampling, while computationally efficient, oversimplifies the data distribution by averaging all data points, leading to a loss of critical information needed for robust adaptation. Random sampling performs almost as well as cluster sampling in some scenarios. This may be due to the fact that random sampling, by chance, captures sufficient variations within each class, providing a diverse enough representation to improve OOD accuracy. However, it may not be as reliable as cluster sampling because it lacks systematic selection of prototypes and could miss important subgroups within the class distribution.

## Appendix C More Details about Experiment Implementation

### C.1 Basic Setup

#### Global classification head generation.

We generate the global classification head by utilizing the manually designed prompts and class names of the dataset, calculated by the pretrained text encoder in the CLIP model used in our experiments. The global classification head is obtained by averaging the ’prompt+label’ embeddings from all different domains. Following common fine-tune settings, we only train the adapter, while freezing the entire image encoder and global classification head.

#### Training.

Our simulations are conducted on a $\text{Google}^{\circledR}$ Compute Platform (GCP) equipped with 47 $\text{Intel}^{\circledR}\text{Xeon}^{\circledR}$ CPUs and 4 $\text{NVIDIA}^{\circledR}$ L4 GPUs. For global adapter training, we employ cross-entropy loss with an AdamW optimizer, setting the learning rate to 0.001. We set the maximum global rounds to 200 and implement an early stopping strategy to evaluate the convergence rounds. It is important to note that our method achieves convergence in just one global round, rendering the early stopping strategy primarily applicable to other FL methods. For simplicity, we assume that all clients can participate in every communication round in all experiments.

### C.2 Details about Implementation of Baselines

For the baseline models used in the experiments, we identified the best parameters for our dataset within the recommended parameter ranges provided in their original texts. For FedAvg, we followed the settings in the original article and used the dataset sizes of different clients as the basis for the weighted average. In FedProx, we set the regularization coefficient to 5, which is lower than the usual settings of 10, 100, or 1000. This adjustment was made because a higher regularization coefficient made it difficult for the model to converge to the global equilibrium point due to data heterogeneity. In Ditto, we used a local round number of 1 and set the regularization term to 2. For MOON, we set the coefficient $\mu$ to 0.001 and the temperature coefficient $\tau$ to 1, both within the recommended ranges of the original text. For FedProto, we used 50 as the regularization coefficient. In DBE, we adopted 0.01 as the momentum coefficient and 1 as the regular Xiang coefficient, both within the recommended ranges of the paper. For all baselines, we use the same set of hyperparameters, as shown in Table 5.

Table: Table 5: The hyperparameters used in all baselines.

### C.3 Details about Early Stopping Strategy

Each client completes one local epoch per global round. We set the total number of global rounds to 200 and implement an early stopping strategy to evaluate the convergence rounds of each algorithm. The criterion for early stopping is based on validation loss; specifically, we select the results from the round that achieves the best validation loss as the final outcome. The patience parameter is set to 10 rounds, meaning that if the validation loss does not decrease below the best recorded loss within a span of 10 consecutive rounds, the training process is terminated. By implementing the early stopping strategy, we can more easily test the convergence round of each method and use this strategy to find the round with the best result.

### C.4 Details about Knowledge Distillation in Local Adaptation

We use the most basic form of knowledge distillation strategy in our framework (Hinton et al., ), which is response-based knowledge distillation:

$$ $A^{\mathcal{L}}_{i}=\arg\min\mathcal{L}_{\text{KD}}({\mathbb{D}}^{\mathcal{F}} _{i};\Theta^{\mathcal{G}};A^{\mathcal{L}}_{i};A^{\mathcal{G}}).$ (6) $$

Here, $A^{\mathcal{L}}_{i}$ represents the local adapter for client $i$, ${\mathbb{D}}^{\mathcal{F}}_{i}$ is the local dataset, $\Theta^{\mathcal{G}}$ are the global model parameters, and $A^{\mathcal{G}}$ represents the global adapter. Then, we have:

$$ $I^{\mathcal{G}}=\{A^{\mathcal{G}}(f(\phi;x_{1})),\dots,A^{\mathcal{G}}(f(\phi; x_{n}))\},\quad\{(x_{1},y_{1}),\dots,(x_{n},y_{n})\}\in{\mathbb{D}}^{\mathcal{ F}}_{i}.$ (7) $$

Here, $I^{\mathcal{G}}$ denotes the set of outputs from the global adapter for the local dataset ${\mathbb{D}}^{\mathcal{F}}_{i}$.

$$ $I^{\mathcal{L}}_{i}=\{A^{\mathcal{L}}_{i}(f(\phi;x_{1})),\dots,A^{\mathcal{L}} _{i}(f(\phi;x_{n}))\},\quad\{(x_{1},y_{1}),\dots,(x_{n},y_{n})\}\in{\mathbb{D} }^{\mathcal{F}}_{i}.$ (8) $$

Similarly, $I^{\mathcal{L}}_{i}$ denotes the set of outputs from the local adapter $A^{\mathcal{L}}_{i}$ for the local dataset ${\mathbb{D}}^{\mathcal{F}}_{i}$.

$$ $\mathcal{L}_{\text{KD}}=\text{KL}(I^{\mathcal{G}}\parallel I^{\mathcal{L}}_{i})$ (9) $$

The knowledge distillation loss $\mathcal{L}_{\text{KD}}$ is computed as the Kullback-Leibler (KL) divergence between the outputs of the global adapter and the local adapter.

$$ $\text{KL}(p\parallel q)=\sum_{i}p_{i}\log\left(\frac{p_{i}}{q_{i}}\right)$ (10) $$

Here, $p$ and $q$ represent the probability distributions output by the global and local adapters, respectively, and $\text{KL}(p\parallel q)$ denotes the KL divergence.

## Appendix D Sensitivity Analysis of Global Convergence Threshold

During the global adapter initialization phase, we set a threshold to ensure the model stops training the prototypes when the variance of loss over multiple rounds decreases to a low value, indicating that the global adapter has converged. To test the sensitivity of the experimental results to the threshold, we test different thresholds for the global adapter initialization process, specifically 0.1, 0.01, 0.001, and 0.0001. Figure 6 shows the impact of these thresholds(^3^33Note that the rounds in the figure represent the convergence epochs for the global prototype training in the server, not the communication rounds (global rounds).). As the threshold decreases, the out-of-distribution (ood) and in-distribution (ind) performance initially increase and then decrease. In contrast, the convergence epochs and training time consistently increase. This trend is intuitive because a lower threshold requires more rounds for the model to converge. Overall, we recommend using a threshold of 0.01 or 0.001 to minimize training time and reduce the risk of overfitting.

Figure: Figure 6: Impact of different convergence thresholds on global adapter initialization.
Refer to caption: x6.png

## Appendix E More Details about results on each domain

Figure 7 and Figure 8 further illustrate the effects of random sampling and cluster sampling compared to average sampling within the MPFT framework across different sizes of the DomainNet subset. They reveal similar trends: random sampling and cluster sampling achieve more balanced performance in the Quickdraw domain, with curves approaching circular shapes and covering larger areas. This suggests that these improved sampling methods enhance the model’s ability to handle balance and diversity across various data domains.

We also observe that as the sampling rate increases in the random or cluster sampling methods, the model’s performance in the Quickdraw domain improves, leading to more globally optimized results. However, the increase in sampling results in more training convergence time consumption and higher data transmission between the server and clients, which raises both computation and communication costs, as shown in Table 3. This trade-off needs to be considered in practical applications.

Figure: Figure 7: Comparison of cluster and average sampling in MPFT framework.
Refer to caption: x7.png

Figure: Figure 8: Comparison of random and average sampling in MPFT framework.
Refer to caption: x8.png

## Appendix F Results of Multiple Clients with Same Domain

In this section, we present the results of experiments where multiple clients belong to the same data domain, a scenario commonly encountered in FL. Specifically, we partition the data from the same domain into multiple subsets, each maintaining the same class labels, and distribute these subsets across multiple clients. This setup results in a far greater number of clients than in the experiments discussed in Section 5.1. As the number of clients increases, the data distribution among clients becomes more similar, potentially reducing domain heterogeneity.

In particular, we compare the results for scenarios with 6, 18, 24, and 30 clients on the DomainNet Subset-50 dataset. Table 6 shows the ind acc, ood acc and the number of communication rounds required for different FL methods.

As the number of clients increases, we observe that methods such as FedAvg, FedProx, and MOON experience a decline in ood accuracy. This could be due to the reduced heterogeneity among clients, leading to less domain-specific knowledge being aggregated into the global model. Conversely, Ditto, FedProto and DBE show a reduction in ind accuracy, because these more personalized methods are less effective at preserving in-domain knowledge when client heterogeneity decreases.

In contrast, MPFT manages to maintain both ind acc and ood acc, similar to the original experimental setup, even as the number of clients increases. Notably, in certain cases, MPFT even improves accuracy. For example, with cluster sampling at a rate of 0.1, MPFT achieves a significant performance boost with 18 clients compared to the original experiment. This suggests that MPFT’s adaptive aggregation mechanism is robust to changes in client numbers and data distribution, making it more scalable and effective in scenarios with multiple clients from the same domain.

Table: Table 6: Results of multi clients with same domain on DomainNet Subset-50 dataset.

## Appendix G Feature space hijacking attack

Figure: Figure 9: Results of the feature space hijacking attack on our model. Each prototype was constructed from a single image. Even after one million iterations of training, the original image could not be recovered, demonstrating the security of using prototypes.
Refer to caption: x9.png

In the MPFT framework, the communication of local clients’ prototypes and a trained global adapter may present potential security vulnerabilities. We hypothesize an attack vector utilizing the architecture of our approach, to design a feature space hijacking attack. The attacker could leverage the pretrained model’s image encoder $f(\phi)$ and the uploaded prototype $p^{(k)}$ to attempt restoration of the original training data $x$:

- 1.
An estimated input $x^{*}$ is constructed and processed through the pretrained image encoder $f(\phi)$ to obtain an estimated prototype $p^{*}$.
- 2.
The mean squared error (MSE) loss is used to iteratively refine $x^{*}$, aiming to minimize the discrepancy between $p^{*}$ and the actual prototype $p$, thereby approximating $p^{(k)}_{i}$.

To demonstrate the resistance of our method to feature space hijacking attacks, we randomly selected a picture from each client to form a prototype and attempted to simulate an attacker trying to restore the picture. One picture is chosen as the prototype because if a prototype attack composed of a single picture cannot be restored, a prototype composed of multiple average representations of the same category will be even more challenging for an attacker to restore and exploit.

Figure 9 illustrates the process of restoring a single image by multiple clients. Intuitively, we observe that even after one million gradient descent iterations, the attacker still cannot restore the salient features of the original image. It is important to note that on our device, it takes nearly 8 hours to complete such iterative training for each image, imposing a significant time cost on attackers attempting large-scale attacks.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The experimental settings are somewhat vague, particularly regarding whether competitive approaches, such as FedAvg and FedProx, are trained from scratch or from the same pretrained backbone model as the MPFT. Could the authors specify this information in the paper, ideally in the experimental setup or implementation details section?

2. The privacy protection measures appear informal. While Gaussian noise is added to prototypes (i.e., embeddings) as noted in Section 5.6, it is important to assess whether the noise level provides adequate privacy protection for sensitive attributes or labels. To strengthen this section, could the authors provide a more rigorous analysis of the privacy guarantees associated with their approach? For example, quantifying the level of privacy protection using established metrics or frameworks would be beneficial.
soundness: 3
strengths: 1, Experiments on two datasets indicate a significant performance boost from the proposal, increasing accuracy by 2%.

2. The approach is intuitive, sharing prototypes using three sampling strategies.

3. The proposal is communication-computationally efficient, as it involves a low number of interaction rounds and requires only prototypes for communication.
summary: This work studies domain adaptation in federated learning scenarios, employing prototype-based fine-tuning to leverage knowledge from other clients. The fine-tuning procedure requires only one round of communication between clients and the server, making it communication-computationally efficient.
weaknesses: The motivating scenario may be impractical, as the proposal assumes that clients already possess a well-pretrained model. This assumption can be particularly challenging in sensitive areas where federated learning is essential, such as healthcare or finance, where data privacy concerns limit access to robust pre-trained models. 

Can the authors discuss specific applications or scenarios within these sensitive domains where obtaining a pre-trained model might be more feasible? Additionally, it would be valuable to explore potential solutions or adaptations of their method for situations where a pre-trained model is not available.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: I suggest that the authors remove the overly powerful pre-trained feature extractor and instead use a simpler, more foundational model for feature extraction, such as ResNet or ConvNet. This would more accurately showcase the contribution of the prototype and methodology itself.

The authors should provide a validity proof to demonstrate that training the adapter on aggregated prototypes achieves comparable performance to training on aggregated raw client data or features—or, alternatively, establish an upper bound on the performance gap between these approaches.

Although differential privacy (DP) experiments were conducted, a theoretical analysis is also necessary. Specifically, the authors should clarify the noise variance conditions required for their method to satisfy DP. In particular, the statement “Furthermore, we observe that specific noise configurations can reduce bias across heterogeneous datasets, enhancing the robustness of prototype data” requires a more robust theoretical explanation.

I would encourage the authors to further explain, both theoretically and experimentally, the differences, advantages, and potential complementarities among various prototype selection mechanisms.
soundness: 3
strengths: The main strength of the paper lies in its comprehensive experimentation, which provides a detailed view of MPFT’s performance across various scenarios, showing superior computational and communication efficiency.
summary: The paper proposes Multi-domain Prototype-based Federated Fine-Tuning (MPFT), a framework designed for Federated Domain Adaptation (FDA) by utilizing domain-specific prototypes. Instead of relying on traditional model aggregation techniques, which often falter due to data heterogeneity, MPFT transfers prototypes (compressed domain representations) to the server. This approach allows the server to learn a global adapter that improves both in-domain and out-of-domain performance. MPFT also incorporates differential privacy to protect prototypes from potential data leakage.
weaknesses: The Multi-domain Prototype-based Federated Fine-Tuning (MPFT) method proposed in this paper is overly simplistic, primarily relying on basic prototype sampling strategies—such as mean, cluster, and random sampling—to generate client prototypes. Additionally, because the model does not undergo training on each client’s data, it fundamentally contradicts the original design principles of federated learning (FL). The performance gains observed in the experiments likely stem from the strong pre-trained feature extractor, which does not require training on all client data but only a few representative prototypes to achieve reasonable performance. If the pre-trained feature extractor were removed and the model was trained solely on prototypes, the results would likely be poor. If only prototype fine-tuning is needed, then why even employ a federated learning framework? Thus, despite achieving certain results, this approach introduces no complex mechanisms or innovative architectures to enhance the FL system, lacking fresh or original design. It merely follows a standard process: feature extraction with a general pre-trained model, prototype selection, server training, and client fine-tuning, without presenting any real novelty.

The paper does not provide a thorough theoretical analysis of the different prototype sampling methods, nor does it uncover the fundamental differences between these sampling strategies in handling heterogeneous data distributions. Although the paper presents experimental results comparing mean, cluster, and random sampling methods, it lacks detailed explanations on the core differences among these strategies in terms of learning mechanisms, communication efficiency, and personalization effects. Consequently, the paper falls short in theoretical depth, failing to provide any deep insights into the impacts of these sampling strategies.

The theoretical analysis in this paper primarily focuses on convergence. However, training the adapter on aggregated prototype data is no different from centralized data training, so convergence is naturally expected. To provide valuable theoretical insights, the paper would need to show that the adapter’s performance on aggregated prototypes is comparable to that on aggregated raw client data or offer theoretical performance bounds. Furthermore, transmitting prototypes instead of models introduces a higher risk of privacy leakage, so the analysis should prioritize differential privacy (DP) guarantees rather than focusing solely on convergence. The current paper fails to address theoretical guarantees for privacy protection, which is particularly critical in privacy-sensitive federated learning contexts.

Although the experimental section is comprehensive, covering multiple datasets, various sampling strategies, and different DP configurations, it does not clearly reveal the functional differences and applicability of each prototype sampling approach across different scenarios. While the experiments test different prototype configurations, they lack an in-depth discussion on how these configurations perform under different data distribution patterns and sample sparsity conditions. This results in a set of experiments that, while extensive, fails to provide a robust summary of the unique characteristics of the prototype methods, thus missing an opportunity to elevate the research value.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Would the authors explain how you chose the parameter $s$ in Section 5.6? 

2. Why is the class $k$ missing in the proof in Appendix B (while formulated before proof)?

3. Can authors make more comparisons with recent (>=2023) works? For example,
Fedcp: Separating feature information for personalized federated learning via conditional policy. In KDD 2023.
FedFed: Feature Distillation against Data Heterogeneity in Federated Learning. In NeurIPS 2023.
Fedgh: Heterogeneous federated learning with generalized global header. In MM 2023.

It would be better to compare aggregation-based improvements in recent works and the new mechanism proposed in this paper. Since the authors stressed the drawbacks of these works, it would be better to give detailed support or explanation.
soundness: 3
strengths: + They propose an interesting idea -- MPFT as a one-round federated fine-tuning approach for multi-domain environments, demonstrating its performance improved over previous methods. 
+ A new metric is introduced to evaluate model adaptability, assessing both out-of-domain and in-domain accuracy to balance knowledge retention and domain adaptation.
+ The writing and logic are clear and easy to follow. The problem formulation is clear and friendly to readers.
summary: This paper introduces Multi-domain Prototype-based Federated Fine-Tuning (MPFT), a framework designed to address data heterogeneity and data privacy in federated learning (FL) by using prototype-based training rather than traditional averaging methods. In MPFT, each client generates a set of representative data embeddings (prototypes) that capture essential domain-specific characteristics without transferring raw data. These prototypes are then aggregated at the server, allowing for a simulated centralized learning approach and enabling fine-tuning of a global adapter. This method aims to achieve performance on par with centralized learning while solving the challenges incurred by aggregating client models.
The efficiency of MPFT lies in that it requires only a single round of global communication, significantly reducing computational and communication costs compared to multi-round FL methods. Furthermore, by selectively sampling prototypes, the framework limits data transfer volumes. To ensure privacy, MPFT integrates differential privacy mechanisms, mitigating risks of data exposure and rendering prototype-based data reconstruction ineffective—even when the prototype encoder is known to potential attackers.
weaknesses: - The proof of convergence in Section B lacks the differential-privacy-related analysis. A supplemental analysis of "MPFT with DP" should be added for theoretical completeness. Intuitively, the addition of differential privacy introduces a bounded randomness during the convergence process.

- The selected experimental counterparts for comparison are mostly in 2017-2022, which are relatively not new. In Section 2, the authors think the "averaging-based aggregation results in poor out-of-domain adaptation performance." It would be better to show the performance of out-of-domain adaptation from the recent advances atop model/parameter aggregation. Some recent works (with or without aggregation) relevant to data heterogeneity in federated learning are expected to compare, such as,

Fedcp: Separating feature information for personalized federated learning via conditional policy. In KDD 2023.
FedFed: Feature Distillation against Data Heterogeneity in Federated Learning. In NeurIPS 2023.
Fedgh: Heterogeneous federated learning with generalized global header. In MM 2023.

Meta-review: This paper proposes to enhance federated domain adaptation and, in particular, to mitigate the challenge of data heterogeneity. The authors propose a framework known as Multi-domain Prototype-based Federated Fine-Tuning (MPFT). It finetunes a pre-trained models using multidomain prototypes. MPFT enables supervised learning on the server to derive a globally optimized adapter that is subsequently distributed to local clients, without the intrusion of data privacy. It is shown empirically that MPFT improves in- and out-of-domain accuracy over conventional federated domain adaptation methods. Privacy is also considered. 

Generally, the reviewers felt positive about the paper, which uses some logical and natural approaches to solve the problem. While nothing is overly surprising, the theoretical and experimental results are sound. MPFT is low cost, efficient and robust. I believe that MPFT may be useful in future federated domain adaptation tasks.

Decision: Accept (Poster)