Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning
Authors: Jingyuan Zhang
Yiyang Duan
Shuaicheng Niu
YANG CAO
Wei Yang Bryan Lim
Venue: ICLR 2025 Poster (2025)
Keywords: Federated Learning; Federated Domain Adaptation; Federated Fine-Tuning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Federated Domain Adaptation (FDA) is a Federated Learning (FL) scenario where models are trained across multiple clients with unique data domains but a shared category space, without transmitting private data. The primary challenge in FDA is data heterogeneity, which causes significant divergences in gradient updates when using conventional averaging-based aggregation methods, reducing the efficacy of the global model. This further undermines both in-domain and out-of-domain performance (within the same federated system but outside the local client), which is critical in certain business applications. To address this, we propose a novel framework called \textbf{M}ulti-domain \textbf{P}rototype-based \textbf{F}ederated Fine-\textbf{T}uning (MPFT). MPFT fine-tunes a pre-trained model using multi-domain prototypes, i.e., several pretrained representations enriched with domain-specific information from category-specific local data. This enables supervised learning on the server to create a globally optimized adapter that is subsequently distributed to local clients, without the intrusion of data privacy. Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation in FDA. Notably, MPFT achieves convergence within a single communication round, greatly reducing computation and communication costs. To ensure privacy, MPFT applies differential privacy to protect the prototypes. Additionally, we develop a prototype-based feature space hijacking attack to evaluate robustness, confirming that raw data samples remain unrecoverable even after extensive training epochs. The complete implementation of MPFL is available at \url{https://anonymous.4open.science/r/DomainFL/}.

Published as a conference paper at ICLR 2025
ENHANCING FEDERATED DOMAIN ADAPTATION WITH MULTI-DOMAIN PROTOTYPE- SOASEE FEDERATED FINE- TUNING
Jingyuan Zhang', Yiyang Duan', Shuaicheng Niu', Yang Cao", Wei Yang Bryan Lim!
I College of Computing and Data Science, Nanyang Technological University 2 Department of Computer Science, Institute of Science Tokyo
(jzhang149, yiyang007]ee ntu. edu. sg, shua cheng ni niunntu. edu. sg caoßc. t witech.cc....................... jp, bryan. 1 limwyentu.eer
ABSTRACT
Federated Domain Adaptation (FDA) is a Federated Learning (FL.) scenario where models are trained across multiple clients with unique data domains but a shared category space, without transmitting private data. The primary challenge in FDA is data heterogeneity, which causes significant divergences in gradient updates when using conventional averaging- weeseging-based based aggregation methods, reducing the efficacy of the global model. This further undermines both in-domain and out-of-domain performance (within the same federated system but outside the local client). To address this, we propose a novel framework called Multi-domain Prototype-based Federated Fine-Tuning (MPFT). MPFT fine-tunes a pre-trained model using multi- domain prototypes, i.c.. pretrained representations enriched with domain -ppecii information from category-specite local data. This enables supervised learning on the server to derive globally optimized adapter that is subsequently distributed local clients, without the intrusion of data privacy. Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation FDA. Notably, MPFT achieves convergence within single communication round, greatly reducing computation and communication costs. To ensure privacy. MPFT applies differential privacy to protect the prototypes. Additionally, develop prototype- based feature space hijacking attack to evaluate robustness, confirming that raw data samples remain unrecoverable even after extensive training epochs. The complete implementation of MPFL is available at hhttps://ndt zjy github. Withub.io/Domee wat.oiii/DDmmann..
INTRODUCTION
Federated Learning (FL) is privacy-preserving distributed machine learning paradigm designed to protect the data of participating clients (McMahan et al., 20176). In FL., only models trained on local data are shared between clients and servers, rather than the raw data itself, mitigating the risk of data leaks. Mainstream FL research primarily focuses on optimizing each client's 's performance within its local data domain (in-domain performance). However, Federated Domain Adaptation (FDA) scenarios, clients need to perform well on the collective data domains shared by all participants to meet certain business requirements. For instance, consider a consortium banks training a model to detect fraudulent transactions. Each bank has distinct customer bases and transaction patterns, leading to variations their data (domains). A good out-of-domain performance is essential for anticipating new risks that bank may not be exposed to yet. The challenge is to ensure the global model performs well across banks, achieving good in-domain accuracy (within each bank's 's typical cases) and out-of-domain accuracy (generalizing across the federated financial system).
Studies have shown that FL achieves results similar to centralized training only when the datasets among clients are independently and identically distributed (i.i.) and share similar domain char-
"Coreesonding author

---

Published as a conference paper at ICLR 2025
colors: domain
class
e
prototype
aggregated model
local model
centralized leamming
(b) Fl. with averaging based aggregation
(c) MPFT with average sampling
(d) MPFT with cluster sampling
(e) MPFT with random sampling
Figure 1: Comparison MPFT to centralized learning and previous averaging- based FL approaches.
acteristics (Liu et al. 2023 Seol & Kim 2023 Shaheen et al.) 2022). In typical scenarios where all clients' models are trained on local data from the same domains, update directions are aligned, making it feasible to use averaging-basee aggregation algorithms such FedAvg (McMahan et al. 2017b). However, in FDA, in which clients have unique domains but shared category space, the significant dissimilarities of feature space among local model updates mean that averaging of weights may not yield an optimal global model (Su et al. 2024 Sun et al. (021).
To address this issue, propose novel framework called Multi-domain Prototype-base Federated Fine-Tuning (MPFT). MPFT tackles this issue by having each client generate specific proportion of data embeddings (i.e., prototypes) to be transmitted the server create a prototype training dataset. This allows us to simulate a centralized learning approach without transferring raw data, as prototypes encapsulate sufficient domain-specific features represent the entire data domain. The server then fine-tunes a global adapter using this comprehensive set of prototypes, with the goal of approximating the performance of centralized learning without relying on conventional averaging approaches to derive a global model. Figure illlustrates how our aggregation method compares to centralized learning and traditional averaging- baase FL approaches.
Since only specific proportion of local data embeddings are sampled as prototypes, and MPFT requires only single round of global communication to converge, our framework incurs significantly lower computation and communication costs compared to other multi-round FL frameworks. MPFT also incorporates a differential privacy mechanism to mitigate the risk that the original data of specific prototypes exposed during the prototype transmission process. Furthermore, simulations of feature space hijacking attacks on MPFT demonstrate that attackers cannot reconstruct the original data from the uploaded prototypes, even when the pretrained prototype encoder is known.
Our contributions are as follows: a) We propose MPFT, a one-round federated fine-tuning framework with convergence guarantees that outperforms previous methods multi-domain environments. b) We introduce a novel metric to evaluate the performance of the FDA, specifically assessing out- of-domain and in-domain accuracy consider the trade-off between knowledge preservation and adaptation. c) We demonstrate empirically that MPFT incurs lower computational and communication overheads as compared other FL methods while ensuring privacy through differential privacy protection of the prototypes and maintaining robustness against feature space hijacking attacks.
2 RELATED WORK
Efforts to enhance performance FL. across heterogeneous datasets (FDA) have been extensive. Regularization methods during training, such as FedProx (Li et al. (0200) and FedDyn (Jin et al. 2023). or knowledge distillation techniques like FedGen /Venkateswaran et al. 2023) and FedNTD (Lec et al. 2022), aim to align local models more closely with the global model. Such approaches help mitigate the divergence among local models but rely on the assumption that the averaged global model is well-suited for the entire data distribution which often does not hold in FDA.
Updating only parts of the model during aggregation is another strategy, exemplified by personalized FL. (PFL.) (Tan al. 2022a), which aims to customize local models to enhance in-domain perfor-
2

---

Published as a conference paper at ICLR 2025
mance. Some methods involve using a personalized aggregation base to select specific portions of global information for model aggregation, or updating only parts of the model during the aggrega- tion phase, e.g., APFL (Deng et al. 2020). FedFomo (Zhang et al., 2020), FedAMP (Huang et al.) 2021). FedPHP (Li et al. c021c). , APPLE (Luo & Wu 2022), and FedALA (Zhang et al.) 2023a). Additionally, some methods split model layers segregate global and local components, such in
FedPer (Arivazhagan et al. 2019). LG-FedAvg (Liang et al. 2020), FedRep (Husnoo et al. 2022).
FedRoD (Chen & Chao 2021). FedBABU (Oh al. 2021), FedCP (Zhang al. 2023b). FedGH Yi et al. 2023), and DBE Zhang et al. 2024). However, key drawback is these methods still rely on averaging- based aggregation, which results poor out-of-domain adaptation performance, despite achieving some in-domain improvements.
Some methods incorporate prototype learning. For instance, FedProto (Tan et al. 2022b) utilizes averaged local prototypes to train as the global prototype on local clients. FPI. (Huang et al. (023) transmit all the prototypes (data embeddings) averaging, which is then sent back to the clients for further optimization. This method introduces significant communication overhead and potential privacy leakage during the first phase. FedNH (Dai et al. 2023) leverages class prototype transmission to address class imbalance across clients, rather than tackling the more complex issue of domain heterogeneity.
the server during the initial phase for clustering and
Unlike the above methods, do not assume that the average aggregation of different client models or prototypes can represent the global model or global prototype in scenarios involving heterogeneous client data. MPFT uses prototypes from different clients as basic units that replace data distribution for centralized training on the server, thus obtaining an approximate global model capable of fitting all heterogeneous client domains distribution, as demonstrated in Appendix A
3 PROBLEM STATEMENT
Scenario assumption. Consider scenario involving N clients, where each client possesses private training dataset D..... .., Dy each drawn from a unique data domain and shared category space]] Our goal is to train model that balances domain knowledge preservation and domain knowledge adaptation. Domain knowledge preservation refers to the ability of the model to retain each client's unique domain insights. Domain knowledge adaptation is defined the model's 's ability to indirectly extract and transfer knowledge from each participating client's 's domain to others, even client cannot directly train on another client j's data.
Optimization goal. Building on this, we define the Domain Knowledge Preservation loss L and the Domain Knowledge Adaptation loss A as follows:
C. = C(OP:DD00))
C CA=
Σ Σ 1)) トー
(1)
where L()) denotes the loss function, O denotes the local (personalized) model parameters for client i, and 0 denotes the global model parameters. conventional FL, the local models . are periodically synchronized with the global model e
The optimization goal is to decrease both the L" and L. Thus, we formalize the FDA optimization
goal as follows:
(OP. O.. [ arg min(a) of
where at and of are client-defined weight parameters which balance the trade-off between domain knowledge preservation and domain knowledge adaptation, in line with the "no free lunch" theorem.
Evaluation metrics. To quantify the effectiveness of the optimization, propose two metrics:
in-domain accuracy (ind acc) and out-of-domain accurucy (ood acc). Denoting ACC!i) to be the accuracy for a client i when tested on domain j. ind acc and ood acc are defined as follows:
'These domains may be largely isolated with minimal overlap, but MPFT also generalizes scenarios where each client may have data from multiple domains, as demonstrated in Section
3

---

Published as a conference paper at ICLR 2025
ind Ee ii ACC
Σ E Σ Di n2
jz ACC
ood acc
(3)
Si=l n4
where is the number of test samples for client The ind acc measures each client's performance on its own domain data, while the ood acc evaluates adaptation performance when tested on data from other domains.
4 METHODOLOGY
Overview. We introduce Multi-domain Prototype- Federated Fine-Tuning (MPFT), which consists of three main components: Prototype Generation, Global Adapter Initialization, and Few-shot Local Adaptation, as depicted Figure 2 During the Prototype Generation phase, we generate domain-specifie prototypes for each client based specific sampling methods and ratios. The clients subsequently transmit their local prototypes to the server. In the Global Adapter Initialization phase, we utilize these prototypes train global adapter designed handle the multi-domain distribution of all clients, thereby improving the ood performance of the global adapter across clients. The global adapter then sent back to the clients for local inference. While the global adapter performs well in ood accuracy, some clients may require better ind performance. In such cases, they can proceed to the Few-shot Local Adaptation phase, where few-shot dataset is sampled locally to further fine- the local adapter. Knowledge distillation is employed to mitigate catastrophic forgetting of global knowledge during this phase.
Server
Client
Prototype Generation
Globaal Adletter Initidiiation
protegge
Tiann Data
Baskbcoe
Sead protutppi Eoe differet diects
Testure
Prontype
Few--soo Local Adaotaation
UM
Ttw shat
1xput
one-tre
Clotal Adaaleer
Cloba Adapter
glvbll prokespe frgut
Clnn Data
local prodittion
lced Alkgpdaccii Lom bail pavi
Send optiiized ginzal adliptor
giobel grecstoo
Global
Globel Hoad
Figure 2: An overview of MPFT.
Prototype Generation. A prototype is a compact representation specific class feature that unique to each domain. To synchronize the consistency of the prototype's embedding space across domains, each client utilizes the same pretrained image encoder to generate the prototype set (P(1) p(()), }. where K. represents the total number of classes. Although all clients share the same label space, each label manifests uniquely within its respective feature domain.
To generate these prototypes, different sampling methods can be applied, including Mean Sampling, Cluster Sampling, and Random Sampling, as described in Algorithm [ The choice of sampling method depends on the desired trade-off between computational efficiency and prototype representa- tional robustness. In Mean Sampling, each client generates prototype for class k: by calculating the mean of the pretrained embeddings for that class. In Cluster Sampling, clustering (c.g., k-means) performed on pretrained embeddings of each class, and certain number of cluster centers are then selected based on predefined sampling rate to form the prototype set. In Random Sampling, fixed number of pretrained embeddings of each class are randomly selected according to the sampling rate, and these selected embeddings constitute the prototype set.
Once each client has generated their prototypes, they transmit these to the server. Consequently, the server accumulates N domain-specific representations subset (prototypes subset) for each class k
which are collectively represented as P
f
p
4

---

Published as a conference paper at ICLR 2025
Algorithm Different Sampling Methods
1: function MEAN SAMPLING(D:)
2: for class 1,..., do
3: p(k) | (x.))D f();z)
Compute mean embeddings for each class
4: end for
5: return P, f (P(1) ,...,, !!!]
6: end function
7: function CLUSTER SAMPLING(D,;
8: for class in ..... do
9:
Ch - r
x
D 1
Set the number of cluster centers b Perform cluster sampling
10: (p(k) Cluster (f(b:
11: end for
12: return P3 U P((i)
13:
end function
14: function RANDOM SAMPLING(D;, for class in 1,. I..... do
16: C²
> Set the number of randomly selected embeddings
TP(A) Randoml SandomlySelect ySelect )
17:
Perform random sampling
18: end for
return P. + - p())
end function
Global adapter initialization. Algorithm 2 outllnes the global adapter initialization process. Note that we avoid averaging-based aggregation within each class across different clients, this would distort the global distribution. Utilizing D'P. we train the global adapter A to adapt to the entire system's 's data distribution with cross cross-entropy loss L:
(0*.4% = arg min (1;:;0;;9))
Upon successful training, the global adapter A is sent to the clients, replacing their local adapters.
Algorithm 2 Federated Learning with Global Adapter Initialization
Input: N clients, L: loss function, O (f()), ((0)}
pretrained CLIP model O with image encoder f()) and text encoder g(Ø). A random initialized adapter, ry: learning rate, K: number of data classes, iD; client training data, (z, y): data sample, method: sampling method (e.g., mean, cluster, random), sampling rate.
Output: Reasonable global adapter A
1: Generate linear probe classification head H by labels and pretrained text encoder g
2: Server sends (() 1 and H to all clients initialize local models.
3: for client in 1,..., N in parallel do
4: IP: + hod rD,,r)
5: end for
6: Clients send the prototype to the server.
7: Server constructs the prototype training dataset D by D + I.
8: while A does not converge do
9: Server optimizes A by A - L
10: end while
return AG
Few-shot local adaptation. While global adapter A performs well in ood accuracy, it may not be sufficient for ind accuracy. To address this, clients can use their local few-shot data D. further fine-tune AG adapting it to their local domain and improving ind accuracy, as shown in Algorithm
5

---

Published as a conference paper at ICLR 2025
Algorithm 3 Few-shot Local Adaptation
Input: N joining clients, C: loss function, θ
(f()), ( g())): pretrained CLIP model t with image encoder f()) and text encoder g A global initialized adapter, a: local learning rate, I:: number of data classes, F: few-shot number, M. client training data, (x. y): data sample.
Output: Reasonable local adapters
1: Server sends A to all clients to initialize local adapter A
2: for client in 1,...,,. in parallel do
1...., FF} 3: Df {US- f(om, Mnn) E DD; 3m
D. Generate few-shot data
4: Client i obtains Af by Af A. v/ LID A9).
Local adaptation
5: end for
6: return {AF A
To avoid catastrophic forgetting (French 1999) of global knowledge during the local adaptation process, knowledge distillation (KD) method is employed regularize the locally trained adapter A ensuring it does not deviate too much from the global adapter A The loss function for local adaptation is defined as:
L = Lce + BLKD.
(5)
where LCE represents the cross-entropy loss, and LKD denotes the KD loss. The hyperparameter balances the influence of the KD loss in the overall objective. In Section 5.4 of different KD weights 3 on balancing ind accuracy and ood accuracy.
compare the effects
Convergence. We analyze the convergence of MPFT, with detailed proof's provided in Appendix B
Theorem (Convergence of fine-tuning with prototypes) For a smooth, non-convex loss function with a Lipschitz continuous gradient with constant L, the global fine-tuming using prototypes D converges. The sequence of updates for the global adapter A achieves monotonic decrease in the loss function L(DP; 09,A9 ,A),. Specifically, choosing a learning rate 1) such that < fj ensures:
C(DP;05 AlT1) O - AO 08, )1
N
where c = Lui is a positive constant, ensuring that the step size is appropriately bounded for convergence. Here, A is the maximum prototype divergence across clients.
Corollary 1.1 (Convergence to stationary point and rate) As T increases, the average gradient norm decreases, indicating convergence to a stationary point:
T
CAs 12) L(DP;09 ;0, L(DP; AA))
TA
N
cT
This indicates that as T increases, the right-hand side approaches zero, confirming that the gradient norm diminishes and the updates converge to stationary point.
5 EXPERIMENT
Datasets and Models. To simulate a FDA enviromment, We use the DomainNet (Peng et al. 2019) and PACS (Li et al. 2017) datasets which are widely used in multi-domain data adaptation. For these datasets, we employ pretrained CLIP models from OpenCLIP (Cherti et al.) 2023 Radford al. 2021, Schuhmann al. 2022). The image encoder of CL.IP for NomainNet ViT-B-32 pretrained on the LAION-2B dataset, while for PACS, we use a ConvNeXT-Bass pretrained on the LAION-400M dataset as the image encoder.
Implementation details. We implement various representative FL algorithms baselines, includ- ing FedAvg (McMahan et al.] (0177)), FedProx (Li et al. 2020). Ditto (Li et al.) 20216). MOON (Li al. 2021a), FedProto (Tan et al. 2022b), and DBE (Zhang al. 2024), using the Py Pyrchh library

---

Published as a conference paper at ICLR 2025
and based the integrated FL library PFLlib (Zhang al. 2023c). To simulate the common FL scenario where data resides only on clients, we split the data for each client into a training set (70%), test set (20%), and validation set (10%). We evaluate in-domain (ind) and out-of-domain (ood) accuracy following Equation 3 Too demonstrate the scalability of our method, we partition the DomainNet dataset, which includes 345 categories, into subsets containing 50, 100, and 150 classes, respectively. To more conveniently compute the convergence time for each FL method and compare the computational and communication costs of them, we introduce an early stopping strategy during training. More details about our experimental setup and baselines can be found in Appendix C.1 and Appendix C.2 with details on the early stopping strategy provided in Appendix
Diverse FDA Scenarios. We conducted experiments and analyses from different perspectives on various potential FDA scenarios. Section [5.] presents the performance of different FL methods in a basic scenario, where each client is assigned a unique data domain, with minimal overlap between domains. Section provides prrovidss detailed analysis of the global model's performance each client, exploring the fairness of different methods in FDA. In Section 5.3 we design a more realistic scenario where each client may hold data from multiple domains, reducing the domain heterogeneity between clients. Appendix sets up another realistic scenario where multiple clients belong to the same data domain, significantly increasing the number of clients compared the original setup. In Section (5.4. we investigate the role of local adaptation. Section 5.5 compares the computational and communication costs of different methods. Finally, in Section 5.6 and Appendix [ we explore the differential privacy mechanism in MPFT and its robustness against feature space hijacking attacks.
5.1 PERFORMANCE ON MULTI-DOMAIN
We evaluate method alongside other FL approaches in Table [ including local training (i.e., each client fine-tunes the pretrained model separately). Empirical results show that local training excels in ind accuracy but performs poorly in ood accuracy. A reason is that local fine-tuning results in catastrophic forgetting (Luo et al. 2023). Personalized FL. methods such as FedProto and DBE, which generally maintain a personalized local model for each client, have higher ind accuracy but compromise ood accuracy. In contrast, methods like FedAvg, MOON, and Ditto demonstrate more balanced improvements in both ind and ood accuracies. FedProx, which introduces a regularization term between the global and local models, improves ood accuracy at the expense of ind accuracy. In comparison, our method consistently achieves superior performance in both ood and ind accuracy across all DomainNet subsets. As the subset size of DomainNet increases, we observe variable convergence stability across methods such as FedAvg, FedProx, and Ditto, while DBE demonstrates accelcrated convergence. In contrast, method requires only one global communication round, which significantly reduces both computational and communication costs. This benefit is further elaborated in section 5.5
Table 1: Test accuracy and communication rounds for different FL methods on DomainNet subsets and PACS. The communication rounds are determined using an early stopping strategy, where fewer rounds indicate faster convergence. Additionally, we compare the sensitivity of MPFT to different global convergence thresholds to ensure the robustness of the results across various hyperparameters. as refer to Appendix D
DomainNet: Subver-501
DomainNer: Subset-100
DomainNet: Subsei-150
PACS
ood acc
ind
rounds
ood acc
ind acc
rounds
ood
ind ace
rounds
ood acc
ind ncc
rounds
0.7361
U.8600
0.6554
0.8310
0.6003
0,8067
0.6547
0.9984
FodAvg
(MeMalun et al 701
0.7902
0.7345
24
0.7628
0.6966
49
0.7263
0.6700
0.9725
0.9887
FedProx I si 2000
0.7752
0.7178
10
0.7499
0.6827
9
0.7131
0.6569
0.9219
0.9659
13
Ditt
0i 20110
0.7811
0.7621
20
0.7511
0.7182
30
0.7149
0.6904
0.9172
0.9830
35
MOXX
0.7902
0.7344
28
0.6952
0.7267
0.9763
FedProlo Lan clal 20122
0.7296
0.7696
5
0.6732
0.7385
0.6321
0,7073
0.8627
0,9963
32
DBE
[Zaing al. 2024
0.7421
0,7622
22
0.7179
0.7233
0.6820
0.6956
0.971
0.984
12
MPFT
0.8077
0.7813
0.7674
0.7399
0.7294
0. 0,7099
0.9486
0.9703
MPFT (Cluster, 1)
0.7951
0.7957
0.7692
0.7171
0.9880
MPFT (Cl Clusser, rate=#0.3)
0.8201
0.8294
0.7766
0.7791
0.7430
0.7514
0.9541
0.9896
MPPT
(Random, rate-0.1 0.1)
0.7953
0.7899
0.7566
0.7509
0.7194
0.7233
0.9829
0.9888
MPFT (Random, rate=0.3)
0,8236
0.8294
0.7803
0.7811
0.7469
0.7542
0.9887
0.9919

---

Published as a conference paper at ICLR 2025
5.2 DETAILS ABOUT RESULTS ON EACH DOMAIN
To further explain why MPFT achieves better ind and ood accuracy compared to the baselines, we visualize the performance of each domain Figure Each axis of the radar chart represents specific data domain (e.g.. Real or Painting), with the shape and coverage area of the curves illustrating the global model's performance across these domains. Empiricallyy, the roundness of the curve could reflect the fairness of the model across different clients (domains) the rounder the curve, the more fair the method is in the global distribution, leading to better ood accuracy.
Compared other FL baselines across different DomainNet subset sizes, MPFT with average sampling method performs exceptionally well in the Quickdraw domain, with a more balanced curve shape. Additionally, MPFT maintains strong performance across other domains relative the baselines, thereby achieving better overall fairness. For more details about the effects of random and cluster sampling compared average sampling on each domain, please refer Appendix B
Fechugg
FecXrox
Ditto
MOON
FeciProto
OBE
MPFT (overage)
Clipurt
Cligart
Clipart
Infograph
Painting
Infograph
Painting
Infograph
Painting
Quickdraw
Real
Quickdraw
Quickdraw
Roal
Skitch
Skitch
Skatch
DomainNat, Subset-50
DomainNet, Subset-100
DomainNet, Subset-150
Figure 3: Comparison of different FL methods across various DomainNet subset sizes.
5.3 IMPACT OF MULTI-DOMAIN DIFFERENCES ON PERFORMANCE
In real-world scenarios, a client may contain data from multiple domains rather than single specific domair We simulate a situation where each client contains 1 mrr percent of data from its original domain, mixed with mr percent of data from another domain. Here, mr represents the mixed ratio, indicating the level of domain diversity on the client side. We evaluate our method alongside other FL approaches under this scenario, as shown in Table where DomainNet subset-50 used. We observe reduction in the performance advantage of our method compared to others, as the mixed ratio increases. This decline is due to the reduced heterogeneity within the FL system, which diminishes the strengths of our approach. However, it is still evident that our method outperforms most FL algorithms, particularly when using Random and Cluster sampling strategies.
Table 2: Results of mixed multi-domain situation DomainNet Subset-50.
original
mixed ratlo 0.3
mixed ratio 0.4
mixed mili D.5
rounds
rounds
0.7361
0.3600
0.7316
0.8477
B.7339
0.844
0.74UK
RodAvg (MeMalun et
2017
0.7902
0.7345
24
0.7975
0.7639
19
B.790
0.7691
0.8090
0.7645
FedProx
0.7752
0.7178
10
0.7798
0.7386
67
0.7813
0.7148
0.7903
0.7588
0.7811
0.7624
0.8027
MOON
L1 20cial
0.7902
0.7344
28
0.7984
0.7639
0.8043
0.7763
0.8109
0.7865
FedPpplo
Tdn cl 2012b
0.7296
0.7696
5
0.7110
0.7395
0.7075
0.7421
0.7019
0.7293
DBE
Zrang cl
0.7421
0.7622
22
0.7348
0.7371
0,7293
0,7349
0.7286
0.7331
MPPT (Average)
0.7813
MPFT (Clusser, rate=0.11
0.7951
0.7957
0.8213
0.8169
0.7887
0.7897
0.8007
0.7912
MPFT (Cluster, ratc=00.3)
0.8201
0.8294
0.8276
0,8253
0.8209
0.5142
0,8142
0.8071
MPFT (Random, rate-0.
0.7953
0.7899
0.7928
0.7856
0.8040
0.7958
0.7919
0.7801
rate==.3)
0.5141
Additioonlll, is common for multiple clients share the same data domain, refer Appendix
8

---

Published as a conference paper at ICLR 2025
5.4 PERFORMANCE WITH LOCAL ADAPTATION
We compare the few-shot performance of local adaptation with different knowledge distillation (KD) weights in Figure As the KD weight increases, there is less out-of-domain knowledge forgetting but worse in-domain knowledge alignment. With an increase in the number of few-shot samples, the ood and ind accuracy show similar trend. We provide more details about the experiment implementation of KD in local adaptation in Appendix C.4
zubset 50, ldd weight=O
Subset 50, kd Weight=10
subset 50, ldd welght-100
subset 50, kd weight-1000
1.800
3850
0.850
1.800
2.000
2.773
2373
0.773
3.50
9.55
0.750
4.9%
2325
0.725
6.70%
2.775
2613
0.675
2.575
afvest
Buteet
abvees det
hww-shot Number
heweshtt A Kumber
Pesnaho: Aumber
Peonaho: Number
subset 100, kd W
Subset 100, kd weight=10
subset 100, kd weight=100
subset 100, kd 1000
3.800
3.50
0.850
1.000
3.800
2.800
Acourayy
2,775
2373
0,775
3.50
2.50
0.250
2.75
2725
0.725
8.723
1.675
2.755
0.675
1,675
Rideet
Boteet
aketit
Tew-shot Number
Feowshot Kumber
Peonshot Kumber
Pegrshor Number
subset 150, kd W
Subset 150, weight=10
gubset 150, kd weight=100
subset 150, kd weight- 1000
3.800
3.50
0.050
7.000
3.000
3.000
Squuccy
3.775
2375
2.50
2350
0.750
0,250
2.73
2325
8.729
1.775
2.75
0.455
4,475
adtet
Lindel
ghatet
stebet
that
situfio
Botee
okebol
Tew-shot Number
Tewetho Kumber
Feeeshot Kumber
Teg-shot Number
Figure 4: Few-shot performance comparison of local adaptation with different KD weights.
5.5 COMPUTATION COST AND COMMUNICATION COST
We further analyze the computation cost and communication cost in Table Computation cost is the total training time of FL, which is related to the number of communication rounds and the computational complexity within each round. Among the methods compared, Ditto is the most time-consuming due to the additional local training epoch it requires. In contrast, the MPFT, which converges in just one global round, significantly reduces training time, particularly when the sampling method is set to average. For other sampling methods, such Random and Cluster, our approach trades a modest increase in training time for substantial improvements in both ind and ood accuracy. Communication cost is the number of parameters transmitted, which theoretically influenced by the number of communication rounds R. the model (adapter) parameters Σ. and the prototypes II. The number of communication rounds R directly contributes to a linear increase the overall communication cost. Furthermore, the model adapter parameters Σ and the size of the prototypes II determine the communication cost per round in these specific algorithms. In our empirical results, MPFT with average sampling achieved the lowest communication cost across all experiments. However, while random or cluster sampling slightly increases communication overhead, it also significantly improves MPFT's performance (see Table T).

---

Published as a conference paper at ICLR 2025
Table Computation cost and communication cost of FL methods. R represents the convergence round, Σ represents the number of model (adapter) parameters, II represents the size of prototypes, N represents the number of clients. The subsets are derived from the DomainNet dataset.
Computation (tocal truining time)
Communication Cost (total parameter transmission)
Subset-50
Subser-100
Subset-130
150
PACS
Subset-50
Subsct-101
Subsci-130
PACS
Theoretical
lcall
240.8s
834.8s
1575.Is
311.2s
0
TedAvg
1933.5s
5812.4s
3824.6s
663.9s
144MB
294MB
102MB
128MB
x
FedProx
811.96
1253.5s
1120.14
270.0s
60MB
S4MB
30MR
52MB
R
x
N
8365.8s
5862.3s
1455.68
120MB
180BB
N
MOON
2164.24
2237.46 44
7027 4s
872.76
168MB
96MB
186M
168MB
R
FedProto
393. Is
1155.4s
1608.0s
683.3s
5.9MB
188MB
24.6MB
3.5MB
R x
1693.6s
1160.3s
132MB
30MB
x
MPFT (Average)
1.9%
7.3s
10.7s
0.1s
3.6MB
4.2MB
4.MM
2MB
N
(II
MPFT (Cluster, mite-D.)
44.7%
302.44
208.0s
0.6s
12.5MR
20.9MB
30.7B
3.1MB
N
MPFT
(Cluster, ratc=0.3)
525.00 1%
624.18
1344.46
2.25
30.9MB
5588BB
84,8MB
5.3MB
MPFT (Random, rate=0.1)
33.15
99.9s
451.9s
0.45
12.5MBB
20,9MB
30.7MB
3.1MB
N
MPFT (Random, rate=0.3)
454.5s
478.9s
30.9MB
84.8BB
5.6 PRIVACY PRESERVATION ANALYSIS
Following DBE (Zhang et al. 2024). add Gaussian noise /V to client prototypes Pi:.... PN with perturbation coefficient for the noise and scale parameter for the noise distribution, the perturbed prototype P. of client can be defined as P. Pt + q (0, s²), where p; is the original prototype of client i. The relationship between noise and privacy budget can be found in Appendix H
Table 4 shows the results of applying this differential privacy method on the DomainNet subset-50, with the sampling method set to average under various noise parameter combinations. This approach effectively mitigates attackers from inferring individual data points even when they possess the pretrained model and most of prototypes. Furthermore, we observe that specific noise configurations can reduce bias across heterogeneous datasets, enhancing the robustness of prototype data. some cases, this even leads to improved performance compared to models without noise. For instance, the combinations of (q = 0.5, s = 0.1), (g = 0.1, 0.05), and (q = 0.5, = 0.05) exhibit such effects. According to DBE, setting q 0.2 and s 0.05 is sufficient to ensure privacy protection. However, excessively large noise can degrade model performance.
Table 4: Performance of differential privacy with varying noise parameters configuration
q 0.1
0.05
original
0.1
S 0.5
==1
5
q= 0.1
q 0.2
9=5.5
4 0.8
ood acc
0.8077
0.8064
0.8083
0.8065
0.7898
0.8078
0.8064
0.8083
0.8055
ind acc
0.7813
0.7806
0.7806
0.7747
0.7437
0.7824
0.7806
0.7820
0.7782
To further evaluate the robustness of MPFT against adversarial attacks, we simulate a feature space hijacking attack (Vepakomma et al. 2021 on MPFT, please refer Appendix G
CONCLUSION AND FUTURE WORK
We propose an adaptive and lightweight FDA framework, MPFT, designed align global model with heterogeneous domains by fitting prototypes from different domains. Extensive experiments demonstrate the effectiveness, low cost, and robustness of MPFT. This study may inspire further research in FDA that focuses generalizing across heterogeneous domain prototypes, rather than relying on model parameter averaging for aggregation.
While MPFT achieves strong performance, it has some limitations. First, the quality of the prototypes highly dependent on the pretrained model's 's ability to extract meaningful features. Second, although attackers cannot reconstruct specific raw data from the prototypes, they may still be able to perform membership inference attacks (Shokri et al.) 2017) or attribution inference attacks (Fredrikson et al. 2015) by exploiting statistical information contained within the prototypes. Addressing the aforementioned limitations could further enhance the viability and effectiveness of this approach in practical FI. applications, making this method viable alternative of averaging- based FL. methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported by the NTU startup grant and the RIE2025 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) (Award E0026), administered by STAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL). This research project is supported by A*STAR under its Japan-Singapore Joint Call: JST-A* 2024 (Project ID: R2416TR 139). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the A*R. The authors would also like to thank our lab members Trantong Wang, Tiantong Wu, Yurong Hao, Mengpu Liu and Wenjie Li for their valuable feedback.
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint arXiv: 2019.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. arXiv preprint arXiv:2107.00778 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Yutong Dai, Zeyuan Chen, Jumnan Li, Shelby Heinecke, Lichao Sun, and Ran Xu. Tackling data heterogeneity in federated learning with class prototypes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7314-7322, 2023.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. /3461, 2020.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9:211 2014. httpps//s///pp semant casholaa ar
Dxg/CorpussD:220177822
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1322 1333, 2015.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-133) 1999.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. URL http:: www.aeseccnansccaaaoooo...... ott.aattiiccoool....oooooooooopp 7200347
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-336)) 366, 1989.
Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16312-16322. IEEE, 2023.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangehuan Liu, Jian Pei, and Yong Zhang Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 7865-7873, 2021.
Muhammad Akbar Husnoo, Adnan Anwar, Nasser Hosseinzadeh, Shama Naz Islam, Abdun Naser Mahmood, and Robin Doss. Fedrep: Towards horizontal federated load forecasting for retail energy providers. In 2022 IEEE PES 14th Asia-Pacifi Power and Energy Engineering Conference (APPEEC), pp. 1-6. IEFE, 2022.
11

---

Published as a conference paper at ICLR 2025
Cheng Jin, Xuandong Chen, Yi Gu, and Qun Li. Feddyn: A dynamic and efficient federated distillation approach on recommender system. In 2022 IEEE 28th International Conference on Parallel and Distributed Systems (ICPADS), pp. 783-793. IEEE, 2023.
Gihun Lee, Minchan Jeong, Yongjin Shin. Sangmin Bac, and Se- Se- Young Yun. Preservation of the global knowledge by not-true distillation in federated learning. Advances in Neural Information Processing Systems, 35:38461-38477 2022.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pp. 5542 5542-5550, 2017.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 10713-10722, 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pp. 6357-6368. PMLR, 2021b.
Xin-Chun Li, De-Chuan Zhan, Yunfeng Shao, Bingshuai Li, and Shaoming Song. Fedphp: Federated personalization with inherited private models. In Joint European Conference Machine Learning and Knowledge Discovery Databases, pp. 587-602. Springer, 2021c
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis- Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.00 20.100000,.,,,,,. 2020.
Bingyan Liu, Nuoyan Lv, Yuanchun Guo, and Yawen Li. Recent advances on federated learning: systematic survey. arXiv preprint arXiv:2301.01299 2023.
Jun Luo and Shandong Wu. Adapt to adaptation: Learning personalization for cross-silo federated learning. In IJCAI: proceedings of the conference, volume 2022, pp. 2166. NIH Public Access, 2022.
Yun Luo. Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effiieee learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp.
PMLR, 2017a.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effccieett learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017b.
Jaehoon Oh, Sangmook Kim, and Se- Young Yun. Fedbabu: Towards enhanced representation for federated image classification. arXiv preprint 1XXi::100600644 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICMI, 2021.
12

---

Published as a conference paper at ICLR 2025
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk. and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks http:///ppnneeeee..eeeee
Track, 2022. URL
M377 4 4
Mihye Seol and Taejoon Kim. Performance enhancement in federated learning by reducing class imbalance of non-iid data. Sensors, 23(3):1152, 2023.
Momina Shaheen, Muhammad Shoaib Farooq, Tariq Umer, and Byung-Seo Kim. Applications of federated learning; taxonomy, challenges, and research trends. Electronics, 11(4):670, 2022.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18.
IEEE, 2017.
Shangchao Su, Mingzhao Yang, Bin Li, and Xiangyang Xue. Federated adaptive prompt tuning for multi-domain collaborative learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15117-15125, 2024.
Benyuan Sun, Hongxing Huo, Yi Yang, and Bo Bai. Partialfed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems, 34:23309- 23320, 2021.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022a.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogencous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8432-8440. 2022b.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
Praveen Venkateswaran, Vatche Isahagian, Vinod Muthusamy, and Nalini Venkatasuubramanian. Fedgen: Generalizable federated learning for sequential data. In 2023 IEEE 16th International Conference on Cloud Computing (CLOUD), pp. 308-318. IEFF, 2023.
Praneeth Vepakomma, Abhishek Singh, Emily Zhang, Otkrist Gupta, and Ramesh Raskar. Nopeek- infer: Preventing face reconstruction attacks distributed inference after on-premise training. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 1-8. IEEE, 2021.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q S. Quek, and H. Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15:3454-3469, 2020. doi: 10.1 10../TIFS.0000.298885.
Liping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi, and Han Yu. Fedgh: Heterogeneous federated learning with generalized global header. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 8686-866 8696,6666200 2023.
Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 1 1127--1124, 2023a.
Jangqng Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedep: Separating feature information for personalized federated learning via conditional policy. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3249-3261, 2023b.
Jinnqing Zhang, Yang Liu, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Jian Cao. Pfllib: Personalized federated learning algorithm library. arXiv preprint arXiv:23/2.0499 2023c.
13

---

Published as a conference paper at ICLR 2025
Jianqing Zhang, Yang Hua, Jian Cao, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Eliminating domain bias for federated learning in representation space. Advances in Neural Information Processing Systems, 36, 2024.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. arXiv preprint urXiv:2012.08566, 2020.
A VISUALIZATION OF PROTOTYPE
In order to allow readers to intuitively understand the relationship between prototypes and original data, we visualize the data embeddings (all prototypes) and averaged prototypes in two-dimensional coordinates using the t-SNE (Van der Maaten & Hinton 2008) algorithm, as shown in Figure S Different colors represent different domains: for example, blue indicates the "Painting" domain, while orange signifies the "Real" domain. Different markers represent various categories of data samples. The darker markers located within each sample cluster represent the prototypes of "domain-class". It is evident that each prototype effectively reflects the distribution of its specific domain-class information. However, the mean prototype, represented by the large red star, is distant from each individual prototype. This observation underscores why do not average the prototypes, as it fails to accurately reflect the overall data distribution, particularly in FDA scenarios.
t-SNE Visualization of Multiple Datasets with Prototypes
Legend
Real
Quickdraw
Datasel-Clas
Painting aircraft carrier
Painting airplane
Painting alarm clock
Dimension
alamm
Real aarplane
Real aircrafl carrier
Quickdraw airplane
-20
Quickdraw aircraft carrier
Quickdraw - alarm clock
Average Prototype
Dimension 1
Figure 5: t-SNE visualization of multiple datasets with their corresponding prototypes.
The Universal Approximation Theorem Horrikk et al. 1989) suggests that neural networks act as "universal" data distribution fitters, effectively fitting the distribution of given data samples. However, this also leads to parameter space deviations between different client models in heterogeneous FL. To address these issues, we introduce more reasonable aggregation method rather than overaging-basee aggregation. We treat the same category data in different domains with significant differences as independent data prototypes. We then the collection of these data prototypes, which form more general data distribution that covers the entire FL system, to train global model that better fits the global distribution.
14

---

Published as a conference paper at ICLR 2025
B CONVERGENCE ANALYSIS
Setup:
The global loss function for fine-tuning using prototypes is:
COD":0 N ==1
Σ
= ,44),
where IPP is the set of prototypes aggregated from all clients.
The standard assumptions follow those of (Li et al. 2020: Tan al. 2022b : The loss function L is non-convex but smooth, 2) The gradient of the loss function is Lipschitz continuous with constant L. 3) The divergence between the prototypes of different clients for a given class is bounded. Let A be the divergence between client i's prototype for class & and the average prototype across all clients:
Assumptions:
A (k)
lp: (K)
((K)
(K) P
and assume that A A for all and k. Since the prototypes are derived from pretrained image embeddings, this assumption likely holds due to the consistency provided by the pretrained encoder in synchronizing the embedding space across domains.
Proof: Given the update rule A/41 A - No C(DP; A for smooth, non-convex loss
function L, the Lipschitz continuity implies:
Ln
A
C(D": & C(D"; (D";:0, (49,49)-00 - vo 000 (.0,,4911 1²+
I
2
N
Rearranging the terms, we obtain:
L(D?:09 14_1) - L(D?: A9) - (1- 7 2
La7²
A
| no C 00 ,/
Choosing such that 0 < 1 < E. we have:
L(D": 00 ( < 09, - cll V A L(DD;
: :09 11
N
Lad is a positive constant. This ensures decrease in the loss function at each step, where с 1] 2
leading to convergence, with an additional term accounting for prototype divergence.
B.1 COROLLARY 1: ON CONVERGENCE TO STATIONARY POINT AND CONVERGENCE RATE
To analyze the convergence rate, we sum both sides of the inequality from t = 1 to T:
T
T
TA
E 14 <
L(D:: 09 ,4)
V No C(DD :09 o +
N
1=1 This simplifies to:
1=1
T
IV Aoo 00 19)1 TA
L(DD: 09, A9) L(D:; 09 1 2
N
Rearranging the terms and dividing by cT' we obtain the convergence rate:
A 08, = L(DP;08 A - L(DP; 00 A
TA N
cT
As T' increases, the average gradient norm decreases, indicating convergence stationary point, ad- justed for prototype divergence. Note that the size the divergence A affects the rate convergence. A smaller A implies that the prototypes across clients are more similar, leading to faster convergence. Conversely, a larger suggests greater variability between client data, which may slow down the convergence rate.
15

---

Published as a conference paper at ICLR 2025
B.2 REMARK: ON ERROR BOUNDS
By incorporating the average prototype divergence, the error bound for client i is:
& clcaal + aA," mM
where eqqual is the local model error without federation, a is constant that measures the sensitivity of the error the prototype divergence, and A. t Σx- (1) the average prototype divergence for client i. Thus, the in-domain enror bound can be written as:
ind EN (droal + az, ( n;
Ei n4
For out-of-domain accuracy (ood acc), we are interested in how well the global model generalizes across different client domains. The maximum divergence of prototypes captures the worst-case divergence between any two domains.
Given the metric:
ΣN Ejjs ACC( Σ Di
ood acc
By incorporating the maximum prototype divergence, the error bound for client on domain j is:
and - local + AA max
where docal the local model error without federation, 33 is a constant that measures the sensitivity of the error the prototype divergence, and A max; l|p -p Thus, the overall out-of-domail error bound is:
(k)
N Ei=l
1z4 (clocll
31 max
n2
c .ood
E
We note that the in- and out-of-domain error bounds are directly influenced by average prototype divergence and maximum prototype divergence (worst-case scenario), respectively.
Impact of higher sampling rate: A higher sampling rate generally leads to better accuracy because more prototypes are generated per class, providing richer representation of the feature distribution across domains. This helps the global model better generalize to out-of- domain data by capturing diverse set of variations.
Cluster sampling is particularly effective for OOD accuracy because it captures the underlying structure of the data distribution within each class by selecting multiple representative prototypes (e.g., cluster centers). Mean sampling, while computa- tionally efficient, oversimplifies the data distribution by averaging all data points, leading to a loss of critical information needed for robust adaptation. Random sampling performs almost as well cluster sampling in some scenarios. This may be due to the fact that random sampling, by chance, captures sufficient variations within each class, providing a diverse enough representation improve OOD accuracy. However, may not be as reliable as cluster sampling because it lacks systematic selection of prototypes and could miss important subgroups within the class distribution.
Sampling methods:
B.3 CONVERGENCE ANALYSIS WITH DIFFERENTIAL PRIVACY NOISE
The convergence analysis changes slightly when differential privacy (DP) noise is added. The key difference is the adjustment in the divergence term:
16

---

Published as a conference paper at ICLR 2025
The divergence between client prototypes for class lt is given by:
A A lpp; (1)
p
p
Suppose Gaussian noise B(p) (k)), with an upper bound of 3, is added to each prototype p² Since the Gaussian noise has mean of zero, by the law of large numbers, we can assume that 5l remains
approximately unchanged when N is large. The new divergence term becomes:
A (k)
A (k) +
S
leading to an updated upper bound:
A < A 8.
The new convergence inequality, adapted from Theorem is:
clV Ao (9) A + 8
C(D":09 . s
This indicates that the constant term of convergence increases linearly with the noise level. However, since the coefficient of / is N. the impact of DP noise becomes negligible as grows larger, meaning that the addition of DP noise does not significantly hinder convergence in theory.
C MORE DETAILS ABOUT EXPERIMENT IMPLEMENTATION
C.1 BASIC SETUP
Global classification head generation. We generate the global classification head by utilizing the manually designed prompts and class names of the dataset, calculated by the pretrained text encoder in the CLIP model used in our experiments. The global classification head is obtained by averaging the ppropptlabell' embeddings from all different domains. Following common fine-tune settings, we only train the adapter, while freezing the entire image encoder and global classification head.
Training. Our simulations are conducted on Google Compute Platform (GCP) equipped with 47 Intel @ CPUs and 4 NVIDIA @ L4 GPUs. For global adapter training. we employ cross- entropy loss with an Adam W optimizer, setting the learning rate to 0.001. We set the maximum global rounds to 200 and implement early stopping strategy to evaluate the convergence rounds. It important to note that our method achieves convergence in just one global round, rendering the early stopping strategy primarily applicable other FL methods. For simplicity, assume that all clients can participate every communication round in all experiments.
C.2 DETAILS ABOUT IMPLEMENTATION OF BASELINES
For the baseline models used in the experiments, we identified the best parameters for our dataset within the recommended parameter ranges provided in their original texts. For FedAvg, we followed the settings in the original article and used the dataset sizes of different clients as the basis for the weighted average. In FedProx, we set the regularization coefficient to 5, which is lower than the usual settings of 10. 100, or 1000. This adjustment was made because higher regularization coefficient made it difficult for the model to converge to the global equilibrium point due to data heterogeneity. In Ditto, we used a local round number of and set the regularization term to 2. For MOON, we set the coefficient μ 0.001 and the temperature coefficient to 1, both within the recommended ranges of the original text. For FedProto, we used 50 as the regularization coefficient. In DBE, we adopted 0.01 the momentum coefficient and as the regular Xiang coefficient, both within the recommended ranges of the paper. For all baselines, we use the same set of hyperparameters, as shown in Table
To ensure fairness in the comparison, all baseline methods, including FedAvg and FedProx, were trained using the same pre-trained feature extractor as MPFT. Furthermore, the training setup for all
17

---

Published as a conference paper at ICLR 2025
Table The hyperparameters used in all baselines.
Optimizer
AdamW
Learning rate
0,001
Batch size
32
Gradient clip
1
Local epoch
Maximum global rounds
200
Warm-up global rounds
Patience global rounds
10
baselines was aligned with MPFT's training paradigm: the pre-trained feature extractor and the global head were frozen, and only the adapter was trained. For consistency, operations other baselines (e.g., average aggregation in FedAvg) were applied specifically to the adapter parameters instead of the entire model parameters. This adaptation ensures a fair comparison of the performance of MPFT and the baselines under the same conditions.
C.3 DETAILS ABOUT EARLY STOPPING STRATEGY
Each client completes one local epoch per global round. We set the total number global rounds to 200 and implement early stopping strategy to evaluate the convergence rounds of each algorithm. The criterion for early stopping is based on validation loss; specifically, we select the results from the round that achieves the best validation loss the final outcome. The patience parameter is set 10 rounds, meaning that if the validation loss does not decrease below the best recorded loss within span of 10 consecutive rounds, the training process is terminated. By implementing the early stopping strategy, we can more easily test the convergence round of each method and use this strategy to find the round with the best result.
C.4 DETAILS ABOUT KNOWLEDGE DISTILLATION LOCAL ADAPTATION
We use the most basic form of knowledge distillation strategy in our framework Hinton et al. 2015).
which is response-based knowledge distillation:
A = arg min LKD (D/: AF;AF;A))
(6)
Here, Af represents the local adapter for client i, Dr is the local dataset, 0 are the global model parameters, and A represents global adapter. Then, we have:
1 (A9 (f())()())) (f
{(11, 81)...., (En, yn)) E DFF.
(7)
Here, 1 denotes the set of outputs from the global adapter for the local dataset D.".
(Af((( (f [(21, ..o))....,,,,,,...... (Ins vnn) E D...
(8)
Similarly, If denotes the set of outputs from the local adapter Af for the local dataset D...
LKD KL(19 1.9)
(9)
The knowledge distillation loss LKD is computed as the Kullback-Leible (KL) divergence between the outputs of the global adapter and the local adapter.
kL 11 q)
Σ pt log
Pi
(10)
qi
Here, p and q represent the probability distributions output by the global and local adapters, respec- tively, and KL q) denotes the KI. divergence.
18

---

Published as a conference paper at ICLR 2025
D SENSITIVITY ANALYSIS OF GLOBAL CONVERGENCE THRESHOLD
During the global adapter initialization phase, we set a threshold to ensure the model stops training the prototypes when the variance of loss over multiple rounds decreases low value, indicating that the global adapter has converged. To test the sensitivity of the experimental results to the threshold, we test different thresholds for the global adapter initialization process, specifically 0.1, 0.01, 0.001, and 0.0001. Figure 6 shows the impact of these threshold As the threshold decreases, the out-of-distribution (ood) and in in-distribution (ind) performance initially increase and then decrease. In contrast, the convergence epochs and training time consistently increase. This trend is intuitive because a lower threshold requires more rounds for the model to converge. Overall, we recommend using a threshold of 0.01 0.001 to minimize training time and reduce the risk of overfitting.
cuteen 53, Dvdd
<Livse 52, IVd brt
01eem 50, Aound
anae 50. Thin
3.000
1480
0,30
1.99
1204
0C4
C39
8.00
200
Thrmshale
Threshldd
Trrashedd
Threshal
<:beee-100.
absn 130,
sibeeb-100
Suzeet 00, Thi
0.194
7528
0.72
0.344
3.20
Tanimm
3750
E.08
560
Theerhal
Bheehhid
Theehels
satset.550
200sct-50 Taa
3.390
$390
6.12
C.50
Hound
1225
6/96
1008
75
Theetals
Thredril
Thretral
Figure 6: Impact of different convergence thresholds on global adapter initialization.
E MORE DETAILS ABOUT RESULTS ON EACH DOMAIN
Figure and Figure & further illustrate the effects random sampling and cluster sampling compared to average sampling within the MPFT framework across different sizes of the DomainNet subset. They reveal similar trends: random sampling and cluster sampling achieve more balanced performance in the Quickdraw domain, with curves approaching circular shapes and covering larger areas. This suggests that these improved sampling methods enhance the model's 's ability to handle balance and diversity across various data domains.
We also observe that as the sampling rate increases the random or cluster sampling methods, the model's performance in the Quickdraw domain improves, leading more globally optimized results. However, the increase in sampling results in more training convergence time consumption and higher data transmission between the server and clients, which raises both computation and communication costs, as shown in Table 3 This trade-off needs to be considered in practical applications.
Note that the rounds in the figure represent the convergence epochs for the global prototype training in the server, the communication rounds (glohal rounds).
19

---

Published as a conference paper at ICLR 2025
MPFT 0.1)
MPFT MPFFT (awamga)
Clipart
Cilpart
Cligart
Infograph
Paimtig
Infograph
Painting
Infograph
Painting
Quickdraw
Real
Quickdraw
Real
Quickdraw
Roal
Skatch
Sketch
Sketch
DomainNet, Subset-50
DomainNet, Subset-100
DomainNet,
Figure Comparison of cluster and average sampling in MPFT framework.
MPFT (random, rate 0.1)
MPFT (jandom, rato 0.3)
MPFTT (randiom, rato 0.5)
MPFT (random, rate
71
MPFT (everage)
Clipart
Clipart
Clipart
Infograph
Painting
Infograph
Painting
Infograph
Painting
Quickdraw
Real
Quickdraw
Real
Quickdraw
Real
Sketth
Skitch
Sketchh
DomainNet, Subset-50
DomainNet, Subset-100
NomainNet, Subset-150
Figure 8: Comparison of random and average sampling in MPFT framework.
F RESULTS OF MULTIPLE CLIENTS WITH SAME DOMAIN
In this section, we present the results of experiments where multiple clients belong to the same data domain, scenario commonly encountered in FL. Specifically, we partition the data from the same domain into multiple subsets, each maintaining the same class labels, and distribute these subsets across multiple clients. This setup results in a far greater number of clients than in the experiments discussed in Section 5.1 As the number of clients increases, the data distribution among clients becomes more similar, potentially reducing domain heterogeneity.
In particular, we compare the results for scenarios with 6, 18, 24, and 30 clients on the DomainNet Subset-50 dataset. Table shows the ind acc, ood acc and the number of communication rounds required for different FL methods.
As the number of clients increases, we observe that methods such as FedAvg, FedProx, and MOON experience decline ood accuracy. This could be due to the reduced heterogeneity among clients, leading to less domain-specifir knowledge being aggregated into the global model. Conversely, Ditto, FedProto and DBE show a reduction in ind accuracy, because these more personalized methods are less effective at preserving in-domain knowledge when client heterogeneity decreases.
In contrast, MPFT manages to maintain both ind acc and ood acc, similar to the original experimental setup, even as the number of clients increases. Notably, in certain cases, MPFT even improves accuracy. For example, with cluster sampling at a rate of 0.1, MPFT achieves a significant performance boost with 18 clients compared to the original experiment. This suggests that MPFT's adaptive aggregation mechanism is robust to changes in client numbers and data distribution, making it more scalable and effective in scenarios with multiple clients from the same domain.
20

---

Published as a conference paper at ICLR 2025
Table 6: Results of multi clients with same domain on DomainNet Subset-50 dataset.
clients (original)
18 clicuts
30 clients
ood acc
ind acc
rounds
ond acc
ind acc
rounds
ood acc
ind noc
rounds
ood acc
ind acc
rounds
loca
0.7361
0,8609
0.7330
0.8557
0,7437
0,8479
0,7434
0.8451
FodAvg
0,7902
0,7345
24
0.7708
0,7197
25
0.7693
0.7186
24
0.7685
0.2192
FedProx
0,7752
0.7178
10
0.7685
0.7165
0
0.7675
0.7163
0
0.7667
0.7172
Dito
0,7811
0.7624
20
0.7643
0.7386
15
0.7639
0.7397
37
0.7619
0.7357
MOON
0.7902
0.7344
28
0.7709
0.7197
27
0.7691
0.7186
20
0.7682
0.7188
FedProto
0.7296
0.7696
0.7299
0,7530
9
0.7305
0.7476
0,7308
0.7562
11
0.7421
0,7622
MPFT (Average)
0,8077
0.7813
0.8062
0.7853
0.8032
0,7820
0,8052
0.7839
MPFT (Cluster, rate=0.)
0.7951
0,7957
0.8091
0.8128
0.8002
0.8011
0.8217
0.8232
(Cluster, rate-D.3)
0.8126
MPFT (Randmm, rate=0.1)
0.7953
0.7899
0.7911
0.7880
0.7909
0.7838
0.7975
0.7971
MPFT (Random, r.te=0.3)
0,8236
0.8294
0.8218
0.8267
0.8119
0,8138
0.8219
0.8257
G FEATURE SPACE HIJACKING ATTACK
Oiginal
IsrwO
1500220000
Nur-000303
1a-0755000
100000
Orgginal
Mrr=
1ar0026000
2cp=500002
fp-0750000
120000
origial
1er. 75ten
origina
Br--0
ter- 251cn0
Bnc- 500335
Eer. 752000
Ter. 1000003
Original
Mar=0
1ar-036000
2ur-500003
ts-0050000
Iar- 150000
0
Figure 9: Results the feature space hijacking attack on our model. Each prototype was constructed from single image. Even after one million iterations of training, the original image could not be recovered, demonstrating the security of using prototypes.
In the MPFT framework, the communication of local clients' prototypes and trained global adapter may present potential security vulnerabilitics. We hypothesize an attack vector utilizing the archi- tecture of our approach, to design feature space hijacking attack. The attacker could leverage the pretrained model's image encoder f() and the uploaded prototype (kk) to attempt restoration of the original training data r::
1.
An estimated input x² is constructed and processed through the pretrained image encoder
f (Φ) obtain an estimated prototype p".
21

---

Published as a conference paper at ICLR 2025
2. The mean squared error (MSE) loss is used to iteratively refine x², aiming minimize the discrepancy between p" and the actual prototype p. thereby approximating P3
To demonstrate the resistance of our method feature space hijacking attacks, we randomly selected picture from each client to form a prototype and attempted to simulate attacker trying to restore the picture. One picture is chosen as the prototype because a prototype attack composed single picture cannot be restored, a prototype composed of multiple average representations of the same category will be even more challenging for an attacker to restore and exploit.
Figure 9 illustrates the process of restoring single image by multiple clients. Intuitively, we observe that even after one million gradient descent iterations, the attacker still cannot restore the salient features of the original image. important note that on our device, takes nearly 8 hours to complete such iterative training for each image, imposing a significant time cost on attackers attempting large-scale attacks.
H RELATIONSHIP BETWEEN NOISE AND PRIVACY BUDGET
We perform differential privacy (DP) analysis in the average prototype sampling stage:
Proposition 1 (Post-Processing) Let f N + R be a randomized algorithm that is (c, 6)- differentially private. Let R + be an arbitrary randomized mapping. Then g o f N - R (e, diddifferryttlllly private.
In MPFT, can regard sampling as f and learning from the prototype g, which output is the learned model. Then the whole learning process is (s, d)-differennially private provided that the sampling function (e, diffeerrettll private.
Definition (Gaussian Mechanism) Let f NK + Rd be an arbitrary d-dimensional function, and define its &g sensitivity to be:
A2f
max 11f(x) f(v)l2-
adjacent 2.31
The Gaussian Mechanism with parameter adds noise scaled (O, o²) to each of the d components of the output.
Theorem (Relationship between Gaussian Mechanism and privacy budget) Let e E (0, 1) be arbitrary. For e² > 221(11.5//)), the Gaussian Mechanism with parameter
A2f
(e, 8) diifferrentll private.
For average sampling, the sampling procedure for class involves the data embeddings D; (dl (a(0) mn where n; is the number of data points in class i. For cluster sampling, simply replace D; by data embeddings inside the cluster, with other analysis unchanged.
The average sampling function f is
J(D.)
We define f: as the restriction of f taking only data points in class i as input. Two adjacent datasets differ by exactly one data point. For two adjacent datasets Dp. Dy. suppose they differ at datapoints
de E Dp, de) Dg. then:
Eg EDDED
d0
EDDED,
d
(6)
3)
ED,
- - f(D,)l2
n2
n2;
22

---

Published as a conference paper at ICLR 2025
(E)
(8)
dy
22fi
max
74
For (6,, (fi.e))-difeerentialll private guarantee:
2ln(1.25/6.) A2f
2
strong differentially private guarantee achieved when E; < 1, 6, < according to the theorem
in (Dwork & Roth 2014). In federated learning settings such as (Wei et al., 2020), a medium differentially private guarantee is achieved when & is around 10.
We calculate our differentially private guarantee as
& 22 (21.(1))5m.) A2fi
For our differential privacy experiment in Section 5.6 the a value used is the multiplication gs. Table 7 shows empirical results of average privacy budget E with different values, where take 5,
Table 7: Average privacy budget E for different a values.
0.01
0.05
0.1
a 0.5
DomainNet, Subset=50
54.76
10.95
5.48
1.10
DomainNet, Subset=100
65,48
13.10
6.55
1.31
NomainNet, Subset=150
60.73
12.15
6.07
1.21
The table demonstrates that, as the noise scale a increases, the average privacy budget È decreases, indicating stronger privacy protection. When above 0.05, the privacy budget sufficient to provide robust privacy guarantees. Furthermore, as shown in Table 4 the performance of MPFT experiences minimal decline when a in the range of 0.05-0.1. These results, both theoretical and empirical, demonstrate that MPFT can effectively utilize differential privacy mechanisms to protect prototypes without significant performance loss.
23


Reviews:
[{"review_id": "OS7EtXKTt8", "reviewer": "Reviewer_ZWk6", "summary": "This work studies domain adaptation in federated learning scenarios, employing prototype-based fine-tuning to leverage knowledge from other clients. The fine-tuning procedure requires only one round of communication between clients and the server, making it communication-computationally efficient.", "questions": "1. The experimental settings are somewhat vague, particularly regarding whether competitive approaches, such as FedAvg and FedProx, are trained from scratch or from the same pretrained backbone model as the MPFT. Could the authors specify this information in the paper, ideally in the experimental setup or implementation details section?\n\n2. The privacy protection measures appear informal. While Gaussian noise is added to prototypes (i.e., embeddings) as noted in Section 5.6, it is important to assess whether the noise level provides adequate privacy protection for sensitive attributes or labels. To strengthen this section, could the authors provide a more rigorous analysis of the privacy guarantees associated with their approach? For example, quantifying the level of privacy protection using established metrics or frameworks would be beneficial.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, Experiments on two datasets indicate a significant performance boost from the proposal, increasing accuracy by 2%.\n\n2. The approach is intuitive, sharing prototypes using three sampling strategies.\n\n3. The proposal is communication-computationally efficient, as it involves a low number of interaction rounds and requires only prototypes for communication.", "weaknesses": "The motivating scenario may be impractical, as the proposal assumes that clients already possess a well-pretrained model. This assumption can be particularly challenging in sensitive areas where federated learning is essential, such as healthcare or finance, where data privacy concerns limit access to robust pre-trained models. \n\nCan the authors discuss specific applications or scenarios within these sensitive domains where obtaining a pre-trained model might be more feasible? Additionally, it would be valuable to explore potential solutions or adaptations of their method for situations where a pre-trained model is not available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZjD5qOBYJF", "reviewer": "Reviewer_xVyS", "summary": "The paper proposes Multi-domain Prototype-based Federated Fine-Tuning (MPFT), a framework designed for Federated Domain Adaptation (FDA) by utilizing domain-specific prototypes. Instead of relying on traditional model aggregation techniques, which often falter due to data heterogeneity, MPFT transfers prototypes (compressed domain representations) to the server. This approach allows the server to learn a global adapter that improves both in-domain and out-of-domain performance. MPFT also incorporates differential privacy to protect prototypes from potential data leakage.", "questions": "I suggest that the authors remove the overly powerful pre-trained feature extractor and instead use a simpler, more foundational model for feature extraction, such as ResNet or ConvNet. This would more accurately showcase the contribution of the prototype and methodology itself.\n\nThe authors should provide a validity proof to demonstrate that training the adapter on aggregated prototypes achieves comparable performance to training on aggregated raw client data or features—or, alternatively, establish an upper bound on the performance gap between these approaches.\n\nAlthough differential privacy (DP) experiments were conducted, a theoretical analysis is also necessary. Specifically, the authors should clarify the noise variance conditions required for their method to satisfy DP. In particular, the statement “Furthermore, we observe that specific noise configurations can reduce bias across heterogeneous datasets, enhancing the robustness of prototype data” requires a more robust theoretical explanation.\n\nI would encourage the authors to further explain, both theoretically and experimentally, the differences, advantages, and potential complementarities among various prototype selection mechanisms.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The main strength of the paper lies in its comprehensive experimentation, which provides a detailed view of MPFT’s performance across various scenarios, showing superior computational and communication efficiency.", "weaknesses": "The Multi-domain Prototype-based Federated Fine-Tuning (MPFT) method proposed in this paper is overly simplistic, primarily relying on basic prototype sampling strategies—such as mean, cluster, and random sampling—to generate client prototypes. Additionally, because the model does not undergo training on each client’s data, it fundamentally contradicts the original design principles of federated learning (FL). The performance gains observed in the experiments likely stem from the strong pre-trained feature extractor, which does not require training on all client data but only a few representative prototypes to achieve reasonable performance. If the pre-trained feature extractor were removed and the model was trained solely on prototypes, the results would likely be poor. If only prototype fine-tuning is needed, then why even employ a federated learning framework? Thus, despite achieving certain results, this approach introduces no complex mechanisms or innovative architectures to enhance the FL system, lacking fresh or original design. It merely follows a standard process: feature extraction with a general pre-trained model, prototype selection, server training, and client fine-tuning, without presenting any real novelty.\n\nThe paper does not provide a thorough theoretical analysis of the different prototype sampling methods, nor does it uncover the fundamental differences between these sampling strategies in handling heterogeneous data distributions. Although the paper presents experimental results comparing mean, cluster, and random sampling methods, it lacks detailed explanations on the core differences among these strategies in terms of learning mechanisms, communication efficiency, and personalization effects. Consequently, the paper falls short in theoretical depth, failing to provide any deep insights into the impacts of these sampling strategies.\n\nThe theoretical analysis in this paper primarily focuses on convergence. However, training the adapter on aggregated prototype data is no different from centralized data training, so convergence is naturally expected. To provide valuable theoretical insights, the paper would need to show that the adapter’s performance on aggregated prototypes is comparable to that on aggregated raw client data or offer theoretical performance bounds. Furthermore, transmitting prototypes instead of models introduces a higher risk of privacy leakage, so the analysis should prioritize differential privacy (DP) guarantees rather than focusing solely on convergence. The current paper fails to address theoretical guarantees for privacy protection, which is particularly critical in privacy-sensitive federated learning contexts.\n\nAlthough the experimental section is comprehensive, covering multiple datasets, various sampling strategies, and different DP configurations, it does not clearly reveal the functional differences and applicability of each prototype sampling approach across different scenarios. While the experiments test different prototype configurations, they lack an in-depth discussion on how these configurations perform under different data distribution patterns and sample sparsity conditions. This results in a set of experiments that, while extensive, fails to provide a robust summary of the unique characteristics of the prototype methods, thus missing an opportunity to elevate the research value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PZ8YkptHl3", "reviewer": "Reviewer_ckUs", "summary": "This paper introduces Multi-domain Prototype-based Federated Fine-Tuning (MPFT), a framework designed to address data heterogeneity and data privacy in federated learning (FL) by using prototype-based training rather than traditional averaging methods. In MPFT, each client generates a set of representative data embeddings (prototypes) that capture essential domain-specific characteristics without transferring raw data. These prototypes are then aggregated at the server, allowing for a simulated centralized learning approach and enabling fine-tuning of a global adapter. This method aims to achieve performance on par with centralized learning while solving the challenges incurred by aggregating client models.\nThe efficiency of MPFT lies in that it requires only a single round of global communication, significantly reducing computational and communication costs compared to multi-round FL methods. Furthermore, by selectively sampling prototypes, the framework limits data transfer volumes. To ensure privacy, MPFT integrates differential privacy mechanisms, mitigating risks of data exposure and rendering prototype-based data reconstruction ineffective—even when the prototype encoder is known to potential attackers.", "questions": "1. Would the authors explain how you chose the parameter $s$ in Section 5.6? \n\n2. Why is the class $k$ missing in the proof in Appendix B (while formulated before proof)?\n\n3. Can authors make more comparisons with recent (>=2023) works? For example,\nFedcp: Separating feature information for personalized federated learning via conditional policy. In KDD 2023.\nFedFed: Feature Distillation against Data Heterogeneity in Federated Learning. In NeurIPS 2023.\nFedgh: Heterogeneous federated learning with generalized global header. In MM 2023.\n\nIt would be better to compare aggregation-based improvements in recent works and the new mechanism proposed in this paper. Since the authors stressed the drawbacks of these works, it would be better to give detailed support or explanation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ They propose an interesting idea -- MPFT as a one-round federated fine-tuning approach for multi-domain environments, demonstrating its performance improved over previous methods. \n+ A new metric is introduced to evaluate model adaptability, assessing both out-of-domain and in-domain accuracy to balance knowledge retention and domain adaptation.\n+ The writing and logic are clear and easy to follow. The problem formulation is clear and friendly to readers.", "weaknesses": "- The proof of convergence in Section B lacks the differential-privacy-related analysis. A supplemental analysis of \"MPFT with DP\" should be added for theoretical completeness. Intuitively, the addition of differential privacy introduces a bounded randomness during the convergence process.\n\n- The selected experimental counterparts for comparison are mostly in 2017-2022, which are relatively not new. In Section 2, the authors think the \"averaging-based aggregation results in poor out-of-domain adaptation performance.\" It would be better to show the performance of out-of-domain adaptation from the recent advances atop model/parameter aggregation. Some recent works (with or without aggregation) relevant to data heterogeneity in federated learning are expected to compare, such as,\n\nFedcp: Separating feature information for personalized federated learning via conditional policy. In KDD 2023.\nFedFed: Feature Distillation against Data Heterogeneity in Federated Learning. In NeurIPS 2023.\nFedgh: Heterogeneous federated learning with generalized global header. In MM 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)