Enhancing Foundation Models with Federated Domain Knowledge Infusion
Authors: Jiaqi Wang
Jingtao Li
Weiming Zhuang
Chen Chen
Lingjuan Lyu
Fenglong Ma
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Foundation Model
Primary area: applications->computer_vision

Abstract: Vision foundation models (FMs) like CLIP have exhibited exceptional capabilities in visual and linguistic understanding, particularly in zero-shot inference tasks. However, these models struggle with data that significantly deviates from their training samples, necessitating fine-tuning, which is often infeasible in centralized settings due to data privacy concerns. Federated learning (FL) combined with parameter-efficient fine-tuning (PEFT) offers a potential solution, yet existing methods face issues with domain-specific characteristics and out-of-domain generalization. We propose a cross-silo Federated Adapter Generalization (FedAG), a novel federated fine-tuning approach that leverages multiple fine-grained adapters to capture domain-specific knowledge while enhancing out-of-domain generalization. Our method uses quality-aware in-domain mutual learning and attention-regularized cross-domain learning to integrate domain-specific insights effectively. Experiments of the CLIP model on three domain-shifting datasets, ImageCLEF-DA, Office-Home, and DomainNet, demonstrate the superior performance of FedAG in both in-domain and out-of-domain scenarios. We envision this work as a milestone for generalizing CLIP to handle the challenge of out-of-domain knowledge under federated learning setting.

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Jiaqi Wang' Jingtao Li Weiming Zhuang? 2
Chen Chen
Lingjuan Lyu² Fenglong Ma
Abstract
ployed in wide range of vision tasks, including image classification, especially for zero-shot inference, thanks to their large number of parameters and the extensive train- ing data. However, these models still face challenges when confronted with input data significantly different from their training data. Therefore, fine-tuning becomes essential. Tra- ditional fine-tuning strategies are typically conducted in centralized manner. However, this approach is often imprac- tical, particularly for sensitive data like medical informa- tion, which is often distributed among clients and cannot be shared. This distributed scenario significantly complicates the fine-tuning process for CLIP.
Vision foundation models (FMs) like CLIP have exhibited exceptional capabilities in visual and linguistic understanding, particularly in zero-shot inference tasks. However, these models strug- gle with data that significantly deviates from their training samples, necessitating fine-tuning. which is often infeasible in centralized settings due to data privacy concern.. Federated learn- ing (FL. combined with parameter-efficient fine- tuning (PEFT) offers potential solution, yet ex- isting methods face issues with dopain-specific characteristics and oft-of-domain generalization. We propose a cross-silo Federated Adapter Gener- alization (FedAG), a novel federated fine-tuning approach that leverages multiple fine-gr giee-griieed adapters to capture domain-specific knowledge while enhancing nut-of-domain generalization.
Recent studies addressed this challenge by combining feder- ated learning (FL) with fine-tuning techniques, technique known as federated fine-tuning, Existing approaches (Xiao et al., 2023; Wang et al., 2024b; Marchisio et al., 2023; Chua et al., 2023; Khalid et al., 2023; Wang et al., 2024a) typically fine-tune these models without utilizing the en- tire model. Instead, they often employ layer-drop tech- niques (Sajjad et al., 2023) to compress a full model into a sub-model. The sub-model and an emulator are distributed to clients. Clients then update this compressed sub-model with their private data with the help of the emulator iter- atively. The resulting sub-model is eventually incorpo- rated into the full model to complete fine-tuning. However, these compression techniques fail to maintain alignment between the fine-tuned compressed layers and the remain- ing ones, leading to performance degradation. Federated parameter-efffcient fine-tuning (PEFT) techniques, such as FedCLIP (Lu al., 2023) and FedPETuning (Zhang et al., 2023), have emerged to address the aforementioned prob- lem. These approaches involve deploying the foundation model with an additional adapter on each client, which is then collaboratively trained like FedAvg (McMahan et al., 2017). The aggregated adapter is subsequently integrated into the foundation model to achieve fine-tuning. However, they have several issues:
Our method uses quality-award -aware in-domain mutual learning and dntention-regularizee cross-domain learning to integrate Nomain-speeciff insights effectively. Experiments of the CLIP model three domain-shifting datasets, ImageCI.FF. DA. Office-Home, and DomainNet, demonstrate the superior performance of FedAG in both in- domain and out-of-domain scenarios. We envi- sion this work as a milestone for generalizing CLIP to handle the challenge of out-of-domain knowledge under federated learning setting. The source code can be found at ht https: //t::/giiii git..b. com/ com/JackgoWang/ fedag.
1. Introduction
CLIP (Contrastive Language-Iman Image Pre-training) (Radford al., 2021) and its variants (Li al., 2023), have demon- strated superior capabilities in understanding visual con- cepts and their linguistic descriptions. They have been em-
Indistinguisaable in domain-specifir characteristics. In real-worke applications, the data collected by clients may exhibit different characteristics even for the same task. For instance, the stylistic realism of an image can vary across different forms of visual art, such as painting, photogra- phy, and digital art, leading to unique artistic expressions. However, existing models typically employ single adapter
'The Pennsylvania State University "Sony AL Correspondence to: Lingjuan Lyu <Lingjuan com>, Fenglong Ma <fen- glong @ puu.eduu.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Clipart
Painting
Real
work model domain-specific characteristics. Sub-model fine-tuning approaches encounter difficulties in compressing multiple domain-specific sub-models and aggregating them. Similarly, PEFT approaches face challenges in aggregating adapters with diverse knowledge. On the other hand, equip- ping the capability of out-of-domain generalization with federated fine-tuning is an open challenge in this domain and largely underexplored in existing studies. Thus, it is urgent to develop a new method tackle these challenges simultaneously.
70
67
85
ACC (%)
66
69
83
65
68
CLIP CLIPo
64
CLIPs CLIPM
81
CLIPo
CLIPM
Figure 1. In-domain preliminary results.
Infugraph
Quickdraw
Sketch
68
ACC (%) (%)
42
62
CLIP2 (T.PP CLTP
12
CLIPx PLIP, CLP
880
56
CL,P, 0.P, CLU
In this paper, we propose a novel federated fine-tuning ap- proach named Federated Adapter Generalization (F (FedAG), as shown in Figure 3. Intuitively, the types of domains for a specific task are usually limited, and the data belonging to a domain is usually collected by a specific client. This moti- vates us design new model under the cross-silo federated learning setting and allow all clients to be involved in the learning at each communication round. Besides, this setting also allows us to employ multiple fine-grained adapters to in- ject domain-speciff knowledge into corresponding adapters while enhancing the capability of out-of-domain knowledge generalization by jointly combining these adapters. Unlike existing work, which either compresses a sub-model for each client deploys foundation model, we enable clients to have their domain-specing models representing the char- acteristics of their data. These client models are trained with private data (Figure 3(b)) and uploaded to the server to inject their domain-specific knowledge into CLIP.
Figure 2. Out-of-domain preliminary results. to capture knowledge from mixed domains, resulting in a performance gap compared to domain- -eemain-ppecfficc adapters. We conducted a preliminary experiment on the DomainNet dataset to validate this observation, using three domains: "clipart", "painting", and "real", Following FedCLIP (Lu et al., 2023), we employed an attention-base based adapter to fine- tune CLIP in two ways. The first way is use all the data from three domains to fine-tune one adapter, denoted as CLIPo. The other is to fine-tune three individual adapters only using each domain's 's data, denoted as CLIPM- M-
The results are shown in Figure 1. It can be observed that despite using data from all three domains to fine-tune the adapter, CLIPo still performs worse than CLIP M. which fine-tunes each adapter using only domain-specific data. This issue is expected to exacerbate in the federated fine- tuning setting due to the heterogeneity of clients, leading to an aggregated adapter inferior to centralized fine-tuning. These initial findings motivate us to develop domain-specifti adapters for use in federated PEFT.
2. Related Work
Fine-tuning foundation models is essential for specific downstream tasks, especially parameter-efffiient fine-tuning (PEFT) (Ding et al., 2023; He et al., 2022; Han et al., 2024). However, the data privacy issue limits the fine- tuning foundation models in a centralized way. Offsite- tuning (Xiao al., 2023) incorporating federated learning techniques (MeMahan et al., 2017) has recently been pro- posed to address this problem by incorporating federated learning techniques. Existing works can be roughly divided into three categories. Federated full model tuning (Deng et al., 2023; Fan et al., 2023) uses client outcomes as feed- back to guide the fine-tuning of the foundation model. Fed- erated partial model tuning (Peng et al., 2024; Marchisio et al., 2022; Khalid et al., 2023) compresses a submodel from the foundation model, then sends the compressed one to clients for extracting client knowledge, and finally aggre- gates the parameters learned from clients into the foundation model. Federated PEFT (Lu et al., 2023; Zhang et al., 2023) techniques use an extra adapter to learn and exchange client knowledge.
Incapable to out-of-domain generalization. While exist- ing federated fine-tuning approaches can improve perfor- mance compared to zero-shot inference the original mod- els, they still struggle when faced with new or data. To explore the out-of-domain generalization ability, we use the fine-tuned CLIPo to directly conduct the in- ference on three new domains: "infograph", " "quickdraw", and "sketch", comparing with the zero-shot inference with CLIP, denoted CLIPz,We We fine-tune three new individ- ual adapters using each domain's 's data denoted as CLIP" M which can be treated as the performance upper bound.
Figure 2 shows the preliminary results of the out-of-domain testing. We can observe that fine-tuning with a shared adapter (CLIPo) does improve performance compared to CLIPz, but the degree of improvement is limited, as the results are far from the performance achieved by CLIP Therefore, it is crucial to enhance the adapters' capability for out-of -ott------minnggoo generalization, especially in the federated fine-tuning setting.
Our work falls into the federated PEFT category, and only a few studies have begun exploring this challenging task. In
However, addressing the aforementioned issue is challeng- ing. On the one hand, it is hard to directly extend existing
2

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Client
Stable Diffusion
Server
S
Image Embedding
D1
W
Synthetic Data
A
Image Encoder
Local D2
W
A
Text Embedding
Data
Upload
Text Encoder
D3
W
A
Download
(a) Model Overview (Sec. 3.2) and
Logits
Adapters
Logit Learning (Sec. 3.4.1)
Momentum Update
W
Nynthetic Duta
Suftmax
Φ
Φ
In-domain Prediction
S,, Wt
Mutual Learning
Ot
32
Dn
W!
n
s n
3
Regularizer
a
Cross-domain
Private Data
Client Model
Estimated Data Quality Score
77 77
Prediction
(c) Quality-aware In-domain Mutual Learning (See 3.4.2)
(d) Attention-regularizze
(b) Local Training (Sec 3.3)
Cross-domain learning (Sec 3.4.3)
Figure 3. The overview of proposed FeaAG. We use three domains (N = 3) as an example.
(Lu et al., 2023), each client has foundation model and ex- changes the adapters with the server in each communication round. The server conducts the basic FedAvg on the adapter and returns it to the clients. Similarly, FedPETuning (Zhang et al., 2023) provides a parameter-efficien tuning approach on pre-the trained language models via sharing part of the client models federated learning. The aforementioned studies typically require clients possess foundation mod- els. However, this is impractical since small clients may be unable to fine-tune large foundation model. Our approach places the foundation model on the server side, representing a more practical setting. Moreover, our objective is to enable clients to collaboratively contribute to the foundation model learning using their specific domain knowledge, without accessing local data.
In practice, clients will share the style information (text prompt or the generated textual inversion token) so that domain-specife synthetic data (S), SS) can be gener- ated on the server'. Once synthetic data is generated, they will be transferred to the corresponding clients to perform the quality estimation. The communication of the synthetic data is only a one-time cost and is often negligible, The de- tails of synthetic data generation can be found in $Sec. 4.1.
3.2. Model Overview
As shown in Figure 3, the proposed FedAG model com- prises two main updates: the client update and the server update. The client update module (SSec. 3.3) is designed to train local model fn for each client C, using their respec- tive data D.,, where the parameters of fn (i.e., W at the (-th communication round) encapsulate the domain-specific knowledge. Additionally, it estimates data-quality score a E a for each synthetic instance s E S,, The client model parameters W and the estimated quality scores of, are then uploaded to the central server for further processing.
3. Methodology
3.1. Model Input
The proposed model FedAG aims to iteratively inject domain knowledge into the vision foundation model deployed on the server through collaboration with N mutually exclusive and independent domain-specific clients (C.,... .Cy) } without sharing their private data (Di,... } guarding clients' data privacy, the conventional approach involves data-free knowledge transfer, where often random Gaussian noise is utilized to distill knowledge from one model to another (Chen et al., 2019). Despite recent ad- vancements (Raikwar& & Mishra, 2022), noise-based knowl- edge transfer still encounters performance degradation com- pared to using real data. To conduct effective knowledge transfer, we leverage the open-source text-to-image model, Stable Diffusion 2.0 (Rombach et al., 2022), to generate domain-specific data S, for each client C..
During the server update at the t-th communication round, FedAG first learns the logits of synthetic data using the CLIP framework in $Sec. 3.4.1. It then integrates the do- main knowledge from W. into the corresponding domain- specific attention-bast logits through a quality-aware in-domain mutual learning module that is detailed in fSec. 3.4.2. Furthermore, it ex- tends the model's is capability to out-of-domain knowledge us- ing an attention attention-regulariize regularized cross learning module, as described in Seec. 3.4.3. Afterward, the updated client
}. To facilitate knowledge transfer while safe-
adapter A based on the learned
Althooghh it is possible to generate synthetic data on the client side and upload it to the server, some clients may lack the capacity to deploy the Stable Diffusion model. Furthermore, uploading data is less efficient than downloading it. Therefore, based on these considerations, we enable the server to generate synthetic data.
3

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
models (denoted as (Wi. w are redistributed to
q through W We then calculate the cosine similar- ity of between q: and the corresponding prototype pk represented as a = cos(q p): The vector of these similarity scores, a for all generated data in S, on the n-th client, is then compiled and prepared for uploading to the server along with W
their respective clients for another round of the client update.
3.3. Client Update
3.3.1. CLIENT MODEL TRAINING
At the f-th communication round, client C, will receive an updated model W from the server, which is trained using the synthetic data S, in the server update. Since the generated synthetic data S, is different from the real domain data n.. directly using W the initialized client model at the t- communication round (i.e., W W will be unsuitable To mitigate this issue, we propose the of momentum update initialize the client model as follows:
This methodology offers significant advantages: it ensures that uploading synthetic data quality scores does not com- promise the confidentiality of client data, and it allows each client model to provide specific data-quality scores, thus enhancing the precision of the mutual learning process ($Sec. 3.4.2).
3.4. Server Update
W W + (1-7)W;>
(1)
Upon receiving the uploaded client models (Wi, ,W)) and their corresponding estimated data-quality scores {at, ... ,,y,, the server integrates speciff knowledge into CLIP. This achieved by incorporating domain-specific attention based adapters (Ai, ) each consisting of an identical multi-layer block positioned after the feature extractor of CLIP.
where y is the hyperparameter. We then use the traditional cross-entropy (CE) loss to train the client model's parame- ters W for the n-th client using D, as follows:
min C W
Σ
CE(fn(X)) W.). (2)
D
(x JED,
3.4.1. CLIP-BASED LOGIT LEARNING
where fn is a ViT-Tiny model (Wu et al., 2022), [Dxr is the total number of private training data, x", is the i-th data feature, y E {0, }}10) is the corresponding label, and y is the set of distinct labels, which shared by all domains. The trained model W via Eq. (2) contains the knowledge of the n-th domain.
The goal of FedAG is to inject domain knowledge included in client model parameters into the CLIP model in PEFT way. Let Encing () represent the frozen image encoder and Encrze( be the frozen text encoder of CLIP. Let Ly denote the description of class label y, i.e., "a photo of [class]" To learn the logit for an image si, Sn, we follow the CLIP pre- traininii framework and take the image s and all the label descriptions | as the input. In particular, we first obtain the representations of s and Ly using the corresponding encoders as follows:
3.3.2. QUALITY ESTIMATION FOR DOMAIN-SPECIFIC SYNTHETIC DATA
The synthetic dataset S,,, generated through stable diffusion, is essential for the server update but presents an unknown quality challenge. To address this, we propose estimating data quality using a prototype-based similarity measurement for each domain-specine set of generated data Sn, utilizing the trained local model W
I = =EnCingg (5
) Ty, = EEneant (Ly)-
Following FedCLIP (Lu et al., 2023), the image represen- tation I" RRd will pass an attention- based adapter A,, to obtain a fine-tuned domain- specific representation as fol- lows:
Label-aware Prototype Representation Learning. Let DH C Dn, denote the subset of training data with labels E Vn. For each data instance x within DR, we first derive its feature representation rff using the encoder layers of Wt. prototype representation p for each label category y by averaging the representations of all data in Dy. specifically, p P.NT Extepg
n before the prediction layer. We then compute a
The = AC, Softmax(ML.P/ (Tanh(MLPP2 SSTTaa (T.))))
(4)
where i Rd, d is the dimension size, and O denotes the element-wise dot product. MLP is the multi-layer percep- tion.
Similarity- -based Data Quality Estimation. For the gen- erated data subset Sy C S, labeled y in the 1-th domain, each instance si E Sy also receives a feature representation
Finally, we can obtain the domain-specific logit for the input image as follows:
2 An empirical experiment has been conducted to demonstrate this issue, as shown in Appendix D.
1 T [
(5)

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
3.4.2. QUALITY-AWARE IN-DOMAIN MUTUAL
the aggregated logit for each data instance follows:
LEARNING
To transfer domain-specific knowledge from the client model W to the CLIP model (i.e., the corresponding adapter A,), an intuitive way is to conduct knowledge dis- tillation (Hinton et al., 2015) by treating W n. as the teacher network and the adapter-based CLIP as the student network. However, this simple strategy presents several limitations: it overlooks the quality of domain domain-specifie synthetic data S, involved in the distillation process and only allows uni- directional knowledge transfer, which does not update the local model W thus underutilizing the potential of the federated learning framework.
Ok
is
(9)
18 .... 344 = softmax softmax(MMLP(@ ,
MLP( I)- (10)
The domain index n is known for each training data during the training. Thus, the attention weight 8 should be larger than those obtained from the other adapters. We use this intuition as prior knowledge to guide the model learning via an attention-based regularizer as follows:
R = max(0, 5+ m+aax(( ,... ""." Shc -
To overcome these shortcomings, we introduce quality- aware in-domain mutual learning strategy. This approach not only ensures effective integration of commai--ppecfic knowledge into A but also facilitates dynamic updates of the local model, leveraging the quality assessments of the synthetic data to enhance the overall learning process.
(11)
where is the margin hyperparameter.
3.4.4. SERVER OPTIMIZATION
Based on Eqs. (7), (8), (9), and (11), we obtain the final loss function for the server update as follows:
Note that we use W to distinguish the updates the client model W on the server. The loss function is defined as follows:
min g A A
T
Σ
CE(yp vt)
N
(8) M EES
In-domain Prediction
min Tn
Σ
di of (KL((0001103
(6)
CB( yn)
+XR+"]
(12)
AC, W:
2 2) ar 68,
Cross-dornate Prediction
*KL(P (1²)))
where A {Ai.... .... W (WI, W N Ko softmax and X is the hyperparameter. The updated client models W = (Wi.- w will be re- distributed to the corresponding Domain-specifer clients for update in the next communication round.
0¹2²" = (n(hi wh). = softmax (+))))
(8)
where 02C( is the predicted probabilities by the client model W on each data instance s on the server, and sp is prob- abilities ouputed by the CLIP model using Eq. (5). (L-(-I-) is the Kullback-Leiller divergence.
3.5. Inference
FedAG will be trained iteratively using Eqs. (2) and (12) until convergence. We then conduct the inference on the testing data. For the in-domain scenario, where the do- main index is known, we use the label index with the maximum value in di, as the predicted label, i.e., ih arg (1........I1)) 1213 (Ø)) via Eq. (5). For the out-of-domain scenario where the domain is unknown, we use the label index with the maximum value in v/ as the predicted label, i.e., ÿ'
3.4.3. ATTENTION-RFGULLRRZAN CROSS-DOMAIN LEARNING
Using Eq. (7), we can update the adapters and client models simultaneously. However, such a design may only work for data belonging to existing domains, i.e., there is lack of generalization ability for out-of-domain data. We propose novel attention attention-regullarized cross-domain learning strategy to equip the proposed F eedAG with the capability for dealing with out-of-domain data.
arg max (1,,),)))))))))))) (17()))))) via Eq. (9).
4. Experiments
4.1. Experimental Setups
In particular, for a synthetic data instance s E Sn, we not only generate its logit t:. via Eq. (5) with the domain-specife adaptor A but also from other adaptors {Ai, A Af A A. We calculate the atten- tion score 8 R (k E NJ) for each adaptor using a softmax function on top of an MLP layer and then obtain
Datasets. To fairly validate the proposed model FedAG in our experiments, we focus on the image classification task on three commonly used domain- shifting datasets: Domain- Net, Office-Home and ImageCLEF-DA More details can be found Appendix. We also incorporate synthetic data
5

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Table 1. The results (mean and standard deviation) of in-domain evaluation of three runs.
ImageCLEF-DA
Office-Home Product
DomainNet Painting
Setting
Method
Caltech
Image Vel
Art
Real
Clipart
Real
Zero-shot
CLiPz
97.25 + 1.03
96.87 = 1.67
78.12 1.21
85.11 0.84
86.33 = 1.25
62.67 4 2.68
59.77 = 2.50
78.07 2.07:11 1.90
Centr.
Classical
ViT
85.41 + 3.87
82.06 + 2.89
62.81 = 3.08
83.97 = 3.91
76.32 + 3.08
53.44 4 3.97
58.32 + 2.23
77.01
2.65
CLIP,
98.49 + 1.70
95.45 + 2
85.01 = 2.78
87.92 = 2.06
88.41 1.25
65.15 4: 1.89
65.66 + 1.97
83.28 4 2.05
PEFT
A
98.11 + 2.01
95.32 +: 1.42
84.17 + 2.22
88.02 = 1.76
88.26 = 1.86
68,5
65.13 :: 1.66
82.79 1: 2.37
FodAvg
95.11 + 3.90
83.33 + 2.74
75.62
1.76
86.85
58
82.07
2.96
51.66 4: 3.80
53.02 :: 2.74
69.31
FodAvg y
90.06 +: 2.08
80.25 +: 2.30
61.33 1.71
75.51
2 09
2.15 74.68
48.27 +: 3.00
43.87 +: 1.77
62.06 1 2.08
Classical
Federated
FedProx
96.75 + 3.62
84.16 +: 2.85
76.98 + 1.52
87.26
= 1.47
83.15
1.70
50.4 + 2.76
53.45 + 1.05
69.87 + 1.45
F
91.30 + 1.85
80.34 +: 2.11
62.47 + 1.86
75.65 1.38
74.98 + 1.76
48.89 + 3.10
44.92 + 1.62
63.77 + 1.10
FedCLIP
97.34 4 3.11
97.89 + 2.06
82.14 = 1.35
84.33
2.31
87.62 + 1.89
67. 96. 4: 2.05
65.78 + 1.66
$2.93 +: 1.21
CLIP
FedOT
97.26 + 2.85
97.91
+ 1.73
82.56 + 1.91
85.47
2.98
86.61
+ 3.26
67. 668
+ 2.74
65.85 + 1.78
83.20 + 1.40
FedNG
98.62:1 1.34
98.56 +: 1.78
84.97 = 1.78
88.69
1.06
88.79 = 1.57
70.36 +: 1.96
66.29 + 1.08
84,92 + 0.85
during the model training. The number of synthetic data for each training domain equals 10% of the real domain data.
will learn a shared adapter, but the number of parame- ters in the adaptor is the same as that of FedAG, although FedAG is equipped with several domain-specific adapters. We set the rank for CLIP as 32.
The details of synthetic data generation for different datasets are follows. When training the proposed FedAG, we also incorporate domain- synthetic data generated by Sta- ble Diffusion V2. The number of synthetic data for each training domain equals 10% of the real domain data. For the style-distinetive datasets, DomainNet and OfficeHome, synthetic data can be readily generated using text prompts following the template "a photograph/drrwwin of Sclass in Sstyle style". However, for ImageCLEF-DA, where the style information is implicit and challenging to articulate using text prompts, we resort to generating synthetic data using textual inversion (Gal et al., 2022). Textual inversion entails deriving an appropriate text token corresponding to the implicit style. We sampled 10 instances from each of the 12 classes within the real ImageCLEE dataset and employed the Diffuser library to perform textual inversion. Once the style token is derived, the server utilizes a similar template, "a Sclass in Sstyle token style", to generate synthetic images for ImageCLEF-DA Implementation Details can be found in Appendix C.
Federted Learning. We use two classical federated learn- ing approaches, FedAvg (McMahan et al., 2017) and Fed- Prox (Li et al., 2020), as baselines. These approaches are trained only with client data without interacting with CLIP. Since our model FedAG uses synthetic data for fine-tuning the client models, in the experiments, we also fine-tuned FedAvg and FedProx the same epochs as our approach the server at each communication round. The fine-tuned models are denoted as FedAvgz: and Fed- Prox The most relevant baselines are FedCLIP (Lu et al., 2023) and FedOT (Xiao et al., 2023). FedCLIP deploys a CLIP model for each client and fine-tunes the adapter on the local side. The adapters are uploaded to the server for aggregation, similar FedAvg. To conduct fair comparison, we conduct the same fine-tuning pro- cess on the adapter embedded in the same CLIP model on the server side as our own approach. FedOT (Xiao et al., 2023) is a federated version of Offsite-Tuning, where the CLIP model generates a compressed model and an emulator, which are shared with clients for their training.
Baselines. We compare the proposed F FedAG with several baselines in different settings, including zero-shot inference, centralized training, and federated learning.
4.2. In-domain Result Analysis
Zero-Shot Inference. We directly use the original CLIP model to predict the labels for given images in the testing data denoted as CLIP2- 2-
We train the models using the domains shown in the table and conduct the testing with the remaining domain data. Table 1 presents the average accuracy of three runs for the in-domain evaluation. We can observe that the proposed F FedAG performs best on all domains in all datasets. CLIP₂ 2 is a zero-shot learning model with CLIP, which does not use any training data. It performs better than the classical federated learning models like FedAvg and FedProx. These comparisons prove the predictive power of foundation mod- els for downstream tasks. The centralized PEFT approaches (CLIP, and CLIP A) achieve comparable performance but outperform the zero-shot model CLIP z, which confirms the necessity of fine-tunig foundation models for boosting per- formance. Although they are trained a centralized manner
Centralized Learning. Since F FedAG uses private do- main data (Di,... .... DNn } for client training and synthetic data {S,,... ,SN} } for server training, for fair compari- son, we also use them together for the centralized training baselines. This setting involves two kinds of centralized training: classical centralized training and fine-tuning CLIP. For the classical training, we directly train ViT with all data, denoted as ViT cen- We also choose two commonly used parameter-efficient fine-tuning methods, adapter fine-tuning and LoRA (Hu et al.. 2022) as base- lines, which are denoted as CLIPA a and CLIPP, L. CLIPA
6

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Table 2. The results (mean and standard deviation) of out-of-domain evaluation of three runs.
ImageCLEFe Pascal
Office-Home Clipart
DomainNet Quickdraw
Setting
Method
Infograph
Sketch
Zero-shot
CLIPz
82.13 + 0.58
61.07 + 1.09
39.34 + 2.24
13.06 + 1.87
58.11 + 1.56
Centra.
Classical
ViT oen
71.66 + 3.40
42.66 + 2.74
20.15 + 3.66
10.67 + 1.62
40.75 + 2.93
C
81.22 + 1.63
07.15
42.10 + 2.87
14.38 2.70
59.48 + 2.60
PEFT
CLIPA
81.08 + 2.97
67.31 2.51
42.22 + 3.51
13.85 1.55
59.01 + 2.74
FedAvg
78.33 + 4.33
43.58 3.65
26.75
10.78 + 2.95
40.56 + 3.11
FedAvg
73.02 + 3.88
11.12
24.27 3.14
10.33 + 2.40
37.91 + 2.25
Classical
Federated
FedProx
78.69 + 3.06
45.88 2.87
27.50 3.55
12.04
2.29
40.97 + 2.84
FedProx ft
72.68 2.74
40.75 + 2.44
24.63 2.81
11.89
2.48
38.54 + 2.36
FedCLIP
82.45 2.46
64.44
41.65 3.18
12.89
2.70
59.23 2.52
CLIP
FedOT
82.10 + 2.89
65.27 + 2.69
40.70 2.90
15.51 + 2.64
60.30 2.70
FedAG
83.78 + 1.94
68.15 + 1.08
45.56 + 2.26
21.04 + 2.20
63.29 + 1.76
and perform the best among all baselines, their performance worse than that of FedAG. The reason is that these two approaches only use one adapter or two low-rank matrices to model uses
Table 3. Ablation study on the Domain dataset. C for Clipart, P for Painting, R for Real, I for Infograph, Q for Quickdraw. S for Sketch.
store mixed domain knowledge. However, domain-speciff adapters to capture the characteristics of do- mains, thus leading to the best performance in the in-domain evaluation. These results also validate the design of multiple domain adapters. When comparing with the federated fine- tuning approaches, we can find they also perform better than CLIPz but have performance gaps with centralized PEFT approaches CLIP, and CLIPA- A. These results demonstrate the efficacy of injecting domain knowledge into foundation models in federated way.
In-domain
Out-of-domain
Method
P
R
Q
FedAGmome
68.54
65.60
83.00
44.38
20.14
62.85
FedAGquality
68.12
65.13
83.11
44.79
20.58
63.15
FedAGkd
64.27
82.55
42.62
18.40
62.03
FedAGeroas
70.04
66.11
84.13
40.63
15.70
59.04
FedAGrey
68.26
64.05
81.08
42.01
17.55
61.69
FedAC
70.36
66.29
84.92
45.56
21.04
63.29
best baseline FedOT. These results strongly indicate that our model effectively handles out-of-domain knowledge.
4.3. Out-of-domain Result Analysis
The ultimate goal of training a foundation model is to ap- ply it to various downstream tasks, including inference on unseen data. To assess this capability, we conduct an out-of- domain evaluation using the trained models with in-domain evaluation to validate the unseen domains. The results are shown in Table 2.
4.4. Abaltion Study
We use the following baselines to validate the effectiveness of our model design. FedAGmome does not use momentum update (i.e., Eq. (1)) for the local model after receiving the learned global model. FedAGquality denotes removing data quality estimation in Eq. (7). FedAGeross denotes removing the module of attention-regularized cross-domain learning. F Fed FedAGreg means that we remove the designed attention- based regularization term R in Eq. (12). The results of the ablation studies are presented in Tables 3. It is evident that removing each designed module results in a performance drop, underscoring the necessity of each module. Inter- estingly, the in-domain results suggest that cross-domain learning may not be as crucial compared to momentum updates and data quality estimation. However, in the out- of-domain evaluation, F FedAG plays significant role, as its removal leads to a dramatic performance drop. These findings align with the motivations behind our model design, emphasizing the importance of the cross-dae doos--domann learning module addressing the out-of oft-oandomain issue.
We observe similar trends to those in the in-domain evalu- ation. Specifically, FedAG outperforms all baselines, and CLIP performs better than classical models. However, compared to the in-domain evaluation results, the perfor- mance gaps between the centralized PEFT models (i.e., CLIP and CLIPA) and CLIPz are not as significant. In fact, their performance even worse than that of FedOT in several domains. These results highlight the limitations of existing models in generalizing out-of- knowledge. In contrast to existing approaches, our proposed F FedAG consistently achieves superior performance, leading to sig- nificant improvements in accuracy. For instance, the Quickdraw domain of the DomainNet dataset, our approach demonstrates 36% performance increase compared to the
7

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
4.5. Momentum Update for Clients
Table 4.
Results with different training domains.
Figure displays the empirical experiment results of mod- els trained with real and synthetic data the DomainNet dataset in centralized manner using three domains: "Cli- part", "Painting", and "Real", where the model used in this preliminary experiment is the same as our client model, which is TinyViI (Wu et al., 2022). The testing data used in the experiment are the head-out in-domain data. The data details can be found in &eec. 4.1.
Setting
Clipart
Painting
Real
Infograph
Quickdraw
Sketch
2 1
69.23 v
64.33 v
81.15 x
42,90 x
16.20 X
61.00 x
84.92
45.56
63.29
4
70.21 V
66.57 v
85.17
45.48
21.89 x
65.08 x
70.35 v
66.48
85.68
46.50
23.84
66.78
4.7. Generalization Study
It is evident from Figure that models trained with real data outperform those trained with synthetic data by significant margin. Therefore, replacing the well-trained client model W with the distributed W arbitrarily would disrupt the clients' training. Thus, we propose use the momentum update for the client training.
assess the effectiveness of various image encoders and to validate the generalizability of our proposed method, conducted additional tests using different CLIP image encoders the server side, including ResNet-50 (RS) and ViT-Tiny (VT). The outcomes of these tests are shown in Figure 5, In our main experiments, we employed the ViT- B-32 (VB) encoder. We can observe that ResNet-50 the weakest encoder, but its performance comparable that of the best baseline FedOT listed in Tables and Using other encoders can significantly enhance the performance. validating the generalizability of FedAG.
75
1I Synthetic Data
Real Data
ACC (%)
55
35
4.8. Performance Upper-bound Exploration
15
To further validate the efficacy of proposed method, FedAG, conducted comparative analysis against an upper bound benchmark. In the upper bound scenario, all training and synthetic data are combined to fine-tune a CLIP adapter, with each adapter tailored to a specific domain. This process, referred to as CLIP Mr involves tuning domain- specific adapters using the corresponding domain data. The comparative results are presented in Table 9, Our findings reveal that the performance of eddAG closely approaches, and in some instances surpasses, this upper bound. This fur- ther demonstrates the effectiveness of our proposed FedAG.
Clipart
Painting
Real
Figure 4. Performance comparison with synthetic data and real data.
4.6. Influence of Domain Knowledge
To investigate the impact of domain knowledge on in- domain and out-of-domain performance, we adjust the set- ting of in-domain and out-of-domain data utilization. We selected DomainNet for our experiments because it encom- passes wide variety of domains, and its zero-shot per- formance across these domains exhibits significant vari- ability. In our primary experiment, we assigned three do- mains Clipart, Painting, and Real to different clients for training. further extended our study to include three additional training configurations: (1) two training domains: Clipart and Painting, (2) four training domains: Clipart, Painting, Real and Infograph, and (3) five training domains: Clipart, Painting, Real, Infograph, and Quickdraw. For all the scenarios, we always keep the Sketch domain for out-of-domain testing. The in-domain and out-of-domaan test results from these configurations are subsequently re- ported. From the analysis presented in Table 4, it is evident that increasing the number of domains enhances the per- formance in both in-domain and out-of-domain tests. This improvement is particularly noticeable in domains that ini- tially demonstrated weaker performance, such as Infograph and Quickdraw. Such enhancements can be attributed to our proposed method's ' ability to effectively utilize knowledge from various domains.
Table Upper bound analysis.
Dataset
Domain
CLIP M
FedAG
Caltech
98.55
98.62
ImageCIEFF-DA
ImageNet
95.86
98.56
Pascal
83.67
83.78
Art
85.14
84.97
Product
88.78
88.69
Office-Home
Real
88.98
88.79
Clipart
69.59
68.15
Clipart
69.44
70.36
Painting
66.47
66.29
Real
84.86
84.92
DomainNet
Infograph
46.78
45.56
Quickdraw
21.80
21.04
Skectch
65.44
63.29
8

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
(i) In-domain Comparison
(b) Out-of-domain Comparison
90
Clipart
90
Painting
Real
90
Infograph
90
Quickdraw
90
Sketch
ACC (%)
45
45
45
45
0
RS VT
VB
RS
VB
RS
VB
VB
RS
VT
VB
VI
VB
Figure Results on the DomainNet dataset with different image encoders of CLIP. The blue dashed line denotes the performance of the best baseline FedOT using ViT-B-32 (VB) the encoder.
visualize the attention weights 3 learned by Eq. (9) with the four fine-tuned models various domains, mentioned in Table 4. These variations are depicted in Figure 6. The first image, originating from the Sketch domain, shows that within the context of out-of-domain data, Clipart receives the highest attention score, followed by Real and Painting, Conversely, for the in-domain example from the Real do- main (as depicted the second image), including identical domain data in the training regimen results in it receiving the highest attention score. These observations, supported by both Table 4 and Figure 6, effectively illustrate the dy- namic nature of domain knowledge in influencing model performance. They also highlight how the attention score adjusts in response to various domain knowledge configu- rations, emphasizing the adaptability and efficiency of our method in leveraging diverse domain insights.
Table The performance of FedAG with different sizes of syn- thetic data. "in" means the in-domain results, and "out" means the of results.
Data Volume
Dataset
Domain
100%
75%
50%
25%
Image
CLEP-
Caltech (in)
98.62
98.88
97.51
97.86
DA
ImageNet (in)
98.56
98.47
97.22
97.31
Painting (out)
83.78
83.04
82.50
82.43
Art (in)
84.97
84.30
82.46
82.04
Office -Home
Product (in)
88 69
88.21
87.51
86.77
Real (in)
88.79
88.63
87.18
86.45
Clipart (out)
68.15
67.30
66.78
66.44
Clipart (in)
70.36
68.41
67.96
66.12
DomainNet
Painting (in)
66.29
65.15
64,04
61.50
(in)
84.92
84.50
83.09
81.96
Infograph (out)
45.56
44.21
43.76
40.05
Quickdraw (out)
21.04
20.76
18.89
16.76
Sketch (out)
63.29
62.33
61.07
59.53
4.9. Synthetic Data Volume
Heatmap
Heatmap of
In this subsection, we examine the influence of synthetic data volume on the performance of our proposed algorithm. We sampled subsets of 75%, 50%, and 25% from the syn- thetic data used our main experiments while keeping other settings constant. The results for both in-domain and out-of-domain evaluations are presented in Table 10. From the analysis, we observe the following: (1) As the volume of synthetic data decreases, there is corresponding decline in performance across the three datasets for both in-domain and out-of- of-ommii-onnn scenarios. (2) The performance degrada- tion from reducing synthetic data from 50% to 25% more pronounced than the drop from 100% to 50%. (3) Notably, even with a minimal amount of synthetic data (25%), our ap- proach maintains reasonable performance in both in-domain and out-of-domain settings for all datasets. In summary, our investigation into the effects of synthetic data volume confirms its impact on algorithm performance; however, our approach demonstrates resilience to reduced data volumes within certain limits.
0.75
0.25
0.62
0.48
Dossnin: Skerids Birdd
Pk
0.70
0.17
0.13
0.20
0.68
0.15
0.09
0.94
Labeli Serbet
0.61
0.12
0.13
0.05
0.08
0.03
0.11
C
R
Figure The visulization of B3 image samples.
5. Conclusion
In this study, we introduced FedAG, an innovative federated fine-tuning approach designed to address the challenges of domain-specific characteristics and out-of-domain gener- alization with CLIP. Using multiple fine-grained adapters and novel learning modules, FedAG effectively integrates domain domain-specific knowledge and enhances generalization across diverse domains. Our extensive experiments ious datasets validate the efficacy of FedAG, showing per- formance improvements over wate-of-the-as methods. Our proposed approach has the potential to be applied to cross- device federated learning if a large number of domains are available, which we plan to extend our future work.
4.10. Case Study
To further illustrate the effectiveness of the proposed F FedAG for out-of-domain generalization, we use two case studies to

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Impact Statement
Hinton, G., Vinyals, O., and Dean, Distilling the knowledge in neural network. arXiv preprint
This paper contributes to advancing federated learning, foun- dation models, and domain adaptation by enabling efficient collaboration among multiple parties. Our work facilitates the development of foundation model generalization while addressing challenges in distributed data utilizatioon, domain adaptation, and garameter-efficier fine-tuning. These con- tributions have broad implications for sustainable comput- ing, improving the adaptability and efficiency of foundation models across diverse and decentralized environments.
arXiv: 1xXiv:50030023) 550,02251,, 2015.
Hu, E. J., Wallis, P., Allen-Zhu, Z,,Li, L.. Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Li, Y., Wang, S., Wang,
Khalid, U.. Iqbal, H., Vahidian, S., Hua, J., and Chen, C. Cefhri: communication efficient federated learning framework for recognizing industrial human-robot in- teraction. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10141- 10148. IEEE, 2023.
References
Chen, H., Wang, Y., Xu, C.,, Yang, Z.,, Liu, C., Shi, B., Xu, C., Xu, C., and Tian, Q. Data-free free-free learning of student networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3514-3522, 2019.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 1 17th International Conference on Machine Learning (ICMI. 2000), pp. 1207-1216, Stanford, CA, 2000. Morgan Kaufmann.
Chua, T. J., Yu, W., Zhao, J., and Lam, K.-Y. Fedpeat: Con- vergence of federated learning, parameter-efficien fine tuning, and emulator assisted tuning for artificial intelli- gence foundation models with mobile edge computing. arXiv preprint arXiv:2310.17491 2023.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-iman -inagggee-mggeeeee pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Deng, Y., Qiao, Z., Ren, J., Liu, Y., and Zhang, Y. Mu- tual enhancement of large and small language mod- els with cross-silo knowledge transfer. arXiv preprint arXiv:2312.05842,, 2023.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A.. and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems. 2:429 450, 2020.
Lu, W., Hu, X., Wang. J., and Xie, X. Fedelip: Fast general- ization and personalization for clip in federated learning. arXiv preprint arXiv:2302.13453 2023.
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., Yi, J., Zhao, W., Wang, X., Liu, Z., Zheng, H.-T., Chen, J., Liu, Y., Tang, J., Li, J., and Sun, M. Parameter-efficiee fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023.
Marchisio, K., Lewis, P., Chen, Y., and Artetxe, M. Mini- model adaptation: Efficiently extending pretrained mod- els to new languages via aligned shallow training. arXiv preprint arXiv:2212.10500. 2022.
Fan, T., Kang, Y., Ma, G., Chen, W., Wei, W., Fan, I.,, and Yang, Q. Fate-llm: A industrial grade federated learning framework for large language models. arXiv preprint 1rX0v22310.10049, 2023.
Marchisio, K., Lewis, P., Chen, Y., and Artetxe, M. Mini-model adaptation: Efficiently extending pretrained models new languages via aligned shallow train- ing. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 5474-5490, 2023.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H.. Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation us- ing textual inversion. arXiv preprint arXiv:2208.0166 016611 2022.
McMahan, B., Moore, E., Ramage, D.,, Hampson, S., and y Arcas, A. Communication--fffcti efficient learning of deep networks from decentralized data. In Artificial intelligence and statisties, pp. 1273-12823 PMLR, 2017.
Han, Z., Gao, C., Liu, J., and Zhang, S. Q. Parameter- efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.I5 146.6880 2024.
Peng, Z.., Fan, X., Chen, Y., Wang, 7.,, Pan, S., Wen, C., Zhang, R., and Wang, C. Fedpft: Federated proxy fine-tuning of foundation models. arXiv preprint arXiv:2404. 2024.
He, X., Li, C. Zhang, P., Yang, J., and Wang, X. E. Parameter-effieieet fine-tuning for vision transformers. arXiv preprint arXiv:2203.163: 16329, 3, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark,
10

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
J., al. Learning transferable visual models from natu- ral language supervision. In International conference machine learning, pp. 8748-8766 PMLR, 2021.
Raikwar, P. and Mishra, D. Discovering and overcom- ing limitations of noise-engineered data-free knowledge distillation. Advances in Neural Information Processing Systems, 35:4902-49912 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695. June 2022.
Sajjad, H., Dalvi, Durrani, N., and Nakov, P. On the ef- fect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429 2023.
Wang, J., Wang, X., Lyu, L., Chen, J., and Ma, E. Fedmeki: A benchmark for scaling medical foundation models via federated knowledge injection. Advances in Neural Information Processing Systems, 2024a.
Wang, X., Wang. J., Xiao, H., Chen, J., and Ma, Fed- kim: Adaptive federated knowledge injection into med- ical foundation models. Empirical Methods in Natural Language Processing, 2024b.
Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., and Yuan, Tinyvit: Fast pretraining distillation for small vision transformers. In European Conference on Computer Vision, pp. 68-85. 85. Springer, 2022.
Xiao, G., Lin, J., and Han, S. Offssite tuning: Transfer learn- ing without full model, arXiv preprint arXiv:2302.048700 2023.
Zhang, Z., Yang, Y., Dai, Y., Wang, Q. Yu, Y., Qu, L., and Xu, Z. Fedpetuning: When federated learning meets the paraieter-efffcient tuning methods of pre-trained lan- guage models. In Annual Mceting of the Association of Computational Linguistics 2023, pp. 9963-9977. Asso- ciation for Computational Linguistics (ACL), 2023.

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Appendix
75
Synthetic Data
Real Data
A. Pseudo-code of FedAG
55
Algorithm 1 shows the pseudo-code of the proposed FedAG model, which contains two main updates: the client update (lines 14) and the server update (lines 15 32).
35
15
Clipart
Painting
Real
B. Dataset
DomainNet'. It totally has 569,010 images from 6 domains, in- cluding clipart, infographics, painting. quickdraw, real, and sketch. Each domain contains 48K 172K images, categorized into 345 classes.
Figure 7. Performance comparison with synthetic data and real data.
Office-Home". : It has 15,500 images from different dimensions: artistic images, clip art, product images, and real-world images. Each domain has 65 object classes.
ing. Thus, we propose the momentum update for the client training.
ImageCLEF-DA S It is benchmark for the ImageCLEF 2014 domain adaption challenge, including Caltech-256 ImageNet ILSVRC 2012, and Pascal VOC 2012. There are 12 categories and 50 images in each category.
E. Details of Baselines
Table 7 shows the approaches used in the experiments, including training ways, deployment details, and used data.
F. Performance Upper-hound Exploration
C. Implementation Details
To further validate the efficacy of our proposed method, FedAG, we conducted comparative analysis against an upper bound benchmark. In the upper bound scenario, all training and syn- thetic data are combined to fine-tune tune a CLIP adapter, with each adapter tailored to specific domain. This process, referred to as CLIP us involves tuning domain-specific adapters using the corresponding domain data. The comparative results are presented in Table Our findings reveal that the performance of FedAG closely approaches, and in some instances surpasses, this upper bound. This further demonstrates the effectiveness of our proposed FedAG.
For each dataset, we assign each in-domain data to one client. We utilize T-inyppttcch6-224** for the client model and ViT-B- 32' for the image encoder for the server side. Our experimental setup involves 10 communication rounds. For the local update, we set the local training epoch as 10, the local learning rate as Ic-4, the hatch size as 32, y = 0.9, and the optimizer used the optimization as Adam. For the server update, we set À = 0.1, 5 1e-3, the epoch of quality-aware in-domain mutual learning as and the epoch of adapter initialization S. We keep the shared parameter settings consistent across our method and the baselines. The unique hyperparametens for the baselines are adopted as per the specifications in their respective original papers. All experiments NVIDIA A6000 with CUDA version 12.0, running on a Ubuntu 20.04.6 LTS server. All baselines and the proposed FedAG are implemented using PyTorch 1.0.
conducted
G. Additional Experiment Results of Ablation Study
In order to more thoroughly evaluate the effectiveness our de- signed modules, we conducted ablation studies on two additional datasets: ImageCLEF-DD and Office- The results of these studics detailed in Table 8. Our observations indicate that the performance trends on ImageCLEF--A and Office-Home datasets are consistent with those observed on the DomainNet dataset. This consistency across different datasets further validates the effec- tiveness of the individual components of our proposed method, FedAG, reinforcing the robustness and generalizability of ap- proach.
D. Momentum Update for Clients
Figure 7 displays the empirical experiment results of models trained with real and synthetic data on the DomainNet dataset a centralized manner using three domains: "Clipart", "Painting", and "Real", where the model used in this preliminary experiment is the same as our client model. which is Tiny VIT (Wu et al., 2022). The testing data used in the experiment are the head-out in-domain data. The data details can be found in $Sec. 4.1.
H. Study on Key Hyperparameters
is evident from Figure that models trained with real data out- perform those trained with synthetic data by significant margin. Therefore. replacing the well-trained client model W with the distributed W arbitrarily would disrupt the clients' train-
We examined the impact of key hyperparameters À and on our model's performance, with results illustrated in Figure 8 for X and Figure 9 for 8. Our observations from the experiments are as follows: (1) Increasing values and correspond to improved in-domain performance. This enhancement can be attributed to the mechanisms described in equations 11 and 12, where higher values of À and 8 focus the training more intensively knowledge acquisition. (2) Within certain range, elevating À and also enhances out-of out-of-domain learning, potentially due to the ef- fectiveness of in domain adapters, which facilitate inference through the cross-domain learning module discussed in Sec. 3.4.3. However, excessively high values of À 8 might overly concentrate the learning process on in-domain aspects, thereby
of afficeliomCCatassee.tttt
in-domain
htttp::/www.inngeeeeee..co
httppe///uuggigggaae...///tiitip base_
/nstpss///uggingggeec./////////s clip-vit-bbae--aacchh3
12

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Table Summaries of approaches used in this work.
Name
Training Way
Deployment Details
Used Data
CLIPz
No training
Zero-short CLIP
No data
CLIPO
Centralized
Fine-tuning adapter CLIP
In-domain data
CLIP M
Centralized
Fine-tuning multiple individual adapters
In-domain data
CLIP M
Centralized
Fine-tuning multiple individual adapters
Out-of-domain data
CLIPt
Centralized
Fine-tuning one adapter CLIP
In-domain and synthetic data
CLIPA
Centralized
Fine-tuning LoRA on CLIP
In-domain and synthetic data
ViT
Centralized
Fine-tuning ViT-B-32
In-domain and synthetic data
Client: Fine-tuning ViT-Tiny
In-domain data
FedAvg
Federated
Server: No training
No data
Client: Fine-tuning ViT-Tiny
In-domain data
FedProx
Federated
Server: No training
No data
Client: Fine-tuning ViT-Tiny
In-domain data
FedAvgft
Federated
Server: Fine-tuning the aggregated model
Synthetic data
Client: Fine-tuning ViT-Tiny
In-domain data
FedProx
Federated
Server: Fine-tuning the aggregated model
Synthetic data
Client: CLIP + Adapter
In-domain data
FedCLIP
Federated
Server: No training
No data
Post-training: Fine-tuning one adapter
Synthetic data
Client: Adapter + emulator
In-domain data
FedOT
Federated
Server: Fine-tuning one adapter on CLIP
Synthetic data
Client: Fine-tuning ViT-Tiny
In-domain data
FedAG
Federated
Server: Fine-tuning multiple adapters on CLIP
Synthetic data
Table 8. Ablation study experiment results on ImageCLEF-DA and Office-Home datasets. v denotes the in-domain and X denotes the
out-of-domain.
ImageCLEF-DA
Office-Home
Method
Caltech
ImageNet v
Pascal X
Art
Product v
Real
Clipart X
FedAGmome
98.28
98.17
83.11
84.88
88.09
88.20
66.15
FedAGquality
97.45
98.21
83.26
82.47
87.14
87.24
67.20
FedAG
97.41
98.04
83.21
81.89
86.54
86.57
67.31
FedAGeross
98.44
98.36
82.55
83.09
87.88
88.62
63.89
FedAGreg
97.30
97.13
81.17
81.55
85.69
86.11
67.36
FedAG
98.62
98.56
83.78
84.97
88.69
88.79
68.15
degrading out-of-domain performance. In summary, careful ad- justment of and å offers a means to tailor performance across different domain configurations, achieving a balance between in- domain focus and out-of-domain generalization.
more pronounced than the drop from 100% to 50%. (3) Notably, even with minimal amount of synthetic data (25%), our approach maintains reasonable performance in both in-domain and out-of- domain settings for all datasets. In summary, our investigation into the effects of synthetic data volume confirms its impact algo- rithm performance; however, approach demonstrates resilience to reduced data volumes within certain limits.
I. Synthetic Data Volume
In this subsection, examine the influence of synthetic data vol- ume on the performance of our proposed algorithm. We sampled subsets of 75%, 50%, and 25% from the synthetic data used in our main experiments while keeping all other settings constant. The results for both in-domain and out-of-domain evaluations presented in Table 10. From the analysis,
observe the fol- lowing: (1) As the volume synthetic data decreases, there is corresponding decline in performance across the three datasets for both in-domain and out-of-domadl scenarios. (2) The performance degradation from reducing synthetic data from 50% 25% is
13

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
74
Clipart
68
Paint
90
Real
72
ACC%
66
85
64
68
66
75
0
0.1
0,5
0.9
0.1
0.5
0.9
0
0.1
0.5
0.9
46
Infograph
22
Quickdraw
64
Sketch
ACC%
44
20
62
42
0.5
0.9
0.1
0.1
À.
Figure 8. Hyperparameter study of on the DomainNet dataset. out-of-domain.
Red is the color for the in-domain, and blue is the color for the
72
Clipart
Paint
Real
71
ACC%
66
84
70
64
82
69
80
0.5
0.9
46
Infograph
Quickdraw
Sketch
63
ACC%
20
62
18
43
42
60
0
0.1
0.5
0.9
0
0.1
0.5
0.9
0
0.1
0.5
0.9
8
8
Figure 9. Hyperparameter study of on the DomainNet dataset.
Red is the color for the in-domain, and hlue is the color for the
oft-of-domain.
Table 9. Upper bound analysis.
Table 10. The performance FedAG with different sizes of syn- thetic data. "in" means the in-domain results, and "out" means the out-of-domain results.
Dataset
Domain
CLIP
FedAG
Caltech
98.55
98.62
Data Volume
ImageCLEF-DA
ImageNet
95.86
98.56
Dataset
Domain
100%
75%
50%
Pascal
83.67
83.78
Image
CLEF-
Caltech (in)
98.62
98.88
97.51
97.86
Art
85.14
84.97
DA
ImageNet (in)
98.56
98.47
97.22
97.31
Product
88.78
88.69
Painting (out)
83.78
83.04
82.43
Office-Home
Real
88.98
88.79
Office -Home
Product (in)
Art (in)
88.69
84.97
84,30
88.21
82.46
87.51
86.77
82.0M
Clipart
69.59
68.15
Real (in)
88.79 79
88.63
87.18
86.45
Clipart
69.44
70.36
Clipart (out)
68.
67,30
66.78
66.44
Painting
66.47
66.29
Clipart (in)
70.36
68.41
67.96
66. 12
Real
84,86
84,92
DomainNet
Painting (in)
65.15
64.04
61.50
DomainNet
Infograph
46.78
45.56
Real (in)
84.92
84,50
83.09
81.96
Infograph (out)
45,56
44.21
43.76
40.05
Quickdraw
21.80
21.04
Quickdraw (out)
21.04
20.76
18.89
16.76
Skectch
65.44
63.29
Sketch (out)
63.29
62.33
61.07
59.53
14

---

Enhancing Foundation Models with Federated Domain Knowledge Infusion
Algorithm 1 Algorithm Flow of FedAG.
Input:
Local data {Di,... ), Stable Diffusion V2, domain descriptions, task label descriptions, communication rounds T, local training epoch E., server training epoch E.,, and hyperparameters: A, 7, and 8
Server Initialization
Use Stable Diffusion V2 to generate domain-specific synthetic data {Si, .... .NN) based on domain descriptions and
task label descriptions:
Use CLIP training to initialize domain domain-specific adapters using {Si, SN) individually;
Distribute each Nommann-specifer synthetic dataset S, to the corresponding client Cx; for each communication round .......TTT do
Client Update
for each client n E ,...... .V do
Momentum update W"; i >
for each local epoch E [1, Eal do
Update W with Eq. (2);
end
Calculate quality scores a using Wt, for Sn
end
Upload (W,,a)): (W x to the server;
Server Update
Wi- -W.
w
for each server epoch [1. E, do
Initialize the total loss G 00 0;
for each domain E . No do
// Quality-aware In-domain Mutual Learning
Obtain domain-specific logits t5 for each synthetic data st, Sn using Eq. (5);
Obtain predicted propobalities 0i. for each synthetic data s/. E S,, using the upload client model W n Calculate J, using 0%*t and ti using Eq. (7) for each data;
I/ Attention Attention-regularieed Cross-domain Learning
Caluate the aggregated logits t using Eq. (9) each synthetic data st, E Sn Obtain the attention based regularize Ri, using Eq. (11) each synthetic data s E Sn; Obtain the loss of specific domain ;;:;
g + ;
end
Optimize g using Eq. (12);
end
Distribute the updated local models W = (W;. W to the corresponding local clients;
end
15


Reviews:
[{"review_id": "g3H0VAgMZ8", "reviewer": "Reviewer_9HNy", "summary": "This manuscript addresses the challenge of fine-tuning large-scale vision-language models in a federated learning setting under domain shifts. The authors propose FedAG (Federated Adapter Generalization), a method that introduces multiple domain-specific adapters to capture heterogeneous domain knowledge while maintaining out-of-domain generalization working under a federated setting. \n\nThe FedAG took a two step approach A) Quality-Aware In-Domain Mutual Learning step which utilizes client models’ knowledge to refine the adapter for each domain, weighting synthetic data by estimated quality; B) Attention-Regularized Cross-Domain Learning step which synthesize logits from all adapters for inference on out-of-domain inputs, guided by an attention-based regularizer to help identify which adapter’s knowledge is most relevant to new data.\n\nExperimental evaluations on DomainNet, Office-Home, and ImageCLEF-DA demonstrate improvements in both in-domain accuracy and out-of-domain generalization over existing baselines such as FedCLIP, FedOT, and classical parameter-efficient fine-tuning approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s constructive suggestions and comments. We would like to reply respectively as follows.\n\n`>>> Response to Other Comments Or Suggestions`\n\nThank you for the feedback. We will follow your suggestion to improve the figure and fix the typos in the final version of the paper.\n\n`>>> Q1`\n\nThank you for the question. In our proposed approach, the number of adapters is equal to the number of domains. If we have more clients, we can group them into different groups based on the domains to which they belong. Then we aggregate the model parameters into one for each domain before we conduct the mutual learning process in our design. As for the computational cost, we freeze the encoders of the foundation models and only allow the parameters of adapters and the local model to be trainable and updated. Furthermore, we put most of our designed operations on the server side to release the burden of the local clients, as it is more common that the server side may have more flexible computation constraints than the local side.  As for the communication cost, as we mentioned in Sec 3.1, we may need to transfer the synthetic data from the server to the clients in a one-time manner, which can be negligible.  We appreciate the reviewer’s question and will investigate how to further reduce the computation and communication cost in future work. We hope our response can sufficiently address your concern.\n\n`>>> Q2`\n\nThank you for the comment. It is possible that the real-world scenarios could be more subtle, but we believe that our method can work for these different scenarios as demonstrated with three datasets having different levels of domain shifts. In our approach, we proposed a cross-domain learning mechanism to capture the relationship among different domains and quantify it with a weight score $\\beta$. Besides that, we add a regularizer to further adjust the attention based on the domains of the data.   As the reviewer commented, given a subtle domain shift in the real-world scenario, PEFT could be a more effective way. However,  we may think our method is still advantageous because we do not directly access the local data, while centralized PEFT does not have such advantages. We will explore this direction with some real world data and compare it with PEFT in our future work. We hope our reply can sufficiently answer your question.\n\n`>>> Q3`\n\nThanks for the observation and the valuable question.  We would like to provide our explanations as follows. First of all, as described in Appendix C, we use the ViT-B-32 for the image encoder on the server side. As it is pretrained, it has the basic zero-shot inference capability for image-related tasks. Secondly, for the baseline ViT_cen, we put the data from the domains together and tune all the parameters, which shows the degraded performance. One possible reason could be that we fine-tune all the parameters of this large ViT-based model with not enough training data. This may harm the previous well-pretrained model because of the under-training. This can also be verified by the results of CLIP_L and CLIP_A, where we conduct PEFT approaches with training LoRA and adapters only. With these two approaches, the performance will get boosted compared with ViT_cen.\n\nWe appreciate the reviewer's valuable comment. We will add the analysis above to the result analysis in Sec 4.2 and 4.3, respectively, in the final version of our paper.", "claims_and_evidence": "The authors claim that the proposed method will be an important step for generalising CLIP especially in handling out-of-domain predictions. \n\nThe method was evaluated on DomainNet, Office-Home, and ImageCLEF-DA to demonstrate improvements in both in-domain accuracy and out-of-domain generalization over existing baselines such as FedCLIP, FedOT, and classical parameter-efficient fine-tuning approaches.", "methods_and_evaluation_criteria": "The evaluation method and criteria were proper.", "theoretical_claims": "The manuscript did not take any theoretical claims. It was an empirical research.", "experimental_designs_or_analyses": "The experimental design was overall sound and the data analyses was accurate.", "relation_to_broader_scientific_literature": "This paper presents a novel federated fine-tuning approach for large pretrained models, specifically CLIP. Unlike traditional methods like FedAvg (McMahan et al., 2017) and FedProx (Li et al., 2020) which aggregate fully trained local models, or Offsite-Tuning (Xiao et al., 2023) and FedCLIP (Lu et al., 2023) which distribute compressed or partial models, this work proposes a system where the server hosts CLIP and multiple adapters, while clients have a lighter ViT-Tiny. Clients collect domain-specific knowledge and transfer it to the server's adapters through a structured process, contrasting with prior single-adapter or sub-model compression strategies.", "essential_references_not_discussed": "Not I am aware of.", "other_strengths_and_weaknesses": "The manuscript identifies the drawbacks of using a single adapter for data aggregated from multiple (and potentially very diverse) domains. Multiple domain-specific adapters align well with the stated aim to capture each domain’s particularities while still leveraging each other for out-of-domain generalization.\n\nThe authors present experiments on three domain-adaptation benchmarks (DomainNet, Office-Home, ImageCLEF-DA). They provide ablation studies (momentum, quality estimation, cross-domain learning, attention regularization) and hyperparameter sensitivity analyses, giving the reader a detailed view of how each component influences final performance.", "other_comments_or_suggestions": "The figure 3 was not very easy to understand. I assume the purpose of the chart is to give readers an intuitive overview before diving into the details in the text. I would suggest to keep it high-level to explain the steps in the training cycle.\n\nThere are some typos in the manuscript, e.g. in pseudo code of the algorithm \"Caluate the aggregated logits...\".", "questions_for_authors": "1. while the FedAG will maintain domain-specific adapters for each client, what will be additional computation and communication cost in comparison to the other methods?\n2. In the real-world scenario, the domain shift might be more subtle than the difference in experimental dataset. When will it still be advantageous to take the proposed approach? Are there scenarios that the proposed approach may perform worth than prior approaches, e.g. PEFT?\n3. The performance of Zero-shot inference was stronger than some classical approaches in both in-domain and out-of-domain inference, is there any explanation on this observation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YBSa2CP50", "reviewer": "Reviewer_agi1", "summary": "This paper introduces a federated learning approach to enhance the capability of foundation models to handle in-domain and out-of-domain tasks. In particular, the authors designed quality-aware in-domain mutual learning and attention-based cross-domain learning to capture the knowledge effectively. In this paper, they provided extensive experiment results and compared them with other baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the reviewer’s valuable comments and suggestions. We would like to address them as follows for your review.\n\n`>>> W1`\n\nIn our main experiment, the amount of the synthetic data is equal to 10% of the real data for each domain. To further examine how the amount of synthetic data affects the results, we conduct a further study about the synthetic data volume in Appendix I. In particular, we randomly sample 75%, 50%, and 25% from the existing synthetic data in our main experiment and repeat the experiments while keeping all the settings unchanged. We observe that the performance will degrade with the reduction of the synthetic data. With very limited synthetic data (25%), our approach is still able to maintain an acceptable performance. The results demonstrate how the amount of the synthetic data affects the final results.\n\n `>>> W2`\n\nDue to the limited page number in the main paper, we put the pseudo algorithm in the last page of the Appendix. To further enhance the readability of our work, we will try to put a simplified version of the pseudo code in the main paper.  \n\n `>>> Q`\n\nIn our proposed framework, the clients share very limited and vague information with the server. We let the clients provide the style information via the text prompt or the generated textual inversion token. In particular, for easily distinguishable styles such as “Ghibili cartoon”, “Pablo Picasso”, direct text prompt can be used. For vague or ambiguous styles, one can use textual inversion, a technique that enables Stable Diffusion to learn a new embedding vector to represent this style from just a few sample images. This involves a brief training process, the system optimizes only this new embedding vector. It adjusts the vector so that when it's fed into the frozen Stable Diffusion model along with descriptive prompts (like \"a photo in <my-new-style>\"), the model generates images that look like the sample images. After the style tokens are gathered from clients, the server can simply apply the template such as “a clock in <style-token> style” as text prompt input to Stable Diffusion to generate synthetic data.\n\nBesides that, to avoid the effects of the low-quality synthetic data, we further design a quality estimation module to measure the quality of the generated data and add that score to our optimization equation. We will add a more detailed description to clarify this part in the final version of our paper.", "claims_and_evidence": "The claims in this paper are well supported by statements, formulation, experiments, and discussion.", "methods_and_evaluation_criteria": "The designed method fits the problem setting. First of all, federated learning is able to solve the challenge of distributed data and the foundation model deployment. Secondly, the design of the adapter cluster considers the in-domain and out-of-domain respectively.", "theoretical_claims": "The methodology part is formulated and described clearly.", "experimental_designs_or_analyses": "The authors validate the method via appropriate experimental design and results. In Table 1 and 2, they reported the in-domain and out-of-domain results along with other baselines and settings. They also conduct ablation study, generalization study, and case study. The visualization in Figure 5 facilities the understanding of out-of-domain generalization with the proposed approach.", "relation_to_broader_scientific_literature": "This paper explores a more practical setting where the foundation models are put at the server side and the local models interact with the large models via the adapter cluster. This setting can be extended to broader applications and scenarios.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths \n1, This paper studies a more practical setting to enhance the capability of foundation models via federated learning. It does not require the clients to be equipped with large models. \n2, The in-domain and out-of-domain learning methods consider the different features of the knowledge from the data. The design is able to capture the information effectively. \n3, This approach is straightforward and easy-to-implement. Based on the experiments and design, it can be well generalized to other scenarios and applications. \n4, The whole pipeline only trains very limited parameters, which consider the efficiency as well. \n\nWeaknesses\n1, What amount of the synthetic data is used and how do they affect the results?\n2, The paper lacks a pseudo algorithm for the whole pipeline.", "other_comments_or_suggestions": "No", "questions_for_authors": "1, Could you please clarify how the stable diffusion generates the synthetic data in this work?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3kXlw9gxT", "reviewer": "Reviewer_otAj", "summary": "This paper introduces FedAG, a federated learning method to enhance vision foundation models (e.g., CLIP) by fine-tuning them across distributed domains while preserving data privacy. FedAG employs multiple domain-specific adapters, synthetic data generation via Stable Diffusion, and quality-aware mutual learning to capture domain knowledge. It also uses attention regularization to improve out-of-domain generalization. Experiments on ImageCLEF-DA, Office-Home, and DomainNet show FedAG outperforms centralized and federated baselines, achieving higher accuracy in both in-domain and out-of-domain settings.\n\n---\nI appreciate the authors for their rebuttal and will keep my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the reviewer’s valuable comments and questions. We would like to address them as follows for your review.\n\n`>>> W1` and `>>> Q1`\n\nThanks for your constructive comment and question. If we have a dynamic or numerous clients, we will group clients by the domains they belong to. In Sec 3.4.2 or Figure 3.c, we can conduct a basic model aggregation approach to get the aggregated $\\bar{W}_n$ for each domain group n, e.g., average-based approach. Then we can replace the original $W_n^t$ with the domain group-level \\$\\bar{W}_n$ The aggregation process will merge the knowledge from the same domain and can be generally plugged into the current proposed framework. Also, as it is a basic average operation of the model parameters, it may not raise too much extra computational burden. In our proposed approach, the number of the adapters is equal to the number of domains, and each domain adapter can only interact with one aggregated model parameter, as we discussed above.  However, we totally agree with the reviewer’s comments about the setting with a large number of domains. Our current work is based on the cross-silo setting and we have stated that we will further explore the potentials of extending this approach to the cross-device setting in the conclusion part (line 434-439). We hope our reply can sufficiently answer your question and address your concern.\n\n`>>> W2` and `>>> Q2`\n\nThank you for the comment. We have actually considered the issues of synthetic data quality problems and discussed them in Sec. 3.3.2. We totally agree that the quality of generated data would have an effect on the performance. To solve this, in our proposed work, we design an estimation mechanism to estimate the quality of the generated data in Sec 3.3.2. In particular, we first obtain the prototype representation for each label category. After that, we calculate the cosine similarity of the representation of the generated data and the prototype to obtain the score \\alpha. With this estimation, we add it as a weight to equations (6) and (7) (lines 245 to 248) to conduct mutual learning using quality assessments of the generated data. With a low quality of the synthetic data, the weight will be small and help control its effect on the overall performance. \n\n\nFurthermore, to examine the effectiveness of this quality evaluation module, we also provide the ablation study in Sec 4.4, and the results are shown in Table 3. We remove the quality estimation module for the synthetic data and name the setting as FedAG_quality. We observe that the performance drops in the evaluation of the in-domain and out-of-domain data, which demonstrates the effectiveness of the quality evaluation module to help control the utilization of the synthetic data. \n\n\nWe do appreciate that the reviewer mentioned the possibility of adversarial samples in the synthetic data. We will explore this direction to further improve the security and bias of our proposed approach in future work. We hope our response fully clarifies your question and resolves your concern.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This paper is related to vision foundation models and federated learning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strengths:\n1. The integration of multiple domain-specific adapters within the federated framework is interesting and seems novel.\n2. The proposed method has practical value since it focuses on cross-silo federated learning and aligns with real-world privacy constraints.\n3. The idea of using a diffusion model for synthetic data generation for privacy-preserving is interesting.\n\n### Weaknesses:\n1. The paper assumes a fixed number of domains, how to deal with dynamic or numerous clients? The computational overhead of managing multiple adapters is also unexplored.\n2. The paper assumes the synthetic data is high-quality and representative, which lacks in-depth analysis. Poorly generated data could bias adapters or harm generalization.", "other_comments_or_suggestions": "The mutual learning and attention regularization modules (Sec. 3.4.2–3.4.3) are overly technical; intuitive diagrams or simplified explanations would improve accessibility.", "questions_for_authors": "1. How does FedAG handle scenarios with a large or dynamic number of domains (e.g., 100+ clients)? Does the server-side adapter aggregation scale efficiently?\n2. How does the quality of Stable Diffusion-generated data impact performance? Are there safeguards against adversarial or biased synthetic samples?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WAYx5cRyuT", "reviewer": "Reviewer_373f", "summary": "This paper proposes an efficient federated fine-tuning approach that enhances out-of-domain generalization. In the proposed framework, each client utilizes a lightweight ViT model, which is trained on local data. Subsequently, data quality scores are computed using synthetic data and transmitted to the server along with the locally trained model. The server then performs mutual learning to distill knowledge from a large model (CLIP) and applies attention-regularized cross-domain learning to improve out-of-domain generalization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the comments and questions.\n\n`>>> W1`\n\nYes, the MLP layers are optimized during the training. We will clarify it in the final version.\n\n`>>> W2`\n\nWe do not share any label information with the server.  As introduced in Sec 3.1, clients only share the style information via the text prompt (style is clear) or the generated textual inversion token ( style is vague), for the data generation.\n\n`>>> W3`\n\nWe clarify that the domain shift among clients also represents the data heterogeneity in FL [1,2]. In our experiments, the data from each client represents a unique domain and there is no crossover/mixup between clients. Under such a heterogeneous setting, our approach is still able to help enhance the capability of the foundation models with the help of local clients’ domain knowledge. Furthermore, we add more experiments of the data class-heterogeneity. With ImageCLEF-DA maintaining settings unchanged in Table 1 and Table 2,  we partition the dataset with respect to the classes using the Dirichlet distribution in [3] to simulate the class heterogeneity.  The performance of the results decreases slightly but is still higher than the classical methods.\n\n|        | Caltech | ImageNet | Pascal |\n|--------|---------|----------|--------|\n| FedAG  | 97.61   | 97.15    | 82.66  |\n\n[1] Heterogeneous federated learning: State-of-the-art and research challenges. 2023.\n\n[2] A review of federated learning methods in heterogeneous scenarios. 2024.\n\n[3] Harmonizing Generalization and Personalization in Federated Prompt Learning, ICML 2024\n\n`>>> W4`\n\n[CVPR’24] designs a prompt-tuning approach to solve the data heterogeneity challenge in FL. They learn shared prompts at the server side and match them with the local groups and assign group prompts to them. The main design covers how to learn shared prompts, group prompts, prompt selection, and an optimization method to iteratively learn different knowledge. In [ICML’24]  finds the balance between generalization and personalization for the data heterogeneity. The approach is mainly a prompt-based approach, which is based on utilizing local personalized prompts with the help of a global prompt and an adaption term. Also, most of their designed modules are at the local side and they focus on the evaluation on the local clients.\n\nCompared with them, we have 2 key differences: 1, motivation part: we focus more on how to enhance the capability of foundation models at the server side with the help from the local clients by knowledge infusion; 2, method part: our key method is not a prompt-based approach and most operations happen at the server side to release the computation burden at the client side.\n\n`>>> W5`\n\nThe results in Table 1 and 2 are already cover the setting of 3--->3 in Table 4. Due to the limited time during the rebuttal period to run experiments, we report the 2--->4 setting as below. (We omit the decimal part due to limited space)\n \n|        | Clipart | Painting | Real  | Info | Quick | Sketch |\n|-------------|--------|---------|------|-------|--------|--------|\n| FedAvg      | 48  | 48   | 66 | 22 | 10 | 35  |\n| FedAvg_ft   | 46  | 47   | 66 | 21 |  9 | 33  |\n| FedProx     | 48  | 50   | 67 | 23| 10 | 35  |\n| FedProx_ft  | 46  | 49   | 66 | 22 |  9  | 34  |\n| FedCLIP     | 65  | 61   | 72 | 37 | 11  | 46  |\n| FedOT       | 64 | 61   | 73 | 36 | 12  | 47  |\n| FedAG       | 69  | 64   | 81 | 42 | 16  | 61  |\n\nWe see (1) given 2 in-domain combinations for training, the overall performance decreases compared with the setting where we have 3. This is because of the less training data, which causes the performance degradation; (2) Our approach still outperforms other baselines. \n\n`>>> W6`\n\nOur method can also work even without knowing the in-domain and out-of-domain of a given task and it can be scaled with more domains. Given the initialized number of adapters equal to the number of domains, our designed modules are capable of handling out-of-domain tasks. In particular, we report the results of different training and testing domains in Table 4, where we scale our approach to different numbers of domains. As the reviewer mentioned, given the assumption if we may not be able to access to the domain information that it is in-domain or out-of-domain and minimize the change of current approach, we could equally treat the data and use the label index with the maximum value in $\\eta^{i}$ as the predicted label via eq 9. We provide the results in this case following the setting in Table 4 as below.\n\n|         | Clipart | Painting | Real  |\n|---------|---------|----------|-------|\n| Known   | 70.36   | 66.29    | 84.92 |\n| Unknown | 68.31   | 65.77    | 84.07 |\n\nThe performance has a slight decrease in a certain range but still outperforms the baselines. The results show the effectiveness of our method given the condition without knowing the domain information during the inference process. \n\nWe hope our reply sufficiently answers your question and addresses your concern.", "claims_and_evidence": "By leveraging synthetic data generated by a diffusion model, the proposed method effectively distills knowledge from the large model to the client model. Experimental results validate the effectiveness of this approach.", "methods_and_evaluation_criteria": "The approach of adopting a small model on each client while utilizing a large model on the server is reasonable. Regarding evaluation criteria, this paper employs standard benchmark datasets (e.g., DomainNet) to demonstrate the effectiveness of the proposed method.", "theoretical_claims": "There is no theoretical claim in this paper.", "experimental_designs_or_analyses": "It would be beneficial if the authors conduct additional experiments with various combinations of training domains while comparing the proposed method against other baselines.", "relation_to_broader_scientific_literature": "The unique contribution of this paper lies in utilizing a small model for each client and enhancing its performance by distilling knowledge from the large model on the server.", "essential_references_not_discussed": "Some relevant related works are missing. Efficient fine-tuning in FL has already been explored in federated prompt tuning (e.g., [ICML'24], [CVPR'24]). A comparison with these methods should be included in the paper.\n\n[CVPR'24]] Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning\n\n[ICML'24] Harmonizing Generalization and Personalization in Federated Prompt Learning", "other_strengths_and_weaknesses": "### Strengths\n- The approach of utilizing a small model on each client while leveraging a large model on the server is practically important.\n- The idea of enhancing the local model by distilling knowledge from the large model on the server is promising.\n\n### Weaknesses\n- Some parts of the description of the proposed method are unclear. An additional MLP layer is required to compute $\\beta$ in Eq. (10). Are these layers optimized during training?\n- If the label set differs across clients, this label information should be shared with the server. However, this may lead to the leakage of each client's private information.\n- One of the key challenges in FL is addressing data heterogeneity. However, this paper does not take this aspect into account. It would be beneficial to include experiments that consider data heterogeneity (e.g., using a Dirichlet distribution).\n- It would be beneficial to clearly differentiate the unique contributions or advantages of this method from existing federated prompt learning approaches (e.g., [ICML'24], [CVPR'24]).\n- To demonstrate the robustness of the proposed method across various scenarios, it would be beneficial to compare the results from different training domain combinations in Table 4 with existing baselines.\n- The inference process of this method assumes that the model has to know whether a given test task belongs to an in-domain or out-of-domain scenario, which is impractical in real-world applications. During test time, the model does not have access to this information.\n\n[CVPR'24]] Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning\n\n[ICML'24] Harmonizing Generalization and Personalization in Federated Prompt Learning", "other_comments_or_suggestions": "The cross-silo FL setting targeted in this paper may have sufficient computing resources, as it generally assumes participation from large institutions such as hospitals. Therefore, wouldn't it be more appropriate to target the cross-device FL setting instead?", "questions_for_authors": "- See the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes FedAG, a federated learning method to fine-tune large-scale vision-language models (e.g., CLIP) across distributed domains while preserving data privacy. FedAG introduces multiple domain-specific adapters to capture heterogeneous domain knowledge and maintain out-of-domain generalization. It happens in two-steps: Quality-Aware In-Domain Mutual Learning and Attention-Regularized Cross-Domain Learning. The former utilizes client models’ knowledge to refine the adapter for each domain, and the latter synthesizes logits from all adapters for inference on out-of-domain inputs. Experimental evaluations demonstrate that FedAG outperforms existing baselines in both in-domain accuracy and out-of-domain generalization.

Strengths:
1. The proposed method has practical value since it focuses on cross-silo federated learning and aligns with real-world privacy constraints.
2. The whole pipeline only trains very limited parameters, and the approach is scalable.
3. The reviewers highlight the quality of the empirical study and point out that the ablation studies are appropriate and informative.

Weaknesses:
1. Assumes a fixed number of domains; unclear how it handles dynamic or numerous clients.
2. Assumes high-quality and representative synthetic data, lacking in-depth analysis.
3. The main results do not sufficiently address the case in which domain information is not given

The authors address W1 satisfactorily and provided an intuitive modification to the approach; exploring this further is outside of the scope of this work.  Many weaknesses that were pointed out by reviewers where clarified by the authors in their rebuttal, namely, the comment about the this paper not taking into account data heterogeneity.  Regarding W3, the authors outline how they will address this in the camera ready, however, it is unclear if this will be sufficient.  Overall, the strengths outweigh the weaknesses, and the paper is suitable for ICML.
Decision: Accept (poster)