Learning to Prompt Your Domain for Federated Vision-Language Models
Authors: Guoyizhe Wei
Feng Wang
Anshul Shah
Rama Chellappa
Venue: Accepted by TMLR ()

Abstract: The prompt tuning paradigm, with its great advantages of low parameter count and stable training, has recently inspired numerous applications of CLIP-like vision-language models in federated learning. However, in this work, we posit that under significant domain gaps across federated participants, prompt-based CLIP may easily collapse to non-optimal solutions due to the neglect of domain-aware knowledge. We present a novel prompt tuning method, termed ADAPT, to address this issue by learning both intra- and inter-domain prompts. Specifically, we assign each federated participant a domain-specific prompt and use the image's visual features as a condition to guide the generation of language features, with the underlying idea that the prompted CLIP should detect the input image's domain correspondence before making the prediction of its category. Extensive experiments demonstrate ADAPT's significant efficiency and effectiveness in federated learning. For example, by learning and sharing only 2.1M parameters, ADAPT attains a 69.8% average accuracy over the six domains of DomainNet, which improves the original CLIP accuracy by 16.2%.

Published in Transactions on Machine Learning Research (11/2025)
Learning to Prompt Your Domain for Federated Vision- Language Models
Guoyizhe Wei
Johns Hopkins University
Feng Wang
Johns Hopkins University
Anshul Shah
Johns Hopkins University
Rama Chellappa
Johns Hopkins University
Reviewed on OpenReview: https: // openreview. net/ forun? id- 0S7sP02jr3
Abstract
The prompt tuning paradigm, with its great advantages of low parameter count and stable training, has recently inspired numerous applications CLIP-like vision-language models in federated learning. However, in this work, we posit that under significant domain gaps across federated participants, prompt-based CLIP may casily collapse to non-optimal lutions due to the neglect of domain-aware knowledge. We present novel prompt tuning method, termed ADAPT, to address this issue by learning both intra- and inter-domain prompts. Specifically, we assign each federated participant a domain-specific prompt and use the image's visual features as a condition to guide the generation of language features, with the underlying idea that the prompted CLIP should detect the input image's 's domain correspondence before making the prediction of its category. Extensive experiments demon- strate ADAPT's significant efficiency and effectiveness in federated learning. For example, by learning and sharing only 0.35M parameters, ADAPT attains 69.8% average accuracy over the six domains of DomainNet, which improves the original CL.P accuracy by 16.2%.
1 Introduction
Contrastive Language Image Pretraining (CLIP) (Radford al., 2021) has recently been proven to be powerful framework for multi-modal representation learning. By connecting the latent spaces of visual and textual inputs, CLIP offers convenient approach of open-vocabulary inference for various downstream tasks. Building upon CLIP, the prompt learning technique, which freezes the encoders and introduces learnable tokens the input side, can help CLIP adapt to downstream domains with minimal cost. Compared the traditional finetuning paradigm, prompt learning approaches can produce competitive results with much fewer learnable parameters (e.g., 0.1% of encoder parameters). This significant advantage in parameter efficiency has motivated numerous recent works to explore the potential applications of prompt- based CLIP in Federated Learning, where communication efficiency and training stability play important roles (Lu et al., 2023; Guo et al., 2023; Yang et al., 2023; Su et al., 2024; Li et al., 2024).
However, this work, we identify that prompt learning approaches are sensitive in processing domain information, with prompts learned from specific domain often struggling to transfer effectively new data with significantly different features. This issue has greatly limited the applications of prompt tuning in federated learning scenarios. As in federated learning, one of the primary objectives is to learn from

---

Published in Transactions on Machine Learning Research (11/2025)
heterogencous data, where this heterogeneity may manifest as domain gaps when feature distributions become significantly diverse. To thoroughly examine the potential of prompt tuning in federated learning, we consider a challenging yet realistic scenario in which we assume there are multiple participants intending to train a shared model with their local data originating from different domains. Following the previous practice (Peng et al., 2020), we formulate this scenario using domain-aware datasets like DomainNet (Peng et al., 2019), where there are labeled images sourced from six distinct domains with different styles like real- real-world, paining, and sketch. Due to large diversity in imput, conventional domain-agnostic federated learning approaches often struggle to generalize well in this scenario.
Under this challenging scenario, we find most parameter-cfficien federated learning methods pre- viously worked well in simpler heterogeneous settings with label-wise non-i.i.d fail to produce favorable results. This observation suggests that, federated learning tasks involving significant domain gaps cannot be addressed solely by collaborative training methods; instead, they require the careful incorporation of domain awareness into the model. To this end, we propose a new foderated learning framework termed ADAPT (Feder Ated Domain-A ware Prompt. Tuning), to enhance prompt tuning's ability to handle highly heteroge- neous data in federated learning by incorporating domain- knowledge. Specifically, ADAPT is built upon pretrained CLIP model and consists of three key components: 1) visual and 2) textual prompts are set up for each domain, where each visual prompt denotes a single learnable token appended to eateheembedded image tokens, and each text prompt consists of several learnable tokens representing textual descriptions in- dicative of each domain's 's style information; and 3) a vision-language coupling module done by AdaLN (Perez et al., 2018). We highlight that ADAPT effectively incorporates domain knowledge, achieving significantly superior predictive performance over traditional prompt tuning baselines in domain-aware federated learning tasks. Our contributions can be summarized as follows:
Domain-aware prompt learning. We assign domain-specifie textual prompts to each federated participant, enabling the model to make predictions by input images' corresponding domain infor- mation, which effectively addresses the widespread issue of domain gaps in federated learning. Our experimental results demonstrate a significant performance improvement with this design: based on a pretrained CLIP model equipped with a ViT-Base image encoder, ADAPT achieves average accuracy of 69.8% on DomainNet, which significantly outperforms the zero-shot CLIP's 53.6%, the basic prompt learning's 63.2%, and PedProx's (Li et al,, 2020b) 55.3% with a same image encoder.
Efficient communication. ADAPT requires training and sharing only a small fraction of pa- rameters, significantly reducing the communication overhead in federated learning. For instance, in our domain-aware federated learning experiments, ADAPT, with just 0.35M trainable pa- rameters, achieved state-of-the-art performance on the DomainNet (Peng et al., 2019), Office- Home (Venkateswara et al., 2017), and PACS (Li et al., 2017) datasets.
Superior privacy preservation. Due to the reduction in the number of trainable parameters in ADAPT, traditional federated learning attack algorithms (Zhu et al., 2019; Geiping et al., 2020) struggle to reconstruct the local data of participants from model gradients. Additionally, the learn- able prompts themselves do not leak customer privacy- we have attempted to decode our prompts but found it difficult to extract any interpretable information from them.
2 Related Work
Federated learning was first introduced in the Federated Averaging (FedAvg) paper (McMahan et al., 2017), addressing machine learning problems with massively distributed private data. To enhance the learn- ing potential of FedArg, FedProx (Li et al., 2020b) adds l2 regularization term into the FedAvg's objective, Following FedAvg's 's success, many follow-up works improve foderated learning in terms of privacy-presorving potentials (Wei et al., 2020; Truex et al., 2019), robustness to heterogeneous data (Karimireddy el al., 2020; Li et al., 2019), communicatinn efficiency (Konečný et al., 2016; Sattler et al., 2019), and compatibility to model architoctures (Li et al., 2020a; Qu et al., 2022). Unlike general federated learning methods that simulate non-i.idd data by partitioning datasets in the label space, many recent works consider federated learning in the more realistic context of domain adaptation (Yao et al., 2022a; Shenaj et al., 2023; Peng
2

---

Published in Transactions on Machine Learning Research (11/2025)
et al., 2020). Recently, based on advances in multi-modal contrastive learning (Radford et al. 2021), various works develop CI.IP-based federated learning methods (Lu et al., 2023; Tú et al., 2024; Su et al., 2024; Qiu et al., 2024). For example, FedCLIP (Lu et al., 2023) leverages a pre-trained CLIP model with an additional adaptor layer for federated training. FedOPT (Li et al., 2024) trains both global and local prompts, while FedAPT (Su et al., 2024) creates global prompts from local ones. FedTPG (Qiu et al., 2024) develops a global prompt generator that transforms class names into prompt vectors. Similarly, PromptFI. (Guo et al., 2023) employs simple prompt learning techniques to enhance federated optimization.
Vision-language models. Following the success of contrastive pre-training in visual modality (He et al., 2020; Chen ct al., 2020; Grill et al., 2020; Caron et al., 2021; Chen & He, 2021; Chen et al., 2021), multi- modal contrastive pre-training has become a common paradigm in recent years as well. A representative work is CLIP (Radford et al., 2021), which jointly pre-trains a visual and a textual encoder using an InfoNCE objective (Gutmann & Hyvärinen, 2010) with around 400 million curated image-text pairs. ALIGN (Jia et al., 2021) improves CLIP by scaling up the training datasot BASIC (Pham el al., 2021) further increases the scale of both data and model. As result, such CLIP-like models allow zero-shot inference when it comes to transfer learning downstream tasks.
1.8 billion noisy image-text pairs, and
Prompt tuning. While fine-tuning a pre-trained model for downstream machine learning tasks has tra- ditionally dominated the field of transfer learning, recent progress in prompt learning offers a compelling alternative. Specifically, the prompt tuning techniques fine-tune learnable prompt tokens attached to CLIP's inputs instead of training the entire model (Zhou et al., 2021; 2022; Wang et al., 2023; Yao et al., 2023). There also exist prompt tuning protocols for visual modality (Jia et al., 2022) and both visual and textual modalities (Yao et al., 2021; Zang et al., 2022). Similarly, there are adapter-based methods designed for CLIP-like models, which also freeze the encoders and only fine-tune several newly attached layers on top of them (Gao et al., 2021; Zhang et al., 2021).
3 Preliminaries
3.1 Contrastive Language-Image Pre-training
CLIP is a weakly supervised learning paradigm that combines visual and language encoders to solve image recognition problems. Formally, CLIP has an image encoder Fv: : R3xwxb - Rd where and h denotes the imput image's spatial resolution and d denotes the dimension of the latent space, and text encoder Fr: gxdd. + Rd where / is the length of input sentence and de is the dimension of word embedding. CLIP is trained by image-text pairs, in which the text briefly describes the information in the image. By encoding both image and text into the same latent space, CLIP can learn an alignment between visual and textual input with a contrastive loss (Gutmann & Hyvärinen, 2010). During inference, CLIP supports zero-shot classification by matching the visual representation of input image and the textual representation of target class names.
3.2 Prompt Tuning for Vision and Language
Despite CLIP's impressive zero-shot inference capabilities, there remains a noticeable accuracy gap in com- parison to in-domain fine-tuning, However, fine-tuning the CLIP model may easily break the well-established alignment between vision and language, and CLIP will therefore lose the ability of opea-vocabylary infer- ence. Instcad, prompt tuning attaches learnable tokens to the input, leaving the feature encoders fixed, which allows the model to retain its zero-shot and open-set inference abilities while significantly improving its in-domain accuracy.
Textual Prompt Tuning (TPT). As previously mentioned, CLIP's text query consists of hand-crafted prompt (also referred to as prefix) such as "A photo of a" and a class name such as "dog". TPT replaces the prefix by learnable vectors (Zhou et al., 2021). During training, both CLIP's vision and language encoders are frozen and only the prompt vectors are optimized.
Visual Prompt Tuning (VPT). The prompt tuning protocol also works for visual input if the image encoder is a transformer-like model such as the Vision Transformer (Dosovitskly al., 2021). Specifically,
3

---

Published in Transactions on Machine Learning Research (11/2025)
gradient O
client (local):
Ea
class name
momentum O
Text Encoder
Block 0-3
Text Eneoder Block 4-7
Text Encoder
Block 8-11
Light weight adapter
from client #
t2
Em
class
name
from client
AddUN
AdiN
AdhLN
t2
Gm
class
name
Object class- ification loss
gradient
visual prompt output
Image Encoder
(ViT)
class token's output
[cts]
gradient O: gradient update
momentum momentum update
parameters frozen
Local training framework. We load a pre-trained CLIP model and freeze both its image and text encoders. For each client, we feed the text encoder with text prompts followed by class names, where one is optimized by the gradients and the rest n - are loaded from other clients with momentum update. We feed the image encoder with n learnable prompt tokens followed by patch-wis embedded images, where the prompt tokens optimized by gradients.
Figure 1:
this method attaches trainable vectors to the patch-wise embedded image, and uses an additional head project the output. In VPT, only the prompt tokens and the head are optimized.
4 Methodology
Problem Formulation. Suppose there are clients that desire to deal with the same machine learning problem, e.g., image classification with the same target categories. The clients possess their own training data that originate from n distinct domains. We simulate this scenario using domain adaptation datasets like DomainNet (Peng et al., 2019), which encompass images from six different domains including clipart, information graph, painting, quickdraw, real- world images, and sketch. As the image feat exhibit significant variations across different domains, it is indeed challenging task for federated optimization. However, it is realistic scenario because many times, the data heterogeneity between clients arises from differences in feature distributions rather than label distributions. Notably, our setting is compatible with the task that clients have non-i.i.d labels. In our ablation study, we also further divide each domain into five splits with non-i.i.d. categories.
other words, each client stands for a specific domain.
4.1 Local training
Textual prompts. With CLIP, a very simple way to deal with domain shift is to use domain-aware prompt contexts for text queries. For example, in DomianNet, when we use prefix "a painting of a" for the painting domain, and use "a skotch of a" for the sketch domain, the predictions can be more accurate and robust. This idea also referred as domain-specific prompts (Ge al., 2022), while employing learnable text prompts further improve the predictive performance. Inspired by this observation, propose domain-specific prompts for CLIP's text encoder. Formally, define text prompt by sequence of learnable tokens:
Pr
t) [ ... [t],. Rmxd.
where the length of prompt and each token [t] E Rdd has the same dimension CLIP's word embedding.
4

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 1 illustrates ADAPT's local training framework and process. We initialize ADAPT by loading the same CLIP model for each client and freezing the parameters of both the image encoder Fv and the text encoder Fr. For our task, we have n text prompts Pt. PT..... PT corresponding to the n domains. During local training, the n text prompts are shared among the clients, yet the i-th prompt Pi- can only be trained by the i-th client. (we will detail this mechanism later). We separately leed the encoder Fr with all the n text prompts followed by a class name, leading to n representation vectors ft. ff..... ..p....,fff. f8, where
f = Fr Prr, [class name)).
(2)
Note that we assume each f stands for the representation of the class name in the i-th domain.
Visual prompts. We define visual prompts by n learnable tokens [u]1, [o]2,,..,, [v]n which also correspond to the n domains. During local training, we feed the visual encoder Fv (ViT architecture) with a class token [cls] (directly loaded from CLIP), n visual prompts, and the patch-wise wise embedded image, leading an image representation vector
fv = Fv(clss [o]1,
..., (")n, [image]]-
We oblain the final lextual representation through a cross attention layer. To minimize the number of parameters as much as possible, here we replace the query, key, and value projection matrices in the cross attention block with identity matrices, which we empirically find does not affect performance too much. Formally, denoting Iels as the query vector of the class token, and k; as the key vector of the i-th prompt token in Fv's 's last self-attention block, we have =[w1, W2, Wa] with
exp(< Acls K; TTa)
Wi Σ, expl< 9cls k; > /Ta)*
(4)
where Ta is a temperature coefficient. We regard each component wy as the visual feature's correlation to the i-th domain, and compute the final text output by
n
fT
wi w
(5)
During training, we optimize the model (actually learnable parameters only appear in prompts) using object classification loss, which is a cross-entropy function applied between fy and fr, and a domain correspondence loss which another cross entropy function applied between each pair of visual and lextual outputs. Here we explain how we optimize these parameters. We desire the i-th text prompt PA to represent the features of the i-th domain in the latent space of textual embeddings. However, the i-th client only possesses images from the i-th domain, so we cannot train Pj (j / i) yet. instead load them from other clients. We introduce visual prompts to detect the correlations between an input image and the n domains, so it is fine to optimize all of them. A detailed comparison of different training strategies can be found in our ablation study (see Table 5a and 5b for details).
Guiding textual output by visual clucs. To facilitate domain-speeific representations, we additionally introduce conditioning modules to guide textual outputs. The underlying idea of this strategy is that through the image encoder, we can effectively extract the domain information from the visual input; this domain information can help the text encoder to produce domain-speeific outputs so that to accurately match the visual representation. In this work, we follow the prior practice of diffusion models (Peebles & Xie, 2023) to perform conditioning by AdaLN (Perez et al., 2018) which is efficient in parameter count. To process the textual features after the CLIP's text encoder, we employ a light-weight adapter which comprises a self- attention layer. We concatenate the text fcatures with the output of visual prompts to further encourage domain-aware representations.
4.2 Parameters Aggregation
As mentioned above, for the i-th client, we optimize P by using gradients and load P/- (j f i) from other clients, so the aggregation of text prompts does not involve parameter merging processes (e.g. averaging).
5

---

Published in Transactions on Machine Learning Research (11/2025)
DomainNet
Method
# Com. cost
clipart
info g-
paint.
quick d.
real
sketch
Hero-Shot CLIP (Radford et al., 2021)
66.1
40.6
62.3
13.5
80.4
58.5
53.6
Single-Domain Domain Tuming
72.3
47.2
67.1
18.8
83.6
65.8
59.1
Conventional domain generalization methods:
MIROi (ResNet-50) (Cha al., 2022)
25.7M
40.6
54.3
34.5
75.0
61.5
PCLI (ResNet-50) (Yao al., 2022b)
25.6M
41.2
50.9
32.9
74.1
61.5
Fishrf (RosNet-50) (Rame al., 2022)
25.6M
39.8
54.5
53.6
33.7
73.9
59.7
52.5
Conventional federated learning methods:
FedAvg (ResNet-50 (McMahan et al., 2017)
25.6M
40.2
61.1
57.6
33.5
75.6
60.3
54.7
FedAvg [VET-B/16 (McMahan al., 2017)
85.8M
42.4
60.7
57.0
30.4
79.8
61.1
55.2
FedProx (ReaNet-50 (Li al., 2020b)
25.6M
41.5
62.0
56.8
34.9
79.2
62.6
56.2
FedProx (VTT-B/16) et al., 2020b)
85.8M
40.5
63.1
57.4
29.7
81.2
FodBNt (RcsNot-50) (Li et al., 2021)
45.4M
35.7
43.6
41.0
30.1
67.7
44.6
FPLt (Res (Huang al., 2023)
25.6M
40.5
49.2
56.9
35.0
70.2
60.7
52.1
Domoin-ognostzz vision-language tuning methods:
PromptFLL (Guo et al,, 2023)
<5M
76.0
50.2
70.4
33.5
81.2
67.8
63.2
FedCLIP (Lu al., 2023)
<5M
74.1
48.3
68.5
31.8
80.5
58.6
60.3
pFedPG (Yang al., 2023)
<5M
73.9
49.2
69.8
32.2
81.4
61.5
FedOPT (Li et al., 2024)
<5M
76.1
60.1
65.2
34.2
81.3
59.2
FedAPT (Su al., 2024)
76.3
49.8
69.2
35.7
81.5
68.2
FedTPG (Qiu et al., 2024)
21M
75.4
50.3
69.3
33.5
81.0
67.4
62.8
ADAPT (ours)
<5M
78.5
64.7
72.5
44.5
85.8
73.2
69.8
Table 1: Test accuracy (%) on DomainNet. The info 9., paint., and quick d. denote the domains of infogragh, painting, and quickdraw, respectively. Communication cost is measured by the number of learnable parameter shared at each round. t indicates reproduced results as experimental setup differs. Our results are marked in blue. The best results in each domain are bolded.
Suppose there is centralized parameter server although ADAPT also works for decentralized communi- cation and the clients upload their corresponding text prompt to it in each communication round. The server concatenates the n uploaded text prompts and sends to every client. For visual parameters, as all visual prompts are optimized by every client, we perform federated averaging in the server and then send the merged parameters to each client. Note that we do not need to share the CLIP encoders' parameters as each client initialized with the same CLIP model and its parameters are frozen during training.
This parameter aggregation paradigm works well for ADAPT, yet may create minor problem for the text encoder. Specifically, after each communication round, the external text prompts of the i-th client, i.e., P7 ( will be re-loaded. We observe that this sudden change of parameters often negatively affects our model. To address this issue, we propose to apply momentum update (also referred to as exponential moving average) to the external text prompts. Formally, we have
t =
[0===[]]"** a[t]--1 + - a)]]
(6)
where [t]*, [ denote the prompt tokens at the and step, and [t] denotes the vector received from other clients, and a E [0, 1] is coefficient to control the smoothness. The details of our ablation study related to momentum update are presented in Table 5a.
5 Experiments
Datasets and Basclines. We evaluate the proposed ADAPT and bascline methods on three domain adaptation image classification benchmarks: the DomainNet (Peng el al., 2019), OfficeHome (Venkateswara et al., 2017), and PACS (Li et al., 2017) datasets, presented in the Appendix. We first consider the baselines of CLIP and its adapted models to federated learning. The Zero-shot CLIP, which infers by aligning images class names with hand-crafted prompt, a direct baseline evaluate whether in-domain tuning is necessary for vision-language models in federated learning. We also introduce Single-domain tuning, which

---

Published in Transactions on Machine Learning Research (11/2025)
applies textual prompt tuning (Zhou et al., 2021) to CLIP only in the local domain, another baseline to testily whether it is helpful to combine the information across multiple domains. There are also domain- agnostic federated learning approaches based on CLIP such as PromptfL (Guo et al., 2023), pPedPG (Yang et al., 2023), FedAPT (Su et al., 2024), FedOPT (Li ct al., 2024), FedTPG (Qiu et al., 2024) and FedCLIP (Lu et al., 2023), which train lext prompt and an adapter layer in federated learning fashion, respectively. To further validate the effectiveness of ADAPT, comparisons were made with established federated learning algorithms such as Fed.An (McMahan al., 2017) and FedPror (Li et al., 2020b), as well as recent approaches like FedBN (Li et al., 2021) and FPL (Huang et al., 2023) that address domain shift challenges not based CLIP. Additionally, we compared traditional domain generalization methods including MIRO (Cha et al., 2022), PCL (Yao et al., 2022b), and Fishr (Rame et al., 2022), which also do not utilize CLIP. For these comparisons, we equipped the baselines with a 50-layer ResNet (He et al., 2016) and a base-scale Vision Transformer with a 16x16 patch size (Dosovitskiy et al., 2021), both pretrained on ImageNet-Il Ik (Deng et al., 2009).
Implementation details. We employ pre-trained CLIP model(Raddord et al., 2021) with ViT-Base/ '66 image encoder, so each textual and visual prompt token has the dimension of 512 and 768, respectively. We set the length of cach textual prompt sequence m = 16 for better robustness, which follows the practice of TPT (Zhou et al., 2021). By default, the number of clients is determined by the mumber of domains for each dataset, i.e. n = 6 for DomainNet and n = 4 for OfficeHome and PACS. We train both our model and the baseline models for 200 epochs and execute the aggregation or broadcast process after every one epoch, We train the ResNet- based models and prompt tokens by a SGD optimizer with 0.01 learning rate, 0.9 momentum, and 0.005 weight. decay. ADAPT instead uses AdamW (Toshchilov & Hutter, 2019) optimizer with 31 0.9, B2 0.999, 5e-4 lcarning rate, and 0.01 weight decay for transformer-basee models. We set the lemperature coefficient Ta 0.1 in Equation 4, and set the momentun update ratio o 0.99 in Equation 6. If not specified, all reported results are average numbers over three trials.
5.1 Main Results
Table shows that ADAPT significantly outperforms baseline methods on DomainNet, with notably high improvement in the 'quickdraw' domain at 44.5% accuracy. This underscores the effectiveness of our prompt learning approach, which requires fewer trainable parameters, enhancing robustness even with larger models. In contrast, traditional methods like FedAvg and FedProx show minimal or negative gains, especially when upgrading from ResNet-50 to ViT-Base. ADAPT also achieves higher average accuracy and lower standard deviation compared to domain-aguostic methods, demonstrating better resilience against domain shifts. We further evaluate the models on OfficeHome and PACS datasels, and the results are summarized in Table 2. The experiments on these benchmarks also support our conclusion of ADAPT's effectiveness by demonstrating higher average accuracy and lower deviation across domains. Specifically, we improve the zero-shot CLIP by 16.2% average accuracy and 0.82% standard deviation over four domains in OfficeHome. We also observe that overall, the prompt-based methods consistently outperform the conventional federated learning algorithms that require to train the entire model. This confirms the benefits of employing parameter- efficient approaches in federated learning, and validates our choice of using prompt tuning to address the domain shift issues.
Communication Costs. ADAPT markedly reduces communication overhead in federated learning by only transferring domain prompts, contrary to standard methods that share all trainable parameters. To provide clear comparison, we have included the following results in Table 1. An additional benefit of this approach is its ability to produce favorable results without requiring a substantial volume of training data. As shown in Table 8 (in the supplementary material), we obtain compelitive few-shot results by our prompt tuning technique. In practice, We avoid fine-tuning the CLIP model to maintain its lingal-language alignment. Fine-tuning large models such as CLIP escalates communication expenses and impodes the rate of convergence. With an equivalent number of training iterations, the fine-tuning protocol often falls short to prompt learning.
7

---

Published in Transactions on Machine Learning Research (11/2025)
OfficeHome
PACS
Method
Ar
Cl
Pr
Rw
Avg.
P
A
S
Avg.
Zero-Shot CLIP
79.5
63.1
85.3
86.5
78.6
99.8
96.9
98.8
87.7
95.8
Single-Domain
80.0
65.2
87.5
86.9
79.9
99.8
97.2
99.1
88.9
96.3
Conventional domain generalization methods:
MIROI (ResNet-50)
64.2
48.9
77.3
72.3
65.7
90.5
53.3
78.3
77.2
74.8
PCL1 (ResNet-50)
61.8
45.2
75.9
73.6
64.1
89.4
52.7
77.0
75.9
73.8
Conventional federated learning methods:
FedAvg (ResNet-50)
66.3
49.4
77.1
77.9
67.7
89.6
52.5
78.6
76.1
74.2
FedAvg (VIT-B/16)
67.9
49.6
77.5
81.0
69.0
91.3
54.8
79.2
77.9
75.8
FedProx (ResNel-50)
68.8
50.5
78.6
80.3
69.6
91.7
57.0
81.8
80.2
77.7
FedProx (VTT-B/16)
70.4
51.3
80.3
82.4
71.1
92.0
59.4
83.5
81.6
79.1
FedBN (ResNel-50)
71.2
51.6
81.5
83.6
72.0
92.5
63.8
85.2
84.3
81.5
FPL (Huang et al., 2023)
63.7
54.6
76.5
75.4
67.6
90.2
51.3
86.4
84.7
78.2
GA (Zhang et al., 2023)
67.4
52.9
78.3
78.5
69.3
91.1
52.4
99.1
78.2
80.2
Domain-agnostic assion-language tuning methods: PromptFL (Guo et al., 2023)
79.8
65.6
89.5
89.1
81.0
99.9
97.1
99.0
90.6
96.7
FedCLIP (Tu et al., 2023)
79.1
65.0
88.6
88.4
80.3
99.8
97.4
98.9
89.0
96.3
Ours
83.1
69.6
90.5
90.4
83.4
99.9
98.3
99.2
91.7
97.3
Table 2: Test accuracy (%) on OfficeHome and PACS. Domains include art, clipart, product, and real- world for OfficeHome, and photo, art painting, cartoon, and sketch for PACS. Our results are marked in blue The best results are bolded.
Method
Fed.
VPT
TPT
domain
AdaLN
aCC.
Zero-Shot CLIP
X
X
X
53.6
Single-Domain
59.1
Visual Only
X
54.2
Textual Only
/
X
X
63.2
Domsin-Agnosti
/
/
X
X
63.5
Prompt-only
68.4
ADAPT
69.8
Table 3: Ablation study to model components. We report the average accuracy (%) over six domains in DomainNet. VPT and TPT denote whether using visual or textual prompts.
5.2 Ablation Studies
We first dissect the ADAPT model to ablate its performance gains. ADAPT comprises two primary com- ponents: visual prompts and domain-specific text prompts. By dissecting these components, we get three more variants of our method: 1) Visual Only, which leverages learnable prompt tokens for only image imput and uses CL.IP's hand-crafted prompt for texts. 2) Terfual Only, which discards the visual prompt tokens of ADAPT and uses learnable text prompts only. Note that in the absence of visual prompts, we cannot get the weight w; (see Equations and for each domain, the text prompts from external clients should also be discarded. We instead aggregate the textual prompts by federated averaging (MeMahan et al., 2017). 3) Domain-Agmostic,, which retains both ADAPT's visual and textual prompts but decouples them, i.c., we do not perform the weighted sum process in Equation 5, which can be considered as a simple combination of the modes Textual Only and Visual Only-
We summarize the results Table 3. Since we introduce visual prompt tuning to combine domain information rather than enhancing the visual feature extraction abilities, do attach an additional head for the image encoder in (Jia et al., 2022). Therefore, the Visual Only mode cannot yield significant performance
8

---

Published in Transactions on Machine Learning Research (11/2025)
Input images
Reconstructed images (FedAvg)
Reconstructed images (ours)
Figure 2: Examples of reconstructed images produced by Deep Leakage from Gradient (Zhu et al., 2019). It shows that the gradient attack algorithm cannot reconstruct semantically meaningful information from ADAPT as only share few parameters.
improvements. We also observe that tuning textual prompts results in a 5.5% increase in accuracy, and when tuning them in federated learning fashion, we achieve an additional 4.1% improvement (Teztual Only). Notably, compared to the simple visual-and-textuue prompt tuning with 63.5% accuracy, ADAPT achieves much higher result of 69.8%, demonstrating the crucial significance of our domain-aware design.
Momentum update, prompt length, and communication frequency. We consider three more factors that may affect results. As mentioned in Section 4, update the external text prompts by exponential moving average to prevent parameters' sudden change. Table 5a presents comparisons regarding the update mechanism for text prompts, where the accuracy drops by 2.6% in the absence of momentum update. If we train all text prompt tokens in every client, i.c., we disregard the relationship between text prompts and domains, the accuracy drops by 4.7% as makes ADAPT a domsin-agnostie approach.
By default, we aggregate the visual prompt tokens by federated averaging, as separately training each token in specific domain does not yield better performance (sce Table 5b). As shown in Table 5e, set cach textual prompt length to we further increase the length, the model tends to overfit. and accuracy drops. Notably, the visual and textual prompt lengths are consistent. In Table 5d we also assess the impact of communication frequency by varying it to 0.5, 1, and 2 training epochs per communication round. It shows that compared to our default setup of one epoch per communication round, more frequent aggregation (0.5 epoch/ /round) does not lead to improved performance, while conversely, infrequent communication (2 epochs/round) results in a 0.7% accuracy degradation.
16, as it works more robust than shorter prompt (m = 4), and when
AdaLN component. We evaluated the effect of varying the number of AdaLN layers in model. Beyond the default three layers, tested configurations with an AdaLN layer after every two blocks in the vision encoder (six layers total) and after each block (12 layers total). As shown in Figure 5f, incrcasing from three to 12 AdaLN layers resulted in only a 0.2% accuracy improvement, which is minimal considering the additional network parameters. Thus, we retained the default three-layer configuration. Replacing the AdaLN layers with cross-attention mechanisms improved performance by 0.7% compared to the model without AdaLN. Using AdaLN layers provided an additional 0.7% improvement over the cross-attention approach, as ralustrated in Figure 5c.
9

---

Published in Transactions on Machine Learning Research (11/2025)
#
clipart
infograph
painting
quickdraw
real
sketch
fe
N/A
N/A
kd
2
N/A
#
dng
with
3
1h
bh some
?
N/A
N/A
4
and
N/A
lh
N/A
the
pjf
Table Nearest Words of uextual prompts learned by ADAPT in DomainNet dataset. N/A means non-Latin characters. shows that our prompts tend capture high-level and abstract semantics that are difficult summarize using standard natural language words found in the dictionary.
mode
mode
mode
acc.
w/ mtm.
69.8
average
69.8
w/ AlaLN
69.8
w/o
67.2
split w/ mtm.
69.6
w/o AlaLN
train all
65.1
split w/o mtm.
68.5
cross attention
69.1
(a) Text prompt update.
(b) Visual prompt update.
Visual conditioning.
#eps/round
acc.
#tokens
acc.
numbers
acc.
0.5
69.8
4
67.8
68.4
69.8
9
68.5
3
69.8
16
69.8
68.4
32
68.3
12
70.0
(d) Comm. frequency
(e) Prompt length.
(1) AlaLN amount.
learn. rate
acc.
weight: decay
acc.
epochs
acc.
5 x 10-5
66.9
0.001
69.0
100
64.1
x 10-4
67.2
0.005
69.5
200
69.8
5 x 10
69.8
0.01
69.8
300
70.2
1 x 10-3
69.1
0.05
69.4
400
70.4
x 10-3
67.4
0.1
68.8
600
70.3
(g) Tearning rate.
(h) Weight decay.
(1) Training epochs.
We report the average accuracy over six domains DomainNet. The mtm. denotes momentum update. Our default sctup is marked in blue. The best results of cach ablation study is bolded.
Table 5: Ablation studies.
Privacy Preservation. In ADAPT, there are two potential ways to expose participants' private data. First, similar to most federated learning algorithms, ADAPT shares gradient information across all participants, so some private information might be able to be reconstructed by gradient-based attacking algorithms such as Deep Leakage from Gradient (DLG) (Zhu et al., 2019). However, as ADAPT has relatively small number of learnable parameters, such attacks cannot extract sufficient information from gradients reconstruct participants' local data, which gives ADAPT significant privacy advantage over traditional federated learning algorithms. Figure presents examples of DLG applied to our model and FedAvg. While ADAPT shows significant information leakage, attacks FedAvg reveal extensive details from the original image.
Another potential way to expose privacy is decoding the trained text prompts, which might contain some statistical information of participants. However, our experiments showcase that this is difficult as well. we follow CoOp (Zhou et al., 2021) to decode each text prompt by finding a standard vocabulary word with minimum Euclidean distance to it in the embedding space, and summarize the interpretation results for DomainNet in Table 4. It shows that our prompts tend capture some high-level and abstract semantics that are difficult to be summarized to standard natural words.
10

---

Published in Transactions on Machine Learning Research (11/2025)
6 Conclusion
This work introduces ADAPT, a novel federated learning approach explicitly designed to address the key challenges of domain shift and communication efficiency. Our method strategically combines CLIP and prompt learning techniques for both visual and textual inputs, thereby enhancing parameter-effiiencc and minimizing communication costs, while maintaining robustness in federated optimization involving hetero- gencous data. Furthermore, we address the pervasive issue of domain shift across clients by introducing domain-specific prompts and facilitating correlations between visual and textual representations through self-attention mechanisms. These innovations result in domain-aware federated learning methodology that consistently demonstrates outstanding effectiveness. Notably, our experiments reveal & remarkable achieve- ment-an an average accuracy of 69.8% across six domains in the DomainNet dataset, marking an impressive 16.2% improvement over the original CLIP model. In comparisons with traditional federated learning meth- ods like FedAvg and FedProx, as well as cxisting domain-agnostic CLIP-based approaches such as PromptFL. and FedCLIP, our ADAPT consistently outperforms them across three benchmark scenarios.
References
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Junbum Cha, Kyungjae Lee, Sungrac Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In ECCV, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Feii ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Pong Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Belter vision-language models with feature adaplers. arxiv preprial ar xrxiv:2110. 200:2110.00. 10.:22100.4444,, 2021.
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adap- tation via prompt learning. arXio preprint ar Xiw:2200 06687, 2022.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients- how easy is to break privacy in fodcrated learning? NeurIPS, 2020.
Yean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In NewrIPS, 2020.
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated participants coopcratively learn prompts instcad of models-foderated Icarning in age of foundation model. IEEE Transactions on Mobile Compuling, 2023.
Michael Gutmann and Aapo Hyvärinen. Noiso-contrastive estimation: A new estimation principle for un- normalized statistical models. JML.R, 2010.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast. for unsupervised visual representation learning. In CVPR, 2020.
Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In CVPR, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy lext supervision. In ICML, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020.
Jakub Konečný, Brendan McMahan, Felix
Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:160000544 2016.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain gener- alization. In ICCV, 2017.
Hongxia Li, Wei Huang, Jingya Wang, and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In CVPR, 2024.
Qinbin Li, Zeyi Wen, and Bingsheng He. Practical federated gradient boosting decision trees. In AAAI, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2020b.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang, On the convergence of fedavg on non-iid data. arXiv preprint arXio:1907.02189, 2019.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning non-iid features via local batch normalization. ICLR, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
Wang Lu, Xixu Hu, Jindong Wang, and Xing Xie. Fedelip: Fast generalization and personalization for clip in federated learning. arXiv preprint 2xXiv22302.34483 2023.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-eficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, 2017.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaplation. In ICCV, 2019.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaplation. 2020.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for open-vocabulary image classification. arXiv preprint arX:::2111.10050, 2021.
Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Lin. Federated lext-driven prompt generation for vision-language models. In ICLR, 2024.
Lianggiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feilei Wang, Ehsan Adeli, Li Fei-Fei, and Daniel Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. In CVPR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Alexandre Rame, Corentin Dancette, and Matthieu Cord, Fishr: Invariant gradient variances for out-of- distribution generalization. In ICML, pp. 18347-18377, 2022.
Felix Sattler, Simon Wiedemann, Klaus- Robert Müller, and Wojeiech Samek. Robust and commmication- efficient federated learning from non-iid data. IEEE transactions on neurul networks and learning systems, 2019.
Donald Shenaj, Fros Fani, Marco Toldo, Debora Caldarola, Antonio Tavera, Umberto Michieli, Marco Ciccone, Pietro Zanuttigh, and Barbara Caputo. Learning across domains and devices: Style-driven source-free domain adaptation in clustered federated learning. In Proceedings of the IEEE/CVE Winter Conference Applications of Computer Vision (WACV), pp. 444-454 Jamuary 2023.
Shangchao Su, Mingzhao Yang, Bin Li, and Xiangyang Xue. Federated adaptive prompt tuning for multi- domain collaborative learning. In AAAI, 2024.
Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and Zhou. A hybrid approach to privacy-pres preaervin federated learning. In Proceedings of the 12th ACM workshop on artificial intelligence and security, 2019.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alexander G Schwing, and Heng Ji. Learning to decompose visual features with latent textual prompts. ICLR, 2023.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 2020.
Fu-En Yang, Chien-Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in federated learning via client-specific prompt generation. In Proceedings of the FEF//CVF International Conference on Computer Vision, pp. 19159 19168, 2023.
Chun-Han Yao, Boqing Gong, Hang Qi, Yin Cui, Yukun Zhu, and Ming- Yang. Federated multi-target domain adaptation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1424-1433 2022a.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guide guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6757 6767, 2023.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In CVPR, 2022b.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. orXiv preprinl arXiv:2109. 11 11797, 2021.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXin:2200. 0.07225, 2022.
Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training free clip-adapter for better vision-language modeling, orXiv preprint arXiv:2111.03990, 2021.
Ruipeng Zhang, Qinwei Xu, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang, Federated domain generalization with generalization adjustment. In CVPR, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint or orxi::209 09.000990110000 134, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In CVPR, 2022.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. NewrIPS, 2019.
14


Reviews:
[{"review_id": "pmcy19HID7", "reviewer": "Reviewer_X3s3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a way to finetune CLIP on domain-specific tasks via training a per-domain prompt in a federated way. When using non-i.i.d. data from different domains, the performance of traditional federated learning algorithms suffers, with even methods designed specifically for the heterogeneous setting showing much worse performance than Zero-shot CLIP (e.g. FedProx vs Zero-shot CLIP in Table 1). The paper answers this concern by developing ADAPT, a method that learns per-domain textual and visual prompts and where the text-vision components are coupled via AdaLN (which is just an affine transformation connection). Assuming that each client $i$ has data from domain $i$, ADAPT learns the prompts for domain $i$ using data from that client and shares it with the other clients for inference. Finally, the authors also introduce a momentum update to guard against performance drops from switching to using other clients' prompts after each update.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The algorithm is quite simple, is shown to have good performance (Table 1) in the paper, and is explained well. \n2. My main complaint is that the setting is quite narrow: we have $n$ clients with $n$ different domains neatly divided up, and I cannot really think of a realistic setting where this would be the case. \n3. Data reconstruction is pretty difficult because there are so few parameters learned per client anyway.", "audience_yn": "Yes", "audience_explanation": "I believe the paper is relevant to the literature on learning prompts for CLIP. I'm not really sure if it has wide applicability due to the limitations described above, but it could be useful down the line for the development of more sophisticated algorithms.", "requested_changes": "Typos:\n- \"paining\" in p.1\n\nClaims:\n- \"However, fine-tuning the CLIP model may easily break the well-established alignment between vision and language, and CLIP will therefore lose the ability of open-vocabulary inference\" can you provide a citation for this phenomenon?\n\nOther:\n- Will you share the code for this paper for reproducibility?", "broader_impact_concerns": "N/A", "additional_comments": "1. How would you modify your method to handle the case where the client-domain correspondence isn't perfect? e.g. let's say client 1 had data from domains 1,2 and client 2 had data from domains 2,3.\n2. How about if each client had only one domain but it could be repeated? e.g. clients 2, 3 each have data from domain 2. Would you average the prompts? Would you use SGD with both?", "extra_scores": {}, "extra_text": {}}, {"review_id": "aZgkjYBWyG", "reviewer": "Reviewer_Vcy8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies an important problem of prompt tuning and proposes a novel federated method. The method is sound and the paper is well-written. Experiments show the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments on the baseline comparison, ablation study, and case studies show the effectiveness of the propsoed solutions.", "audience_yn": "Yes", "audience_explanation": "This paper focuses on the prompt tuning, which is attractive to existing communities.", "requested_changes": "1. It would be better to include more experiments regarding the parameter sensitity, which will provide more guidance for others about the parameter configuration.\n2. It would also be better to provide more experiments in terms of efficiency, e.g., training time, test time, and the number of parameters.\n3. It is suggested to discuss the future directions in the conclusion.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hSaElVv3AH", "reviewer": "Reviewer_RceZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a CLIP-based federated prompt-tuning technique called ADAPT for vision-language models. \n\nThey introduce domain-specific tuning of textual prompts, making their approach domain-aware. In addition to learnable textual prompts, they also allow for visual prompts to be learned and combine the representations using attention mechanisms. Since it is a prompt-tuning technique, the authors state that it has the advantage of parameter efficiency and lower communication cost compared to other fine-tuning techniques.\n\nIn their experiments, they empirically demonstrate improved performance over federated baselines and other domain-agnostic tuning methods on three different datasets. Furthermore, they provide a comprehensive ablation study of the different components of their technique. Finally, they provide empirical evidence suggesting that clients private data cannot be easily recovered when using their technique in this federated setting.\n\n\n### Strengths:\n\nS.1. Proposing an efficient domain-aware tuning method for federated learning is an interesting direction.\n\nS.2. The empirical results are comprehensive and show improved performance when using the proposed technique. The authors compare their technique with various baselines and other approaches across $3$ different domains.\n\nS.3. The authors include an extensive ablation study, showcasing the importance of its component and different design choices with respect to final performance.\n\n\n### Weaknesses:\n\nW.1. Given the various CLIP-based federated techniques already presented in related work, I would have expected a more direct comparison highlighting how the proposed method differs from them. This comparison will help clarify the contributions of this paper.\n\nW.2. My main concern is the scalability of the technique to a larger number of clients in federated settings. In the experiments, the authors assume one client per domain, which allows for only $6$ clients in the DomainNet dataset. However, in real-world federated scenarios, the number of clients is typically much larger. It would be helpful if the authors could discuss that.\n\nW.3. I believe the claim of reduced communication cost is slightly overemphasized, since a similar communication cost is shared across other CLIP-based tuning methods.\n\nW.4. It is not very clear to me whether prompt-tuning provides any advantage over other parameter-efficient tuning techniques, such as LoRA. For example, would ADAPT be comparable to LoRA-based finetuning in a federated setting?  [1]\n\nW.5. The paper lacks a discussion of potential disadvantages of domain-specific tuning in federated learning. It would be beneficial to add such a discussion.\n\n\nReferences\n\n[1] Kuo et al., Federated LoRA with Sparse Communication, 2024.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most of the claims the authors make are supported by the empirical evidence. The results in the experiments and ablation section are convincing. See Strengths above.\n\nSee W.3 in the Weaknesses list. \n\nW.3. I believe the claim of reduced communication cost is slightly overemphasized, since a similar communication cost is shared across other CLIP-based tuning methods.", "audience_yn": "Yes", "audience_explanation": "I believe that proposing an efficient domain-aware tuning method for federated learning is an interesting direction to certain individuals in TMLR's audience. However, it would be helpful if the authors could address any weaknesses listed above.", "requested_changes": "I would like to ask the authors to address and clarify the aforementioned weaknesses (W1-W5). I believe addressing those will strengthen the paper.\n\nMoreover, I have a few additional minor questions and suggestions:\n\n- Regarding the implementation, why are different optimizers used? SGD optimizer for ResNet-based models and AdamW optimizer for the proposed technique.\n- In Section 5.1, the authors mention that ADAPT achieves lower standard deviation compared to other methods, demonstrating better resilience in domain shifts. However, the standard deviation across domains is not reported.\n- As a formatting recommendation, I suggest reordering Table 5 with Figure 2 and Table 4. Based on paper’s structure, it makes more sense to move Table 5 to page 9 and Figure 2 as well as Table 4 to page 10.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WBt2HlAeZW", "reviewer": "Reviewer_3GXs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes ADAPT (Federated Domain-Aware Prompt Tuning), a method that applies domain-specific prompt learning to CLIP models in federated learning scenarios. The approach addresses domain shift issues by using separate visual and textual prompts for each domain, with a cross-attention mechanism to match images to appropriate domain-specific text representations.\n\n**strengths**\n\n1. addresses realistic federated learning scenario with domain heterogeneity\n2. only 0.08M trainable parameters\n3. Consistent improvements across three datasets (DomainNet, OfficeHome, PACS)\n4. Comprehensive ablation studies and privacy analysis\n\n**weaknesses**\n\n1. only combines existing techniques (CLIP, prompt tuning, federated learning)\n2. assumes one client per domain, limiting real-world applicability\n3. Incomplete comparison with recent federated domain adaptation methods", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental validation contains proper baselines, ablation studies, and consistent results across datasets, e.g. the 14.8% improvement over zero-shot CLIP on DomainNet. Privacy claims are supported by gradient reconstruction experiments. However, the domain-client correspondence assumption limits the generalizability of findings.", "audience_yn": "Yes", "audience_explanation": "The federated learning community would find this work relevant, particularly for applications with domain heterogeneity. The parameter efficiency results are practically important for communication-constrained scenarios. The combination of vision-language models with federated learning addresses a timely intersection of active research areas.", "requested_changes": "**critical**\n\nExpand evaluation to scenarios where multiple clients share domains (not one-to-one mapping)\n\nCompare against recent federated domain adaptation methods beyond basic FedAvg/FedProx\n\nProvide theoretical analysis or bounds on the approach's convergence properties\n\n**non-critical**\n\nDiscuss computational overhead during inference (cross-attention mechanism)\n\nEvaluate on larger-scale datasets beyond the current benchmarks\n\nAdd analysis of failure cases or method limitations\n\nInclude comparison with domain-agnostic prompt learning in centralized setting", "broader_impact_concerns": "No significant ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)