Federated Text-driven Prompt Generation for Vision-Language Models
Authors: Chen Qiu
Xingyu Li
Chaithanya Kumar Mummadi
Madan Ravi Ganesh
Zhenzhen Li
Lu Peng
Wan-Yi Lin
Venue: ICLR 2024 poster (2024)
Keywords: Vision-Language Models
Prompt Learning
Federated Learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Prompt learning for vision-language models, e.g., CoOp, has shown great success in adapting CLIP to different downstream tasks, making it a promising solution for federated learning due to computational reasons. Existing prompt learning techniques replace hand-crafted text prompts with learned vectors that offer improvements on seen classes, but struggle to generalize to unseen classes. Our work addresses this challenge by proposing Federated Text-driven Prompt Generation (FedTPG), which learns a unified prompt generation network across multiple remote clients in a scalable manner. The prompt generation network is conditioned on task-related text input, thus is context-aware, making it suitable to generalize for both seen and unseen classes. Our comprehensive empirical evaluations on nine diverse image classification datasets show that our method is superior to existing federated prompt learning methods, achieving better overall generalization on both seen and unseen classes, as well as datasets.

Published as a conference paper at ICLR 2024
FEDERATED TEXT-DRIVEN PROMPT GENERATION FOR VISION-LANGUAGE MODELS
Chen Qiu' Bosch Center for AI, USA
Xingyu Li Tulane University
Chaithanya Kumar Mummadi & Madan Ravi Ganesh & Zhenzhen Li Bosch Center for AI, USA
Lu Peng Tulane University
Wan-Yi Lin Bosch Center for AI, USA
ABSTRACT
Prompt learning for vision-languagg models, e.g., CoOp, has shown great success in adapting CLIP to different downstream tasks, making it promising solution for federated learning due to computational reasons. Existing prompt learning techniques replace hand-crafted text prompts with learned vectors that offer im- provements on seen classes, but struggle to generalize to unseen classes. Our work addresses this challenge by proposing Federated Text-driven Prompt Generation (FedTPG), which learns a unified prompt generation network across multiple re- mote clients scalable manner. The prompt generation network conditioned on task-related text input, thus is context-aware, making it suitable to generalize for both seen and unseen classes. Our comprehensive empirical evaluations on nine diverse image classification datasets show that our method is superior to ex- isting federated prompt learning methods, achieving better overall generalization on both seen and unseen classes, as well as datasets.
INTRODUCTION
Vision-language models (VLMs) have recently emerged transformative technology for machine learning applications. Seminal contributions like Contrastive Language-Iman Pretraining (CLIP) Radford et al. (202I) have demonstrated unprecedented capabilities in diverse image classification tasks. One often leverages manually-engineere text prompts, such as "a photo of a [class]," to utilize CLIPP's rich semantic features (Jia al. 202T). CLIP has shown its robustness and versatility in handling wide range of image distributions. These properties make CLIP naturally aligned with the objective of Federated Learning (FL), a decentralized approach to train machine learning models with data privacy. However, high computational and communication costs associated with server-client interaction make the training of CLIP impractical the FL setting. This motivates us to explore more efficient and effective methods to adapt the advantages of CLIP in FL.
Emerging prompt learning methodologies based on CLIP such as Context Optimization (CoOp) have revealed that fine-tuning CL.IP can be made more efficient by substituting hand-crafted prompts with learnable soft prompt vectors in few-shot learning paradigm (Perez et al. (02T) for one down- stream task in centralized learning (Zhou et al. 2022b a: Zhu et al. [022] Yao et al. (223) Existing federated prompt learning method, Federated Context Optimization (FedCoOp) (Guo et al. 2023b), adapts the learning paradigm of CoOp FL by learning unified set of prompt vectors across mul- tiple clients with different datasets. FedCoOp improves over CLIP on the seen (during training) classes in each client, but it struggles generalize on the unseen classes (not included in training). Similarly, prompt vectors optimized on seen classification tasks fail to generalize to new tasks of
"Equal contribution. Correspondence to: Chen. QiaquQuuuee Qiu@us.buu tiuu@u.bbssh.com Work done during internship Bosch Center for AT, USA.

---

Published as a conference paper at ICLR 2024
coop
Seen (Tratining Time)
Text-driven Prosst Ceneration (ours)
Smnagg
caceset
Plane Ard Berrel
Pasege
Prodiction
Presietion
Broet
Fryor
ambestins
Parm
Hvgllssi
Uads
Prorpt peneration
Eypge Faelass
Prodidion
Pred ctton
Unseon Tasks (Interence Time)
Roce
Class mames
Jasndra
Suntiower
in cataset
Our proposed prompt generator generates prompt vectors conditioning on the target clas- sification task-related text input. Leveraging contextual awareness, the generated prompt vectors enrich CLIP with context information in the text input and can generalize to unseen classes.
Figure I:
different contexts (c.g., from object recognition to texture classification). Unless otherwise noted, we refer to "task" as an image classification dataset within the context of this work.
Instead of learning one unified set of prompt vectors for different classification tasks, we propose to convert text input containing task-specific semantic information to context-aware prompt vectors. Benefiting from context information in text input, we aim to generate prompt vectors that generalize well to classification tasks that have not been previously observed (refer Figure [ for an illustration of the concept). Following that, we propose Federated Text-driven Prompt Generation (FedTPG), which learns a lightweight unified prompt generator across multiple clients collaboratively. Each client optimizes the prompt generator locally for its classification task described by few-shot image- text pairs, followed by the FL server-client communication obtain the global prompt generator model. An overview of our FedTPG with two remote clients is shown in Figure 2 on various image classification tasks, our prompt generator learns generate prompt vectors condi- tioned on context-related text inputs. Leveraging contextual awareness, the generated prompt vectors differentiate themselves across various tasks and enrich CLIP with context information of the target task. Our comprehensive evaluation on nine diverse image classification datasets demonstrate that FedTPG has improved generalization over the existing prompt learning method FedCoOp on unseen classes by 4.32% and unseen datasets by 1.82%, on average.
By training
We summarize the contributions of our work as follows: (1) We develop a text-driven prompt gener- ation (TPG) technique to improve the generalization performance from observed classification tasks to new classification tasks with different contexts. Instead of learning fixed prompt vectors, the prompt generator converts tesk-related text input to context-aware prompt vectors for various image classification tasks. (2) We propose FedTPG, a scalable way of learning a unified, generalized text- driven prompt generator across multiple clients with various classification tasks collaboratively. (3) We undertake exhaustive empirical analysis using nine datasets to validate the efficacy of FedTPG. Our comparative studies with existing federated prompt learning methods demonstrate FedTPG's superior generalization performance on image classification tasks encompassing a range of domains.
2 RELATED WORK
Visual-Language Model Prompt Learning. Prompt learning, a variation of fine-tuning VLMs, has shown considerable promise in enhancing the task-specific performance of existing pre-trained models under few-shot settings. A significant advancement in this direction was CoOp (Zhou et al. 2022b), which introduced the notion of optimizing continual prompt context vectors for better task adaptation. CoCoOp (Zhou et al.] 2022a) generates prompt vectors conditioning on images with neural network. Several other works have also explored the interplay between textual prompts and visual inputs (Zang et al. 2022; Li et al.) 2023b Xing et al.) 2022). Specifically, MaPLe (Khattak) ctt al. 2023) extends the prompt learning paradigm by integrating both visual and textual information for more robust task adaptation. [ulltt & Tzimiropoulos y: Yao et al. (2023) explore text-to- text optimization to encourage natural language-awar aware soft prompting in VLMs. Chen et al. (2022) 2023b) extend CoOp by learning multiple prompts to describe different characteristics of category-
Concurrent with our work, Wang et al. (2023b): Udandarao et al. (2023) propose to utilize auxiliary images to improve the model performance unseen tasks. Different than learning one model for
2

---

Published as a conference paper at ICLR 2024
Proept
Preest - (0. 1.
(9.0.W, M6)
Contral Server
Class
Cless name tokens:
tokens:
Queey
Query
Q.E
65
Roart 2
Wi
Retantio ion
Ietention
W2
Text enbedding
Wy
Text exbedsing
72 R
%5 R
R
7
Proept vectons Folans
Proept vectors [class]
Frest
Couise Virilelly
Emnarr ER
Foraag
Caeate sirileit
EQB
Backword
Contrastion losu
Figure 2: Our proposed FedTPG learns a unified prompt generator over the frozen CLIP model for converting task-related text input TT to context-aware prompt vectors. The prompt generator is learned across multiple clients with different classification datasets collaboratively.
task in all prior works, we focus learning one unified model that generalizes well across broad spectrum of tasks and domains with task-related text inputs.
Federated Learning with Visual-Language Models. Federated Learning (FL.) (MeMahan al., 2017) has emerged as pivotal paradigm for decentralized training of machine learning models on heterogencous data (Li et al. 2023a). Recently, fine-tuning of VLMs has been extended to the federated setting to reduce the computational burden on a single FediLIP (Lu et al., 2023) proposes an extension of standard fine-tuning CLIP to the FL setting to enable strong performance and personalization. Halbe et al. (2023) provides continual lifelong prompt learning mechanism to mitigate the effect of client drift. Wang al. (2023a) further showcase the corrective attribute of prompts in the snapshot compressive imaging application domain while Chen et al.) (2023a) high- light the adaptability of federated prompt-based methods to diverse data landscapes beyond visual and textual data for weather forecasting. Of relevance to approach is PromptFL (Guo et al. 2023b which proposes a FL framework for prompt learning that enables participants to coopera- tively learn common prompt vector. [Su et al. (0022) who delve into the cross-domaie applicability of federated prompt learning in VLMs, and Guo et al. (2023a) who combine a federated prompt learning scheme with personalized spacial visual features. A key distinction between these meth- ods and our approach is our use of a learnable text-conditioned prompt generator which improves generalization performance on both seen and unseen tasks, typically unexplored setting for VLMs under the FL scheme. Concurrent with our work. (ang et al. (2023) propose prompt generator with a cross- attenttooo mechanism similar to our approach. In addition to their focus on using frozen VIT backend, we hypothesize that their dependence on fixed client-specific features learned from seen clients would limit their generalization to unseen tasks. comparison, our prompt generation depends text inputs and has no hurdles in generalizing to unseen tasks.
METHOD
In this section, we present our problem setup of FL in Section 3.1 followed by our text-driven prompt generation technique Section 3.2 and finally propose our FedTPG algorithm that deploys text-driven prompt generation in FL in Section 3.3
3.1 PROBLEM SETUP
We consider a federated network setting with one central server for model aggregation and multiple remote clients, where each client has a private classification dataset with labeled images (z, y) D, from n4 classes with class name tokens (Cij) ) (a sample setting with two remote clients is depicted Figure 2 Data distribution across the federated network follows a non-IID setup where clients contain samples from disjoint set of classes. The goal of FL framework in our setup is
For the sake of presentation, name PromptFL. FedCoOp, as PromptFL. adapts CoOp to the FL. setting.
3

---

Published as a conference paper at ICLR 2024
to jointly learn one model that not only solves different image classification tasks spanning across multiple remote clients but also attains generalization ability to unseen classes and datasets. In contrast to the setting in FL literature (Kairouz et al., 202T), our consideration of generalization to unseen classes and datasets makes the setup more challenging. Following the recent success of VLMs like CLIP across broad range of tasks (Radford et al. 202T), we look into the adaptation of CLIP models in our FL framework to achieve our goal of generalization.
CLIP is large VLM with an image encoder Eimage and a text encoder Etezt, and can classify images utilizing linguistic knowledge. In our FL setup, we consider each client to have access to an off-the-shellf pretrained CLIP model. We focus on adapting the pretrained CLIP model collabora- tively across all clients. However, updating large models like CLIP requires extensive computational power and bandwidth, making it impractical for Fl. applicaatssns. Recently, prompt learning has been used to offer computation and communication efficient FI. framework e.g.. FedCoOp (Guo et al.) 2023b) for adapting a frozen CLIPP across multiple clients. Specifically, hand-crafted text prompts (e.g., "a photo of [class]") for Etent are replaced with trainable prompt vectors 11, 02; ...,... u.. while keeping CLIP weights unaltered. In federated prompt learning. lightweight trainable prompt vectors shared across clients at each communication round and updated with local training on client data.
In this work, our goal is to learn a FL prompt model that can solve various image classification tasks across multiple clients and also generalize to novel classes image classification tasks from new clients, which can be challenging to existing methods like FedCoOp. Zhou et al. (2022a) have shown that CoOp's 's prompt vectors, optimized for observed classes, fail to generalize to novel classes. We notice a similar generalization issue in FedCoOp i.e., learned unified prompt vectors perform well on the seen classification tasks across remote clients, but fail to generalize to tasks with different contexts (e.g.. from object recognition to texture classification). We attribute this behavior to the fixed nature of soft prompts and not being able to adjust to the context of the task. To address this, we propose a novel strategy that alters how the soft prompt vectors are obtained. Instead of directly learning the soft prompts, we learn a text-driven prompt generation module that takes task-related text input and transforms it into context-aware prompt vectors, which we detail in the next section.
3.2 TEXT-DRIVEN PROMPT GENERATION
We develop a prompt generation module fe that generates context-aware prompt vectors conditioned on the target classification task-related text inputs, as shown in Figure The text input is translated to text embeddings T and the prompt generator So converts these text emheddings TT to a set of m-length input prompt vectors P 1mxdd for Etezt as:
P Pm = [e(
(1)
Context-related text input can be obtained from the natural language description. We find that avail- able candidate class names naturally represent context-related text for the classification task ([class 0]. [class 1], [class n)). We translate the natural language class names to text embeddings as T = Ereet (c))}²=
Raaxd set of embeddings of n class name tokens ej from CLIP text
encode Besides, prompt generator So is lightweight cross-attention module comprising of learn- able parameters di Q E Rixdd WK E Raxd Wv E Rdddd Given the text embeddings T we have:
(T( ho (CrsssAttentio((() (Q, K7, V7)) with KT=T WK. VT = T Wv. (2)
The prompt generator transforms context information from the text embeddings T into key and value vectors KT and VT respectively. Cross-attention layer merges these vectors with the learnable query vector Q, and hidden layers hep projects cross cross-attention attention layer output to prompt vectors P.
Prompt vector for each class is defined as t, P U {cy}, concatenating generated context prompt vectors P and text token of class name cj. Given an input image x and prompt vectors for all n candidate classes, the prediction probability of CLIP for classification task is computed as follows:
exp(cos(EEmmage (x)) (x), Erewt brewe((t))) (t,))/T) polu j|æ,, T) Σ exp(cos( Prrmage (I); Freert (1+))/T)
Text embeddings T produced from a peetrpret text encoder like CLIP provides rich and mean- ingful context information for given text. The prompt generator fo should serve extract and
For simplicity, we consider single client with index 1, and remove the client's index in notations.
4

---

Published as a conference paper at ICLR 2024
Algorithm 1: FedTPG Algorithm
Input: No. of communication rounds R. No. of local epochs K, initialization parameters 0". Server executes:
Initialize prompt generator fo parameters with 8
for r 4- R do
Pick a random subset of remote clients as S
for i 8 in parallel do Send the current global model 0" to client i.
Receive locally updated 0 from Local Client Training.
end
Aggregate the updated model parameters g*+1 187 Eies 0.+1
end
Obtain the final model parameter 0
Local Client Training:
Obtain the set class name embeddings T. = (tend(Crs))))
for k + 0 to K do
Generate the context prompt vectors P. = So: (Ti).
Get the prompt vectors for each class f/,5 Pi Ci.j 1-
Update parameters 8" to 0,+11 locally using eqs.
to on (z, y) ~ Dj.
end
transfer context-critical information from the already meaningful embeddings T to prompt vectors P. Training fo different classification tasks from diverse contexts would facilitate its conver- gence to produce generalized context-aware prompt vectors, and thus improve prediction precision of (o = j|, T) on unseen classes. In practical scenarios, the data encompassing wide range of classification tasks is typically distributed across different clients. Addressing this, we next present a scalable way of learning the prompt generator across multiple clients collaboratively.
3.3 LOCAL TRAINING AND SERVER AGGREGATION
We incorporate prompt generation module in FL settings, where multiple remote clients han- dling diverse image classification tasks train the prompt generator fo collaboratively. We refer this approach Federated Text-driven Prompt Generation (FedTPG). We outline the training pipeline of our FedTPG in Algorithm [
Initially, the server initializes fo parameters randomly with 6 and then at each communication round, random subset of remote clients S" retrieve the up-to-date So parameters for local training. Below we describe the training steps of FedTPG at each round r:
Step I: Remote client in S receives current up-to-date parameters 6" configure the local fer.
Step II: At each client, the frozen CLIP text encoder Etent provides text embeddings of the local available class name tokens T, {Eress (ci.))) 1 The prompt generator før, the frozen CLIP model, the context text embeddings Tr, and the dataset D, together define the local objective
: = =-E(Z.,)DD log Pe- (yla, Te),
(4)
where per (2) , Tx) is defined in eq. 33 By utilizing an optimizer, e.g. SGD, we can estimate the unbiased gradient of (,(;; Ts) with respect to or and get the updated parameters 0 +1 after K iterations with learning rate 17" as:
g-+1 SGDx (1"" 8", "T,T,, L.)
(5)
Step III: After local few-shot training, all the remote clients in S" send back their locally updated prompt generator 0, +1 the server for aggregation: 0r+1
T Eiesr +1
After performing Step I-III for R communication rounds, FedTPG obtains the final model parame- ters f We argue that the proposed FedTPG achieve the generalization goal from two aspects: (1) unlike existing prompt learning techniques that directly learn fixed prompt vector, our TPG method captures richer contextual and semantic information for each local classification task; (2)
5

---

Published as a conference paper at ICLR 2024
Potth hom
ForPi NOAAA Zond eff how
3030/0 huabudyy
(a) CoOp and FedCoOp's prompts
(b) FedTPG's prompts
(c) Zero-shot FedTPG's prompts
Figure 3: 3D visualization (after PCA) soft prompt vectors. (a) CoOp learns diverged prompt vec- tors on each dataset individually, while FedCoOp learns one unified set of prompt vectors for tasks with various contexts (b) FedTPG's prompt generator learned on bases classes generates context- aware prompt vectors for each task. (c) FedTPG's 's prompt generator learned on ImageNet generates context- ware prompt vectors for nine unseen datasets aligned with the generated vectors in (b).
through the FL collaboration framework, diverse contextual and semantic information across multi- ple remote clients with different tasks benefit the model learning well. Multiple clients encode text embeddings based on their distinct tasks, enabling the global model to serve variety of contexts without overfitting to a specific task. Overall, the federated model can potentially learn a richer set of semantic features, and facilities better "transfer learning" capabilities, enabling the model to generalize well to both seen and new unseen tasks (that includes both classes and datasets).
4 EXPERIMENTS
We evaluate the proposed method FedTPG mainly on two benchmarks: (1) generalization unseen related classes in Section 4.1 (2) generalization unseen datasets in Section 4.2 We also provide
ablation studies to evaluate FedTPG's robustness under various settings in Section 4.3
Baselines. We compare FedTPG with (i) CLIP with hand-crafted text prompt template, c.g., "a photo of a [class]": (ii) CoOp (Zhou et al.) 2022b) with learnable prompt vectors replacing hand-crafted text prompts. CoOp is trained on each client individually to provide a baseline of local training. (iii) FedCoOp (Guo al., 2023b), FL variant of CoOp. The unified prompt vectors learned across multiple clients with federated averaging. (iv) FedKgCoOp. a FL variant of KgCoOp Yao al. 2023). KgCoOp improves over CoOp on generalization performance by adding reg- ularization of minimizing the discrepancy between learned prompts and the hand-crafted prompts. (v) FedCoCoOp, FL variant of CoCoOp [Zhou al., [202aa]. CoCoOp generates prompt vectors conditioning on images with neural network. (vi) FedMaple, FL variant of Maple (Khattak et al.) 2023). Maple learns prompt vectors for both vision and encoders. We develop all FL variants of existing prompt learning approaches with FedAvg (McMahan et al.) (017)). For all FL methods, one unified model learned across clients is used for the evaluation of different datasets.
Implementation Details. All methods are built
frozen CLIP with VIT-B/16 backbone. FedTPG learns unified prompt generator parameterized by a four-head cross-attention layer with layer norm and MLP (ha) consisting of two linear layers with ReL.U. The dimension of vectors Q. Kr, VT in the cross-attention layer, and linear layers in hi is 512. The length m of prompt vectors is 4, and the dimension d is 512. Please refer to Appendix A and Appendix B for more details.
4.1 GENERALIZATION TO SEEN AND UNSEEN CLASSES
Datasets. We employ nine image datasets including Caltech101 (Fei-Fei al. 2004), OxfordPets (Parkhi et al. alla|22.111 20122) StanfordCars (Krause et al. 2013), Flowers 102 (Nilsback & Zisserman 2008)
Food101 (Bossard et al. 2014), FGVCAireraft (Maji al. 2013), SUN397 (Xiao al. 2010 UCF101 (Soomro et al. 2012). and DTD (Cimpoi et 2014).
Experimental setup. We split the classes of each dataset equally into two groups, classes and the other as new classes. Images from base classes are available for training, while the images from classes used for evaluating the generalization performance. We perform all
base

---

Published as a conference paper at ICLR 2024
Table Accuracies (%) on clients' local tasks (seen), base (seen) classes, and new (unseen) classes. FedTPG achieves the superior generalization performance over existing prompt learning methods and their FL variants, and the highest harmonic mean (HM) of three benchmark results.
(a) Average 9 datasets.
(b) Caltech101.
Local
Base
New
HM
Local
Base
New
HM
CLIP
76.72
70.52
75.78
74.24
CLIP
97.57
96.97
93.89
96.12
CoOp
83.67
71.49
71.15
75.01
CoOp
97.79
94.02
93.1
94.93
OpdCoOp
81.75
74.50
71.70
75.75
FedCoOp
96.97
96.69
92.79
95.45
PedKgCoOp
78.38
72.18
75.87
75.39
FedKgCoOp
97.65
97.24
94.79
96.54
FedCoCoOp
81.41
73.87
65.98
73.21
FedCoCoOp
96.71
94.41
91.59
94.19
FedMaple
81.63
74.44
70.59
75.28
FedMaple
97.00
95.41
90.06
94.06
FedTPG
80.75
73.68
76.02
76.70
FedTPG
97.59
97.08
95.24
96.62
(c) 100
(d) FGVCAireraft.
Local
Base
New
HM
Local
Base
New
HM
CLIP
82.58
72.18
77.94
77.33
CLIP
30.59
27.55
35.81
30.96
CoOp
97.27
69.37
71.95
77.73
CoOp
36.88
28.30
28.59
30.79
FedCoOp
94.44
76.40
70.12
79.16
FedCoOp
36.29
32.41
30.95
33.07
FedKgCoOO
84.59
72.11
77.06
77.59
FedKgCoOp
33.68
29.79
34.01
32.37
FedCoCoOp
94.00
77.49
65.63
77.31
FedCoCoOp
35.21
31.93
22.67
28.89
FedMaple
94.28
76.44
68.51
78.35
FedMaple
35.83
31.39
32.34
33.08
FedTPG
90.76
71.80
77.76
79.35
FedTPG
34.68
30.82
35.18
33.44
(c) UCF101.
(f) OxfordPets.
Local
Base
New
HM
Local
Base
New
HM
CLIP
80.75
70.58
77.5
76.04
CLIP
91.33
91.33
97.04
93.16
CoOp
88.37
69.62
68.09
74.32
CoOp
94.89
94.89
96.60
95.46
FedCoOp
86.13
75.65
70.60
76.93
FedCoOp
93.31
93.32
95.39
94.00
FpdKgCoOp
82,66
73.14
76.36
77.19
FedKgCoOp
91.58
91.58
96.53
93.17
FedCoCoOp
84.92
75.23
64.25
73.83
FedCoCoOp
92.34
92.34
87.36
90.61
FedMaple
84.17
75.12
68.68
75.46
FedMaple
95.00
95.00
97.09
95.68
FedTPG
85.64
74.89
76.64
78.79
FedTPG
94.70
94.69
95.79
95.06
(g) Food101.
(h) DTD.
Local
Base
New
HM
Local
Base
New
НM
CLIP
94.39
90.16
91.25
91.90
CLIP
53.13
53.01
58.21
54.68
CoOp
93.98
88.20
88.
90.05
CoOp
72.34
72.34
54.99
65,46
FedCoOp
93.52
88.63
88.47
90.15
FedCoOp
68.67
68.67
52.74
62.39
FedKgCoOp
94.19
89.94
91.81
91.95
FedKgCoOp
58.76
58.75
59.61
59.04
FedCoCoOp
93.24
87.57
84.95
88.45
FedCoCoOp
68.63
68.63
45.77
58.83
FedMaple
93.95
89.43
89.60
90.94
FedMaple
68.28
68.28
46.61
59.11
FedTPG
94.09
89.87
91.64
91.83
FedTPG
63.62
63.62
60.51
62.55
(i) StanfordCars.
(i) SUN397.
Local
Base
New
HM
Local
Base
New
HM
CLIP
71.51
63.44
74.9
69.61
CLIP
88.66
69.41
75.46
77.05
CoOp
78.65
61.34
70.17
69.33
CoOp
92.83
65.29
68.62
73.78
FedCoOp
74.53
66.16
72.32
70.82
FedCoOp
91.93
72.34
71.89
77.70
FedKgCoOp
71.89
64.33
75.71
70.32
FedKgCoOp
90.38
72.72
76.94
79.34
FedCoCoOp
76.62
66.51
66.40
69.52
FedCoCoOp
91.44
69.76
65.36
73.94
FedMaple
74.76
66.26
71.33
70.60
FedMaple
91.40
72.66
71.33
77.47
FedTPG
74.54
66.34
74.26
71.50
FedTPG
91.11
74.01
77.13
80.10
methods under a non-lID FL setting, where the base classes of nine datasets are distributed multiple clients. Each client owns 20 completely disjoint classes where each class has eight labeled images for few-shot training. We report the classification accuracies on clients' cal classification tasks, on the base classes (combining classes from multiple clients), the new

---

Published as a conference paper at ICLR 2024
Table 2:
Accuracies (%) on ImageNet (seen) and domain-shi ImageNet variants (unseen).
FedTPG consistently outperforms other baselines on both source dataset and domain-shifed datsets.
ImageNet
ImageNetV2
ImageNct-S
ImageNet-A
ImageNet-R
Average
CLIP
66.75
60.79
46.
47.79
74.00
57.18
FedCoOp
67.80
61.59
45.61
48.78
74.49
57.62
FedKgCoOp
67.53
61.60
46.69
48.37
74.71
57.84
FedCoCoOp
68.51
62.29
46.90
50.33
76.49
59.00
FedMaple
66.96
60.65
44.69
46.24
74.62
56.55
FedTPG
69.51
62.90
47.65
49.97
76.35
59.22
Table 3: Accuracies (%) on source (seen) and target (unseen) datasets. FedTPG consistently outper- forms other federated prompt learning methods both source dataset and unseen target datsets.
Source
Target
FGVCAmcaat
ImageNet
Caltechtor
Flowers
UCFI01
OxfondPetss
Food101
DTD
Stanford
SUN397
EuraSAT
Average
CLIP
66.75
92.90
71.29
24.72
66.75
89.15
86,09
44.33
65.29
62.59
47.68
65.08
FedCoOp
67.80
91.87
68.13
21.44
64.13
88.70
85.85
42.43
63.59
62.77
43.26
63.22
FedKgCooOp
67.53
93.63
69.31
23.06
64.46
88.55
85.37
44.74
64.99
63.85
43.29
63.13
FedCoCoOp
68.51
94.11
66.34
20.79
62.75
89.04
85. 40
43.20
63.98
64.02
55.40
64.50
FedMaple
92.49
68.25
23.52
83.52
44.68
61.85
45.38
62.98
FedTPG
69.51
93.75
70.04
23.22
64.72
90.60
85.91
46.25
63.98
66.78
47.86
65.31
classes in Table [ We report the harmonic mean (HM) of these three accuracies showing the overall performance. All results averaged over three independent runs.
Quantitative results. As shown Table a). the proposed FedTPG achieves the best average accuracy new classes, showing its advanced generalization ability. FedTPG also achieves the highest harmonic mean which averages the accuracies on clients' local tasks, base classes, and new classes. Although the prompt generator is trained on local tasks consisting of few classes, it generalizes well to a more complex classification task one the base classes (combining classes from multiple clients), and novel classification task on the unseen classes. Due the non-III setting, CoOp outperforms the FL methods on the corresponding local task but fails to generalize to other base classes and new classes. Benefiting from learning across multiple clients, FedCoOp. FedCoCoOp and FedMalpe improve lot base classes, however, have degraded performance on new classes, highlighting the generalization challenge in federated prompt learning. FedOgCoOp has an improved accuracy on new classes with cost of performance degradation on base classes and local tasks, resulting from the difficulties of balancing the CLIP loss and the regularization term.
Qualitative analysis. We visualize the prompt vectors learned by CoOp each dataset individually and the unified prompt vectors learned by FedCoOp in Figure 3 (a), and the prompt vectors gener- that CoOp learns different optimal prompt vectors each dataset. However, the unified prompt vectors learned by FedCoOp are not flexible enough to fit the context of all different datasets. In comparison, FedTPG learns generate task-specific prompt vectors conditioning on the context-related text input. From Figure 3 (b) we can see the prompt vectors generated on clients (stars) sharing data from the same dataset are automatically clustered together, showcasing that the prompt generator learns to extract context information from the text input. Also, although the model is not trained on base-class classification and new-class classifi cation, their associated generated prompt vectors (triangle for base, square for new) are clustered based on the dataset context accordingly, explaining FedTPG's strong generalization ability.
ated by FedTPG in Figure ( (b). We
4.2 GENERALIZATION TO UNSEEN DATASETS
Datasets. For evaluating the generalization to unseen datasets, we train all models on ImageNet, and test the model on two benchmarks: (1) four variants of ImageNet containing various types of ImageNet-A, and ImageNet-R, (2) ten domain shifting: including ImageNetV2, ImageNet-Skette unseen datasets including nine datasets used in Table I and FuroSAT (Helber et al. 2019).
8

---

Published as a conference paper at ICLR 2024
Table 4: Ablation study: three trials where each client owns n FedTPG consistentl achieves the highest harmonic mean (HM).
{5, 10, 20) disjoint classes.
n=5
n= n=10
n=20
Base
New
HM
Base
New
HM
Base
New
HM
FcdCoOp
69.53
70.05
69.69
72.15
70.61
71.37
74.50
71.70
73.07
FedKgCoOp
70.83
75.55
73.11
71.18
75.81
73.42
72.18
75.87
73.98
FedTPG
71.08
75.51
73.23
72.15
75.84
73.95
73.68
76.02
74.83
76.5
76.5
Hammonic mean (%)
75.0
Harnonic mean
73.5
75.5
FedTPG
72.0
FedCoOp
75.0
FedCoOp
FedkgCoOp
FedKgCoOp
70.5
CLIP
74.5
100
Number of shots
Participation rate (%6
(a) Number shots
(b) Participation rate clients
Figure 4: (a) FedTPG gets improved when increasing the number of shots (for training), and has the best results when using more than one shot. (b) FedTPG robust to the participation rate of clients.
Experimental setup. We consider non-lID setting with 200 clients. Each client owns n = disjoint classes. At each communication round, random 10% clients contribute the model update. We also consider few-shot setting, where eight labeled images are available in each class. We only consider FL baselines this setting. results are averaged over three independent runs.
Results. Our proposed FedTPG improves over other FL. prompt methods ImageNet and other variants of ImageNet consistently as shown Table 2 On the more challenging unseen datasets, while all the existing methods sacrifice their performance, FedTPG avoids the overfitting problem as the compared FL prompt methods, outperforming CLIP as shown Table 3 Although the prompt generator is trained images and class names from ImageNet, the model learns generalizable function mapping the context-related text embeddings T to task-specific prompt vectors as visual- ized in Figure improving the classification accuracy datasets with totally different context, e.g.. from object recognition to texture classification. We can see that the prompt vectors generated by FedTPG trained on ImageNet aligned with the prompt vectors generated by FedTPP trained on these nine datasets, which demonstrates its tross-dataset transfcrability.
4.3 ABLATION STUDIES
Size of clients: To understand the impact of the number of classes owned by the client, we conduct three trials where each client
{5. 10, 20) disjoint classes, and number of shots is 8. As shown in Table FedTPG outperforms FL baselines in all cases in terms of the harmonic mean.
Number of shots:In Figure a). show FedTPG consistently improves
FedCoOp with vary- ing number of shots, and outperforms FedKgCoOp, when the number of shots is larger than one.
Participation rate of clients: In Figure bb) show that FedTPG consistently outperforms Fed- CoOp and FedkgCoOp, when the participation rate of clients varies from 10% to 100%.
CONCLUSION
This paper addresses the challenge of generalization in adapting CLIP to the FL setting, We pro- pose a novel Federated Text-driven Prompt Generation (FedTPG) algorithm, which learns a unified prompt generator across multiple clients. The prompt generator converts task-related text inputs to context-aware prompt vectors. Our comprehensive experiments demonstrate FedTPG's superior generalization performance, outperforming existing FL prompt methods by decent margins.

---

Published as a conference paper at ICLR 2024
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-minins discriminative compo- nents with random forests. In Computer -ccionEECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446-461. Springer, 2014.
Adrian Bulat and Georgios Tzimiropoulos. Lasp: Text-to-text optimization for language- aware soft prompting vision & language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
June 2023.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022.
Shengchao Chen, Guodong Long, Tao Shen, Tianyi Zhou, and Jing Jiang. Spatial-temporal prompt learning for federated weather forecasting, arXiv preprint arXiv:2305. 14244, 2023a.
Ziliang Chen, Xin Huang, Quanlong Guan, Liang Lin, and Weiqi Luo. A retrospect to multi-promp learning across vision and language. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22 22190-22201, 2023b.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Tao Guo, Song Guo, and Junxiao Wang. pfedprompt: Learning personalized prompt for vision- language models in federated learning. In Proceedings of the ACM Web Conference 2023, pp- 1364-1372 2023a.
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing, 2023b.
Shaunak Halbe, James Seale Smith, Junjiao Tian, and Zsolt Kira. Hepco: Data-free heterogeneous prompt consolidation for continual federated learning. arXiv preprint arXiv:2306.099: 099300000999.. 2023.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-223) 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and languag representation learning with noisy text supervision. In International conference machine learning, pp. 4904 4916. PMLR, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Ad- vances and open problems in federated learning, Foundations and Trends@ in Machine Learning, 14(1-2): 2):1-210, 2021.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19113-19122, June 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference computer vision work- shops, pp. 554-561, 2013.
Xingyu Li, Zhe Qu Bo Tang, and Zhuo Lu. Fedlga: Toward learning via local gradient approximation. IEEE Transactions Cybernetics, 2023a.
of federated
10

---

Published as a conference paper at ICLR 2024
Yaowei Li, Ruijie Quan, Linchao Zhu, and Yi Yang. Efficient multimodal fusion via interactive prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 16604-2611, 2023b.
Wang Lu, Xixu Hu, Jindong Wang, and Xing Xie. Fedelip: Fast generalization and personalization for clip in federated learning. arXiv preprint urXiv:2302.13448 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Brendan McMahan, Eider Moore, Daniel Ramage. Seth Hampson. and Blaise Aguera y Arcas. coammuiccaioon-effciiee learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Maria- Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp- 722-729. IEFE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference computer vision and pallern recognition, pp. 3498-3505. IEEE, 2012.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:1 1004-0 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763, PMLR, URL https: http:es:///roocdddngg. mlr wwwppppsssssseeeeeeeeep radford21a..
18 24 Jul 2021. html
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12.204002 2012.
Shangchao Su, Mingzhao Yang, Bin Li, and Xiangyang Xue. Cross-domain federated adaptive prompt tuning for clip, arXiv preprint arXiv:2211.07885
2022.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-free name-only transfer of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2725-2736. 2023.
Jiamian Wang, Zongliang Wu, Yulun Zhang, Xin Yuan, Tao Lin, and Zhiqiang Tao. Cooperative hardware-prompt learning for snapshot compressive imaging. arXiv preprint arXiv:2306.01 176, 2023a.
Zhengbo Wang, Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5032-3042, 2023b.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485- 3492. IEEE, 2010.
Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang- Class-aware visual prompt tuning for vision-language pre-trained model. arXiv preprint arXiv:2208.08344 08340, 2022.
Fu-En Yang, Chien-Y Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in fed- crated learning via client-specific prompt generation. arXiv preprint arXiv:2308. 2023.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-languee prompt tuning with knowledge- guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6757-6767, June 2023.

---

Published as a conference paper at ICLR 2024
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning, arXiv preprint arXiv:2210.07225, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision (ICV), 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligne gradient for prompt tuning. arXiv preprint arXiv:2205. 14865, 2022.
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENT SETUP DETAILS
A.1 DATASET AND HYPER-PARAMETER DETAILS
We follow the settings in Zhou et al. 2022b) to conduct the experiments in this paper with the nine to unseen classes, and four variants of ImageNet
classification datasets for generalization datasets for domain shifting, where the statistical details are presented in Table
For each compared FL approach and each classification task, via grid search, the learning rate of the SGD optimizer was set to 17 = 0.003 with a decay rate 1c-5 and a momentum of 0.9. The local SGD training step is
K 1. The number of communication rounds is 500. The batch size is 200. By default, all the experimental results in the paper are obtained by averaging from three runs with different random seeds.
A.2 FEDERATED LEARNING SETUP DETAILS
Experimental Setup for Seen and Unseen Classes in Table [
To evaluate the generalization ability for the proposed FedTPG and compared FL approaches from in the paper, monitor the model performance on the following three benchmark accuracies: (1) The local classification accu- racy, representing the performance of local clients' classification tasks local available classes; (2) The base classification accuracy. representing the performance against all seen classes (combining classes from multiple clients) in a dataset in the FL network; (3) The new classification accuracy, which indicates the performance unseen classes but within the domain of seen classes. We report the harmonic mean (HM) these three accuracies on each classification task, as shown in Table]
In the FL. data partition process for T T we first split the classes of the considered 9 classification datasets equally into two groups D and D' denotes seen and unseen groups respectively. Then we split the classes within D the 30 remote clients, where each remote client has 20 classes in each local dataset D, For each class, the number of image-texi paired data shots is set to 8. During the FL training process, the participation rate of remote clients is set 100% and the communication round is set to 500.
To evaluate the generaliza- tion ability of FedTPG unseen datasets during training. consider the following two settings: (1) Domain Shifting, where we monitor the performance of model by training with ImageNet and testing four variants of ImageNet, including ImageNetV2, ImageNet-SSetchh, ImageNet-A, and ImageNet-R; (2) Unseen Datasets, where we evaluate the performance trained model in (1) on nine unseen datasets, including Caltech101, OxfordPets, StanfordCars, Flowers 102, Food101, FGV- CAircraft, SUN397, UCF101, and DTD. During the training process, we set the FL network with 200 remote clients where each client has n 5 classes of 8-shots training data disjointly. The par- ticipation rate of remote clients is set to 10% that |S*| = 20 and the global communication round is set to = 500 to obtain RR
Experimental Setup for Unseen Datasets in Table and Table 3
Experimental Setup for Ablation Study in Table and Figure We study the impact of the number of classes owned by each client at Table 4 from the introduced local, base and new classifi- cation accuracies with the same setup in Table | where full client participation performed with R = 500 and number of shots is 8. Specifically, we perform the data partition with the disjoint rule during class splitting: when = 5, we set the number of clients to 119; when n = 10, we set the number of clients to 59; and when n 20, we set the number of clients to 20, respectively.
The study of the number of shots is shown Figure 4 (b, where we set the number clients to 30 with 20 and the client participation rate is 100% in each round where R 500. The study of the participation rate is shown in Figure (b), where we set the number of clients to 30 with n = 20 and the number of shots is 8.
Then, we monitor the impact of the FL client participation rate in each communication round as shown in Figure a). We formulate the FL network with 30 clients where 20 and the number of shots is 8. Four client participation rates in {10% 40%, 70%, 100)% are considered during the model training process with R = 500.
13

---

Published as a conference paper at ICLR 2024
Table Dataset statistical details on class, training and test splits, prompt template.
Dataset
Classes
Train
Test
Hand-crafted prompt template
ImageNet
1000
1.28M
50,000
A photo of a [class]
1altech101
101
4,128
2,465
A photo of a [class]
Flowers102
102
4,093
2,463
photo of a [class], type of flower
FGVCAircraft
100
3,334
3,333
A photo of [class], a type of aircraft
UCF101
101
7,639
3,783
A photo of a person doing [class]
OxfordPets
37
2,944
3,369
A photo of a [class] type of pet
Food101
101
50,500
30,300
A photo of [class] type of food
DTD
47
2,820
1,692
A photo [class] a type texture
StanfordCars
196
6,509
8,041
A photo a [class]
SUN397
397
15,880
19,850
A photo of a [class]
EuroSAT
10
13500
8,100
A centered satellite photo of [class]
ImageNetV2
1000
N/A
10,000
A photo of a [class]
ImageNet-Sketch
1000
N/A
50,889
A photo of [class]
ImageNet-A
200
N/A
7500
A photo a [class]
ImageNet-R
200
N/A
30,000
A photo a [class]
B ADDITIONAL RESULTS
Size clients. Table and Table 7 show the detailed results of FedTPG and the compared FL baselines the benchmark of seen and unseen classes with 5 and 10, respectively. The results of G Table (are the detailed results of Table in the main paper, where we would like claim that the HM results in the main paper are the harmonic mean of the base accuracy and the new accuracy, while the results in Table 6 and Table 7 are the harmonic mean of the local accuracy, the base accuracy and the new accuracy that leads the difference in some columns. The results show that similar to the results of n 20 in Table [ the proposed FedTPG achieves the best average accuracy on unseen classes, and achieves the best new performance for 3 tasks while the second best new performance for most of the other tasks. We can also observe that as n increases, the advantage of FedTPG against other approaches becomes more significant. This supports our theoretical claim that the unified prompt generator in FedTPG generalizes better unobserved classification tasks, especially for challenging scenarios.
14

---

Published as a conference paper at ICLR 2024
Table 6: Accuracies (%) on clients' local tasks (seen), base (seen) classes, and new (unscen) classes. Each client has labeled images from five disjoint classes. The number shot is and n 5.
(a) Average over 9 datasets.
(b) Caltech
Local
Base
New
HM
Local
Base
New
HM
CLIP
86.25
70.52
75.78
76.98
CLIP
97.40
96.97
93.89
96.06
FedCoOp
89.38
69.53
70.05
74.74
FedCoOp
97.19
93.67
92.14
94.28
FedKgCoOp
86.63
70.83
75.55
77.12
FedKgCoOp
97.95
96.57
94.21
96.22
FedTPG
87.78
71.08
75.51
77.51
FedTPG
97.31
94.00
94.43
95.22
(c) Flowers102.
(d)
FGVCAireraft.
Local
Base
New
HM
Local
Basc
New
HM
CLIP
91.12
72.18
77.94
79.66
CLIP
49.04
27.55
35.81
35,45
FedCoOp
97.89
70.65
74.47
79.37
FedCoOp
55.82
25.45
26.57
31.63
FedKgCoOp
89.96
70.27
76.51
78.09
FedKgCoOO
51.98
28.89
33.75
35.93
FedTPG
94.20
70.23
76.77
79.20
FedTPG
53.62
26.38
33.92
34.87
(c) UCF101.
(f) OxfordPets.
Local
Basc
New
HM
Local
Basc
New
HM
CLIP
88.78
70.58
77.50
78.25
CLIP
96.75
91.33
97.04
94.96
FedCoOp
90.71
69.75
65.33
73.77
FedCoOp
98.08
91.92
94.57
94.79
FodKgCoOp
87.68
70.06
76.14
77.29
FedOgCoOp
96.65
91.34
96.16
94.66
FedTPG
88.53
71.20
75.96
77.91
FedTPG
97.96
91.39
96.03
95.04
(g) 1o2.s102.
(h) DTD.
Local
Basc
New
HM
Local
Base
New
HM
CLIP
97.57
90.16
91.25
92.88
CLIP
79.55
53.01
58.21
61.71
FedCoOp
97.17
88.27
86.67
90.48
FedCoOp
86.94
54.40
51.45
60.83
FedKgCoOp
97.42
89.59
91.52
92.72
FedKgCoOp
80.50
55.47
60.26
63.77
FedTPG
97.34
89.24
91.31
92.51
FedTPG
82.72
60.19
61.53
66.73
(i) StanfordCars.
SUN397.
Local
Base
New
HM
Local
Base
New
HM
CLIP
83.06
63.44
74.90
72.90
CLIP
93.02
69.41
75.46
78.10
FedCoOp
86,06
64.84
71.77
73.22
FedCoOp
94.55
66,83
67.44
74.32
FedKgCoOp
83.42
63.84
75.85
73.46
FedKgCoOp
94.12
71.45
75.52
79.23
FedTPG
83.75
63.92
72.35
72.45
FedTPG
94.56
73.17
77.24
80.67
15

---

Published as a conference paper at ICLR 2024
Table 7: Accuracies (%) on clients' local tasks (seen), base (seen) classes, and new (unscen) classes. Each client has labeled images from ten disjoint classes. The number shot is and n = 10.
(a) Average over 9 datasets.
(b) 10ltech
Local
Base
New
HM
Local
Base
New
HM
CLIP
80.57
70.52
75.78
75.40
CLIP
97.83
96.97
93.89
96.20
FedCoOp
85.64
72.15
70.61
75.57
FedCoOp
97.45
94.56
93.46
95.13
FedKgCoOp
81.39
71.18
75.81
75.90
FedKgCoOp
97.64
96.80
93.99
96.12
FedTPG
83.49
72.17
75.84
76.89
FedTPG
98.03
95.83
94.58
96. 13
(c) Flowers102.
(d) FGVCAireraft.
Local
Base
New
HM
Local
Basc
New
HM
CLIP
84.58
72.18
77.94
77.91
CLIP
37.88
27.55
35.81
33.10
FedCoOp
97.17
73.33
71.10
78.96
FedCoOp
44.00
27.23
25.76
30.53
FedKgCoOp
84.77
71.93
76.80
77.48
FedKgCoOO
38.53
26.86
35.06
32.71
FedTPG
90.03
71.58
77.08
78.85
FedTPG
41.74
28.44
35.05
34.21
(c) UCF101.
(f) OxfordPets.
Local
Basc
New
HM
Local
Basc
New
HM
CLIP
83.65
70.58
77.5
76.87
CLIP
93.26
91.33
97.04
93.82
FedCoOp
87.56
73.53
71.76
77.01
FedCoOp
95.95
92.36
91.60
93.27
FodKgCoOp
84.00
71.25
76.11
76.77
FedKgCoOp
92.55
90.32
96.36
93.01
FedTPG
85.78
72.15
76.05
77.59
FedTPG
95.86
93.92
96.73
95.48
(g) Foods102.
(h) DTD.
Local
Base
New
HM
Local
Base
New
HM
CLIP
95.94
90.16
91.25
92.38
CLIP
62.74
53.01
58.21
57.71
FedCoOp
95.18
88.21
89,91
90.72
FedCoOp
78.15
63.11
49.65
61.50
FedKgCoOp
95.81
89.88
91.66
92.38
FedKgCoOp
68.10
57.12
60.26
61,49
FedTPG
95.73
89.93
91.63
92.36
FedTPG
71.41
59.52
60.18
63.26
(i) StanfordCars.
SUN397.
Local
Base
New
HM
Local
Base
New
HM
CLIP
78.29
63.44
74.9
71.62
CLIP
90.96
69.41
75.46
77.61
FedCoOp
81.23
65.76
70,93
72.09
FedCoOp
94.07
71.32
72.10
77.88
FedKgCoOp
78.82
64.13
75.52
72.25
FedKgCoOp
92.28
72.36
76.47
79.51
FedTPG
80.15
65.33
74.62
72.84
FedTPG
92.71
72.90
76.62
79.88
16


Reviews:
[{"review_id": "umLWys6yUr", "reviewer": "Reviewer_KFdC", "summary": "Prompt learning for vision-language models has shown promising results in adapting CLIP to different downstream tasks. The paper proposes to improve the generalizability of the learn prompts to unseen classes, by proposing to solve this problem in a federated learning setting. Specifically, the prompt generation network is conditioned on task-related text embeddings, making the generation process context-aware and generalizes to the unseen classes. The paper evaluates the proposed approach on a diverse set of 9 datasets, as well as different ImageNet variants and demonstrates better generalization to unseen classes and domains than the baseline CLIP and CoOp.", "questions": "Please see the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The generalization of the prompt tuning vision-language models is an important research topic, and the federated learning setting is interesting.\n- The proposed technique effectively improves the generalizability of the learned prompts.\n- The paper is well written.", "weaknesses": "- What is the reason of not comparing to CoOp on EuroSAT/ImageNet in Table 1? This seems to be the setting followed by many works along this direction.\n\n- What is the reason of not comparing to CoCoOp [1] and MaPLe [2] in the base-to-new generalization setting? Will the proposed technique have the same benefit if naively applied to Fed-CoCoOp / Fed-MaPLe?\n\n- Why are the baseline numbers of CLIP/CoOp different between Table 1 of this paper and CoCoOp Table 1? Is it because the different selection of the base class and why?\n\n[1] Zhou et al. \"Conditional prompt learning for vision-language models.\" CVPR 2022.\n[2] Khattak et al. \"Maple: Multi-modal prompt learning.\" CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MVWQGm4uq8", "reviewer": "Reviewer_o1Q3", "summary": "This paper propose a Federated prompt tuning method, where a unified prompt generation network shared by all clients are proposed to generate specific prompt for each client based on the dataset context. The experiments show that the proposed method outperforms existing federated prompt tuning methods on both based-to-new and dataset generalization task.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of a context-aware prompt generation network is reasonable. \n2. The experiments show that the proposed methods outperform existing federated prompt tuning methods.", "weaknesses": "1. Can the author provide a more detailed analysis on the different between the proposed method and CoCoOp, since it also introduces an extra prompt generalization network. \n2. Like CoCoOp, the image may also provide important context informatio. How is the performance of proposed model compared to a model taking image as input? Will using both visual and textual information as context further improve the performance?\n3. The proposed method does not achieve the state-of-the-art performance on 3 out of 8 datasets in terms of the base-to-new task. Can the authors provide a more detailed analysis on the characteristics of these 3 datasets and why does the proposed method performs relatively worse on these datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lcUPIFee39", "reviewer": "Reviewer_qkce", "summary": "Prompt learning for vision-language models has achieved remarkable success in the adaptation of CLIP to different downstream tasks. This paper considers apply this approach to the field of federated learning. Specifically, they present Federated Text-driven Prompt Generation (FedTPG), which enables the scalable learning of a unified prompt generation network across multiple remote clients. Since the prompt generation network is conditioned on task-related text input, it is context-aware and suitable for generalization to both seen and unseen classes. The comprehensive empirical evaluations on nine diverse image classification datasets demonstrate that the proposed method outperforms existing federated prompt learning methods.", "questions": "no", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis is the first paper that solves the challenge of using prompt learning in FL for unseen classes, which is of great importance. \n2.\tThe idea of adopting a prompt generator for context-aware tasks is novel.\n3.\tThe experimental results demonstrate the effectiveness of the proposed method.", "weaknesses": "1.\tThe computational cost is high. Although the CLIP pre-trained model is fixed during local training, it still requires backward propagation over the encoder. Considering the clients may be edge devices with limited resources, this may hinder the wide adoption of the proposed method.\n2.\tThe generator is more likely designed for the centralized settings instead of the decentralized settings. It appears to be a simple prompt strategy in the centralized setting. Distributed properties and NonIID data distribution are not well taken into account. As a consequence, many existing prompt learning strategies for centralized settings may also be directly adopted and should be compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y2gKRq19EU", "reviewer": "Reviewer_RTLu", "summary": "This paper proposed a Federated Text-driven Prompt Generation (FedTPG) network to generalize the classification ability of vision-language models to unseen classes. The prompt generation network is conditioned on task-related text input, which makes it suitable to generalize for both seen and unseen classes. The experimental results on 9 datasets demonstrate that the proposed FedTPG achieves overall better generalization on both seen and unseen classes.", "questions": "1. What scenarios require training vision-language models under the framework of federated learning?\n2. What is the difference between the proposed FedTPG with existing generative prompt learning methods and cross-modal prompt learning methods?\n3. Why can the generated prompt vectors be generalized to unknown categories using a simple attention network?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The structure and the content of this paper are complete. \n\nThe formulation is clear enough. \n\nThe proposed FedTPG is easy to follow.", "weaknesses": "1. The applicable scenario of the proposed method is unclear. What scenarios require training vision-language models under the framework of federated learning? And test on unknown datasets?\n\n2. Many existing works have studied generative prompt learning methods [1] [2] [3] and cross-modal prompt learning methods [4] [5] [6]. What is the difference between the proposed FedTPG and these methods? Why the proposed FedTPG can generalize to unknown datasets but the existing methods mentioned above cannot?\n\n3. More comparison experiments with existing prompt learning methods [1-6] should be included to verify the effectiveness of the proposed FedTPG.\n\n4. In Table I, the proposed FedTPG did not achieve the best performance on 9 datasets while under Local Base and HM settings. Does this mean that FedTPG sacrifices the classification performance of known categories when considering unknown categories? Therefore, the contribution of this article is difficult to evaluate.\n\n5. The prompt generator network proposed in this article is based on a simple attention mechanism, which is too simple. Using such a network, why can the generated prompt vectors be generalized to unknown categories?\n\n[1] Improving Zero-shot Generalization and Robustness of Multi-modal Models\n[2] Improving Zero-Shot Generalization for CLIP with Synthesized Prompts\n[3] SuS-X: Training-Free Name-Only Transfer of Vision-Language Models\n[4] Class-aware visual prompt tuning for vision-language pre-trained model\n[5] Prompt learning with optimal transport for vision-language models\n[6] A Retrospect to Multi-prompt Learning across Vision and Language", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)