Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
Authors: Jun Luo
Chen Chen
Shandong Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Federated learning
prompt learning
vision-language model
mixture of experts
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose pFedMoAP, a novel personalized federated prompt learning framework for CLIP-like VLMs under data heterogeneity from a Mixture of Experts perspective.

Abstract: Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods are habitually restricted to the traditional FL paradigm, where the participating clients are generally only allowed to download a single globally aggregated model from the server. While justifiable for training full-sized models under federated settings, in this work, we argue that this paradigm is ill-suited for lightweight prompts. By facilitating the clients to download multiple pre-aggregated prompts as fixed non-local experts, we propose Personalized Federated Mixture of Adaptive Prompts (pFedMoAP), a novel FL framework that personalizes the prompt learning process through the lens of Mixture of Experts (MoE). pFedMoAP implements a local attention-based gating network that learns to generate enhanced text features for better alignment with local image data, benefiting from both local and downloaded non-local adaptive prompt experts. Extensive experiments on 9 datasets under various federated settings demonstrate the efficacy of the proposed pFedMoAP algorithm. The code is available at https://github.com/ljaiverson/pFedMoAP.

Published as a conference paper at ICLR 2025
MIXTURE OF EXPERTS MADE PERSONALIZED:
FEDERATED PROMPT LEARNING FOR VISION- LANGUAGE MODELS
Jun Luo
Chen Chen
Intelligent Systems Program University of Pittsburgh Pittsburgh, PA 15213, USA jul117@pitt.cm
Center for Research in Computer Vision
University of Central Florida
Orlando, FL 32816, USA
chen. chen Bercv. www.ruc...c..m ucf.edu
Shandong Wu
Intelligent Systems Program Department of Radiology Department of Biomedical Informatics Department of Bioengineering University of Pittsburgh Pittsburgh, PA 15213, USA wus 30 @uppcc...... . edu
ABSTRACT
Federated prompt learning benefits federated learning with CLIP-like Vision- Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods habitually re- stricted to the traditional FL paradigm, where the participating clients are gener- ally only allowed to download single globally aggregated model from the server. While justifiable for training full-sized models under federated settings, in this work, we argue that this paradigm is ill-suited for lightweight prompts. By fa- cilitating the clients to download multiple pre-aggregated prompts as fixed non- local experts, we propose Personalized Federated Mixture of Adaptive Prompts (pFedMoAP), novel FL framework that personalizes the prompt learning pro- cess through the lens of Mixture of Experts (MoE). pFedMoAP implements a local attention-based gating network that learns to generate enhanced text fea- tures for better alignment with local image data, benefiting from both local and downloaded non-local adaptive prompt experts. Extensive experiments on 9 datasets under various federated settings demonstrate the efficacy of the pro- posed pFedMoAP algorithm. The code is available https://github. com/ wwwiieessonppeeeeeeeee
INTRODUCTION
Recent years have witnessed the prosperity of federated learning (FL) (McMahan et al. 2017 Kairouz et al. 2019 Li et al,, (220a) as a potent paradigm for training machine learning mod- els across decentralized data sources. While offering privacy-preserving solution collaborative training settings, this approach faces a critical challenge in the form of heterogeneous data distri- bution across the participating clients (Li et al. 2020a). And due to the data heterogeneity, the generalization capabilities of the trained models may be compromised (Li et al. 2020b; Sahu al.) 2018). In this context, the pre-trained Vision-Lamm Laniuag Models (VLMs) such as CLIP (Radford al. 2021) and ALIGN (Jia et al.) 2021) can provide significant impact with their remarkable applications in learning transterable representations across a wide range of tasks and disparate data distributions encountered in federated settings.

---

Published as a conference paper at ICLR 2025
MrFi-15 36
Server
ResMet18
RusNot50
Toxt feature
Image feature
Train Fix
Pool
Client,
Client,
Client
Prompt,
logits
Prompt,
Prompt,
Prompt
Prompt:
Prompt
Lngthh 16
Pramps
Prompt,
Prompts
Prompt,
Mixture
(a) Model parameter counts
(h) Overall pipeline of predMoAP over three clients
Figure I: Schematic diagram of employing Mixture of Experts into federated learning. We facilitate the sharing of pre-aggregated prompts thanks their lightweight nature. Each client downloads the pre-aggregated prompts trained on the remaining two clients through the server, keeping them fixed locally non-local experts.
However, applying VLMs in FL contexts is not without challenges. Due their millions of pa- rameters, fine-tuning these large-scale models in federated settings incurs prohibitively high com- munication overhead, making it impractical for many real-world applications. This limitation has led researchers explore more efficient adaptation techniques for VLMs, with prompt learning emerging as promising approach to overcome the communication bottleneck.
Prompt learning for VLMs (Zhou al. 2022b), originally proposed to eliminate the need for hand- crafted prompts, replaces the context words with small-scale learnable vectors while keeping the pre-trained model fixed. Such a lightweight approach partieularly benefits FL by tremendously ducing the communication overhead associated with transmitting the entire fine-tuned model. For instance, P PromptFI. (Guo al. 2023b) leverages FedAvg (MeMahan et al. 2017) style aggre- gation of the locally trained prompts. FedPR (Feng et al. (2023) learns the visual prompt within the null space of the global prompt. addition, recent criticism on prompt-based VLMs unseen data distributions (Zhou et al. 2022bb Khattak et al. [2023a b) also facilitates FL researchers to combine prompt learning with personalized FL (PFL). Instead of restricting the global consensus model, PFL (Tan et al.) 2022: Kulkarni et al., 2020) allows tailored models for each individual client, sys- temically mitigating the data heterogeneity issues and enhancing the overall flexibility of the system. However, directly applying PFL techniques, such local fine-tuning (Cheng al. 202T) and per- sonalizing specific layers (Arivazhagan et al. 2019), to prompt learning has demonstrated limited efficacy in training the prompt (Guo et al. (223b) under extreme data heterogeneity. Consequently, some works tailor PFL techniques to prompt learning in CLkP-like VLMs. In this recently emerged field, Guo et al. (023a) trains a global consensus prompt with personalized visual attention modules on locally memorized data. [i et al. (2024) leverages unbalanced Optimal Transport to align visual feature maps with personalized prompts. More details on prior works are presented in Appendix A
Unfortunately, existing federated prompt learning approaches habitually adhere to the paradigm of traditional FL./FIL techniques, wherein the participating clients generally only allowed to download single globally aggregated model from the server. While justifiable for full-sized models, we argue that this paradigm ill-suited for federated prompt learning, as the significantly reduced size of prompts substantially mitigates the potential communication overhead. To contextualize this disparity, consider that while a typical ResNet50 model contains approximately 25.6 million param- eters, a learnable prompt with length 16 and dimension 512 comprises merely 16 x 512 = 8, 192 parameters - a reduction three orders of magnitude. Given this significant parameter reduc- tion, restricting clients to downloading only single globally aggregated model - as in traditional FL/PFL- - unnecessarily limits federated prompt learning. The lightweight nature of prompts provides unique way for more flexible and effective personalization strategies. Therefore, this motivates us to address the following pivotal question:
How can we devise personalized federated learning framework, tailored for prompt learning in CLiP-like VLMs, while fully exploiting the lightweight nature of the prompts?
In light of these challenges and opportunities, propose novel framework: Personalized Federated Mixture of Adaptive Prompts (PFedMoAP) Tailored specifically for prompt learning in CLIP-like VLMs, proposed framework aims unleash the potential of the lightweight prompt
2

---

Published as a conference paper at ICLR 2025
by allowing the clients download multiple pre-aggregated prompts (see Fig. lb) to acquire col- lective knowledge. In this manner, we enhance the generalization capabilities of the PFL system under extreme data heterogeneity, offering more flexible and effective solution that fully leverages the lightweight nature of prompts federated settings.
At its core, pFedMoAP personalizes the federated prompt learning problem through the lens of Mix- ture of Experts (MoE), treating all locally updated prompts as specialized experts. Benefiting from lifting the aforementioned ill-suited restriction, pF e'edMoAP facilitates the sharing of pre-aggregated prompts between clients (through the server). In addition, our proposed framework implements a novel client-specifie, parameter-efficient hanced text features for better alignment with local image data on each client. Through this locally trained gating network, the enhanced text features are generated from the adaptive local and non- local prompt experts via CLIP's text encoder. The local expert is trained exclusively with the client's s data, while non-local experts, trained on other clients, are sparsely selected from a server-maintan maintaina pool based on K nearest neighbors (KNN), and shared with the client without aggregation, fostering collective knowledge sharing across clients beyond the global aggregation.
attention based gating network that learns to generate en-
We summarize our main contributions as follows:
We pioneer a paradigm shift in federated prompt learning for VLMs by challenging the restriction on sharing pre-aggregated prompts between clients. Circumventing this ill-suited restriction for prompt learning unlocks more potential inherent in the lightweight prompts, facilitating more effective cross-client knowledge sharing. As such, pave the way for more flexible federated prompt learning in VL.Ms.
We propose PFecMMAP, novel framework designed specifically for personalizing federated prompt learning in CLIP-like VLMSs under data heterogeneity for image recognition tasks. PFecMoAP personalizes the prompt learning with a unique attention-based gating network. Thanks to its flexibility, this gating network has the potential to extend beyond federated learning for prompt-based VLMs.
We validate the effectiveness of the proposed PFedMoAP through extensive experiments and ablation studics across 9 widely adopted datasets under various federated settings. The results verify the superiority of pF edMMAP over compared kate-of-theeart methods.
2
PRELIMINARIES
2.1 PERSONALIZED FEDERATED LEARNING
Conventional federated learning (FL. aims train glohal consensus model for federation of clients with similar data. As the most notable FL algorithm, FedAvg (McMahan et al., (017) minimizes the global objective over N clients defined as:
min F(0) - P.(O).) i=1
where 0 and F.(-) represents the global model and the local objective of client i, respectively. and the weight pa is often set as p. = nz/n with n = Σ, n4 where n, denotes the number of data samples client i. In FedAvg, the local objective F.(.) measures client i's empirical loss, Ek= L. (m) yk)), where La represents its loss and a is its k-th data sample with ground truth label yk-
Compared to conventional FL, personalized FL (PFL) relaxes the number of models where each client i is allowed have its tailored model O.. The goal of PFL is, therefore, defined as:
min F(O ..., 0N) min Σ p...( O.....Ox
O N
2.2 PROMPT LEARNING FOR CLIP-LIKE VLMS
To efficiently adapt pre-trained CLIP-like VLMs to downstream tasks, prompt learning methods (Zhou al. 2022hla) model prompt's context words with learnable vectors. While zero-shot
3

---

Published as a conference paper at ICLR 2025
transfer of CLIP leverages the fixed word embedding W {W1,.... ..... wi} from hand-crafted prompt templates with / context words (e.g. "a photo of a <<laas>>."), prompt learning replace it with the learnable prompt P {pi, .... pi E Rx where d is the dimension of the word embedding. The full prompt P(e), consisting of the learnable prompt P with the embedding of class label c, is then fed into the fixed text encoder, g()). Together with the fixed image encoder f(-), the classification logit for class is then computed as matching score between the image and text features. And the prediction probability of each class for image x is derived by taking a Softmax, controlled by temperature т, over the logits:
logit (e) = sim (lm()()).g( (P(c))).
(3)
(sp(logit())/1
p c/a) E exp(logit (8) /T)
(4)
where sim(.) denotes metric function (e.g. cosine similarity), ú is the predicted label, and C denotes the number of classes. With the probabilities, we can learn the prompt P by minimizing the cross-entropy loss.
2.3 MIXTURE OF EXPERTS
With similar form to FL's, Mixture of Experts (MoE) aggregates the output of multiple experts or trained models, instead of the model parameters. For a general MoE system with N experts, the output of MoE with input z is defined as:
MoE(a: = Σ G(w), (x)),,
(5)
==1
where E.(-) represents the i-th expert, and G(-)s represents the weight assigned to the output of expert from a N dimensional vector, based on its input. The mechanism for weight assignment is often controlled by network called gating network (also known router). The implementation
of the gating network varies (Clark et al. 2022 Hazimeh et al. 2021 Zhou et al. 2022c), but simple yet effective one is implemented by taking the Softmax over top-K (K N) logits of linear layer with the remaining N K weights set to zero (Shazeer et al. 2017). We propose novel lightweight attention-based gating network devised for CL.PP-like VLMs that extends beyond federated learning. Details are presented in Sec. 3.3
3 PERSONALIZED FEDERATED MIXTURE OF ADAPTIVE PROMPTS
Motivation. While federated prompt learning for pre-trained CLIP-like VLMs offers an approach to efficiently adapt these models to downstream tasks under federated settings, such methods mostly lack personalization for the prompt to generalize with extreme data heterogeneity. Prior PFL meth- ods devised for prompt learning habitually adhere to the restriction for the clients to download only single globally aggregated model from the server, failing to leverage the lightweight nature of the prompt to the fullest. In addition, from an MoE perspective, allowing clients to download pre- aggregated prompts trained other clients naturally provides systemic solution to the dilemma where too many experts on the client incurs prohibitively high communication overhead while too few experts impairs the benefit of employing MoE. Consequently, there a need for tailored PFL approach for prompt learning in CLIP-like VLMs that offers the fiexibility to share trained prompts from an MoE perspective.
Overview.
The rest of this section presents the details of the pF edMoAP by gradually building upon PFL with prompt learning and mixture of adaptive prompts, and how the proposed iiteetiin-bbsed gating network works in DFedMoAP, as well as potentials in extending it beyond federated learning.
3.1 PERSONALIZED FEDERATED PROMPT LEARNING WITH LOCAL PROMPT ONLY
In our study, we presume each client hosts a CLIP model with fixed image encoder f(-) and text encoder g(.) for image recognition tasks. We first suppose, in this subsection, that each client
4

---

Published as a conference paper at ICLR 2025
Nhg <class>
plane
Nhy <classs
Tile
car
Nily <classs
dog
Text Encoder
a.g.
May
Attantion
P PNLx
Take
Fill
Fike
I Gartnng
(33 MoE
G (9x)
bird
T
CLIP'S Encoder
logits
Figure 2: Workflow of pFedMoAP at client i. The client first computes the non-local text features using the non-local prompt experts. As training progresses, then calculates the local text features. Taking class based gating network as both key and value, while image features serve as the query. This process generates enhanced text features. Matching socres are derived from two sources: local text features and MoE-enhancee text features. These scores are then combined through weighted averaging to produce the final logits.
example, both local and non-local text features are input into the attention-
personalizes the local prompt without the MoE. At federated round i, each client in the selected set of participating client S, locally updates its prompt, P. initialized with the global prompt from the last round, p The prompt is then updated through a gradient-based optimization, e.g. Stochastic Gradient Descent (SGD), over a cross-entropy loss for multiple local epochs.
After finishing the local update, the learned prompts P. for all the clients in S, are then aggregated by the server for global prompt pt FedAvg manner:
PJ =E-
n4
P
EkES, 11
While P used for the next round of training. for client P is stored and used for inference purposes, achieving simple personalization for federated prompt learning while MoE is not present.
3.2 PERSONALIZED FEDERATED MIXTURE OF ADAPTIVE PROMPTS WORKFL.OW
Thanks to the lightweight nature of the prompt, clients are enabled to acquire collective knowledge beyond the globally aggregated prompt. In pFedMoAP, we facilitate, rather than discourage, each client to download K pre- pre-aggregated prompts trained on other clients as non-local client experts. These non-local experts The pool functions as a dynamic repository, refreshing at the conclusion of each federated round. It incorporates the newly acquired prompts from the clients who participated the current training round, and overwrites their previous entries (if existing) in the pool, i.e:
sparsely selected from server-maintaine pool of prompt experts, P.
P. Pt-1 (P. "eee. {P))
At the beginning of the current round f,, each client S, first assigned non-local experts from the previous round's pool, Pt-1. This assignment is based on a K nearest neighbors (KNN) algorithm, which identifies the top- K nearest experts from client i's entry in the pool, terms of lp d the rationale behind the KNN-based expert assignment is that clients with similar locally
'The first-time participants with entry in the pool conduct standard local training of the global prompt, and upload its trained prompt the pool.
5

---

Published as a conference paper at ICLR 2025
trained prompts more likely to share comparable data distributions. This similarity- -basdd expert assignment aims to leverage knowledge from clients with potentially similar data characteristics, enhancing the personalization from closer collaborators in terms of local data distributions.
While such a server-side assignment of the non-local experts inherently simulates the sparse gating in MoE, client-side gating network, G,(+), enables dynamic dense incorporation of the output from all selected experts on per-image basis (see Fig. 2
Let Q 1 (NL represents "Non=Local") be the set of clients assigned client é, whose prompts, P. are served as non-local experts on client i. Before the local training of client i initiates, text features for every non-local prompt are computed via the CLIP's fixed text encoder, g(-). Note that since the non-local prompt experts are kept fixed throughout the local training, these text features are only necessary be computed once. Let T( Ni, be the text features for class generated by the j-th non-local prompt expert on client with class c's embedding, P At the
I
beginning of the local training on client i, the text features, N. from every non-local expert for every class с [C], formulated below, are fixed and ready to use for the entire local training process:
E [C], TML
N = o(P N
D, VNL, E ei).
(8)
In PFedMoAP, a novel attention-based gating network, G(-J0.) parameterized by 6:, is proposed. Unlike the G(-) in Eq. (5). the proposed gating network functions to generate mixture of the outputs from the experts, instead of the weights for the outputs in traditional MoF. The weights, however, are still internally computed as intermediate products through the attention mechanism. We discuss the gating network in details in Sec. 3.3] Let I = f(æx)) be the computed image feature of the input image Iy through CLIP's fixed image encoder f()). In pF edMoAP, for each class c E [C], what the gating network, G(-10:), takes as input are three-fold: 1) image feature, I;; 2) text features from the locally updated prompt expert T( = -(p((") where I. represents "Local"; and 3) text features from the all non-local experts, Tx2 defined in Eq. (8) The gating network then outputs the enhanced text feature, Tí better aligned with the image feature Ig for more generalizability that incorporates both collective knowledge and personalization, i.e.,
Ve [C), Mo e G(T,,TIO
NL . O.).
Note that although the gating network could technically be parameterized in a class-speciffr manner, our implementation opts to use shared set of parameters across all classes for parameter efficiency.
As the final step, PFedMoAP computes the logits based on the matching score between the image feature I and text features from two sources, namely T MOE from the MoE, and T(() from the local expert. Although Tr(e) MaE carries the global collective knowledge, we further address T(()) with weight \ as the local prompt P, is the only learnable expert on the client, i.e.
VC [C]. logit()) sim(Is:
T(c) Mog)) A sim ( T
(10)
With the classification logits, the local prompt P. and the parameter 6; in the gating network are updated through optimizing over the cross-entropy loss based on the prediction probability computed as in Eq. After the local training, client i upload P. to the server while maintaining the gating network locally.
3.3 ATTENTION BASED GATING NETWORK: IN PFEDMOAP AND BEYOND
To generate enhanced text features that better aligns with the image feature Iz. the proposed gating network employs a multi-heas attention (MHA) layer [Vaswani 2017) with the query being the Ig. while the key and the value are both text features T N L, T ... NLK- However, since the dimension of these features are usually large, resulting in too many parameters in the MHA layer (e.g. for CLIP with a ResNet50 backbone, dieature 1024, MHA layer will have 4.2M parameters), we force each feature to first go through a pooling layer to reduce the dimension to a fixed dgaaing = 128 (MHA layer will only have 66.0K parameters). Therefore, with Q Pooling It), K = = Pooling(T)) (T (c) TT(e) and h being then number of heads, we have:
((c) MoE G(1,, T(o) T MHA K, K,V) Concat(head), ..... head, WO
(11)
6

---

Published as a conference paper at ICLR 2025
Table I: Few-shot performance on CLIP datasets under pathological non-III setting.
Flowcrs 102
OxfordPets
Food101
Caltech101
DTD
2S-CLIP (Radford et al.
2021
62.17=0.12
84,4720.01
75.27
85.1410.24
40.21:0.12
CoCp (Zhou et al., 2022h
70. 0.776
83.2
7..43+2242
87.37+0.44
44.23++0.63
PromptFl (Guo ct al. |023bb
72.80=1.14
90.79=0.61
17.31=1.64
89.70+1199
54.11=0.22
PromptFL+FT [Cheng et al. 2021
72.31
91.23310.50
77.16.1.566
89.7011..2
5..7411.36
Prompt=FFdPer (Aniaahagan et al. 2019
72.11+1.33
89.50+1.62
1..29+1.87
1..72+1.45
50,23++0.22
Prompt+FedProx [Li et al. 20206
66.40=0.29
89.24=0.41
7..24411.4
44,26=+1111 89.41
Frompt+FedåMP (Huang et al. 2021
69.1010..3
80.2110.44
74.48.1.711
87.3111.60
47.16.10.92
pFedPrompt (Guo al. 20265
86.46=00115
91.84+041
92.26+1.344
96.54+1.31
77.14+00.99
Fedo12 eLi al. 2024]
96.23=0.44
98.8210.11
92.73310.15
97.0210.36
87.6410.70
PFedMoAP ()=0.0)
17.61
94.83100.5
86.7110..15
95.7110.37
85.6410.34
PFedMoXP (A=0.5)
98.41+0.04
99.06+0.09
93.39+0.099
97,95+0.07
89.13+0.54
where head, = Attention KWW K VW)) and W W.. W W. are standard MHA pa- Ty()) MoE E RRquine, computing sim I. MoR) requires pooling on I as well.
rameters in 0,. As Te()
While a linear projection-based gating network, Gpoo(()) in typical MoE is also able to achieve enhanced text features for CLIP (e.g. by assigning weights (an)(as)) E IR++1 for the experts' output based on the image input (Shazeer et al., 2017). the proposed attention-based gating net- work is more favorable for the following FL-agnostic reasons. 1) Since the experts are constantly adapting during the training process, directly learning the assigned weights for the experts, as in projection-based gating, will be less robust than learning the relationship between the image feature and the text features generated by the experts, as in attention-based gating. 2) The output of the proposed attention- based gating is the enhanced text features, which makes the MHA layer serve as a lightweight linear probing with a much larger search space, that functions beyond assigning K + weights to aggregate the experts' output as a linear combination in a K + 1-dimensional search space. Even with a redesigned projection layer to directly produce the enhanced text fea- tures, the parameter count of such a projection based gating network would substantially exceed that of the attention-based gating network. 3) A projection-baset gating network fails to leverage the pre-trained CL.IP's powerful capability to align text features with image features, while the atten- tion mechanism naturally harnesses it through the scaled dot-product operation. 4) Even if we feed both image and text features from multiple experts into a projection- based gating network, the order of these input would affect the final output in uncontrollable way. And the number of experts has to be fixed for a projection- based gating network. In contrast, the proposed iitention-based gating network is order-agnostic with a flexible number of experts it engages with. In pFedMoAP while we fix the number of non-local experts on each client to be K, in practice, K can be client-specific based on the client's memory and communication bandwidth.
As such, our design of the attention- gating network extends beyond federated learning. This versatile network could be adapted general p learning for VLMs where multiple prompts are trained, such as multi-task learning or domain adaptation. We anticipate that our work will inspire future research exploring the broader applications of mixture of adaptive prompts for VLMs. The detailed pF P algorithm can be found in Algorithm
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset and data heterogeneity. We evaluate the efficacy of the proposed pFedMoAP with 9 public benchmark datasets under various federated settings to simulate different types of data het- erogeneity. Following previous research (Guo et al. 2023b), to evaluate pF FFeMMMAP under label heterogeneity, we adopt 5 representative visual classification datasets used to evaluate CLIP (Rad-] Ford et al.) 2021]. 1) namely OxfordPets Parkhi et al. (2012). ). Flowers Nilsback & Zisserman 2008 )
DTD Cimpoi et al. (2014). Caltech101 Fei-Fei (2004). Food101 Bossard et al. (2014). We refer to these datasets collectively as CLIP datasets. On these datasets, test pF e edMoAP's few-shot per- formance under label heterogeneity by employing a pathological non-lID setting, where the classes evenly distributed to the clients with overlapping classes between any two clients. In addi-
7

---

Published as a conference paper at ICLR 2025
tion, we use CIFARIO and CIFARIO0 dataset (Gong et al. 2012) and a Dirichlet distribution with Dir(a = 0.5) to simulate the label shift (Hsu et al. 2019 The Dirichlet distribution with a = represents an extreme case where each chent has only one class and larger simulates stratified and even split of labels. Dir(a 0.5) is on the more heterogeneous side. For feature heterogeneity, adopt DomainNet dataset (Peng et al. 2019) and Office-Caltech10 dataset (Gong et al. 2012), with 6 and 4 inherent domains, respectively. To add on extra heterogeneity, for DomainNet and we split each domain with Dir(a 0.3) for extra label heterogeneity, mimicking real-world FL settings with both label and feature shifts.
Compared baselines. We compare our approach with three categories of baselines. The first cat- egory is local methods, where the clients do not participate in a FL setting. This category includes zero-shot CLIP (ZS-CLIP) (Radford et al., 202T) employing prompt templates: "a photo of a <class>>" and CoOp (Zhou et al. 2022b) with learnable prompt vectors trained locally on each client. The second category comprises existing federated prompt learning methods, which include PromptFL FL (Guo et al., 2023b), which learns a unified prompt across clients, pFedPrompt (Guo ct al. 2023a), which learns a shared prompt with personalized visual attention modules on locally memorized data, and FedOTP (Li al., 2024), which leverages unbalanced Optimal Transport to align visual feature maps with personalized prompts. The third category consists of adapted meth- ods from traditional FL/PFL techniques P PromptFI. namely FT (Cheng et al. 2021), FedP rox Li et al. (020b), FedPer (Arivazhagan et al. 2019), and FedAMP (Huang et al. 2021).
Implementation details. 1) CLIP datasets. Each dataset in CLIP datasets is partitioned into
Table 2: Results on CIFARIO and CIFAR100 with label shift with (Dir(a 0.5)) partition into 100 clients
N 10 clients, each with a dis- joint set of classes evenly and randomly assigned to the clients.
CIFAR1O
CIFARIO
The training proceeds for 10 rounds with
CLTP (Radford et al.
2021
53,46+00.1
32.68=0.00
100% par- ticipation rate. The CLIP uses ResNet50 backbone. 2) CI-
CoOp (Zhou et al. 2022h
00.84++0.39
48.74-0.17
PromptFL (Guo et al. 2023b
73.29:10.37
45.00=0.62
Prompt +FedP rox (Li el al. 20206
73.32+00.4
45.63+0.75
FARIO & CIFAR100.
PFedMcAP
83.46:10.53
53.42.10.22
100 clients results from Dir(a
0.5) partition with T = 120 rounds with r = 10%. CLIP also uses a ResNet50 backbone. 3) Do-
main & Office-Caltech10. Each domain of these datasets partitioned to 5 clients with
Dir(a- = 0.3), resulting in N = 30 for DomainNet and N = 20 for Office-Caltech1.. T = 25 for both datasets, while r 25% for DomainNet and r 50% for Office-Caltech1... The CLIP uses ViT-b-16 backbone. 4) Training specs. For all methods, we use SGD as the optimizer with a learning rate of 0.002 and 5 local epochs (except CoOp is locally trained for 25 epochs without FL.. For APe we use SGD with a learning rate of 0.01 to train the h = 8-head gating network and default = 0.5 in Eq. (00). summary of the dataset setup can be found at Appendix C.I
4.2 PERFORMANCE EVALUATION
We compute as the metric the average accuracy over every client's private test set, drawn from the same distribution as its training set, and report the mean and standard deviation of all methods over three runs with different seeds.
Evaluation on Label Shifts. To assess the performance of our method in handling label shifts, we conducted experiments on the CLIP datasets for few-shot training and CIFAR datasets for standard training. Tab. Tab.[shhow shows the results on the CLIP datasets under a pathological non-III setting, while Tab. presents the results on CIFARIO and CIFAR 100 with Dir(a = 0.5). On the CLIP datasets (Tab. our proposed method PFedMoAP consistently outperforms other baselines across all five datasets, with significant margins over most methods, which proves the superiority of рFedMoAP. For instance, on the Flowers 102 dataset, pFedMoAP achieves an accuracy of 98.41 48.41 with À = 0.5, respectively 18% and 11 11.95% higher than FedoTP and pFedPrompt. the top 2 performant fed- erated prompt learning methods, both recently proposed to achieve personalization through interme- diate visual feature maps. Similar improvements can be observed across other datasets. Notably, on DTD dataset where zero-shot struggles to excel, pF pFedMoAP has little performance drop compared to personalized methods such as pFedP rompt and Prompt FedPer, showing PFedMoAP's
8

---

Published as a conference paper at ICLR 2025
Table 3: Results on DomainNet with feature shift and label shift with Dir(a 0.3) partition into clients/domain
Clipart
Inlograph
Painting
Quickdraw
Real
Sketch
Average
25 CLIP
9.18.1062
10.03:10.16
9.93=0.51
10.25=0.40
9.90+11.30
9.5411.13
9.8110.30
CoOp
43.84=3.51
45.72=0.85
29.94:0.46
36.83=1177
31.64=0.49
33.97=0,78
36.99-00799
PromptFL
27.63=16.41
27.69=#88.07
21.6248.34
23.45=13.49
20.62=11.03
25.90+88110
24.48+12.52
Frompt-FedProx
22.23-15.44 15.42 21.75++17.00
18.58+8.15
19.40+12.59
17.17+10.25
22.49+8.444
20.27+-11.83
pFedMoAP
47.49_0.64
46.73-10.71
32.74.10.84
37.16.1.334
31.02-1.599
37.67.10.72
38 38.8010.111
Table 4: Results on O with feature shift and label shift with Dir(a 0.3) partition
into 5 clients/domain
Amazon
Caltech
DSLR
Wehcam
Average
2S-CLIP Radford el al. 2021
9.83+
10.67:1..9
1..89
6.2013.84
9.40:10.77
CoOp (Zhou et al. 2202hb
30.29+3.664
35.88+1.30
33.43=2.25 29.89+5.115
32,37+
PrompUFL (Guo et al. 20236
21.08=9.60
23.72++1.21
22.9447.96
25.88=7.72
23.41+9.06
Prompt+Fede FeeetFFFFrrrrrr (Li et al. 2020b
18.6441.55
19.56+11.59
20.89+4.338
22.96=7.56
20.51
рPeÍMoAP
35.47++.37
37.45++.33
45.11+3.14
35.22+1.044
38.31+1.21
remarkable ability to adapt to diverse scenarios under few-shot settings. In addition, pF edM with x = 0.5 also beats À = 0.0 over all datasets, demonstrating the effectiveness of addressing the local prompt together with MoE. The results on CIFAR datasets (Tab. 2 further corroborate the efficacy of our method in handling extreme data heterogeneity with Dir(a 0.5) over 100 clients. In this challenging scenario, pF PFedMoAP still consistently outperforms the compared baselines, re- flecting the potency to handle label shifts by introducing non-local experts to the clients.
Evaluation on Feature & Label Shifts. To evaluate the performance of our method in scenarios closer to real-world FL applications, we introduce feature shift on top of label shift via DomainNet and
where each domain is partitioned into 5 clients with Dir(a = 0.3), resulting respectively. The results of these
in 30 and 20 total clients in DomainNet and are presented in Tabs. and respectively. Under two types of heterogeneity, traditional federated learning methods struggle to benefit the clients. On average, the FedAvg style method P PromptFl exhibits worse performance than the local training method CoOp by 12.51 % and 8.96% respectively on DomainNet and Office-Caltechl However, pF BFMMMAP remains better than local training and FL with up to 5.94% accuracy boost under both types of heterogeneity. This validates the effective- ness and robustness of pFedMoAP in scenarios closer to real-world federated settings.
4.3 ABLATION STUDY
Impact of number of shot. Following prior works (Li al. 2024 Guo et al. 2023a), we investigate the impact of shots in the few-shot learning for pF FedMoAF and other FL. techniques implemented to prompt learning, namely P PromptFl FL and P FroppFLL+Fed r rox, across the CLIP datasets. The
number of shots varies from [1, 2, 4, 8, and 16]. As shown in Fig. 3 pFedMoAP consistently outperforms the other two methods across all datasets and shot numbers, demonstrating its superior effectiveness and robustness. While pF edMM outperforms the compared methods from only shot, the gap between pF edMoAB and other methods widens as the number of shots increases. This demonstrates that pF DFeMMcAP particularly adept at leveraging additional training examples to enhance its performance, due to the introduction of MoF and the gating network. The trend of the curves also just that PFedMoAP has a more robust growth of performance when the number of shots increases, where the performances of the compared methods sometimes drop.
Impact of X, the coefficient for logit computed from local prompt. the contribution between the logits generated from the local prompt and from MoE. Fig. displays the results for À values ranging from 0.0 to 5.0 across the CLIP datasets. The trends reveal that although the optimal À value varies across datasets, marginal address on the logits from the local prompt (c.g. 0.5 or 1) can already reach near-best performance. The flat plateau also indicates that the balance between the logits generated from the local prompt and from MoE is easy reach.
In PFecMoAP, balances

---

Published as a conference paper at ICLR 2025
Flowers102
OxfordPets
Food101
100
90
92.5
Acturcyy
00
07.5
144
80
92
Tost
85.0
80.0
60
16
Number Shots
Number Shots
Number Shots
Caltech101
DTD
80
Text Accuraty
-41
12
80
600
Promptfl
Figure Ablation study on the number shots.
Flowers102
OxfordPets
Food101
95
25
95
Accuracy
00
90
90
89
89
85
80
80
80
Caltech101
DTD
pFecMooN
26
06
X 0.0
Accurscy
Accurscy's
90
90
2.0
85
2.5
0.5
1.0
2.0
2.5
0.5
1.0
2.0
Figure Ablation study the coefficient for the logits from local prompt, X.
In addition, Fig. also demonstrate disinclination of too little too much address the local prompt-generate logits. We attribute this to the fact that only the local prompt is being updated on client among all the experts, while immoderately addressing it ignores the benefit from MoE.
CONCLUSION
In this paper, we introduced pFedMoAP, a novel framework for personalized federated prompt learning in CLIP-like VLMs. Our approach leverages the lightweight nature of prompts to enable efficient cross-client knowledge sharing through Mixture of Experts paradigm. By implementing client-specific, attention- baased gating network, pFedMoAP dynamically incorporates both local and non-local expert knowledge, striking a balance between personalization and collaborative learn- ing. Extensive experiments across various datasets and federated settings demonstrate the superior performance of pFedMoAP compared to sfate-of-thh-an alternatives, particularly in handling ex- treme data heterogeneity with label and/or feature shift. As federated learning continues to evolve, PF edMoAP represents a significant step forward in personalizing prompt learning for VLMs, open- ing new avenues for efficient and effective collaborative learning in -preeeciie settings.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by a National Institutes of Health (NIH)/National Cancer Insti- tute (NCI) grant (IR01CA218405) NIH Other Transaction research contract 10T20D037972-01, the grant IR01EB032896 (and a Supplement grant 3RO1EB03 2896-03S1) as part of the National Science Foundation (NSF)NIH Smart Health and Biomedical Research in the Era of Artificial In- telligence and Advanced Data Science Program, a NSF grant (CICI: SIVD: #21 15082), an Amazon Machine Learning Research Award, and the University of Pittsburgh Momentum Funds (a scaling grant) for the Pittsburgh Center for AI Innovation in Medical Imaging. This work used Bridges-2 at Pittsburgh Supercomputing Center through allocation [MED200006] from the Advanced Cyberin- frastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF grants #2 #2138259, #2138286, #2138307, #21 #2137603, and #2 #2138296. The views and conclu- sions contained in this document are those of the authors and should not be interpreted as represent- ing official policies. either expressed or implied, of the NIH NSF.
Ethics Statement. The research presented in this paper adheres to the ICLR Code of Ethics. Our study does not involve human or animal subjects. Every dataset used in our study is publicly avail- able. Our method does not address discrimination/ hattiiionnnas//aaiesss.. concerns.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Fed- erated learning with personalization layers. arXiv preprint 19Xiv:9912.0080 2019.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. (ood-1011mining discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446-461. Springer, 2014.
Zhongyi Cai, Ye Shi, Wei Huang, and Jingya Wang. Fed-COy: Cooperation of online and offline models for severe data heterogeneity in federated learning. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Gary Cheng, Karan Chadha, and John Duchi. Fine-tuning is fine federated learning. arXiv preprint arXiv:2 108.073330 3, 2021.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057- 4086. PMLR, 2022.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International conference on machine learning, pp. 2089-2099, PMLR, 2021.
Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. Harmonizing generalization and personalization in federated prompt learning. In Forty-first International Conference on Machine Learning, 2024. URL
wtt.pp/nwweeiieeeeeeeeeeaaaeeeea net 2oma://oooommdad2/XXXXXXX.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 13461, 2020.

---

Published as a conference paper at ICLR 2025
Chen Dun, Mirian Hipolito Garcia, Guoqing Zheng, Ahmed Awadallah, Robert Sim, Anastasios Kyrillidis, and Dimitrios Dimitriadis. Fedjets: Efficient just-in-timg inst-in-time personalization with feder- ated mixture of experts. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
Cynthia Dwork.
Differential privacy. International colloquium on automata, languages, and programming, pp. 1-12. Springer, 2006.
Li Fei-Fei. Learning generative visual models from few training examples. In Workshop Generative Based Vision, IEEE Proc. CVPR, 2004, 2004.
Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo. Learning Federated Visual Prompt in Null Space for MRI Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8064-8073, 2023.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp- 2066-2073. IEEE, 2012.
Binbin Guo, Yuan Mei, Danyang Xiao, and Weigang Wu. Pil-moe: personalized federated learn- ing based on mixture of experts. In Web and Big Data: 5th International Joint Conference, APWeb- WAIM 2021, Guangzhou, China, August 23-25, 2021, Proceedings, Part pp. 480- 486. Springer, 2021.
Tao Guo, Song Guo, and Junxiao Wang. PFedPrompt: Learning Personalized Prompt for Vision- Language Models in Federated Learning. In Proceedings of the ACM Web Conference 2023, pp. 1364-1374, 2023a.
Tao Guo, Song Guo. Junxiao Wang, Xueyang Tang, and Wenchao Xu. PromptFL: Let federated participants cooperatively learn prompts instead of models-federatte learning in age of foundation model. IEEE Transactions on Mobile Computing, 2023b.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong. and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335-2999 29347, 2021.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.0633 2019.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang- Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 7865-7873, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lanoun-an representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint 0rXiv:2401.04088 2024.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint 1rXiv:1912.04977. 2019.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1911 19122, 2023a.
12

---

Published as a conference paper at ICLR 2025
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- -asuan Yang, and Fahad Shahbaz Khan. regulatgu Prompts: Foundational model adaptation with- out forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15 15190-15200, 2023b.
Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp. 794-797. IERE, 2020.
Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang, Weiping Ding, Chin-Teng Lin, and Ye Shi. FedTP: Federated Learning by Transformer Personalization. IEEE Transactions on Neural Net- works and Learning Systems, 2023.
Hongxia Li, Wei Huang, Jingya Wang. and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, 2024.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and sys- lems, 2:429-450, 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. International Conference on Machine Learning, pp. 6357- 6368. PMLR, 2021a.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Dou. FedBN: Federated learning on non-iid iid features via local batch normalization. arXiv preprint arXiv:2102.07623 202
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis- Philipp Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint 21X1v:2001.01523, 2020.
Jun Luo and Shandong Wu. Adapt to adaptation: Learning personalization for cross-silo federated learning. In IJCAI: proceedings of the conference, volume 2022, pp. 2166. NIH Public Access, 2022.
Jun Luo, Matias Mendicta, Chen Chen, and Shandong Wu. Pgfed: Personalize each client's global objective for federated learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelli- gence Review, 42:275-299 293, 2014.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-fffciene learning of deep networks from decentralized data. Artificial intelli- gence and statistics, pp. 1273-1282, PMLR, 2017.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
Bikang Pan, Wei Huang, and Ye Shi. Federated learning from vision- vision-language foundation models: Theoretical analysis and method. arXiv preprint arXiv:2409. 196100 2024.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1406-1415. 2019.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Matthias Reisser, Christos Louizos, Efstratios Gavves, and Max Welling. Federated mixture of experts. arXiv preprint arXiv:2107.06724 2021.
Anit Kumar Sahu, Tian Li, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, and Virginia Smith. On the convergence of federated optimization in heterogeneous networks. arXiv preprint
arXiv:1812.06122 3:3, 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quac Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gparsely-gated mixture-of-experts layer. 2017.
arXiv preprint
Shangchao Su, Mingzhao Yang, Bin Li, and Xiangyang Xue. Cross-domain federated adaptive prompt uning for clip. arXiv preprint 0rXiv:2211.07866 2022.
Guangyu Sun, Matias Mendieta, Jun Luo, Shandong Wu, and Chen Chen. Fedperfix: Towards partial model personalization of vision transformers in federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4988-4995 2023.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions Neural Networks and Learning Systems, 2022.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Luyuan Xie, Manqing Lin, Tianyu Luan, Cong Li, Yuejian Fang, Qingni Shen, and Zhonghai Wu. Mh-pflid: Model heterogeneous personalized federated learning via injection and distillation for medical data analysis. arXiv preprint arXiv:2405.000 06822, 2024.
Kai Yi, Nidham Gazagnadou, Peter Richtárik, and Lingjuan Lyu. Fedp3: Federated personalized and crivacy-friendl, network pruning under model heterogeneity. arXiv preprint :rXiv:2444009816, 2024a.
Liping Yi, Han Yu, Chao Ren, Heng Zhang, Gang Wang, Xiaoguang Liu, and Xiaoxiao Li. pfed- moe: Data-level personalization with mixture of experts for hodell-hteroggeeous personalized federated learning. arXiv preprint
2024b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16816-1666
2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348, 2022b.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mfxture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7 35:7103-7114 2022c.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A RELATED WORK
Personalized federated learning, The mediocre performance of conventional federated learning (FL) (McMahan et al. 2017) over heterogeneous data calls for a more customized solution. Person- alized federated learning (PFL.) (Tan et al. 2022 Kulkarni et al., (020), allowing tailored models for each client instead of a consensus model, systemically mitigates the data heterogeneity issuc. Var- ious strategies have been proposed to achieve personalization, including local fine-tuning (Cheng 2021), different optimization techniques such as attentive message passing and regularization a), and different personalized layers including feature extractor, classification head, attention, and batch normalization layers (Liang et al.) 2020; Arivazhagan et al., 2019 Sun al. 2023 Li et al., 2023 2021b). Some works use separate models learn global and local knowledge, respectively (Cai et al. 2024, Deng et al. 2020). Some recent PFL works also switch gear to model heterogeneity (Xie et al. 2024 Yi et al. 2024a). While promising address- ing data heterogeneity, these methods primarily ly focus on traditional machine learning models and do not leverage the capabilities of large pre-trained vision-language models.
et al.
(Huang et al. 2021 Li al.
Federated prompt learning for pre-trained models. Prompt learning for pre-trained models com- bines models' generalization capabilities and prompt learning's 's flexibility adapting these models to downstream tasks. As fundamental work for this combination, CoOp (Zhou et al. (022b) models prompt's context words with learnable vectors in CLIP. Such a combination has quickly drawn the attention of the FL community. For instance, PromptFL (Guo et al. 2023b) does F FedAvg (McMahan et al.) 2017) style global aggregation over the locally updated prompts. Build- ing upon this, FedPR (Feng et al. 2023) learns federated visual prompts for MRI reconstruction, while Fed-DPT (Su et al. 2022) applies both visual and textual prompt tuning to facilitate domain adaptation over decentralized data. pFedprompt (Guo et al. 2023a) integrating global consen- sus prompt and local attention over stored few-shot data for each client. FedOTP (Li al. (024) and FedPGP (Cui et al., 2024) employ Optimal Transport and prompt-wise contrastive loss, re- spectively, between global and local prompts, capturing diverse category traits per-client basis. Pan al. (2024) presents theoretical analysis framework for prompt-based federated learning of vision-language models. However, these approaches habitually adhere to the paradigm where lo- cally learned prompts are not allowed for inter-client sharing before aggregation. While sharing full models between clients would be impractical due to the massive communication costs O(N2 M) per round when N models of size M are distributed to N clients (Luo & Wu 2022 Luo et al. 2023) sharing prompts is a different scenario. Since prompts are orders of magnitude smaller than full models (as shown in Fig. [. the communication overhead becomes much more affordable. This makes it feasible for the inter-client sharing of agg-aggeeete prompts under federated settings.
Federated learning with Mixture of Experts. From the proprictary GPT-4 (Achiam et al.) 2023) )
to the open-sourced Mixtral of Experts 7x (Jiang et al. 2024), Qwen 1.5MMo. (Bai al. 2023),
Mixture of Experts (MoE) (Zhou et al. 2022c Masoudnia & Ebrahimpour 2014) is prevailing long after its initial proposal thanks the recent heat in Large Language Models (LLMs). Its application in FL, however, dates back before LLMs. FedMix (Reisser et al. [202]] and FedJETs (Dun et al.) 2023) allow each client to construct an MoE with a shared gating network, selecting specific non- local models more adaptive to the client's 's local data for ensembling, PFL-MoE (Guo et al., 202T) and PPedMoE (Yi et al. 20244b incorporate the global aggregated model as the global expert and its locally fine-tuned model as a local expert to each client, achieving personalized two-expert MoE. However, under federated settings, with too many experts on the client comes prohibitively high communication overhead (Reisser et al. 2021 Dun al. 2023) while too few experts impairs the benefit of employing MoE.
In contrast with the aforementioned methods, our proposed PFedMoAP, is a devised PFL method for prompt learning in CikP-like VLMs. sidesteps the impractical restriction on sharing the prompts without aggregation, while allowing mape-expert (e.g. 10 experts) MoE scenario for each client with negligible communication overhead, achieving efficient and effective personalization for the federated prompt learning.
15

---

Published as a conference paper at ICLR 2025
B FFEDMOAP: THE ALGORITHM
We provide the overall algorithm of pF eFedMoAP below.
Algorithm PFedMMAP
Input: N clients, learning rates 11. 72, number of rounds T,, logit coefficient X, CLIP image/text encoder g-). datasets {D; [ Output: Personalized prompts Pt. P...... PN. gating network weights 02,02..... ON.
ServerExecute:
1: Server initialize Oe and the pool of prompt experts Po empty set
2: Clients intialize 01, ....... 0N-
3: for t 1, do
4: Select a subset of [S|] clients, S,
5: for i E S, in parallel do
6: if Client i does not have an entry in the server-maintained pool, P, then standard=True)
P -CCientUdaate(P;
8: else
9:
Compute 2; by K nearest neighbor, given P = Pt.
10:
PNL = (Pr-ili)bes) // prompt in the pool from selected group of clients
11:
P. =ClientUpdatee ( PN // downloaded as non-local experts
P. P cache to pool 13: end if
12:
14: P =Eies, P.P.
15: end for
16: end for
17: return P.T P.. PN and 01, o...... ON
ClienUUpdate(P) (P Pov:=Noe, standard-False):
1: P + Pg
2: if standard then
3: client does standard fine-tuning
4: else
5:
for (xx: yk) D; do
6:
Tı- (P))
7:
TNL = =g((NN)
8:
In =((x))
9:
TMOE G(Ia, Ti, (,.)
10:
logit sim It. TMOo) + X sim Tz)
11:
P(ğ = c| = Sofimax(ogit,,)) // 7 is the temperature
12:
Lee Σ. Ui p(ū c(=a
(c)
13:
end for
end
15:
return p
C ADDITIONAL EXPERIMENTS
C.I DETAILS DATASET SETUP
We follow the most recent prompt-based FL for VLMs works' standard (Li et al. 2024 Cui et al. 2024 and use the datasets they have used. Tab. 5 lists the details of these datasets and the parti- tioning method used in our experiments for each dataset.
16

---

Published as a conference paper at ICLR 2025
Table 5: The statistics of each dataset and the partitioning details used in our experiments
Dataset
Training set size
Test set size
#Classes
#Chients
Sample Rate
Heterogeneity
Flowers 102
4,093
2,463
102
10
100%
Pathological
OxfordPets
2,944
3,669
100%
Pathological
Food101
50,500
30,300
101
10
100%
Pathological
Caltech101
4.128
2,465
100
10
100%
Pathological
DTD
2,820
1,692
47
10
100%
Pathological
Pllice-Caltech..
2,025
508
10
20
50%
Dir(0.3)
DomainNet
18,278
4,573
10
30
25%
Dir(0.3)
CIFARIO
50,000
10,000
100
10%
Dir(O.5)
CIFARI00
50,000
10,000
10%
Dir(0.5)
C.2 PFELMOAP UNDER DIFFERENTIAL PRIVACY
In PFedMoAP, clients remain unaware of the origin of non-local prompts, including which client they come from or the training round they were last updated in. Besides, the non-local prompts are downloaded after a KNN- based sparse selection process on the server. These factors collectively make it extremely difficult for client to infer the gradient associated with prompt or the data it has been trained with, which largely mitigates the privacy risk.
Table 6: Performance under (e, - privacy on CLIP datasets under pathological non-III setting.
Flowers102
OxfordPets
Food101
Caltech101
DTD
Without differential privacy (from Tab, T
PromptFL (Guo et al. 2023b
12.80=1.14
90.79=0.61
1..31#1.64
89.70+1.99
54.11=0.22
FromptF-FFedPeee rox et 2020b
66.40.10.29
89.2411.41
76.24.1..9
44.26.11.11 89.41
(FedMoXP(ours)
98.41+0.04
99.06+0.09
93.39++0.99
97.95+0.07
89.13+0.54
With differential privacy (c
50)
PromptFL (Guo et al. 2023b
67.07=0.60
88.05=0.32
77.41+0.60
84,83+00.2
38.39+1.25
FromptFL+FedPree Li Lietar al. 2020b
66.22:10.63
87.7810.61
77.27
84.6810.64
39.4311.111
(FedMoXP(ours)
98.34-+0.06
99.08 0.02
93.36++0.04
97.90+00.08
89.99+40449
With differential privacy
25)
PromptFL [Guo et al. 20236
64.25=1.100
86.2611.07
76.8410.66
85.0011.59
18.1960.66
PromptF1.. C1.ppFl.+FedFF. rox [L et al. 2020b
62.87-+0.99
8..82+0.47
7..21+0.64
+
0.55
3FedMoAP(ours)
98.36:0.12
99.02+0.04
93.41+0.13
97.9940..06 89.11
Additionally, we evaluate how (c, d)ddifferential privacy (DP) Dwork 2006 affects the per- formance of pFedMoAP on CLIP datasets and compare its performance with P PromptFL and propptL+FeddPro. This achieved by clipping the gradients to the uploaded prompts. We set the failure probability 8 to be 0.05 and report the accuracy with the privacy budget being set 50 and 25. The results in Tab. lab.ondemotttttte that while the accuracy of PromptFL and PromptFli FL++ edP rox significantly drop with privacy guarantees, DP's impact in accuracy performance to pF edMMAP remains minimal for different privacy budgets. This because the gating network in pF PFdMMAP is maintained locally for the entirety of the training process, and its multi-head attention layer learns proper transformations for the experts to align with the image features, mitigating the impact of the coise-contaminated prompts.
the prompts and adding noise
C.3 ATTENTION-BASEL VS. LINEAR PROJECTION-RASE GATING NETWORK
Table 7: Comparison between the proposed inteetiio-baseen gating network and the linear projection- based gating network
Flowcrs
OxfordPets
Food101
Caltech101
DTD
Linear projection mojeetion-hased based experts)
86.92+1.88
90.54+1.33
18.19+3.007
89,59
61.42+55.3
Linear projection-baser based (10 experts)
69.64+4.57
52.78=6.88
77.39+3.29
86.57+1.96
30.42+7.14
Autentio-bbaeed with aggregation Aitentionhbase. without aggregation (ours)
97.5611007
98.2410.12
91.8911..19
96.17.0.118
87.52.0..9
98.41+0.04
99.06+0.09
93.39++0.99
97.95+0.07
89.13++0.54
17

---

Figure 5: The impact of the number of experts on CIFAR10 with 100 clients Published as a conference paper at ICLR 2025
We compare the proposed attention-bant baatd gating network against the traditional linear projection- based gating network. The results are reported in Tab. 7 which strongly favor the attention- based approach. showing significantly better and more stable performance across all datasets. Notably, the linear projection-basee approach struggles with larger numbers of experts (10 experts vs experts), while the attention-based method maintains high performance. This is because the attention-based gating network 1) is more robust to adaptive experts; 2) serves as linear probing with more capacity; 3) leverages CLIP's feature alignment with attention mechanism; and 4) is agnostic experts' order (detailed strength of attention- ceantio---baedd approach over linear projection is described in Sec. 3.3)
In addition, we choose avoid aggregating the gating network due to its larger size. The gating network can be magnitudes larger than the prompts (see Tab. [9), which would largely increase the communication overhead. This design choice also aims for higher performance. As the deepest parameterized module of the entire model, the gating network should be fully "personalized" as opposed to "globalized" 2021). The process of aggregating the gating network limits the level of its personalization, which yields worse results than our method that keeps the gating network locally (see Tab.
achieve higher performance (Arivazhagan
al.] 2019: Collins et al.)
C.4 ADDITIONAL ABLATION STUDY
Table 8: Ablation study: impact of number of experts (K non-local experts 1 local experts) to pF edMMMAP on DomainNet
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Average
K + 1
47.20-0.30
46 88:0044
32.54.1042
37.67
31.50-10.65
36.09.10.92
38.63-10.29
K 1+11- 10
46.89-1.07
46.15:10.95
32.76.10:22
37.70
31.94=0.3
36.8411.21
38.7110045
<+11 15
47.36=1.12
46.29:0083
32.76:10.57
37.76:20.98
31.78=0.72
36.67+1.03
38.7740.38
20
47.49=0.64
46.73=0..1
32.74:0.844
37.16=0.34
31.02=0.99
37.67=0.72
38.80+10.11
K 25
47.56-0.92
1,389-1130
32.23++0.97
38.03=1.14
31.66=0.77
36.79+1.02
38.86++0.43
K + 30
47.56-0.87
46.64-+1.38
32.19+0.89
37.93+ 1.22
31.80+0.95
37.01+1.08
38.85++0043
Number of experts. We present an ablation study examining how the number of experts (K non- local experts + local expert) affects pF @FedMoAP's performance the DomainNet dataset. The results in Tab. show that the average performance gradually improves as the number of experts increases from 5 30. Fig. 5 plots the trend of performance with different numbers of experts from to 40 on CIFARIO. 10. For both datasets, it plateaus around larger numbers of experts, suggesting that while having multiple experts is beneficial, there's a point of diminishing returns. This is because the extra downloaded experts (selected last by our KNN-based expert selection mechanism) by client are trained from the data that is distinctly distributed from the client's local data distribution. This shows that the proposed method does not require an excessive number of experts to achieve high performance.
83.55
83.50
Accuracy (%)
83.40
03.35
83.30
83.25
5
10
15
20
25
30
35
40
Number Experts
18

---

Published as a conference paper at ICLR 2025
Table Ablation study: the impact of output dimension of CLIP backbone to PFedMMAP on CLIP datasets. For dieature < 1024, pooling layer is added after the dieature backbone to reduce the size of the gating network mentioned Sec. 3.3
1024 feature from the
Gating network size
Flowers102
OxfordPets
Food
Caltech101
DTD
dieatare 32
4.2K
97.28=0.118
98.75+00.2
93.42-0.08
97.37=0.08
88.61=0.89
diearare 64
16.6K
98.55+0.10
98.91+0.23
93.89=0.12
97.75+0.12
89.96=+0.09
dicature 128
66.0K
98.41+0.04
99.06++0.09
93.39-0.09
97.95+00077
89.13+0.54
dicature 256
263.2K
99.01+0.05
98.88+0.21
92.49-0.20
97.93++0.07
90.88+0.16
decature 512
1.IM
18.18-0..38
9.855+0..2
90.34=0.11
96.99:t.11
89.65:+0.10
decoture 1024
4.2M
98.111:..33
95.8111.84
89.20-0.37
96.82:10.26
89.03:1.114
Feature dimension. We investigate the impact of the output dimension (dreatur) of the CLIP back- bone on pF @FedMOAP's performance across CLIP datasets. The study shows that moderate feature dimensions (128-256) generally yield the best results, with dieature = 128 or 256 consistently per- forming well across all datasets. Lower feature dimensions (dieature = 32 or 64) have less capacity, failing to unlock the full potential of the pre-trained VLM. High dimensions (dieature 128 or 1024) significantly increase the size of the gating network. With less training data on a client, high dimen- sions lead to overparameterized gating network, resulting in slightly degraded performance. This suggests that a relatively compact feature dimension is sufficient and possibly preferable for the attention-based gating mechanism.
TIK
NeV-0014
9h
FS
LUU
reviae133
(a) Caltech101, 10 experts
(b) DTD, 10 experts
(c) Food101, 10 experts
neN 2049
11.26
1298
28.4%
1477
2304
112%%
naniic
50 loci
(d) Flowers102, 10 experts
(c) OxfordPets, 10 experts
(f) DomainNet, 20 experts
Figure 6: Contribution of the experts based on averaged attention across all test images. The first five charts are for CLIP datasets, for which there are 10 clients in each dataset. The last chart is for DomainNet with a total of 30 clients.
Visualization of contribution for each expert. To have a clear understanding of how the experts contribute the MoE text feature (TMoE). we visualize the contribution of the local and nonal experts towards it. This is achieved by taking the following steps. For each image in a client's 's test set, we compute the attention score (softmax of scaled dot product between the image and text features [Vaswani (017)) for each expert (local and non-local) through the multi-head attention layer in the proposed gating network. The scores from non-local experts are sorted a descending manner. All scores are averaged over the test set across all clients. We visualize the averaged scores under 6 datasets in Fig. From this figure, we can see that 1) every expert has a non -riviri contribution to TMOE, the text feature generated from the MoE, indicating that each expert is indispensable in the whole design; 2) as a result of the minimization, the contribution from the local experts to the MoE not the highest. This is reasonable since besides the T' TMME, Eq. (10's second term optimizes the
19

---

Published as a conference paper at ICLR 2025
text feature from the local expert itself (Tz) based on the image feature, which directly contributes to the loss; and 3) although not the highest, the contribution from the local expert to the MoE is still among the highest ones as it is the only expert that is updated during local training with fixed non-local prompts.
20


Reviews:
[{"review_id": "kgZdXySLA2", "reviewer": "Reviewer_eFjf", "summary": "This paper introduces pFedMoAP that enables effective federated prompt learning for vision-language models like CLIP. The key innovation is allowing clients to download multiple pre-aggregated prompts as fixed non-local experts rather than being restricted to a single globally aggregated model.", "questions": "Refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe method is novel which allows the client download prompts from others.\n2.\tThe experiments are extensive and show good performance.", "weaknesses": "1.\tIt is very similar to the prompt learning framework. How does your method differ from the classic prompt learning framework if you can directly access prompts from other clients?\n2.\tI noticed that similar work, like pFedPrompt, conducted experiments on large-scale datasets like UCF-101 or ImageNet. Would the scale of the dataset influence the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKrjMdyTXS", "reviewer": "Reviewer_zSUv", "summary": "This paper proposes a lightweight federated prompt learning framework to personalize the prompt learning process through the lens of mixture of experts (MoE). An attention-based gating network is also introduced to achieve efficient cross-client knowledge sharing. Experiments indicate that the proposed **pFedMoAP** performs better than state-of-the-art methods.", "questions": "1. In Section 4.1 **Implementation details.**, for CIFAR10&CIFAR100, why the participation rate is 10%? Does it mean only one client will be selected to join the training process in each communication round?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. Implementing prompt learning into the distributed environment is an important topic for FL applications due to its efficiency in computation and communication.\n\n2. The proposed learning framework is simple and effective.", "weaknesses": "1. The quality of the writing is poor. The expression of the paper is somehow obscure, and not concise enough. Besides, there are many typos in the paper, e.g.,  lacking space between two adjacent words in Line 27, Line 198, Lin 415, and Line 469. \n\n2. The novelty is limited. The **pFedMoAP** seems like a naive combination of PromptFL and MoE. Although the selected non-local experts will be updated in each communication round, the intuition behind it is also not explained well, leading to weak convincingness. \n\n3. it is unclear why the attention-based gate network does not need to be uploaded to the server for aggregation. Please clarify.\n\n4. I think the ablation study of the paper is not enough. \n- To study the effectiveness of MoE-based textual feature enhancement mechanism, the authors should add experiments under different values of $K$ to confirm performance changes.\n\n- The author implements a dimension reduction operation in Section 3.3, please add experiments to show whether the performance will be influenced.   \n\n- How about a larger number of clients, e.g., 100 or more?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Kw6jJjIo8", "reviewer": "Reviewer_CcjC", "summary": "This paper proposes a Mixture of Experts (MoE) approach for prompt-based federated learning, employing a local attention-based gating network to refine text features for better alignment with local image data. It also highlights the lightweight design of the prompt, exploring a configuration where the prompt is updated directly by other clients rather than relying on an aggregated prompt from the server.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The integration of the MoE method into prompt-based federated learning is an innovative concept.\n\n2. The paper is well-organized and effectively communicates its main contributions. The methodology is clearly explained, particularly regarding the incorporation of the MoE approach in the federated learning context.", "weaknesses": "1. While this approach utilizes inter-client sharing, it could expose prompt updates directly to other clients, which may lead to privacy concerns as prompt updates could be tracked, making the model susceptible to certain attack algorithms. \n\n2. In the related work section in the appendix (Line 783), the authors mentioned that previous work utilized inter-client sharing prior to aggregation. However, these works allow prompt aggregation in the server and do not share local prompts with other clients. This sentence is somewhat unclear; please provide a clearer version to avoid misunderstandings. \n\n3. The paper claims that the MoE-based method can leverage prompts from other clients. Please provide experimental evidence to show whether your method’s effectiveness is sensitive to the total number of clients and the value of $K$. \n\n4. Some related works are encouraged to be properly cited in Related work. For example, personalized federated learning [a,b] and prompt-based federated Learning [c]. \n\n[a] Li, et al, FedTP: Federated Learning by Transformer Personalization, TNNLS 2023. \n\n[b] Cai, et al, Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning, NeurIPS 2023. \n\n[c] Pan et al, Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method, NeurIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tq43xPLaTk", "reviewer": "Reviewer_iqPq", "summary": "This work introduces a new paradigm in federated prompt learning for pre-trained Vision-Language Models (VLMs) such as CLIP, challenging the traditional restriction that clients are limited to downloading a single globally aggregated model. This shift is particularly beneficial for lightweight prompts, enabling more flexible and effective knowledge sharing among clients.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. pFedMoAP proposes a novel framework that personalizes the prompt learning process by allowing clients to download multiple pre-aggregated prompts as fixed non-local experts. This leverages the concept of Mixture of Experts, where a local attention-based gating network is implemented to generate enhanced text features that better align with the local image data on the client side.\n\n2. By facilitating a many-expert scenario (e.g., 10 experts) per client with negligible communication overhead, pFedMoAP achieves efficient and effective personalization. This approach overcomes the limitations of having too many or too few experts, which can lead to high communication costs or suboptimal performance, respectively.\n\n3. The algorithm consistently outperforms existing work across various heterogeneous federated settings, as demonstrated through extensive experiments on 9 datasets.", "weaknesses": "1. Determining which prompt experts should be included in the pool and when to update or replace them requires careful management. If the selection process is not optimal, it could lead to suboptimal performance. Additionally, maintaining a diverse and up-to-date pool of experts is crucial but can be complex.\n\n2. The local attention-based gating network, while designed to be efficient, may still introduce optimization challenges. Ensuring that this network converges to a solution that effectively combines local and non-local prompts is not trivial, especially when dealing with heterogeneous data distributions.\n\n2.  As the number of clients increases, managing a dynamic pool of prompt experts and maintaining efficient communication between the server and all clients can become more challenging. The complexity of coordinating updates and ensuring that each client receives the most relevant prompts could grow significantly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)