Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models

## Abstract

Abstract Prompt learning for pre-trained Vision-Language Models (VLMs) like CLIP has demonstrated potent applicability across diverse downstream tasks.
This lightweight approach has quickly gained traction from federated learning (FL) researchers who seek to efficiently adapt VLMs to heterogeneous scenarios.
However, current federated prompt learning methods are habitually restricted to the traditional FL paradigm, where the participating clients are generally only allowed to download a single globally aggregated model from the server. While justifiable for training full-sized models under federated settings, in this work, we argue that this paradigm is ill-suited for lightweight prompts. By facilitating the clients to download multiple pre-aggregated prompts as fixed non-local experts, we propose P ersonalized Fed erated M ixture of A daptive P rompts ( pFedMoAP ), a novel FL framework that personalizes the prompt learning process through the lens of Mixture of Experts (MoE). pFedMoAP implements a local attention-based gating network that learns to generate enhanced text features for better alignment with local image data on the client, benefiting from both local and downloaded non-local adaptive prompt experts. The non-local experts are sparsely selected from a server-maintained pool, fostering collaborative learning across clients.To evaluate the proposed algorithm, we conduct extensive experiments across 9 datasets under various heterogeneous federated settings. The results show that pFedMoAP consistently outperforms the state-of-the-art alternatives, underscoring its efficacy in personalizing prompt learning for CLIP within the federated learning paradigm.

## 1 Introduction

Recent years have witnessed the prosperity of federated learning (FL) (McMahan et al., ; Kairouz et al., ; Li et al., ) as a potent paradigm for training machine learning models across decentralized data sources. While offering a privacy-preserving solution in collaborative training settings, this approach faces a critical challenge in the form of heterogeneous data distribution across the participating clients (Li et al., ). And due to the data heterogeneity, the generalization capabilities of the trained models may be compromised (Li et al., ; Sahu et al., ). In this context, the pre-trained Vision-Language Models (VLMs) such as CLIP (Radford et al., ) and ALIGN (Jia et al., ) can provide a significant impact with their remarkable applications in learning transferable representations across a wide range of tasks and disparate data distributions encountered in federated settings.

However, applying VLMs in FL contexts is not without challenges. Due to their millions of parameters, fine-tuning these large-scale models in federated settings incurs prohibitively high communication overhead, making it impractical for many real-world applications. This limitation has led researchers to explore more efficient adaptation techniques for VLMs, with prompt learning emerging as a promising approach to overcome the communication bottleneck.

Prompt learning for VLMs (Zhou et al., ), originally proposed to eliminate the need for hand-crafted prompts, replaces the context words with small-scale learnable vectors while keeping the pre-trained model fixed. Such a lightweight approach particularly benefits FL by tremendously reducing the communication overhead associated with transmitting the entire fine-tuned model. For instance, PromptFL (Guo et al., ) leverages a FedAvg (McMahan et al., ) style aggregation of the locally trained prompts. FedPR (Feng et al., ) learns the visual prompt within the null space of the global prompt. In addition, recent criticism on prompt-based VLMs to unseen data distributions (Zhou et al., ; Khattak et al., ; ) also facilitates FL researchers to combine prompt learning with personalized FL (PFL). Instead of restricting to the global consensus model, PFL (Tan et al., ; Kulkarni et al., ) allows tailored models for each individual client, systemically mitigating the data heterogeneity issues and enhancing the overall flexibility of the system. However, directly applying PFL techniques, such as local fine-tuning (Cheng et al., ) and personalizing specific layers (Arivazhagan et al., ), to prompt learning has demonstrated limited efficacy in training the prompt (Guo et al., ) under extreme data heterogeneity. Consequently, some works tailor PFL techniques to prompt learning in CLIP-like VLMs. In this recently emerged field, Guo et al. () trains a global consensus prompt with personalized visual attention modules on locally memorized data. Li et al. () leverages unbalanced Optimal Transport to align visual feature maps with personalized prompts. More details on prior works are presented in [Sec. 2](#S2).

Figure: (a) Model parameter counts
Refer to caption: /html/2410.10114/assets/figs/parameter-counts.png

Unfortunately, existing federated prompt learning approaches habitually adhere to the paradigm of traditional FL/PFL techniques, wherein the participating clients are generally only allowed to download a single globally aggregated model from the server. While justifiable for full-sized models, we argue that this paradigm is ill-suited for federated prompt learning, as the significantly reduced size of prompts substantially mitigates the potential communication overhead. To contextualize this disparity, consider that while a typical ResNet50 model contains approximately 25.6 million parameters, a learnable prompt with length 16 and dimension 512 comprises merely $16\times 512=8,192$ parameters — a reduction of three orders of magnitude.
Given this significant parameter reduction, restricting clients to downloading only a single globally aggregated model — as in traditional FL/PFL — unnecessarily limits federated prompt learning. The lightweight nature of prompts provides a unique way for more flexible and effective personalization strategies.
Therefore, this motivates us to address the following pivotal question:

How can we devise a personalized federated learning framework, tailored for prompt learning in CLIP-like VLMs, while fully exploiting the lightweight nature of the prompts?

In light of these challenges and opportunities, we propose a novel framework: Personalized Federated Mixture of Adaptive Prompts (pFedMoAP). Tailored specifically for prompt learning in CLIP-like VLMs, our proposed framework aims to unleash the potential of the lightweight prompt by allowing the clients to download multiple pre-aggregated prompts (see [Fig. 1(b)](#S1.F1.sf2)) to acquire collective knowledge. In this manner, we enhance the generalization capabilities of the PFL system under extreme data heterogeneity, offering a more flexible and effective solution that fully leverages the lightweight nature of prompts in federated settings.

At its core, pFedMoAP personalizes the federated prompt learning problem through the lens of Mixture of Experts (MoE), treating all locally updated prompts as specialized experts. Benefiting from lifting the aforementioned ill-suited restriction, pFedMoAP facilitates the sharing of pre-aggregated prompts between clients (through the server).
In addition, our proposed framework implements a novel client-specific, parameter-efficient, attention-based gating network that learns to generate enhanced text features for better alignment with local image data on each client. Through this locally trained gating network, the enhanced text features are generated from the adaptive local and non-local prompt experts via CLIP’s text encoder. The local expert is trained exclusively with the client’s data, while non-local experts, trained on other clients, are sparsely selected from a server-maintained pool based on $K$ nearest neighbors (KNN), and shared with the client without aggregation, fostering collective knowledge sharing across clients beyond the global aggregation.

We summarize our main contributions as follows:

- •
We pioneer a paradigm shift in federated prompt learning for VLMs by challenging the restriction on sharing pre-aggregated prompts between clients. Circumventing this ill-suited restriction for prompt learning unlocks more potential inherent in the lightweight prompts, facilitating more effective cross-client knowledge sharing. As such, we pave the way for more flexible federated prompt learning in VLMs.
- •
We propose pFedMoAP, a novel framework designed specifically for personalizing federated prompt learning in CLIP-like VLMs under data heterogeneity for image recognition tasks. pFedMoAP personalizes the prompt learning with a unique attention-based gating network. Thanks to its flexibility, this gating network has the potential to extend beyond federated learning for prompt-based VLMs.
- •
We validate the effectiveness of the proposed pFedMoAP through extensive experiments and ablation studies across 9 widely adopted datasets under various federated settings. The results verify the superiority of pFedMoAP over compared state-of-the-art methods.

## 2 Related work

Personalized federated learning.
The mediocre performance of conventional federated learning (FL) (McMahan et al., ) over heterogeneous data calls for a more customized solution. Personalized federated learning (PFL) (Tan et al., ; Kulkarni et al., ), allowing tailored models for each client instead of a consensus model, systemically mitigates the data heterogeneity issue. Various strategies have been proposed to achieve personalization, including local fine-tuning (Cheng et al., ), different optimization techniques such as attentive message passing and regularization (Huang et al., ; Li et al., ), and different personalized layers including feature extractor, classification head, attention, and batch normalization layers(Liang et al., ; Arivazhagan et al., ; Sun et al., ; Li et al., ).
Some recent PFL works also switch gear to model heterogeneity (Xie et al., ; Yi et al., ). While promising in addressing data heterogeneity, these methods primarily focus on traditional machine learning models and do not leverage the capabilities of large pre-trained vision-language models.

Federated prompt learning for pre-trained models.
Prompt learning for pre-trained models combines models’ generalization capabilities and prompt learning’s flexibility in adapting these models to downstream tasks. As a fundamental work for this combination, CoOp (Zhou et al., ) models a prompt’s context words with learnable vectors in CLIP. Such a combination has quickly drawn the attention of the FL community. For instance, PromptFL (Guo et al., ) does a FedAvg (McMahan et al., ) style global aggregation over the locally updated prompts. Building upon this, FedPR (Feng et al., ) learns federated visual prompts for MRI reconstruction, while Fed-DPT (Su et al., ) applies both visual and textual prompt tuning to facilitate domain adaptation over decentralized data. pFedprompt (Guo et al., ) integrating global consensus prompt and local attention over stored few-shot data for each client. FedOTP (Li et al., ) and FedPGP (Cui et al., ) employ Optimal Transport and prompt-wise contrastive loss, respectively, between global and local prompts, capturing diverse category traits on a per-client basis. However, these methods habitually adhere to the
restriction for locally learned prompts from inter-client sharing before aggregation. For full-sized models, this inter-client sharing is evidently prohibitive, as it introduces $O(N^{2}\cdot M)$ communication cost per round if each of the $N$ models with size $M$ are shared to $N$ clients (Luo & Wu, ; Luo et al., ). For the prompt, however, $M$ can be orders of magnitude smaller (see [Fig. 1](#S1.F1)), which substantially lowers the communication cost and frees federated prompt learning from this ill-suited restriction.

Federated learning with Mixture of Experts.
From the proprietary GPT-4 (Achiam et al., ) to the open-sourced Mixtral of Experts 8$\times$7B (Jiang et al., ), Qwen1.5-MoE (Bai et al., ), Mixture of Experts (MoE) (Zhou et al., ; Masoudnia & Ebrahimpour, ) is prevailing long after its initial proposal thanks to the recent heat in Large Language Models (LLMs). Its application in FL, however, dates back before LLMs. FedMix (Reisser et al., ) and FedJETs (Dun et al., ) allow each client to construct an MoE with a shared gating network, selecting specific non-local models more adaptive to the client’s local data for ensembling. PFL-MoE(Guo et al., ) and pFedMoE Yi et al. () incorporate the global aggregated model as the global expert and its locally fine-tuned model as a local expert to each client, achieving a personalized two-expert MoE. However, under federated settings, with too many experts on the client comes prohibitively high communication overhead (Reisser et al., ; Dun et al., ) while too few experts impairs the benefit of employing MoE.

In contrast with the aforementioned methods, our proposed pFedMoAP, is a devised PFL method for prompt learning in CLIP-like VLMs. It sidesteps the impractical restriction on sharing the prompts without aggregation, while allowing a many-expert (e.g. 10 experts) MoE scenario for each client with negligible communication overhead, achieving efficient and effective personalization for the federated prompt learning.

## 3 Preliminaries

### 3.1 Personalized federated learning

Conventional federated learning (FL) aims to train a global consensus model for a federation of clients with similar data. As the most notable FL algorithm, FedAvg (McMahan et al., ) minimizes the global objective over $N$ clients defined as:

$$ $\min_{{\bm{\theta}}}F({\bm{\theta}})=\sum_{i=1}^{N}p_{i}F_{i}({\bm{\theta}}),$ (1) $$

where ${\bm{\theta}}$ and $F_{i}(\cdot)$ represents the global model and the local objective of client $i$, respectively, and the weight $p_{i}$ is often set as $p_{i}=n_{i}/n$ with $n=\sum_{i}n_{i}$ where $n_{i}$ denotes the number of data samples on client $i$. In FedAvg, the local objective $F_{i}(\cdot)$ measures client $i$’s empirical loss, $\sum_{k=1}^{n_{i}}{\mathcal{L}}_{i}({\bm{\theta}}|({\bm{x}}_{k},y_{k}))$, where ${\mathcal{L}}_{i}$ represents its loss and ${\bm{x}}_{k}$ is its $k$-th data sample with ground truth label $y_{k}$.

Compared to conventional FL, personalized FL (PFL) relaxes the number of models where each client $i$ is allowed to have its tailored model ${\bm{\theta}}_{i}$. The goal of PFL is, therefore, defined as:

$$ $\min_{{\bm{\theta}}_{1},...,{\bm{\theta}}_{N}}F({\bm{\theta}}_{1},...,{\bm{\theta}}_{N})=\min_{{\bm{\theta}}_{1},...,{\bm{\theta}}_{N}}\sum_{i=1}^{N}p_{i}F_{i}({\bm{\theta}}_{i}).$ (2) $$

### 3.2 Prompt learning for CLIP-like VLMs

To efficiently adapt pre-trained CLIP-like VLMs to downstream tasks, prompt learning methods (Zhou et al., ; ) model a prompt’s context words with learnable vectors. While zero-shot transfer of CLIP leverages the fixed word embedding ${\bm{W}}=\{{\bm{w}}_{1},...,{\bm{w}}_{l}\}$ from hand-crafted prompt templates with $l$ context words (e.g. “a photo of a $<$class$>$.”), prompt learning replace it with the learnable prompt ${\bm{P}}=\{{\bm{p}}_{1},...,{\bm{p}}_{l}\}\in\mathbb{R}^{l\times d}$ where $d$ is the dimension of the word embedding. The full prompt ${\bm{P}}^{(c)}$, consisting of the learnable prompt ${\bm{P}}$ with the embedding of a class label $c$, is then fed into the fixed text encoder, $g(\cdot)$. Together with the fixed image encoder $f(\cdot)$, the classification logit for class $c$ is then computed as a matching score between the image and text features. And the prediction probability of each class for image ${\bm{x}}$ is derived by taking a Softmax, controlled by temperature $\tau$, over the logits:

$$ $\text{logit}^{(c)}=\text{sim}(f({\bm{x}}),g({\bm{P}}^{(c)})),$ (3) $$

$$ $p(\hat{y}=c|{\bm{x}})=\frac{\exp(\text{logit}^{(c)})/\tau)}{\sum_{k=1}^{C}\exp(\text{logit}^{(k)}/\tau)},$ (4) $$

where $\text{sim}(\cdot)$ denotes a metric function (e.g. cosine similarity), $\hat{y}$ is the predicted label, and $C$ denotes the number of classes. With the probabilities, we can learn the prompt ${\bm{P}}$ by minimizing the cross-entropy loss.

### 3.3 Mixture of Experts

With a similar form to FL’s, Mixture of Experts (MoE) aggregates the output of multiple experts or trained models, instead of the model parameters. For a general MoE system with $N$ experts, the output of MoE with input $x$ is defined as:

$$ $MoE({\bm{x}})=\sum_{i=1}^{N}G({\bm{x}})_{i}\cdot E_{i}({\bm{x}}),$ (5) $$

where $E_{i}(\cdot)$ represents the $i$-th expert, and $G(\cdot)_{i}$ represents the weight assigned to the output of expert $i$ from a $N$ dimensional vector, based on its input. The mechanism for weight assignment is often controlled by a network called gating network (also known as a router). The implementation of the gating network varies (Clark et al., ; Hazimeh et al., ; Zhou et al., ), but a simple yet effective one is implemented by taking the Softmax over top-$K$ ($K\leq N$) logits of a linear layer with the remaining $N-K$ weights set to zero (Shazeer et al., ). We propose a novel lightweight attention-based gating network devised for CLIP-like VLMs that extends beyond federated learning. Details are presented in [Sec. 4.3](#S4.SS3).

## 4 Personalized Federated Mixture of Adaptive Prompts

Motivation. While federated prompt learning for pre-trained CLIP-like VLMs offers an approach to efficiently adapt these models to downstream tasks under federated settings, such methods mostly lack personalization for the prompt to generalize with extreme data heterogeneity. Prior PFL methods devised for prompt learning habitually adhere to the restriction for the clients to download only a single globally aggregated model from the server, failing to leverage the lightweight nature of the prompt to the fullest. In addition, from an MoE perspective, allowing clients to download pre-aggregated prompts trained on other clients naturally provides a systemic solution to the dilemma where too many experts on the client incurs prohibitively high communication overhead while too few experts impairs the benefit of employing MoE. Consequently, there is a need for a tailored PFL approach for prompt learning in CLIP-like VLMs that offers the flexibility to share trained prompts from an MoE perspective.

Overview. The rest of this section presents the details of the pFedMoAPby gradually building upon PFL with prompt learning and mixture of adaptive prompts, and how the proposed attention-based gating network works in pFedMoAP, as well as potentials in extending it beyond federated learning.

### 4.1 Personalized federated prompt learning with local prompt only

In our study, we presume each client hosts a CLIP model with fixed image encoder $f(\cdot)$ and text encoder $g(\cdot)$ for image recognition tasks. We first suppose, in this subsection, that each client personalizes the local prompt without the MoE. At federated round $t$, each client $i$ in the selected set of participating client ${\mathcal{S}}_{t}$ locally updates its prompt, ${\bm{P}}_{i}^{t}$, initialized with the global prompt from the last round, ${\bm{P}}_{g}^{t-1}$. The prompt is then updated through a gradient-based optimization, e.g. Stochastic Gradient Descent (SGD), over a cross-entropy loss for multiple local epochs.

After finishing the local update, the learned prompts ${\bm{P}}_{i}$ for all the clients in ${\mathcal{S}}_{t}$ are then aggregated by the server for a global prompt ${\bm{P}}_{g}^{t}$ in a FedAvg manner:

$$ ${\bm{P}}_{g}^{t}=\sum_{i\in{\mathcal{S}}_{t}}\frac{n_{i}}{\sum_{k\in{\mathcal{S}}_{t}}n_{k}}{\bm{P}}_{i}^{t}.$ (6) $$

While ${\bm{P}}_{g}^{t}$ is used for the next round of training, for client $i$, ${\bm{P}}_{i}^{t}$ is stored and used for inference purposes, achieving a simple personalization for federated prompt learning while MoE is not present.

Figure: Figure 2: Workflow of pFedMoAP at client $i$. The client first computes the non-local text features using the non-local prompt experts. As training progresses, it then calculates the local text features. Taking class 3 as an example, both local and non-local text features are input into the attention-based gating network as both key and value, while image features serve as the query. This process generates enhanced text features. Matching socres are derived from two sources: local text features and MoE-enhanced text features. These scores are then combined through weighted averaging to produce the final logits.
Refer to caption: /html/2410.10114/assets/figs/workflow.png

### 4.2 Personalized Federated Mixture of Adaptive Prompts workflow

Thanks to the lightweight nature of the prompt, clients are enabled to acquire collective knowledge beyond the globally aggregated prompt. In pFedMoAP, we facilitate, rather than discourage, each client to download $K$ pre-aggregated prompts trained on other clients as non-local client experts.
These non-local experts are sparsely selected from a server-maintained pool of prompt experts, ${\mathcal{P}}$. The pool functions as a dynamic repository, refreshing at the conclusion of each federated round. It incorporates the newly acquired prompts from the clients who participated in the current training round, and overwrites their previous entries (if existing) in the pool, i.e:

$$ ${\mathcal{P}}_{t}={\mathcal{P}}_{t-1}-\{{\bm{P}}_{i}^{t-1}\}_{i\in{\mathcal{P}}_{t-1}\cap{\mathcal{S}}_{t}}+\{{\bm{P}}_{j}^{t}\}_{j\in{\mathcal{S}}_{t}}.$ (7) $$

At the beginning of the current round $t$, each client $i\in{\mathcal{S}}_{t}$ is first assigned $K$ non-local experts from the previous round’s pool, ${\mathcal{P}}_{t-1}$. This assignment is based on a $K$ nearest neighbors (KNN) algorithm, which identifies the top-$K$ nearest experts from client $i$’s entry in the pool, in terms of $l_{2}$ distance(^1^11The first-time participants with no entry in the pool conduct a standard local training of the global prompt, and upload its trained prompt to the pool.). the rationale behind the KNN-based expert assignment is that clients with similar locally trained prompts are more likely to share comparable data distributions. This similarity-based expert assignment aims to leverage knowledge from clients with potentially similar data characteristics, enhancing the personalization from closer collaborators in terms of local data distributions.

While such a server-side assignment of the non-local experts inherently simulates the sparse gating in MoE, a client-side gating network, $G_{i}(\cdot)$, enables dynamic dense incorporation of the output from all selected experts on a per-image basis (see [Fig. 2](#S4.F2)).

Let ${\mathcal{Q}}_{i}=\{NL_{j}\}_{j=1}^{K}$ ($NL$ represents “$N$on-$L$ocal”) be the set of clients assigned to client $i$, whose prompts, ${\bm{P}}_{NL_{j}}$, are served as non-local experts on client $i$. Before the local training of client $i$ initiates, text features for every non-local prompt are computed via the CLIP’s fixed text encoder, $g(\cdot)$. Note that since the non-local prompt experts are kept fixed throughout the local training, these text features are only necessary to be computed once. Let ${\bm{T}}_{NL_{j}}^{(c)}$ be the text features for class $c$ generated by the $j$-th non-local prompt expert on client $i$ with class $c$’s embedding, ${\bm{P}}_{NL_{j}}^{(c)}$. At the beginning of the local training on client $i$, the text features, ${\bm{T}}_{NL}^{(c)}$, from every non-local expert for every class $c\in[C]$, formulated below, are fixed and ready to use for the entire local training process:

$$ $\forall c\in[C],\ {\bm{T}}_{NL}^{(c)}\overset{\Delta}{=}\{{\bm{T}}_{NL_{j}}^{(c)}|{\bm{T}}_{NL_{j}}^{(c)}=g({\bm{P}}_{NL_{j}}^{(c)}),\ \forall NL_{j}\in{\mathcal{Q}}_{i}\}.$ (8) $$

Figure: Algorithm 1 pFedMoAP

In pFedMoAP, a novel attention-based gating network, $G(\cdot|{\bm{\theta}}_{i})$ parameterized by ${\bm{\theta}}_{i}$, is proposed. Unlike the $G(\cdot)$ in [Eq. 5](#S3.E5), the proposed gating network functions to generate a mixture of the outputs from the experts, instead of the weights for the outputs in traditional MoE. The weights, however, are still internally computed as intermediate products through the attention mechanism. We discuss the gating network in details in [Sec. 4.3](#S4.SS3). Let ${\bm{I}}_{k}=f({\bm{x}}_{k})$ be the computed image feature of the input image ${\bm{x}}_{k}$ through CLIP’s fixed image encoder $f(\cdot)$. In pFedMoAP, for each class $c\in[C]$, what the gating network, $G(\cdot|{\bm{\theta}}_{i})$, takes as input are three-fold: 1) image feature, ${\bm{I}}_{k}$; 2) text features from the locally updated prompt expert ${\bm{T}}_{L}^{(c)}=g({\bm{P}}_{i}^{(c)})$ where $L$ represents “$L$ocal”; and 3) text features from the all non-local experts, ${\bm{T}}_{NL}^{(c)}$ defined in [Eq. 8](#S4.E8). The gating network then outputs the enhanced text feature, ${\bm{T}}_{MoE}^{(c)}$, better aligned with the image feature ${\bm{I}}_{k}$ for more generalizability that incorporates both collective knowledge and personalization, i.e.,

$$ $\forall c\in[C],\ {\bm{T}}_{MoE}^{(c)}\overset{\Delta}{=}G({\bm{I}}_{k},{\bm{T}}_{L}^{(c)},{\bm{T}}_{NL}^{(c)}|{\bm{\theta}}_{i}).$ (9) $$

Note that although the gating network could technically be parameterized in a class-specific manner, our implementation opts to use a shared set of parameters across all classes for parameter efficiency.

As the final step, pFedMoAP computes the logits based on the matching score between the image feature ${\bm{I}}_{k}$ and text features from two sources, namely ${\bm{T}}_{MoE}^{(c)}$ from the MoE, and ${\bm{T}}_{L}^{(c)}$ from the local expert. Although ${\bm{T}}_{MoE}^{(c)}$ carries the global collective knowledge, we further address ${\bm{T}}_{L}^{(c)}$ with weight $\lambda$ as the local prompt ${\bm{P}}_{i}$ is the only learnable expert on the client, i.e.,

$$ $\forall c\in[C],\ \text{logit}^{(c)}=\text{sim}({\bm{I}}_{k},{\bm{T}}_{MoE}^{(c)})+\lambda\cdot\text{sim}({\bm{I}}_{k},{\bm{T}}_{L}^{(c)}).$ (10) $$

With the classification logits, the local prompt ${\bm{P}}_{i}$ and the parameter ${\bm{\theta}}_{i}$ in the gating network are updated through optimizing over the cross-entropy loss based on the prediction probability computed as in [Eq. 4](#S3.E4). After the local training, client $i$ upload ${\bm{P}}_{i}$ to the server while maintaining the gating network locally.

### 4.3 Attention-based gating network: in pFedMoAP and beyond

To generate enhanced text features that better aligns with the image feature ${\bm{I}}_{k}$, the proposed gating network employs a multi-head attention (MHA) layer (Vaswani, ) with the query being the ${\bm{I}}_{k}$, while the key and the value are both text features ${\bm{T}}_{L}^{(c)},{\bm{T}}_{NL_{1}}^{(c)},{\bm{T}}_{NL_{2}}^{(c)},...,{\bm{T}}_{NL_{K}}^{(c)}$. However, since the dimension of these features are usually large, resulting in too many parameters in the MHA layer (e.g. for CLIP with a ResNet50 backbone, $d_{\text{feature}}=$ 1024, MHA layer will have 4.2M parameters), we force each feature to first go through a pooling layer to reduce the dimension to a fixed $d_{\text{gating}}=128$ (MHA layer will only have 66.0K parameters). Therefore, with $Q=\text{Pooling}({\bm{I}}_{k}),K=V=\text{Pooling}({\bm{T}}_{L}^{(c)},{\bm{T}}_{NL}^{(c)})$ and $h$ being then number of heads, we have:

$$ ${\bm{T}}_{MoE}^{(c)}=G({\bm{I}}_{k},{\bm{T}}_{L}^{(c)},{\bm{T}}_{NL}^{(c)}|{\bm{\theta}}_{i})=\text{MHA}(Q,K,V)=\text{Concat}(\text{head}_{1},...,\text{head}_{h})W^{O},\\$ (11) $$

where $\text{head}_{q}=\text{Attention}(QW_{q}^{Q},KW_{q}^{K},VW_{q}^{V})$ and $W^{O},W_{q}^{Q},W_{q}^{K},W_{q}^{V}$ are standard MHA parameters in ${\bm{\theta}}_{i}$. As ${\bm{T}}_{MoE}^{(c)}\in\mathbb{R}^{d_{\text{gating}}}$, computing $\text{sim}({\bm{I}}_{k},{\bm{T}}_{MoE}^{(c)})$ requires pooling on ${\bm{I}}_{k}$ as well.

While a linear projection-based gating network, $G_{\text{proj}}(\cdot)$, in a typical MoE is also able to achieve enhanced text features for CLIP (e.g. by assigning weights $G_{\text{proj}}({\bm{x}}_{k})\in\mathbb{R}^{K+1}$ for the experts’ output based on the image input (Shazeer et al., )), the proposed attention-based gating network is more favorable for the following FL-agnostic reasons. 1) Since the experts are constantly adapting during the training process, directly learning the assigned weights for the experts, as in projection-based gating, will be less robust than learning the relationship between the image feature and the text features generated by the experts, as in attention-based gating. 2) The output of the proposed attention-based gating is the enhanced text features, which makes the MHA layer serve as a lightweight linear probing with a much larger search space, that functions beyond assigning $K+1$ weights to aggregate the experts’ output as a linear combination in a $K+1$-dimensional search space. Even with a redesigned projection layer to directly produce the enhanced text features, the parameter count of such a projection-based gating network would substantially exceed that of the attention-based gating network. 3) A projection-based gating network fails to leverage the pre-trained CLIP’s powerful capability to align text features with image features, while the attention mechanism naturally harnesses it through the scaled dot-product operation. 4) Even if we feed both image and text features from multiple experts into a projection-based gating network, the order of these input would affect the final output in an uncontrollable way. And the number of experts has to be fixed for a projection-based gating network. In contrast, the proposed attention-based gating network is order-agnostic with a flexible number of experts it engages with. In pFedMoAP, while we fix the number of non-local experts on each client to be $K$, in practice, $K$ can be client-specific based on the client’s memory and communication bandwidth.

As such, our design of the attention-based gating network extends beyond federated learning. This versatile network could be adapted to general prompt learning for VLMs where multiple prompts are trained, such as multi-task learning or domain adaptation. We anticipate that our work will inspire future research exploring the broader applications of mixture of adaptive prompts for VLMs. The detailed pFedMoAP algorithm can be found in [Algorithm 1](#alg1).

## 5 Experiments

### 5.1 Experimental setup

**Table 1: Few-shot performance on CLIP datasets under pathological non-IID setting. The highest accuracy for each dataset is in bold.**
|  | Flowers102 | OxfordPets | Food101 | Caltech101 | DTD |
| --- | --- | --- | --- | --- | --- |
| ZS-CLIP) (Radford et al., ) | 62.17$\pm$0.12 | 84.47$\pm$0.01 | 75.27$\pm$0.05 | 85.14$\pm$0.24 | 40.21$\pm$0.12 |
| CoOp (Zhou et al., ) | 70.14$\pm$0.76 | 83.21$\pm$1.30 | 70.43$\pm$2.42 | 87.37$\pm$0.44 | 44.23$\pm$0.63 |
| PromptFL (Guo et al., ) | 72.80$\pm$1.14 | 90.79$\pm$0.61 | 77.31$\pm$1.64 | 89.70$\pm$1.99 | 54.11$\pm$0.22 |
| PromptFL+FT (Cheng et al., ) | 72.31$\pm$0.91 | 91.23$\pm$0.50 | 77.16$\pm$1.56 | 89.70$\pm$0.25 | 53.74$\pm$1.36 |
| Prompt+FedPer (Arivazhagan et al., ) | 72.11$\pm$1.35 | 89.50$\pm$1.62 | 71.29$\pm$1.87 | 86.72$\pm$1.45 | 50.23$\pm$0.82 |
| Prompt+FedProx (Li et al., ) | 66.40$\pm$0.29 | 89.24$\pm$0.41 | 76.24$\pm$1.94 | 89.41$\pm$0.55 | 44.26$\pm$1.11 |
| Prompt+FedAMP (Huang et al., ) | 69.10$\pm$0.13 | 80.21$\pm$0.44 | 74.48$\pm$1.71 | 87.31$\pm$1.60 | 47.16$\pm$0.92 |
| pFedPrompt (Guo et al., ) | 86.46$\pm$0.15 | 91.84$\pm$0.41 | 92.26$\pm$1.34 | 96.54$\pm$1.31 | 77.14$\pm$0.09 |
| FedOTP (Li et al., ) | 96.23$\pm$0.44 | 98.82$\pm$0.11 | 92.73$\pm$0.15 | 97.02$\pm$0.36 | 87.64$\pm$0.70 |
| pFedMoAP ($\lambda$=0.0) | 97.61$\pm$0.11 | 94.83$\pm$0.65 | 86.71$\pm$0.15 | 95.71$\pm$0.37 | 85.64$\pm$0.34 |
| pFedMoAP ($\lambda$=0.5) | 98.41$\pm$0.04 | 99.06$\pm$0.09 | 93.39$\pm$0.09 | 97.95$\pm$0.07 | 89.13$\pm$0.54 |

Dataset and data heterogeneity. We evaluate the efficacy of the proposed pFedMoAP with 9 public benchmark datasets under various federated settings to simulate different types of data heterogeneity. Following previous research (Guo et al., ), to evaluate pFedMoAP under label heterogeneity, we adopt 5 representative visual classification datasets used to evaluate CLIP (Radford et al., ), namely OxfordPets Parkhi et al. (), Flowers102 Nilsback & Zisserman (), DTD Cimpoi et al. (), Caltech101 Fei-Fei (), Food101 Bossard et al. (). We refer to these datasets collectively as CLIP datasets. On these datasets, we test pFedMoAP’s few-shot performance under label heterogeneity by employing a pathological non-IID setting, where the classes are evenly distributed to the clients with no overlapping classes between any two clients. In addition, we use CIFAR10 and CIFAR100 dataset (Gong et al., ) and a Dirichlet distribution with $\text{Dir}(\alpha=0.5)$ to simulate the label shift (Hsu et al., ).
The Dirichlet distribution with $\alpha=0$ represents an extreme case where each client has only one class and larger $\alpha$ simulates a stratified and even split of labels. $\text{Dir}(\alpha=0.5)$ is on the more heterogeneous side.
For feature heterogeneity, we adopt DomainNet dataset (Peng et al., ) and Office-Caltech10 dataset (Gong et al., ), with 6 and 4 inherent domains, respectively. To add on extra heterogeneity, for DomainNet and Office-Caltech10, we split each domain with $\text{Dir}(\alpha=0.3)$ for extra label heterogeneity, mimicking real-world FL settings with both label and feature shifts.

Compared baselines. We compare our approach with three categories of baselines. The first category is local methods, where the clients do not participate in a FL setting. This category includes zero-shot CLIP (ZS-CLIP) (Radford et al., ) employing prompt templates: “a photo of a $<$class$>$”, and CoOp (Zhou et al., ) with learnable prompt vectors trained locally on each client. The second category comprises existing federated prompt learning methods, which include PromptFL (Guo et al., ), which learns a unified prompt across clients, pFedPrompt (Guo et al., ), which learns a shared prompt with personalized visual attention modules on locally memorized data, and FedOTP (Li et al., ), which leverages unbalanced Optimal Transport to align visual feature maps with personalized prompts. The third category consists of adapted methods from traditional FL/PFL techniques on PromptFL, namely FT (Cheng et al., ), FedProx Li et al. (), FedPer (Arivazhagan et al., ), and FedAMP (Huang et al., ).

Implementation details. 1) CLIP datasets. Each dataset in CLIP datasets is partitioned into $N=10$ clients, each with a disjoint set of classes evenly and randomly assigned to the clients. The training proceeds for $T=10$ rounds with $r=100\%$ participation rate. The CLIP uses a ResNet50 backbone. 2) CIFAR10 & CIFAR100. $N=10$ clients results from $\text{Dir}(\alpha=0.5)$ partition with $T=120$ rounds with $r=10\%$. CLIP also uses a ResNet50 backbone. 3) DomainNet & Office-Caltech10. Each domain of these two datasets is partitioned to 5 clients with $\text{Dir}(\alpha=0.3)$, resulting in $N=30$ for DomainNet and $N=20$ for Office-Caltech10. $T=25$ for both datasets, while $r=25\%$ for DomainNet and $r=50\%$ for Office-Caltech10. The CLIP uses a ViT-b-16 backbone. 4) Training specs. For all methods, we use SGD as the optimizer with a learning rate of 0.002 and 5 local epochs (except CoOp is locally trained for 25 epochs without FL). For pFedMoAP, we use SGD with a learning rate of 0.01 to train the $h=8$-head gating network and default $\lambda=0.5$ in [Eq. 10](#S4.E10). The core source code of the pFedMoAP is included in the supplementary material, and the full implementation will be made publicly available upon acceptance of the paper.

**Table 2: Results on DomainNet with feature shift and label shift with $\text{Dir}(\alpha=0.3)$ partition into 5 clients/domain. The highest accuracy for each domain is in bold.**
|  | Clipart | Infograph | Painting | Quickdraw | Real | Sketch | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ZS CLIP | 9.18$\pm$0.62 | 10.03$\pm$0.16 | 9.93$\pm$0.51 | 10.25$\pm$0.40 | 9.90$\pm$1.30 | 9.54$\pm$1.13 | 9.81$\pm$0.30 |
| CoOp | 43.84$\pm$3.51 | 45.72$\pm$0.85 | 29.94$\pm$0.46 | 36.83$\pm$1.17 | 31.64$\pm$0.49 | 33.97$\pm$0.78 | 36.99$\pm$0.79 |
| PromptFL | 27.63$\pm$16.41 | 27.69$\pm$18.07 | 21.62$\pm$8.34 | 23.45$\pm$13.49 | 20.62$\pm$11.03 | 25.90$\pm$8.10 | 24.48$\pm$12.52 |
| Prompt+FedProx | 22.23$\pm$15.42 | 21.75$\pm$17.00 | 18.58$\pm$8.15 | 19.40$\pm$12.59 | 17.17$\pm$10.25 | 22.49$\pm$8.44 | 20.27$\pm$11.83 |
| pFedMoAP | 47.49$\pm$0.64 | 46.73$\pm$0.71 | 32.74$\pm$0.84 | 37.16$\pm$0.34 | 31.02$\pm$0.59 | 37.67$\pm$0.72 | 38.80$\pm$0.11 |

### 5.2 Performance evaluation

We compute as the metric the average accuracy over every client’s private test set, drawn from the same distribution as its training set, and report the mean and standard deviation of all methods over three runs with different seeds.

**Table 3: Results on CIFAR10 and CIFAR100 with label shift with (Dir($\alpha=0.5$)) partition into 100 clients. The highest accuracies are in bold.**
|  | CIFAR10 | CIFAR10 |
| --- | --- | --- |
| ZS CLIP (Radford et al., ) | 53.46$\pm$0.21 | 32.68$\pm$0.00 |
| CoOp (Zhou et al., ) | 80.84$\pm$0.39 | 48.74$\pm$0.17 |
| PromptFL (Guo et al., ) | 73.29$\pm$0.37 | 45.00$\pm$0.62 |
| Prompt+FedProx (Li et al., ) | 73.32$\pm$0.34 | 45.63$\pm$0.75 |
| pFedMoAP | 83.46$\pm$0.53 | 53.42$\pm$0.22 |

Evaluation on Label Shifts. To assess the performance of our method in handling label shifts, we conducted experiments on the CLIP datasets for few-shot training and CIFAR datasets for standard training. [Tab. 1](#S5.T1) shows the results on the CLIP datasets under a pathological non-IID setting, while [Tab. 3](#S5.T3) presents the results on CIFAR10 and CIFAR100 with $\text{Dir}(\alpha=0.5)$.
On the CLIP datasets ([Tab. 1](#S5.T1)), our proposed method pFedMoAP consistently outperforms other baselines across all five datasets, with significant margins over most methods, which proves the superiority of pFedMoAP. For instance, on the Flowers102 dataset, pFedMoAP achieves an accuracy of 98.41% with $\lambda=0.5$, respectively 2.18% and 11.95% higher than FedOTP and pFedPrompt, the top 2 performant federated prompt learning methods, both recently proposed to achieve personalization through intermediate visual feature maps. Similar improvements can be observed across other datasets. Notably, on DTD dataset where zero-shot struggles to excel, pFedMoAP has little performance drop compared to personalized methods such as pFedPrompt and PromptFL+FedPer, showing pFedMoAP’s remarkable ability to adapt to diverse scenarios under few-shot settings. In addition, pFedMoAP with $\lambda=0.5$ also beats $\lambda=0.0$ over all datasets, demonstrating the effectiveness of addressing the local prompt together with MoE.
The results on CIFAR datasets ([Tab. 3](#S5.T3)) further corroborate the efficacy of our method in handling extreme data heterogeneity with $\text{Dir}(\alpha=0.5)$ over 100 clients. In this challenging scenario, pFedMoAP still consistently outperforms the compared baselines, reflecting the potency to handle label shifts by introducing non-local experts to the clients.

**Table 4: Results on Office-Caltech10 with feature shift and label shift with $\text{Dir}(\alpha=0.3)$ partition into 5 clients/domain. The highest accuracy for each domain is in bold.**
|  | Amazon | Caltech | DSLR | Webcam | Average |
| --- | --- | --- | --- | --- | --- |
| ZS-CLIP (Radford et al., ) | 9.83$\pm$1.63 | 10.67$\pm$0.89 | 10.89$\pm$1.40 | 6.20$\pm$3.84 | 9.40$\pm$0.77 |
| CoOp (Zhou et al., ) | 30.29$\pm$3.64 | 35.88$\pm$1.30 | 29.89$\pm$5.15 | 33.43$\pm$2.25 | 32.37$\pm$1.81 |
| PromptFL (Guo et al., ) | 21.08$\pm$9.60 | 23.72$\pm$12.21 | 22.94$\pm$7.96 | 25.88$\pm$7.72 | 23.41$\pm$9.06 |
| Prompt+FedProx (Li et al., ) | 18.64$\pm$8.58 | 19.56$\pm$11.59 | 20.89$\pm$7.38 | 22.96$\pm$7.56 | 20.51$\pm$8.48 |
| pFedMoAP | 35.47$\pm$1.37 | 37.45$\pm$1.33 | 45.11$\pm$3.14 | 35.22$\pm$1.04 | 38.31$\pm$1.21 |

Evaluation on Feature & Label Shifts. To evaluate the performance of our method in scenarios closer to real-world FL applications, we introduce feature shift on top of label shift via DomainNet and Office-Caltech10where each domain is partitioned into 5 clients with $\text{Dir}(\alpha=0.3)$, resulting in 30 and 20 total clients in DomainNet and Office-Caltech10, respectively. The results of these are presented in [Tabs. 2](#S5.T2) and [4](#S5.T4), respectively.
Under two types of heterogeneity, traditional federated learning methods struggle to keep benefiting the clients. On average, the FedAvg style method PromptFL exhibits worse performance than the local training method CoOp by 12.51% and 8.96% respectively on DomainNet and Office-Caltech10. However, pFedMoAP remains better performance over both local training and FL with up to 5.94% boost in accuracy under two types of heterogeneity. This validates the effectiveness and robustness of pFedMoAP in scenarios closer to real-world federated settings.

Figure: Figure 3: Ablation study on the number of shots.
Refer to caption: /html/2410.10114/assets/figs/ablation-few-shot.png

Figure: Figure 4: Ablation study on the coefficient for the logits from local prompt, $\lambda$.
Refer to caption: /html/2410.10114/assets/figs/ablation-lambda.png

### 5.3 Ablation study

Impact of number of shot.
Following prior works (Li et al., ; Guo et al., ), we investigate the impact of shots in the few-shot learning for pFedMoAP and other FL techniques implemented to prompt learning, namely PromptFL and PromptFL+FedProx, across the CLIP datasets. The number of shots varies from [1, 2, 4, 8, and 16]. As shown in [Fig. 3](#S5.F3), pFedMoAP consistently outperforms the other two methods across all datasets and shot numbers, demonstrating its superior effectiveness and robustness. While pFedMoAP outperforms the compared methods from only 1 shot, the gap between pFedMoAP and other methods widens as the number of shots increases. This demonstrates that pFedMoAP is particularly adept at leveraging additional training examples to enhance its performance, due to the introduction of MoE and the gating network. The trend of the curves also just that pFedMoAP has a more robust growth of performance when the number of shots increases, where the performances of the compared methods sometimes drop.

Impact of $\lambda$, the coefficient for logit computed from local prompt.
In pFedMoAP, $\lambda$ balances the contribution between the logits generated from the local prompt and from MoE. [Fig. 4](#S5.F4) displays the results for $\lambda$ values ranging from 0.0 to 5.0 across the CLIP datasets. The trends reveal that although the optimal $\lambda$ value varies across datasets, a marginal address on the logits from the local prompt (e.g. $\lambda=0.5$ or $1$) can already reach near-best performance. The flat plateau also indicates that the balance between the logits generated from the local prompt and from MoE is easy to reach. In addition, [Fig. 4](#S5.F4) also demonstrate a disinclination of too little or too much address on the local prompt-generated logits. We attribute this to the fact that only the local prompt is being updated on a client among all the experts, while immoderately addressing it ignores the benefit from MoE.

## 6 Conclusion

In this paper, we introduced pFedMoAP, a novel framework for personalized federated prompt learning in CLIP-like Vision-Language Models. Our approach leverages the lightweight nature of prompts to enable efficient cross-client knowledge sharing through a Mixture of Experts paradigm. By implementing a client-specific, attention-based gating network, pFedMoAP dynamically incorporates both local and non-local expert knowledge, striking a balance between personalization and collaborative learning. Our extensive experiments across various datasets and federated settings demonstrate the superior performance of pFedMoAP compared to state-of-the-art alternatives, particularly in handling extreme data heterogeneity. The consistent outperformance of pFedMoAP in scenarios with both label and feature shifts underscores its robustness and efficacy in real-world federated learning applications. Furthermore, our ablation studies highlight the method’s adaptability to different shot numbers and its stability across various hyperparameter settings. As federated learning continues to evolve, pFedMoAP represents a significant step forward in personalizing prompt learning for VLMs, opening new avenues for efficient and effective collaborative learning in privacy-preserving settings.

#### Acknowledgments

This work was supported by a National Institutes of Health (NIH)/National Cancer Institute (NCI) grant (1R01CA218405), the grant 1R01EB032896 (and a Supplement grant 3R01EB03 2896-03S1) as part of the National Science Foundation (NSF)/NIH Smart Health and Biomedical Research in the Era of Artificial Intelligence and Advanced Data Science Program, a NSF grant (CICI: SIVD: #2115082), an Amazon Machine Learning Research Award, and the University of Pittsburgh Momentum Funds (a scaling grant) for the Pittsburgh Center for AI Innovation in Medical Imaging. This work used Bridges-2 at Pittsburgh Supercomputing Center through allocation [MED200006] from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF grants #2138259, #2138286, #2138307, #2137603, and #2138296.

Ethics Statement.
The research presented in this paper adheres to the Code of Ethics. Our study does not involve human or animal subjects. Every dataset used in our study is publicly available. Our method does not address discrimination/bias/fairness concerns.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Refer to the weakness part.
soundness: 3
strengths: 1.	The method is novel which allows the client download prompts from others.
2.	The experiments are extensive and show good performance.
summary: This paper introduces pFedMoAP that enables effective federated prompt learning for vision-language models like CLIP. The key innovation is allowing clients to download multiple pre-aggregated prompts as fixed non-local experts rather than being restricted to a single globally aggregated model.
weaknesses: 1.	It is very similar to the prompt learning framework. How does your method differ from the classic prompt learning framework if you can directly access prompts from other clients?
2.	I noticed that similar work, like pFedPrompt, conducted experiments on large-scale datasets like UCF-101 or ImageNet. Would the scale of the dataset influence the performance?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. In Section 4.1 **Implementation details.**, for CIFAR10&CIFAR100, why the participation rate is 10%? Does it mean only one client will be selected to join the training process in each communication round?
soundness: 2
strengths: 1. Implementing prompt learning into the distributed environment is an important topic for FL applications due to its efficiency in computation and communication.

2. The proposed learning framework is simple and effective.
summary: This paper proposes a lightweight federated prompt learning framework to personalize the prompt learning process through the lens of mixture of experts (MoE). An attention-based gating network is also introduced to achieve efficient cross-client knowledge sharing. Experiments indicate that the proposed **pFedMoAP** performs better than state-of-the-art methods.
weaknesses: 1. The quality of the writing is poor. The expression of the paper is somehow obscure, and not concise enough. Besides, there are many typos in the paper, e.g.,  lacking space between two adjacent words in Line 27, Line 198, Lin 415, and Line 469. 

2. The novelty is limited. The **pFedMoAP** seems like a naive combination of PromptFL and MoE. Although the selected non-local experts will be updated in each communication round, the intuition behind it is also not explained well, leading to weak convincingness. 

3. it is unclear why the attention-based gate network does not need to be uploaded to the server for aggregation. Please clarify.

4. I think the ablation study of the paper is not enough. 
- To study the effectiveness of MoE-based textual feature enhancement mechanism, the authors should add experiments under different values of $K$ to confirm performance changes.

- The author implements a dimension reduction operation in Section 3.3, please add experiments to show whether the performance will be influenced.   

- How about a larger number of clients, e.g., 100 or more?

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness.
soundness: 3
strengths: 1. The integration of the MoE method into prompt-based federated learning is an innovative concept.

2. The paper is well-organized and effectively communicates its main contributions. The methodology is clearly explained, particularly regarding the incorporation of the MoE approach in the federated learning context.
summary: This paper proposes a Mixture of Experts (MoE) approach for prompt-based federated learning, employing a local attention-based gating network to refine text features for better alignment with local image data. It also highlights the lightweight design of the prompt, exploring a configuration where the prompt is updated directly by other clients rather than relying on an aggregated prompt from the server.
weaknesses: 1. While this approach utilizes inter-client sharing, it could expose prompt updates directly to other clients, which may lead to privacy concerns as prompt updates could be tracked, making the model susceptible to certain attack algorithms. 

2. In the related work section in the appendix (Line 783), the authors mentioned that previous work utilized inter-client sharing prior to aggregation. However, these works allow prompt aggregation in the server and do not share local prompts with other clients. This sentence is somewhat unclear; please provide a clearer version to avoid misunderstandings. 

3. The paper claims that the MoE-based method can leverage prompts from other clients. Please provide experimental evidence to show whether your method’s effectiveness is sensitive to the total number of clients and the value of $K$. 

4. Some related works are encouraged to be properly cited in Related work. For example, personalized federated learning [a,b] and prompt-based federated Learning [c]. 

[a] Li, et al, FedTP: Federated Learning by Transformer Personalization, TNNLS 2023. 

[b] Cai, et al, Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning, NeurIPS 2023. 

[c] Pan et al, Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method, NeurIPS 2024.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Please see weaknesses.
soundness: 3
strengths: 1. pFedMoAP proposes a novel framework that personalizes the prompt learning process by allowing clients to download multiple pre-aggregated prompts as fixed non-local experts. This leverages the concept of Mixture of Experts, where a local attention-based gating network is implemented to generate enhanced text features that better align with the local image data on the client side.

2. By facilitating a many-expert scenario (e.g., 10 experts) per client with negligible communication overhead, pFedMoAP achieves efficient and effective personalization. This approach overcomes the limitations of having too many or too few experts, which can lead to high communication costs or suboptimal performance, respectively.

3. The algorithm consistently outperforms existing work across various heterogeneous federated settings, as demonstrated through extensive experiments on 9 datasets.
summary: This work introduces a new paradigm in federated prompt learning for pre-trained Vision-Language Models (VLMs) such as CLIP, challenging the traditional restriction that clients are limited to downloading a single globally aggregated model. This shift is particularly beneficial for lightweight prompts, enabling more flexible and effective knowledge sharing among clients.
weaknesses: 1. Determining which prompt experts should be included in the pool and when to update or replace them requires careful management. If the selection process is not optimal, it could lead to suboptimal performance. Additionally, maintaining a diverse and up-to-date pool of experts is crucial but can be complex.

2. The local attention-based gating network, while designed to be efficient, may still introduce optimization challenges. Ensuring that this network converges to a solution that effectively combines local and non-local prompts is not trivial, especially when dealing with heterogeneous data distributions.

2.  As the number of clients increases, managing a dynamic pool of prompt experts and maintaining efficient communication between the server and all clients can become more challenging. The complexity of coordinating updates and ensuring that each client receives the most relevant prompts could grow significantly.

Meta-review: This paper introduces a lightweight federated prompt learning framework that uses a mixture of experts to personalize prompts and an attention-based gating network for efficient knowledge sharing across clients. It outperforms existing methods and allows direct updates of the prompts by clients. 

There is some appreciation on the novel method allowing clients to download prompts from others, enhancing the personalization of the prompt learning in federated learning. The reviewers also recognize the proposed framework is simple, effective and achieve efficient computation and communication. Extensive experiments across 9 datasets show that the algorithm consistently outperforms existing methods in various setting. 

While some reviewers share the concerns regarding the clarity and writing quality issues. There is also one reviewer zSUv has the methodology and novelty concern, where the reviewer point out pFedMoAP seems like a simple combination of PromptFL and MoE, lack of a strong well explained intuition. In the initial review, there are also criticism regarding the experimental gaps like varying the value of K, and concerns regarding privacy of directly downloading prompt from other clients and management challenges. While most of the concerning points have been addressed after the rebuttal, the issue of missing evaluations on large-scale benchmarks such as ImageNet, which have been used in other recent work, remains unresolved. The authors argue that this is due to a lack of time and resources. I still have some concerns about it. However, the most critical concerns regarding privacy and missing experiments have been addressed. After careful consideration of all the reviews, the authors’ response, and subsequent discussion, I am inclined to accept this paper.

Decision: Accept (Poster)