FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Authors: Chengying Fang
Wenke Huang
Guancheng Wan
Yihao Yang
Mang Ye
Venue: ICML 2025 poster (2025)
Keywords: Federated Prompt Learning
Domain Adaptation
Model Heterogeneity
Singular Value Decomposition
Primary area: deep_learning->large_language_models
TL;DR: FedPHA enables efficient federated personalization through a fixed-length global prompt and flexible local prompts.

Abstract: Federated Prompt Learning (FPL) adapts pre-trained Vision-Language Models (VLMs) to federated learning through prompt tuning, leveraging their transferable representations and strong generalization capabilities. Traditional methods often require uniform prompt lengths for federated aggregation, limiting adaptability to clients with diverse prompt lengths and distribution biases. In this paper, we propose **Fed**erated **P**rompt Learning for **H**eterogeneous Client **A**daptation (FedPHA), a novel framework that combines a fixed-length global prompt for efficient aggregation with local prompts of varying lengths to capture client-specific data characteristics. Additionally, FedPHA designs Singular Value Decomposition (SVD) based projection and bidirectional alignment to disentangle global conflicts arising from client heterogeneity, ensuring that personalized client tasks effectively utilize non-harmful global knowledge. This approach ensures that global knowledge improves model generalization while local knowledge preserves local optimization. Experimental results validate the effectiveness of FedPHA in achieving a balance between global and personalized knowledge in federated learning scenarios.

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Chengying Fang" Wenke Huang"
Guancheng Wan"
Yihao Yang'
Mang Ye
Abstract
Federated Prompt Learning (FPL) adapts pre- trained Vision-Languaae Models (VL.Ms) to fed- erated learning through prompt tuning, leverag- ing their transferable representations and strong gencralization capabilities. Traditional methods often require uniform prompt lengths for feder- ated aggregation, limiting adaptability to clients with diverse prompt lengths and distribution bi- ases. In this paper, we propose Federated Prompt Learning for Heterogeneous Client Adaptation (FedPHA), novel framework that combines a fixed-length global prompt for efficient aggrega- tion with local prompts of varying lengths to cap- ture client-speciffe data characteristics. Addition- ally, FedPHA designs Singular Value Decompo- sition (SVD) based projection and bidirectional alignment to disentangle global conflicts arising from client heterogeneity, ensuring that personal- ized client tasks effectively utilize non-harmful global knowledge. This approach ensures that global knowledge improves model generalization while local knowledge preserves local optimiza- tion. Experimental results validate the effective- ness of FedPHA in achieving balance between global and personalized knowledge in federated learning scenarios. The source code is avail- able at: FedP HA.
Model Heterogeneity
R
Client
Client j
Data Heterogeneity
Frones
Than
CR
CR
Cal-wayy
(b) Heterogeneity Problem
Impede Personalization
(a) Heterogeneous FPL
(c) Conflicting Problem
Figure 1: Problem illustration of heterogeneous federated prompt learning (FPI.). (a) Heterogeneous FPI.: Clients hold different prompts, models, and data distributions. (b) Heterogeneity Prob- lem: Aggregation of heterogeneous prompts and models under non-lID data is inherently challenging. (c) Conflicting Problem: Aggregated global prompts may conflict with client-specific knowl- edge, impeding personalization during local adaptation.
tion. However, existing federated learning approaches face significant limitations due the frequent exchange large volumes of model parameters with central server. This results in high communication overhead, increased train- ing costs, potential performance degradation, and instability during the training process (Wu et al., 2020; Kulkarni et al., 2020; Chen et al., 2022; Wan et al., 2024).
cww/CtFangs/
Fortunately, vision- languae pre-trained models such as Contrastive ingguage-Image Pretraining (CLIP) (Radford et al., 2021) have demonstrated potential in learning robust and versatile representations suitable for various image dis- tributions, aligning well with the objectives of federated learning. However, the substantial communication overhead between the server and clients poses challenges for train- ing CLIP within federated learning frameworks (Lu et al., 2023). Additionally, overfitting concerns may arise when large-scale scale models are trained on limited client data. Prompt learning (Zhou et al., 2022b;a; Khattak et al., 2023; khattak et al., 2023; Li et al., 2024b) offers a flexible approach to adapt pre-trained models to downstream tasks by training only additional parameters. This enables prompts to capture task-specific information while guiding the performance of
1. Introduction
Federated learning (McMahan al., 2017: Yang et al., 2019;
Hong& & Chae, 2021; Qu et al., 2022; Huang et al., 2024), as distributed machine learning paradigm, addresses data silos by enabling participants to collaboratively train models locally, ensuring data privacy while promoting Al collabora-
Eqqual contribution NNational Engineering Research Center for Multimedia Software School of Computer Science, Wuhan University, Wuhan, China. Correspondence to: Mang Ye <ye- mang @ whw.edu.en>.
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
the fixed model. Leveraging its lightweight nature, prior re- search (Guo et al., 2023b; Qiu et al., 2023; Feng et al., 2023; Su et al., 2024) has explored integrating prompt learning into federated learning to address these challenges.
lengths and varying data distributions while preserving in- formation integrity. (2) Developing method mitigate the negative impact of the conflicting parts between global and local knowledge, allowing clients to retain their unique characteristics while benefiting from global knowledge.
As shown in Figure 1, one of the fundamental challenges in federated learning is client heterogeneity (Li et al., 10; Huang et al., 2022; Wang et al., 2023; Huang et al., 2023b; Hu et al., 2024; Tan et al., 2025), which manifests in two key forms: data heterogeneity, where client data distributions are non-IID,, and model heterogeneity, where clients employ diverse model architectures or have varying computational resources. These challenges significantly hinder model con- vergence and system efficiency. Intuitively, different clients should require prompts of varying lengths to more effec- tively capture the characteristics of their local data (proved in Sec 4.3). However, due to aggregation constraints, current federated prompt learning frameworks (Guo et al., 2023b;a; Feng et al., 2023; Yang et al., 2023; Bai et al., 2024) typi- cally enforce uniform prompt lengths across all clients to facilitate the aggregation process. Although some works (Li et al., 2024a; Cui et al., 2024) have proposed dual-layer architectures incorporating both global and local prompts while aggregating only global prompts, the structural con- straints of these methods prevent support for varying local prompt lengths. In such approaches, forcibly expanding or reducing the length of prompts may lead to information loss, further highlighting the challenge of designing methods that can adapt to different prompt length requirements.
In this work, we propose FedPHA (Federated Prompt Learn- ing for Heterogeneous Client Adaptation). a novel method designed to address challenges related data and model het- erogeneity, as well as resolving conflicts between global and local knowledge. FedPHA designs a G-L (Global-Local) architecture to manage the varying prompt requirements of heterogeneous clients. Each client receives a local prompt with a unique length and a global prompt with a uniform length. These prompts are connected through shared to- kens and frozen encoder, establishing an implicit coupling between global and local prompts. To resolve conflicts between global and local knowledge, we integrate an SVD- based projection mechanism, which filters out conflicting parts while preserving essential local information. In addi- tion, we introduce a bidirectional alignment function in the optimization process. This ensures alignment between local and projected features while ensuring a clear distinction between global and local features, preserving their unique characteristics. Our main contributions are summarized as:
We are the first consider the heterogeneity of prompt lengths in federated prompt learning. We design G-L framework to facilitate aggregation and individual client requirements. Shared tokens and a frozen encoder connect the global and local prompts, creating implicit coupling. We use feature-level computation to prevent information loss from prompt length variations.
Furthermore, the complex interplay between shared global knowledge and client-specific local knowledge presents other critical challenge in federated prompt learning, as illustrated in Figure 1(c). While global knowledge aggre- gated from multiple clients can provide valuable general- izable features, it may also contain potentially conflicting information that hinders local optimization (Wang et al., 2020; Li al., 2022: Nguyen et al., 2024). Specifically, in high data heterogeneity scenarios, the knowledge learned from other clients may not align with or even contradict the optimal features required for a specific client's local task. The conventional federated averaging approach (Guo et al., 2023b;a) may force clients to compromise their locally opti- mal representations to accommodate the global consensus, potentially degrading performance on -pient-spccffic tasks. Although some recent works (Guo et al., 2023a; Li et al., 2024a) have attempted to mitigate this issue by separating global and local prompts, they lack explicit mechanisms to resolve knowledge conflicts and ensure effective knowledge transfer between the two.
We devise an -VD-based projection mechanism to disen- tangle conflicting parts between global and local knowl- edge, retaining essential local information while remov- ing inconsistencies. And bidirectional alignment func- tion aligns local and projected features and preserves the unique characteristics of global and local representations.
We evaluate FedPHA against the existing personalized techniques on widely-adopted datasets. Extensive experi- ments and ablation studies demonstrate the superiority of our methods under heterogeneous settings.
2. Related Work
2.1. Heterogeneous Federated Learning
Federated Learning (FL) aims to address the critical chal- lenge of heterogeneity in client data distributions (Xu et al., 2021; Huang et al., 2022; Fang & Ye, 2022; Huang et al., 2023a). Key types of heterogeneity include label shift, where the label distribution P(Y) differs across clients while P(X|Y) remains consistent, and domain shift, where the feature distribution P(X) varies while P(Y) stays un-
To address the challenges in Figure 1, our work is moti- vated by two primary objectives: (1) Designing a framework that balances federated learning aggregation requirements with client-side flexibility, accommodating diverse prompt
2

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
changed. These challenges necessitate specialized meth- ods to ensure effective collaboration across diverse client datasets. To handle such heterogeneity, early approaches in FL often include incorporated regularization terms to the loss function (Li et al., 2020) or fine-tuning the global model on clients' local datasets (Fallah et al., 2020). How- ever, these methods risk local overfitting due to the limited and diverse data on clients. potentially compromising global generalizability. More advanced methods explicitly aim to balance global and local models (Chen & Chao, 2022), or leverage client relationships through weighted aggregation techniques, such as FedPAC (Xu et al., 2023) and FedDisco (Ye et al., 2023). Parameter decomposition has been ex- plored for heterogeneity; e.g., FedTP (Li et al., 2023) learns client-specific self-attention layers. Despite progress, FL methods continue to struggle with balancing personalization and generalization under high data and model heterogeneity. Our proposed FedPHA addresses these challenges by im- proving the balance between global consistency and local personalization under prompt-length heterogeneity.
A = USVT, where U E R V E Ruxn are orthonor- mal matrices, and S E R=xx is a diagonal matrix of sin- gular values. SVD enables dimensionality reduction by retaining only the largest singular values. SVD has been widely explored in large language models (L.LMs) for its ability to decompose matrices into orthogonal components, offering robust mathematical foundations for a variety of ap- plications. It is a powerful tool for dimensionality reduction (Hsu et al., 2022; Yuan et al., 2023; Saha et al., 2023; Wang et al., 2024), enabling the extraction of key features while minimizing redundant information. Additionally, SVD has proven effective in noise filtering (Sharma et al., 2023; Dai et al., 2024), as it isolates signal-dominant components and suppresses less significant, noisy contributions. Further- more, is frequently utilized in subspace projection (Feng et al., 2023; Lan et al., 2024), enabling data representation in lower-dimensional subspaces while preserving essential properties and optimizing computational efficiency. Our work builds on these principles by proposing an SVD-based projection mechanism in the context of federated prompt learning, addressing heterogeneity and reducing potential conflicts between local and global information.
2.2. Federated Prompt Learning
Prompt learning, initially developed for NLP, has been ex- tended to Vision-Language Models to adapt pre-trained mod- els to diverse downstream tasks. Early methods like CLIP (Radford et al., 2021) used manual templates, while newer approaches learn prompts in continuous embedding spaces. For example, CoOp (Zhou et al., 2022b) fine-tune mnnetunes CLIP with continuous vectors, and ProGrad (Zhu et al., 2023) se- lectively updates prompts to preserve essential VLM knowl- edge. To integrate prompt learning into Federated Learn- ing (FL), methods like FedPrompt (Zhao et al., 2023) and PromptFL. (Guo et al., 2023b) accelerate global aggregation and address limited user data. Building on these, pFed- prompt (Guo al., 2023a) employs a non-parametrie per- sonalized attention module for local feature generation, and PFedPG (Yang et al., 2023) designs server-side prompt generator for client-specific personalization. FedOTP (Li et al., 2024a) uses unbalanced Optimal Transport to coordi- nate global and local prompts. FedPGP (Cui et al., 2024) adapts to heterogeneous data via low-rank decomposition of global prompts and contrastive loss to balance personal- ization and generalization. However, the structural limita- tions of these methods prevent them from accommodating varying local prompt lengths and lack a mechanism to sepa- rate conflicting global knowledge from personalized local knowledge. In contrast, our proposed FedPHA leverages a dual-layer architecture and Singular Value Decomposition (SVD) to effectively address these challenges.
3. Proposed Method
In this section, we present the details of FedPHA illustrated in Figure 2. To address the issue that existing methods cannot adapt to heterogeneous prompt lengths, FedPHA in- troduces a G-L heterogeneous federated prompt architecture (Sec 3.2). Meanwhile, to reduce the negative impact caused by the conflict between global prompts and local prompts, we propose SVD-based projection (Sec 3.3) and bidírec- tional alignment (Sec 3.4). The details of our FedPHA are provided in Algorithm 1.
3.1. Preliminaries of Prompt Learning
Prompt learning efficiently adapts pre-trained models like CLIP for downstream tasks by introducing learnable pa- rameters in the text encoder. Unlike zero-shot transfer, which uses fixed word embeddings W = {WI, .., from handcrafted prompts (e.g.. "a photo of a (label)"). prompt learning adds learnable continuous context vectors P. {PL {p1.P2,.- P2.... } E Rxxd where T is the prompt length and d is the embedding dimension. This allows the text encoder to capture task susk-specific information while keep- ing the image encoder fixed. For a class label fclass the textual input is extended as:
p = (Isos, Pt, 1,,11,4 14,,2,,...,,44 L1, 4Cluss, (BOS):
(1)
where tsos and tEos are learnable start/end embeddings, [f,...,tty, ,t,} are fixed word embeddings, and fClsss is the class label embedding. The text encoder g()), composed of transformer layers, generates the prompted textual feature
2.3. Singular Value Decomposition
Singular Value Decomposition (SVD) (Golub et al., 1987) is technique that decomposes matrix A Rmxn into
3

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Prompt Heterogeneous Architecture
SVD-Based Projection
Projected input
Constructing input
decompose
Padding
Adaption
Insert Padding
Client
del
Tort
Isp
Local projected prompt
Server
select longest
len
Acleption
Insert Paddligg
Bidirectional Alignment
Global lem
Projected inpur
K
Local ingut
Constructing
Euc Euclidean distance
Global
Client
Luah
Local
Global
Projected
Zero
Input
Figure 2: Our proposed FedPHA framework for heterogeneous federated prompt learning consists of three main components. The dual-layer architecture (Sec 3.2) assigns each client local prompt of varying lengths and a global prompt of uniform length. The longest prompt forms frozen tokens (Isos, fcluss (satie (ESs). distributed to all clients for input sequence construction, with buudinn filling gaps. The top right shows the SVDD-Based Projection (Sec 3.3), where the global prompt is decomposed via SVD. The last d' components of V form the null space, into which local prompts are projected. The hottom right illustrates the hidirectional alignment (Sec 3.4), which uses Eum align local and projected features, and Lpnch separate local and global features. Only local features used for final inference.
G = gY,, E Rdd, where 0g includes frozen pre-trained parameters and the learnable Pt. For image classification, textual features { are compared with image features f. extracted by the frozen image encoder f(.). The class k probability for an image x is:
Suppose there are N clients indexed by i 1,2...., . N and a central server. Each client holds a local dataset D; of size ni, with (Di...., Dx) collectively denoting the full dataset. Let C, S {L,...., N be the subset of clients selected communication round r. Each selected client performs local training for E epochs using loss L. During local optimization, the global prompt Pr. E xg and the local prompt Pf, T (where Tg fixed and T, may vary across clients) are updated. Let 1) be the learning rate, For each local epoch e, the updates are
exp(sim(āpk: f)/T)
P(j = kxx) E cxp(sim(@px. f)/T)'
where sim(...) is cosine similarity, and regulates the Soft- max sharpness. The learnable prompts Pt are optimized with cross-entropy loss. For a dataset D of input-output pairs (X,y), the objective is:
p = P.T. - VV. C(P PiP: D.).
(4)
where P. denotes either the global prompt g. or the local prompt P.ç
LCE arg min E (Xa)~D ((sim(@p: f).
(3)
P,
At the end of local training, only pr. is uploaded to the server, while p is kept locally to preserve personalized information. The server then aggregates the global prompts from all clients i C,.. The new global prompt is obtained by weighted averaging based on the sample sizes n, of the participating clients:
g.²
3.2. Federated Prompt Heterogeneous Architecture
In federated learning, different clients often exhibit diverse data distributions and task requirements, making it difficult to use a single, fixed-length prompt that balances person- alization and aggregability. To address this issue, we pro- pose G-L heterogeneous federated prompt architecture, which comprises fixed-length global prompt and a variable- length local prompt for each client. Additionally, we employ frozen contextual tokens and zero-padding tokens to ensure the implicit coupling between global and local prompts, providing a consistent structure that facilitates subsequent feature computation.
p(r+1.0) = =E 9
P..F. -EE my 9.
(5)
SEC. Ejec,
where n4 denotes the number of samples in the local dataset DD, This ensures that clients with larger datasets contribute more significantly to the aggregated global prompt. The updated global prompt P (r+1,0) then distributed to all

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
clients in preparation for the next round of communication. Formally, the objective function can be expressed as:
If pree RTxxd be the current global prompt at local epoch e. We directly perform singular value decomposition (SVD) on e obtaining
N
min Po (P.(P
n N Ej
C. P.S.P. (6)
pre USV
(8)
where U E TTTxX and V E Rdxd are orthonormal matri- ces, and S RTyx xd is a diagonal matrix with descending singular values. Because the global prompt has length Tg and the local prompt may have a different length T, choos- ing U to construct the projection could lead to dimension mismatch. Therefore, we utilize V, which naturally resides in the same feature dimension d as both local and global prompts, to form the null space. Let V/2 E Rdddd' be the matrix collecting the columns of V corresponding to the smaller singular values. The number of selected columns d' is determined by the hyperparameter ratio p:
where C 9,4 :D)) denotes the local loss of client i. During training, both global and local prompts are lever- aged to jointly optimize the model. During inference, only local features, derived from the local prompts, are used for final inference to ensure adaptability to personalized data distributions.
Heterogeneous Key. The fundamental distinction of our method from others lies in its unique handling of global and local prompts. Specifically, on the text-encoder side, each client is assigned a local prompt of varying lengths and global prompt of fixed length. We select the longest prompt among all clients to construct the frozen tsos: &Class Suffix tEos via (q.(1). Each client then forms three types of input sequences by concatenating its pre- fix tokens, class name tokens, and suffix tokens with: (1) the global prompt pr.s. (2) the local prompt Pi.. and (3) the projected local prompt e (in (10). If the result- ing sequence is shorter than the maximum encoder length Lmaxs zero vectors (pradinng) are appended. The final input sequence can be represented as:
d' L(1 p)]d]-
These directions typically capture less significant poten- tially conflicting components in the global prompt. The null- space projection matrix Q is then defined as Q = V/222 Then the projection of the local prompt is defined as:
pr.e = PT = Pi; V2V,
(10)
where Pf E RTixd is the local prompt for client i. The projected prompt P/; E Rixdd retains the same dimen- sions as the original local prompt. By projecting PT,:" onto Q, we effectively "filter out" dimensions dominated by the global prompt's 's major components, thereby reducing po- tential conflicts between local and global information. This step is crucial in heterogeneous settings: it preserves local discriminative features relevant to each client's 's data while mitigating interference from global prompt directions that may not generalize to individual client distributions.
Y, {tsos, Pt, &Class, (Suffix teOS, (
The global and local prompts are implicitly coupled and interact through shared prefix and suffix tokens, as well as a common Transformer encoder. This interaction en- sures that while local prompts retain their client-specific distinctions, the overall model still operates within a shared high-dimensional representation space, promoting informa- tion exchange across clients. Finally, the resulting input sequences are fed into the pre-trained CLIP encoder along- side image representations to compute similarity scores and perform classification.
3.4. Bidirectional Alignment
To mitigate conflicts, we project the local prompt Pf; into the null space. However, this may lead to information loss, reducing client-specific expressiveness. To address this, we introduce bidirectional alignment mechanism: "pull" term to retain information by aligning the local prompt with its projection and "push" term to ensure sufficient divergence from the global prompt.
3.3. SVD-Based Projection
Although the above framework achieves personalization with heterogeneous prompt lengths by thoroughly separat- ing global prompts and local prompts, an interaction mech- anism between global and local prompts is still required to facilitate information exchange. Simple alignment or orthogonality between the two may be insufficient in cases of highly heterogeneous data distribution. Therefore, it is necessary to further refine local prompts to mitigate poten- tial conflicts with global prompls. Inspired by subspace projection in matrix factorization, we propose projection mechanism based on Singular Value Decomposition (SVD) to filter out unnecessary or conflicting components from local prompts.
To ensure that the local prompt does not deviate excessively from its projected prompt, we minimize the mean squared error (MSE) between I (P,).) and (PP)))) Formally,
Gpull = ||((7))) - 1(779)
(11)
which encourages Pf to "pull" closer to its null-space- projected version P/, in the feature space. By doing so, we
5

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
retain essential local information while filtering out compo- nents that conflict with the global prompt.
Algorithm Overall Procedure of FedPHA
Data: The random public datasct (D; (D2))=m. sizes k Input: Communication rounds R; Local epochs E; Learning rate vy SVD ratio Margin ox; Initial global prompt p) Initial local prompts PL.
In parallel, to prevent the local prompt from collapsing too closely to the global prompt, we introduce a margin-based based "push" term to maintain a safe distance between f( (P())) and f( (g.)) Specifically,
Output: The final local models M,
// Federated Rounds for o 1,2, do
push ReI.U(a
-
(12)
Participan Side for each client i E C, in parallel do p(ne) p(r. 9. PA
where a > 0 defines the minimal acceptable distance. This ensures that each local prompt remains sufficiently person- alized and does not become overly dominated by the global prompt's major components.
Localt pdate(P 9. PLs
p(r-1E))
end
11 Server Side
In practice, both of these MSE-based terms (pull and push) are combined with standard cross-entropy losses (for both the local prompt and the global prompt), forming unified training objective:
P5 pfrti.) Eiec, Eyec, ns
p(r.E) E) in (5)
end
Local &pochs for e 1,2,..... E do
4 = Len( Pi u:) + LCE( (Pr + Gpall + Lpush- (13)
// Based Projection Pr + USV in (q)
This bidirectional alignment strategy mitigates potential con- flicts arising from heterogeneous data distributions, ensuring that local prompts retain discriminative characteristics while still benefiting from global knowledge.
V/2 V[g, - p) dI via (5)
Q VoV. // Construct projection
PT P. in Eq.10)
4. Experiments
// Compute losses
In this section, we conduct extensive experiments aiming at answer following research questions:
Gpalt + 11(()))) - [(P)))) in Eq.(1)
Q1: Does the proposed method maintain its effectiveness
Lpuch + ReLU (1((P))) - ((27)) Eq.
when the prompt length is fixed? How does it compare to the state-of-the-art (SOTA) methods? (in Sec 4.2)
C. + fee + Lpull + Epssh in (13
Q2: For clients with diverse data distributions, can prompts of varying lengths enhance performance? Does length heterogeneity provide any advantage? (in Sec 4.3)
// Update prompts using gradients p(ne-1), p(net() using VL
Update g.1 Pl
end return MR // Final model after R rounds
4.1. Experimental Setup
Datasets. Following previous research (Guo et al., 2023b;a),
we evaluate our method on multiple public benchmark datasets exhibiting significant data heterogeneity. We use five visual classification CoiassetsFFoddd (Bossard et al., 2014), DTD (Cimpoi al., 2014), Caltech101 (Fei-Fei, 2004), Flowers102 (Nilsback & Zisserman, 2008), and Ox- fordPets (Parkhi et al., 2012) colleccvvvy referred to as the CLIP dataset (1 domain). These datasets are configured using a pathological non-IID setting, where each client is randomly allocated distinct number of overlaaaagg classes to simulate heterogeneous data distributions. In addi- tion, we select two cross domain datasets, Office31 (Saenko et al., 2010) (3 domains) and OfficeHomme (Venkateswara et al., 2017) (4 domains), where the data for each client is drawn from a specific domain, further emphasizing data heterogeneity. Finally, we employ two classic image bench- mark datasets, CIFARIO (Krizhevsky et al., 2010) and
CIFAR- 100 (Krizhevsky & Hinton, 2009), where data is ran- domly partitioned among clients using a symmetric Dirichlet distribution as in (Cao et al., 2023; Shamsian et al., 2021) with 3 = 0.5, further enhancing the diversity in data dis- tribution. Details of these dataset setups are provided in Appendix Section B.I
Baselines. We compare our FedPHA with five baseline methods: (1) Zero-shot CLIP (Radford al., 2021), a lo- cal training approach that utilizes manually designed text prompt templates to generate the model's initial perfor- mance. (2) PromptFL. (Guo et al., 2023b), a prompt-based federated learning method that learns a unified prompt across clients using the federated averaging mechanism (McMahan et al., 2017). (3) PromptFL+Prox (Li et al., 2020), as introduced in (Guo et al., 2023a), which con-
6

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Table I: Comparison with the SOTA methods with same prompt length on single-domain datasets across 10 clients.
Methods
Caltech
Food101
Flowers102
OxfordPets
DTD
Zero-Shot CLIP (Radford al., 2021)
91.43=0.24
85.43+0.05
67.70=0.12
88.95+0.11 12
43,2840. 10
PromptFL (Guo et al., 2023b)
93.47=0.42
86.60#0.10
85.54+0.65
93,44=0.25
55.2440.31
Prompt+Prox (Li et al., 2020)
93.55+0.36
86.65=0.17
85.74+0.25
93.48+0.32
55.6440.23
FedPGP (Cui et al., 2024)
95.86+0.50
88.30#0.41
94,46+2.44
93,87=0.31
62.95+3.01
FedOTP (Li et al., 2024a)
98. 08.004
92.9640.16
98.47=0.07
98.60=0.11
89.97+0.17
FedPHA
99.05=0.08
96.42+0.05
99.25+0.05
99.25+0.06
91.79+0.17
Table 2: Comparison with the SOTA methods with the same prompt length on multi-domain datasets. Each domain consists of two clients, and the table presents both the average accuracy within each domain and the overall global accuracy.
Office31
OfficeHome
Methods
D
W
Avg
P
R
Avg
Zero-Shot CLIP (Radford et al., 2021)
80.96
71.63
74.60
75.73
84.21
66.37
89.16
89.68
82.35
PromptFL (Guo et al., 2023b)
88.20
84.89
91.14
88.08
86.75
75.30
94.38
93.24
87.41
Prompt+Prox (Li et al., 2020)
88.32
85.03
91.43
88.26
86.58
75.65
94.79
93.27
87.57
FedOTP (Li et al., 2024a)
86.62
87.40
93.55
89.19
80.38
76.29
92.49
87.86
84.26
FedPGP (Cui et al., 2024)
89.55
90.70
94.50
91.58
88.34
78.09
95.49
93.86
88.95
FedPHA
90.44
95.80
97.98
94.74
88.70
79.59
95.93
93.83
89.51
strains local prompt updates using a proximal term instead of direct aggregation. Additionally, we include two popular methods that integrate both global and local prompts: (4) FedOTP (Li et al., 2024a), which employs the Unpaired Optimal Transport (UOT) method to align prompts with the most relevant image features, thereby enhancing per- sonalization. (5) FedPGP (Cui et al., 2024), which uses low-rank decomposition and contrastive learning to balance personalization and generalization.
(9)) defaults to 0.8, and alpha (a: in Eq. to More details are provided in Appendix Section B.2.
4.2. Comparison with State-of-the-Art Methods
Evaluation Protocol. We evaluate the models on each client's 's private test data, which follows the same distribution as its training set. The reported results represent the average test accuracy across all clients over three different seeds. For fairness, we use the same prompt length as other models for comparison.
Implementation Details. All methods frozen CLIP model with two backbones: ResNet50 (He et al., 2016) and ViT-B16 (Dosovitskiy et al., 2021), with ViT-B16 as the default. Local training rounds are set to E = and federated communication rounds to
Single-Domain Model Evaluation. To verify that the proposed method remains effective with a fixed prompt length, first evaluate FedPHA against baseline methods single-domain CLIP datasets under pathological non- IID setting. For ease of comparison, Table presents results using the 16-shot setting. As shown in the table, FedPHA consistently outperforms tate-offthhe-rt algorithms across all datasets, demonstrating the effectiveness of our global- local prompt separation mechanism in single-domain sce- narios. Notably, on the Food101 dataset. FedPHA achieves 3.46% performance gain over the best competing method, further highlighting its superiority. An analysis of conver- gence speed is provided in Appendix Section C.I.
50, except for CIFAR-10 and CIFAR-100, where = 25. Final perfor- mance is averaged over the last 10 communication rounds. The number of clients varies by dataset. CLIP datasets (Food101, DTD, Caltech101, Flowers OxfordPets) use N = 10, with each client holding a distinct class subset. Multi-domain datasets 1Office31. OfficeHome) set N to twice the number of domains, assigning each domain's 's data 100,
to two clients. CIFAR-I0 and CIFAR- 100 use with each client randomly assigned 10% of the dataset. For learnable prompts, the default length is 16 with a 512- dimensional representation. In heterogeneous settings, local prompt lengths range from 4 to 32, while the global prompt length remains 16. The batch size is 32 for training and 128 for testing. For hyperparameter settings, the ratio (p
Impact of Number of Shots. Additionally, we explore the impact of the number of shots on FedPHA. To ana- lyze this, we vary the number of shots during training from [1, 2, 4. 8, 16). As shown in Figure 3, FedPHA consistently

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
PromptFL
FedPGP
FedDTP
FodPHA
Caltech101
Food1.01
Flowers102
DxfordPets
100
DTD
99
98
Accuracy
Accuracy
92
Accuracy
85
Accuracy %%
Accuracy %
60
95
88
94
93
84
75
91
80
90
12
Shot Number
Shot Number
Shot Number
Shot Number
Shot Number
Figure 3: Ablation study on the number of shots. The x-axis represents the number of shots, and the y-axis denotes the average test accuracy. Each curve corresponds to a different method, with error hars indicating standard deviations across multiple random seeds.
Table 3: Runtime overhead of SVD-based prompt projection. All results are averaged over 10 communication rounds. Compared to model training, the SVD overhead is negligible
and backward passes- dominates the runtime, taking over seconds per round. These results indicate that the added SVD step does not introduce any significant computational bottleneck, even under large-scale settings. This lightweight design confirms that the benefits of SVD-based global-local prompt separation come at almost no cost, further support- ing the practicality of our method in federated settings where efficiency is critical.
Operation Stage
Time (ms)
Overhead
Global prompt decomposition
4.2
<1%
Local prompt projection
2.8
<1
Local model training
4536.8
4.3. Effectiveness of prompt length heterogeneity
outperforms other methods across all shot settings. In par- ticular, when the number of shots is small, other methods experience significant performance degradation, whereas FedPHA exhibits only a slight decline compared its per- formance at 16 shots. This demonstrates the robustness of approach, enabling effective and rapid adaptation to personalized client requirements even in few-shot scenarios.
Evaluation Protocol. In this set of experiments, each client uses a different prompt length ranging from 4 to 32. For multi-domain datasets, the specified prompt lengths are applied, whereas for CIFAR-10/100 each client is assigned randomly selected prompt length.
Cross Domain Analysis. We investigate the impact of different prompt length combinations on cross-domain per- formance. In the Office31 dataset with three domains, each domain has eight possible prompt length choices: [4, 8, 12, 16, 20, 24, 28, 32]. After evaluating 512 combinations, we identify the optimal combination as [28, 12, 16], achiev- ing accuracy of 95.45%. Figure 4 visualizes the results, where the color intensity of each cell represents the global accuracy for a given prompt length combination. Black- outlined cells indicate cases where all domains use the same prompt length. Notably, most high-accuracy points fall out- side these black-boxed regions, suggesting that the conven- tional approach of assigning the same prompt length to all clients is suboptimal and fails to capture the varying prompt length requirements introduced by data heterogeneity, In contrast, FedPHA enables each client to adopt different prompt lengths, demonstrating its effectiveness in handling heterogeneous data distributions. More details and addi- tional experiments on the OfficeHome dataset are provided in Appendix Section C.2.
Multi-Domain Model Evaluation. We also evaluate the performance of FedPHA in comparison to baseline methods on multi-domain datasets. To simulate client heterogeneity, we partition data within the same domain into two clients using a Dirichlet distribution (3 0.5). We analyze both the average performance of clients within the same domain and the overall mean performance across all clients. Results for the Office31 and OfficeHome datasets are summarized in Table 2. Our method consistently outperforms baseline approaches. For instance, on the Office31 dataset, FedPHA outperforms FedPGP across all domains, further validating its effectiveness in handling heterogeneous data distribu- tions. These results demonstrate the robustness of FedPHA under diverse domain settings.
Computational Cost of SVD. To evaluate the efficiency of the proposed SVD-based prompt projection mechanism, we measure its runtime cost on the client side. As summarized in Table 3, the additional overhead introduced by SVD is minimal compared standard model training. Specifically, global prompt decomposition and local prompt projection take 4.2 ms and 2.8 ms per communication round on aver- age, each contributing less than 1% to the total training time. Importantly, these operations are performed only once per round, not per batch, making their amortized cost negligi- forward
Intra Domain Analysis. Furthermore, explore the im- pact of prompt length on client performance across different length combinations. The two clients within the same do- main use identical prompt lengths. As shown in Figure 5, we illustrate the effect of prompt lengths on the performance
ble. In contrast, local model
8

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Domaini
Domain2
Domain1 16
Domain1
Domaini PLes
Domaini
Domain1
An T-222
Comain2 Lan (4.22)
Coma K2 Jar (632)
Domwint Ler 14221
Demakrd P Len [1220] Deman22 Ler (432))
Domaind Lee (4.32)
Bemain Len - (4.32)
Demain2 J.n (1,72)
Figure 4: Impact of Prompt Length on Domains Performance across different length combinations on overall accuracy. Each 8x8 heatmap illustrates the effect of different Domain 2 and Domain prompt length combinations on model performance when Domain has a fixed prompt length in Office31 dataset. The X-axis represents the average prompt length of Domain while the Y-axis represents that of Domain Color intensity indicates accuracy, with red representing higher accuracy and blue representing lower accuracy. Black-hoxed grids highlight cases where the prompt length of Domain matches that of Domain 2 or 3 under the current length combination.
Client
Client
Client
Table Comparison with the SOTA methods and FedPHA variants (fixed vs. random prompt length) on CIFAR-10 and CIFAR-100 across 100 clients. All baseline methods use fixed prompt length of 16. FedPHA (fixed length) also uses 16 tokens for all clients, while FedPHA (random length) assigns each client random prompt length between and 32.
90
Accuracy
Accuracy
ne.
99
88
91
300
12
20 24 32
12 15 20
12
20 24 28 32
Prompt Length
Prompt Length Client
Prompt Length Clent
Methods
CIFAR-10
CIFAR-100 100
Client
CLIP (Radford et al., 2021)
87,88+0.11 11
64.89+0.19
82
Accuracy
PromptFL (Guo et al.. 2023b)
91.70=0.11
72.58+0.04
Prompt+Prox (Li al., 2020)
91.83+0.12
72.08+0.09
80-
96
FedPGP (Cui al., 2024)
92.1010.21
10.81
00
FedOTP (Li et al., 2024a)
93.43+#041
75.07+0.39
12
12
12
Prompt Length
Prompt Length
Prompt Length
FedPHA (fixed length)
94.11#0.14
75.92+0.13
FedPHIA (random length)
93,80=0.17
75.63+0.17
Figure 5: Impact of Prompt Length on Client Performance across different length combinations. Clients (1,2), (3,4), and (5,6) belong the same domain, respectively. The X-axis represents the prompt length, while the Y-axis shows the client's accuracy. The box depicts the IQR, capturing the middle 50% of values, with the horizontal line inside indicating the median accuracy. Whiskers extend to the min/max within 1.5 times the IQR, and outliers appear points beyond them, highlighting deviations.
(4-32 tokens) prompts. The fixed-length variant slightly outperforms the random one, not due to design limitations, but because random lengths may not align with each client's data distribution, leading to occasional inefficiencies. In con- trast, fixed lengths ensure stable optimization and aggrega- tion. Nevertheless, the random-lengtt setting better reflects real-world client heterogeneity, and FedPHA is uniquely ca- pable of operating under such conditions. Future work may explore adaptive prompt assignment strategies to further improve performance heterogeneous environments.
of each client. We observe that clients within the same do- main exhibit similar sensitivity to prompt length, even when their data distributions differ. For instance, both Client 1 and Client 2 achieve the best performance with a prompt length of 32, while the worst performance occurs at a prompt length of 4. However, the sensitivity prompt length varies across domains. For example, while Client I's optimal prompt length is 32, Client 3 performs best at prompt length of 16 and performs poorly at 32.
5. Conclusion
This paper proposes a novel and effective method of Fed- PHA for federated prompt learning. FedPHA is capable of handling heterogeneity problem and alleviating conflicts between global and local knowledge, In particular, we de- sign a G-L heterogeneous federated prompt architecture to effectively accommodate varying prompt lengths. Mean- while, we introduce SVD-based projection and bidirectional alignment to reduce the negative impact caused by the con- flict between global and local prompts. Experimental re- sults on classification tasks demonstrate that our method outperforms state-of-the approaches and validate the effectiveness of prompt length heterogeneity.
Robustness Analysis. On the CIFAR datasets, we randomly select prompt lengths to evaluate robustness. Table com- pares state-of-the-aa methods under the Dirichlet setting (3 0.5). FedPHA achieves the best performance on both datasets, demonstrating strong generalization in non-III scenarios. Its improvement on C1F further high- lights the effectiveness of the global-local prompt separation mechanism in balancing personalization and global perfor- mance. To examine prompt length adaptability, we compare FedPHA with fixed-length (16 tokens) and random-length

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Acknowledgement
Heigold, G., Gelly, S., et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
This work is supported by the National Key Research and Development Program of China (2023YFC2705700) and National Natural Science Foundation of China under Grant (62361 166629, 62176188, 62225113, 623B2080), the Wuhan University Undergraduate Innovation Research Fund Project. The supercomputing system at the Supercomput- ing Center of Wuhan University supported the numerical calculations in this paper.
Fallah, A., Mokhtari, A., and Ozdaglar, A. Personalized federated learning with theoretical guarantees: A model- agnostic meta-learning approach. In NeurlPS, pp. 3557- 3568, 2020.
Fang, X. and Ye, M. Robust federated learning with noisy and heterogeneous clients. In CVPR, 2022.
Impact Statement
Fei-Fei, L. Learning generative visual models from few training examples. In Workshop on Generative-Modee Based Vision, IEEE Proc. CVPR, 2004, 2004.
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Feng, C.-M., Li, B., Xu, X., Liu, Y., Fu, H., and Zuo, W. Learning federated visual prompt in null space for mri reconstruction. In CVPR, 2023.
References
Bai, S., Zhang, J., Li, S., Guo, S., Guo, J., Hou, J., Han, T., and Lu, X. Diprompt: Disentangled prompt tuning for multiple latent domain generalization in federated learning. In CVPR, 2024.
Golub, G. H., Hoffman, A., and Stewart, G. W. A general- ization of the eckart-young- matrix approximation theorem. Linear Algebra and its applications, 88:317- 327, 1987.
Bossard, L.,, Guillaumin, M., and Van Gool, L. Food-101- mining discriminative components with random forests. In ECCV, pp. 446-461, Springer, 2014.
Guo, T., Guo, S., and Wang, J. Pfedprompt: Learning per- sonalized prompt for vision-language models in federated learning. In Proceedings of the ACM Web Conference 2023, pp. 1364-1374, 2023a.
Cao, Y.-T., Shi, Y., Yu, B., Wang. J., and Tao, D. Knowledge- aware federated active learning with non-iid data. In ICCV. pp. 22279-22289, 2023.
Guo, T., Guo, S., Wang, J., Tang, X., and Xu, W. Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. JEEE TMC, 2023b.
Chen, H., Huang, S., Zhang, D., Xiao, M., Skoglund, M., and Poor, H. V. Federated learning over wireless iot networks with optimized communication and resources. IEEE Internet of Things Journal, pp. 16592-16605, 2022.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Chen, H.-Y. and Chao, W.-L. On bridging generic and personalized federated learning for image classification. In ICLR, 2022.
Hong, S. and Chae, J. Communication--eficciee random- ized algorithm for multi-kerne' online federated learning. IEEE PAMI, 44(12):9872- 9886, 2021.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures the wild. In CVPR, pp- 3606-3613, 2014.
Hsu, Y.-C., Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112 2022.
Cui, T., Li, H., Wang, J., and Shi, Y. Harmonizing general- ization and personalization in federated prompt learning. arXiv preprint ar arXiv2405.00977) 2024.
Hu, M., Zhou, P., Yue, Z., Ling. Z., Huang, Y., Li, A., Liu, Y., Lian, X., and Chen, M. Fedcross: Towards accu- rate federated learning via multi-model cross-aggregation. In IEEE International Conference on Data Engineering (ICDE), pp. 2137-2150. IEEE, 2024.
Dai, S., Zhou, Y., Pang, L.,, Liu, W., Hu, X., Liu, Y., Zhang, X., Wang, G., and Xu, J. Neural retrievers biased towards llm-generated content. In ACM SIGKDD, pp. 526-537, 2024.
Huang, W., Ye, M., and Du, B. Learn from others and be yourself in heterogencous federated learning. In CVPR, 2022.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.,
10

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Huang, W. Ye, M., Shi, Z., and Du, B. Generalizable heterogeneous federated cross-correlation and instance similarity learning. TPAMI, 2023a.
Li, Z., Li, X., Fu, X., Zhang. X., Wang, W., and Yang, J. Promptkd: Unsupervised prompt distillation for vision- language models. In CVPR, 2024b.
Huang, W., Ye, M., Shi, Zz, Li, H., and Du, B. Rethinking federated learning with domain shift: A prototype view. In CVPR, 2023b.
Lu, W., Hu, X., Wang, J., and Xie, X. Fedclip: Fast general- ization and personalization for clip in federated learning. IEEE DEB, 2023.
Huang, W., Ye, M., Shi, Z., Wan, G., Li, H., Du, B., and Yang, Q. A federated learning for generalization, robust- ness, faimess: A survey and benchmark. TPAMI, 2024.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. Communication-efii effiiccinntfii learning of deep networks from decentralized data. In AISTATS, pp. 1273- 1282. PMLR, 2017.
khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi- Modai prompt learning. In CVPR, 2023.
Nguyen, M. D., Le, K., Do,, K., Tran, N. H., Nguyen, D., Trinh, C., and Yang, Z. Towards layer-wise personalized federated learning: Adaptive layer disentanglement via conflicting gradients. arXiv preprint vrXiv:2410.02845. 2024.
Khattak, M. U., Wasim, S. T., Naseer. M., Khan, S., Yang, M.-H., and Khan, F. S. Self-regulating prompts: Founda- tional model adaptation without forgetting. ICCV, pp- 15190-15200, October 2023.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Krizhevsky, A., Nair, V., and Hinton, G. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar: himl, 5(4):1, 2010.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In CVPR, pp. 3498-3505. IEEE, 2012.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
Kulkarni, V., Kulkarni, M., and Pant, A. Survey of person- alization techniques for federated learning. In WorldS4, pp. 794-797, 2020.
Lan, M., Torr, P., Meek, A., Khakzar, A., Krueger, D., and Barez, F. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint arXiv:2410.0698/1 2024.
Qiu, C., Li, X., Mummadi, C. K., Ganesh, M. R., Li, Z., Peng, L., and Lin, W.-Y. Text-driven prompt generation for vision-language models in federated learning. arXiv preprint arXiv:2310.0612. 2023.
Li, H., Cai, Z., Wang. J., Tang, J., Ding, W., Lin, C.-T., and Shi, Y. Fedtp: Federated learning by transformer personalization. IEEE TNNLS, 2023.
Qu, Z., Li, X., Duan, R., Liu, Y., Tang, B., and Lu, Generalized federated learning via sharpness aware mini- mization. In ICML, 2022.
Li, H., Huang, W., Wang, J., and Shi, Y. Global and local prompts cooperation via optimal transport for federated learning. In CVPR, pp. 12151-12161, 2024a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8766 PMLR, 2021.
Li, Q. He, B., and Song, D. learning. In CVPR, pp. 10713-10722, 2021a.
federated
Li, Q., Diao, Y., Chen, Q., and He, B. Federated learning on non-iid data silos: An experimental study. IEEE TKDE, 2022.
Robbins, H. and Monro, S. A stochastic approximation method. AoMS, pp. 400-407, 1951.
Li, T., Sahu, A. K., Zaheer, M.. Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. In MLSys, 2020.
Saenko, K., Kulis, B., Fritz, M., and Darrell, T. Adapting visual category models to new domains. In ECCV, pp. 213-226, 2010.
Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. bbn:: Federated learning on non-(iid) features via local batch normalization. In ICLR, 2021b.
Saha, R., Srivastava, V., and Pilanci, M. Matrix compression via randomized low rank and low precision factorization. NeurIPS, 36, 2023.

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Shamsian, A., Navon, A., Fetaya, E., and Chechik, G. Per- sonalized federated learning using hypemetwwrks. In ICML, pp. 9489-9502, 2021.
Yang, Q., Liu, Y., Chen, T., and Tong, Y. Federated machine learning: Concept and applications. ACM TIST, pp. 1-19, 2019.
Sharma, P., Ash, J. T., and Misra, D. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13556 2023.
Ye, R., Xu, M., Wang, J., Xu, C., Chen, S., and Wang, Y. Feddisco: Federated learning with discrepancy-aware collaboration. In ICMI., pp. 39879-399022 PMLR, 2023.
Yuan, Z., Shang, Y., Song, Y., Wu, Q.. Yan, Y., and Sun, G. Asvd: Activation-awan singular value decomposition for compressing large language models. arXiv preprint arXiv:23/2.05821, 2023.
Su. S., Yang, M., Li, B., and Xue, X. Federated adaptive prompt tuning for multi-domain collaborative learning. In AAAI, pp. 15117-15125, 2024.
Zhao, H., Du, W., Li, F., Li, P., and Liu, G. Fedprompt: communication-efficcie and privacy-preserving prompt tuning federated learning. In ICASSP, pp. 1-5. IEEE, 2023.
Tan, Z. Wan, G., Huang, W., Li, H., Zhang, G., Yang, C., and Ye, M. Fedspa: Generalizable federated graph learning under homophily heterogeneity, 2025.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan, S. Deep hashing network for unsupervised domain adaptation. In CVPR, pp. 5018-5027, 2017.
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. CVPR, 2022a.
Wan, G., Huang, W. and Ye, M. Federated graph learn- ing under domain shift with generalizable prototypes. In Proceedings of the AAA/ Conference on Artificial Intelli- gence, volume 38, pp. 15429-15437, 2024.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. IJV,, 130(9):2337- 2348, 2022b.
Zhu, B., Niu, Y., Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. In ICCV, pp. 15659 15669, 2023.
Wang, B., Li, H., Liu, X., and Guo, Y. Frad: Free-rider attacks detection mechanism for federated learning in aiot. IEEE loT-J, 2023.
Wang, Liu, Q., Liang, H., Joshi, G., and Poor, H. V. Tackling the objective inconsistency problem in hetero- geneous federated optimization. In NeurlPS, pp. 7611- 7623, 2020.
Wang. X., Zheng, Y., Wan, Z., and Zhang, M. Svd- 11m: Truncation-awarc singular value decomposition for large language model compression. arXiv preprint 240.023003.7777. 2024.
Wu, W., He, L., Lin, W., Mao, R., Maple, C., and Jarvis, S. Safa: A semi-asynchronous protocol for fast feder- ated learning with low overhead. IEEE Transactions Computers, 7065):655-668, 2020.
Xu, C., Qu, Y., Xiang, Y., and Gao, L. Asynchronous federated learning on heterogeneous devices: survey. arXiv preprint arXiv:2109.04266 2021.
Xu, J., Tong, X., and Huang, S.-L. Personalized federated learning with feature alignment and classifier collabora- tion. In ICLR, 2023.
Yang, F.E.,, Wang, C.-Y., and Wang, Y.-C. F. Efficient model personalization federated learning via client- specific prompt generation. In ICCV, pp. 19159-19168, 2023.
12

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
A. Method Details
and local prompts, FedPHA avoids negative transfer from global representations to client-specific learning, thereby enhancing the robustness and adaptability of personaliza- tion in federated settings.
A.I. Notations Definition
To facilitate understanding of our proposed FedPHA, we provide a summary of key notations used throughout this paper in Table 5.
B. Experimental Details
Table 5: Summary of Notations.
B.1. Details of Dataset Setup
Symbol
Definition
Symbol
Definition
For our evaluation, we selected nine diverse visual classifi- cation datasets as benchmarks. Table 6 provides detailed overview, including the original task, number of classes, training and test sample sizes, and domain counts.
N
Clients number
Loss function
Client index
Push loss margin
D;
Local dataset
U, S.S.V V
SVD matrices
Dutaset size
P
Null-space ratio
For datasets with multiple domains, we followed the well- established Office-31 benchmarking protocol, which in- cludes three domains: Amazon (A), Webcam (W), and DSLR (D). These domains represent variations in image quality and style, capturing differences between online prod- uct images (Amazon), low-resolution webcam photos, and high-resolution DSLR images. Additionally, we incorpo- rated Home, which consists of four domains: Art (Ar), Clipart (CI), Product (Pr), and Real-World (Rw). These domains encompass diverse image sources, including artis- tic renderings, clipart illustrations, product photos, and nat- ural scene images, effectively capturing distribution shifts across different acquisition methods and environments.
C,
Clients in round
d'
Null-space dim.
Local epochs
V/2
Null-space basis
pre
Global prompt
Q
Projection matrix
9.
P
Local prompt
Y
CLIP text input
Projected prompt
Padding g(-)
Zero padding
P ((401,0)
Aggregated prompt
Text encoder
T
Global prompt length
f(-)
Image eneader
T
Local prompt length
9p
Text feature
Learning rate
f
Image feature
Softmax temperature
sim(.)
Cosine similarity
A.2. Discussion
By leveraging these datasets, our evaluation ensures a com- prehensive assessment of model performance across varying domains and real-world conditions.
Distinction from Existing G-L Prompt Methods. Here, we provide a more detailed analysis of how our approach fundamentally differs from prior work in terms of architec- ture design and personalization flexibility.
B.2. Details of Implementation
FedOTP (Li et al., 2024a) adopts dual-prompt structure consisting of global prompt and a local prompt. How- ever, it requires both prompts to be of equal length due to the constraints of its unbalanced optimal transport frame- work. This architectural constraint significantly limits flexibility and the ability to tailor local representations to client-specific needs.
The optimizer used is Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951) with learning rate of r) = 0.001. All input images are resized 224 x 224 pixels and fur- ther divided into 14 x 14 patches with a dimension of 768. We conducted all experiments with Py Torch (Paszke et al., 2019) on NVIDIA RTX 3090 GPUs.
FedPGP (Cui et al., 2024) employs a global prompt along- side two local adapters. In this setup, the local prompt is generated by adding a local adapter to the global prompt, resulting in a tightly coupled formulation. This additive dependency forces the local prompt inherit features from the global prompt, which may be suboptimal. In scenarios where the global prompt is poorly aligned with a client's local data, this coupling can lead to negative transfer, reducing the effectiveness of personalization.
B.3. Details of Baseline Implementation
To ensure fair and transparent comparison with existing Global-Local prompt methods, we re-implemented both FedOTP (Li et al., 2024a) and FedPGP (Cui et al., 2024) un- der a unified experimental framework. All experiments followed the protocol described in Section 4.1 and Ap- pendix B.2, including identical training schedules, model architectures, and optimization settings.
FedPHA (Ours) introduces a decoupled G-L prompt structure where each client receives a shared fixed-length global prompt and independently configures its local prompt with a variable length. This design explicitly supports heterogeneous prompt configurations, enabling better alignment with diverse data distributions and com- putational capacities. Importantly, by decoupling global
For all methods, we used frozen CLIP backbone (VIT- B/16), with prompt length of 16 and embedding dimension 512. Local training was performed using SGD with a learn- ing rate of 0.001 and a batch size of 32. Each client trained for one local epoch per round, and we ran 50 communication rounds (reduced to 25 for CIFAR- 10 and CIFAR- 100). In ad-
13

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Table Statistical details datasets used experiments.
Dataset
Classes
Train
Test
Domains
Task
Caltech101 (Fei-Fei, 2004)
100
4,128
2,465
1
Object recognition
Food101 (Bossard et al., 2014)
101
50,500
30,300
Finc-grained food recogmition
Flowers102 (Nilshack & Zisserman, 2008)
102
4,093
2,463
Fine-grained flower recognition
OxfordPets (Parkhi al., 2012)
37
2,944
3,669
Fine-grained pel recognition
DTD (Cimpoi et al., 2014)
47
2,820
1,692
Texture classification
Office31 (Saenko et al., 2010)
31
3,292
813
3
Multi-domain image recognition
OfficeHome (Venkateswara et al., 2017)
65
12,475
3,113
Multi-domain image recognition
CIFARIO (Krizhevsky et al., 2010)
10
50,000
10,000
1
General image classification
CIFAR 100 (Krizhevsky Hinton, 2009)
100
50,000
10,000
General image classification
PromptfL
FedRGP
FedOTP
FedPHIA
Cateec101
Food101
Flowers102
OxfardPatt
DTD
100
95-
50
Accurscy
70-
70
60
10
20
10
20
20
10
20
20
20
20
40
10
20
30
Figure 6: Comparison with the SOTA methods of convergence speed on single-domain datasets across 10 clients. The x-axis represents training rounds (from 0 to 50), while the y-axis shows the model accuracy over the course of training.
dition to the shared setup, each baseline has its own method- specific parameters. For FedOTP, used an unbalanced optimal transport formulation (COT), with Sinkhorn pa- rameters set to THRESH = 1 and EPS = 0, and maximum iteration limit of 100. For FedPGP, we followed its original implementation using a bottleneck dimension of 4 and contrastive loss parameters μ = 1 and temperature 0.5. Other hyperparameters were kept consistent across both baselines, including disabling context initialization (CTX TNIT Fall se), disabling class-specific context prompts (CSC False), using mixxd-preciiion training (fp16) 6), and placing the class token at the end of the se- quence.
oog d o0 a0 Boggg do goge
16
05 pooodoasee o 00 pood0s 00
-9..0
99.5
Domain2 Lon
20
16
12 Domaint P Len
12
These configurations ensure that any observed performance differences arise from algorithmic or architectural factors, rather than inconsistencies training conditions. All experi- ments repeated across three random seeds for statistical robustness.
28
Figure 7: Prompt length combination effects Office31. The X. Y. and Z axes represent the average prompt lengths for Domain 1. Domain and Domain 3. respectively. The color intensity of each point indicates the global accuracy achieved under the corresponding prompt length configuration.
C. Additional Experiments Results
C.1. Convergence Analysis
formance of four different training methods: PromptFL, FedPGP, FedOTP, and FedPHA. It can be observed that our FedPHA consistently converges faster than other meth- ods, and its accuracy remains higher than that of other ap- proaches at all stages. This demonstrates the effectiveness of our method in personalization.
Figure 6 presents a comparison of convergence speed across five different datasets (Caltech101, Food101, Oxford Flow- ers, Oxford Pets, and DTTD) over 50 training rounds. Each subfigure represents distinct dataset, illustrating the per-
14

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Domaini
Domain
Domaini Pies
Domainá
Domain1
an 1-222
Comain Lan (4:22)
Coma F La (X32)
Donsnn Len 14321
Demalrd Len [122]
Deman2 Ler (433)
Domain Lee (6.32)
Eemand Len - (432)
Comain2 Jn (1,22)
Figure 8: Impact of Prompt Length on Domain Performance in OfficeHome. Each 8x8 heatmap illustrates the effect of different
Domain 2 and Domain prompt length combinations on global accuracy, when the prompt length of Domain
fixed and Domain 4
set to 16. The X-axis represents the prompt length of Domain 2, while the Y-axis represents that of Domain 3. Color intensity indicates
accuracy, with red representing higher accuracy and blue representing lower accuracy. Black-boxed grids highlight configurations where the prompt length of Domain matches that of Domain 2 3.
C.2. Inter Domain Analysis
Table 7: Comparison Overall Mean values across different domains with different prompts lengths.
89.6
oo oo aon do Ro Begg oogo gogaggg oogo Gogegegoe degse os aoaooooo de 10ooo0o0e d00060000 00t0 90 aeopaeade 00 00 08
28
ID
Domain
Domain2
Domain3
Overall Ace
28
12
16
95.45
-89.2
20
12
28
95.19
24
12
32
95.15
-88.0
4
4
95.13
32
16
95.12
20
8
28
95.12
Domaain2
2g
4
28
95.09
Domain2 Lon
24
88.6
16
32
95.05
32
16
32
95.05
Figure Prompt length combination effects on OfficeHome. The X, and axes represent the average prompt lengths for Domain 1, Domain 2, and Domain 3, respectively, with Domain 4 fixed at length 16. The color intensity of each point reflects the global accuracy.
10
20
4
32
95.04
512
32
28
12
93.04
Mean of 512 combinations
94.36
Additionally, investigated the impact of different prompt length combinations overall performance, as shown in Figure 7. The eight heatmaps in Figure 4 can be regarded as planar slices of Figure 7, providing a more granular view of how prompt length variations influence accuracy across different domains. Table 7, we list the average accuracy of 512 prompt length combinations sorted in descending order. From the table, can observe that the only well- performing combination with uniform lengths is [4,4,4], while all other combinations have varying lengths. This fur- ther validates our research motivation: for clients with dif- ferent data distributions, adopting different adaptive prompt lengths more beneficial for personalization.
rowing the search space a manageable 512 combinations,
The corresponding 3D scatter plot is illustrated in Figure 9, where each point represents a unique prompt length combi- nation for the first three domains. The spatial distribution reveals clear patterns: high-performing configurations tend to cluster in regions where the prompt lengths differ across domains, again suggesting that uniform configurations are not optimal. This observation is consistent with our earlier findings from Office31.
In addition, Figure 8 presents eight heatmaps with fixed prompt lengths for Domain offering a slice-by-slice visu- alization across Domain2 and Domain3. Similar to Office31, -lack-boxed cells indicate uniform prompt lengths across all domains. These boxed areas rarely correspond to the highest accuracy regions, reinforcing the insight that heterogeneity- aware prompt length selection is critical for maximizing performance in multi-domain federated settings.
Beyond Office31, we extended our inter-domain prompt length analysis the OfficeHome dataset, which comprises four distinct domains. Given the increased number of clients, the total number of possible prompt length combinations grows exponentially (8* = 4096), making exhaustive eval- uation computationally infeasible. To reduce complexity while maintaining representative coverage, we constrained the prompt length of the fourth domain 16, thereby nar-
Taken together, these findings provide converging evidence across datasets: the assumption that all clients benefit from the same prompt length is fundamentally flawed. Instead,
15

---

FedPHA: Federated Prompt Learning for Heterogeneous Client Adaptation
Table Sensitivity analysis projection ratio p and push margin a on 5 datasets. FedPHA achieves robust performance across a wide range of settings with shot number 16. Default values are p = 0.8, a = 1.0.
(a) Average over 5 datasets.
(b) Caltech101.
a
0.3
0.5
0.8
1.0
Avg.
a' p
0.3
0.5
0.8
Avg.
0.5
97.05
97.04
96.99
96.92
97.00
0.5
98.97
98.99
98.91
98.97
98.96
1.0
97.04
97.02
97.13
97.08
97.07
1.0
99.01
99.05
99.12
99.04
99.06
1.5
96.94
96.99
96.94
96.95
96.96
1.5
98.97
99.07
98.93
98.95
98.98
2.0
96.88
96.82
96.85
96.87
96.85
2.0
98.89
98.77
98.88
98.84
98.84
Avg.
96.98
96.97
96.98
96.96
96.97
Avg.
98.96
98.97
98.96
98.95
98.96
(c) F0od101.
(d) Flowers
a /
0.3
0.5
0.8
1.0
Avg.
a /
0.3
0.5
0.8
1.0
Avg.
0.5
96.22
96.27
96.23
96.27
96.25
0.5
99.33
99.16
99.23
99.24
99.24
1.0
96.35
96.38
96.42
96.33
96.37
1.0
99.28
99.19
99.23
99.32
99.25
1.5
96.61
96.51
96.49
96.48
96.52
1.5
99.01
99.14
99.14
99.12
99.10
2.0
96.48
96.42
96.43
96.38
96.43
2.0
99.17
98.78
99.08
99.07
99.02
Avg.
96.42
96.39
96.39
96.36
96.39
Avg.
99.20
99.07
99.17
99.19
99.16
(e) OxfordPets.
(i) DTD.
a /
0.3
0.5
0.8
1.0
Avg.
a / /p
0.3
0.5
0.8
1.0
Avg.
0.5
99.15
99.06
99.13
99.08
99.11
0.5
91.59
91.74
91.45
91.02
91.45
1.0
99.24
99.18
99.21
99.14
99.19
1.0
91.31
91.30
91.67
91.57
91.46
1.5
99.08
99.27
99.03
99.06
99.11
1.5
91.02
90.96
91.12
91.16
91.06
2.0
99.06
98.94
98.86
99.01
98.97
2.0
90.82
91.17
90.98
91.07
91.01
Avg.
99.13
99.11
99.06
99.07
99.09
Avg.
91.19
91.29
91.31
91.20
91.25
our FedPHA approach, which allows client-specific prompt length adaptation, effectively captures domain-level vari- ability and leads to superior cross-domain generalization.
deployments where fine-tuning may not always be feasible. For Caltech 101 and OxfordPets, performance remains re- markably stable, exceeding 98.8% across all tested settings, with optimal configurations coinciding with the default val- ues. Food101 exhibits a mild preference for larger a (es- pecially 1.5), indicating that a stronger push margin may help in fine-grained classification tasks. Flowers 102 shows high tolerance to both a and p changes, suggesting that the method generalizes well in image domains with intra- class similarity. In contrast, DTD reveals greater sensi- tivity to hyperparameter shifts. Nevertheless, its accuracy remains within relatively narrow and acceptable range 190.8%-91.794 9.,%), demonstrating that FedPHA retains com- petitiveness even under more challenging visual domains.
C.3. Sensitivity Analysis
Table reports a detailed sensitivity analysis of FedPHA with respect to the projection ratio p in (q) and the push margin in (12) across five datasets. Each cell in the table presents the average accuracy over the last 10 training epochs under a specific configuration.
As shown in Table 8 (a), the average accuracy across all datasets remains consistently high across a broad range of (a, p) combinations. The best overall performance (97.13%) is achieved when = 1.0 and p 0.8, which are also the default values used throughout our main experi- ments. Importantly, performance degradation remains mini- mal-typically within 0.2%-even when deviating from this configuration. This reflects the robustness of FedPHA and its insensitivity to moderate variations in hyperparameter settings, which is a desirable property in practical federated
This table-based analysis confirms that FedPHA achieves strong and stable performance under wide spectrum of hyperparameter choices. The default configuration (a=1.0, 0=0.8) serves as robust and well-balanced setting across diverse data distributions, minimizing the need for extensive tuning in real-world federated learning scenarios.
16


Reviews:
[{"review_id": "bturyzw2TU", "reviewer": "Reviewer_A325", "summary": "This paper introduces FedPHA, a novel Federated Prompt Learning (FPL) approach that enables heterogeneous client adaptation using Vision-Language Models (VLMs). The key contributions include:\nA dual-layer architecture combining a fixed-length global prompt for efficient aggregation and variable-length local prompts for personalization. A Singular Value Decomposition (SVD)-based projection mechanism to resolve conflicts between global and local prompts. A bidirectional alignment strategy to optimize the interaction between global knowledge and local adaptation. Extensive empirical validation on various datasets, demonstrating superior performance over state-of-the-art FL and FPL methods. The paper is technically sound, well-motivated, and presents strong experimental results. However, a few areas need further clarification and improvements, particularly regarding real-world feasibility, communication efficiency, and computational overhead.\n## update after rebuttal\nThanks to the authors for their thoughtful and comprehensive rebuttal. I am satisfied with the authors' responses and believe they addressed the concerns raised in the initial review. I maintain my positive evaluation of the paper and recommend acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer A325:\n\nWe appreciate your recognition of our technical contributions—specifically the dual-layer prompt design, SVD-based projection, and bidirectional alignment—as well as your acknowledgment of our method’s strong performance and adaptability to heterogeneous clients. Below, we address your concerns regarding real-world feasibility, communication efficiency, and computational overhead.\n\n### Weakness\n\n**W1: Communication Overhead**\n\nA1: Thank you for this important observation. While **FedPHA** introduces communication through prompt exchange, we emphasize that prompts are significantly smaller in size compared to full model weights—typically in the order of **KBs vs. hundreds of MBs** for vision-language models (VLMs).\n\nAlthough we do not report the classification accuracy of **FedAvg with full CLIP models** due to **prohibitive computational and communication costs** in our experimental environment, we include it in the comparison table to highlight the dramatic difference in communication overhead. Specifically, FedAvg requires transmitting approximately **497.3 MB per round**, which is over **7,700× larger** than prompt-based methods (64 KB). This made full-model training infeasible within our available resources. Prior studies show that prompt-based methods can match or exceed full-model FL under non-IID settings. We thus focus on efficient prompt-based alternatives better suited for real-world, resource-limited scenarios.\n\nIn the final version, we will add a more detailed quantitative analysis of communication overhead, comparing **FedPHA** with traditional FL and prompt-based FL (FPL) baselines, and demonstrate that FedPHA achieves a highly favorable trade-off between communication efficiency and classification performance.\n\n**Table: Comparison of communication overhead and classification accuracy across different methods**\n|Method|Transmitted Content|Total Comm. / Round|Caltech101 (%)|Office31 (%)|CIFAR-100 (%)|\n|-|-|-|-|-|-|\n|FedAvg+CLIP|Full model weights (CLIP ViT-B/16)|497.3 MB|—|—|—|\n|PromptFL|Prompt (configurable)|64 KB|93.47|88.08|72.58|\n|FedOTP| Global prompt (configurable)|64 KB|98.11|89.19|75.07|\n|FedPGP| Global prompt (configurable)|64 KB|95.86|91.58|74.81|\n|**FedPHA (Ours)**|Global prompt (configurable)|**64 KB**|**99.05**|**94.74**|**75.63**|\n\n**W2: Computational Cost of SVD**\n\nA2: We acknowledge the reviewer’s concern regarding the additional computation introduced by the SVD-based projection. However, this operation is performed only **once per communication round**, rather than per training batch, making the overall overhead negligible across multiple rounds.\n\nOn the client side, both **global prompt decomposition** and **local prompt projection** involve only **low-rank updates**, which are lightweight and computationally efficient. On the server side, since only global prompts are aggregated—not decomposed or projected—the SVD operation incurs no additional computational burden.\n\nMoreover, the global prompt typically has a **modest dimensionality** (e.g., 4–32 tokens × hidden size), making the SVD operation significantly cheaper than standard training procedures such as forward and backward passes in vision-language models (VLMs). To support this claim, we will include the following runtime statistics in the final version.\n\n**Table. Runtime overhead of SVD-based prompt projection.** All results are averaged over 10 communication rounds. Compared to model training, the SVD overhead is negligible, even in large-scale settings.\n\n|Operation Stage|Device|Input Dimension|Avg. Time (ms)|Relative Training Time (%)|\n|-|-|-|-|-|\n|Global prompt decomposition|Client|16 × 768|4.2|**<1%**|\n|Local prompt projection|Client|Local prompt × 768|2.8|**<1%**|\n|Local model training|Client|Image input × model|4536.8|—|\n\n**W3: Potential Privacy Risks**\n\nA3: We thank the reviewer for raising this important point. While privacy is not the central focus of this work, we note that our current implementation exchanges only prompt embeddings—**not raw data or model gradients**—which already mitigates direct privacy exposure to a significant extent. Compared to PromptFL, FedPHA transmits **global prompts** instead of **personalized local prompts**, reducing the risk of leaking client-specific information. Moreover, FedPHA addresses a key limitation in FedPGP, where **local prompts are dependent on global prompts**, potentially increasing inversion risk. By design, our approach improves decoupling and provides a more privacy-preserving prompt structure.\n\nWe acknowledge that embedding inversion attacks remain a broader challenge in the federated learning landscape. Although a comprehensive privacy analysis is beyond the scope of this work, we view this as a valuable direction for future research. FedPHA is also compatible with **differential privacy**, **prompt obfuscation**, and **secure aggregation**, which can be explored in follow-up studies to further enhance privacy protection.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "no", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "This paper introduces FedPHA, a novel Federated Prompt Learning (FPL) approach that enables heterogeneous client adaptation using Vision-Language Models (VLMs). The key contributions include:\nA dual-layer architecture combining a fixed-length global prompt for efficient aggregation and variable-length local prompts for personalization. A Singular Value Decomposition (SVD)-based projection mechanism to resolve conflicts between global and local prompts. A bidirectional alignment strategy to optimize the interaction between global knowledge and local adaptation. Extensive empirical validation on various datasets, demonstrating superior performance over state-of-the-art FL and FPL methods. The paper is technically sound, well-motivated, and presents strong experimental results. However, a few areas need further clarification and improvements, particularly regarding real-world feasibility, communication efficiency, and computational overhead.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n1. Unlike existing federated prompt learning (FPL) methods that enforce uniform prompt lengths, FedPHA accommodates varying local prompt lengths, making it more adaptable to heterogeneous clients.\n2. The SVD-based projection mechanism effectively filters out conflicting components between global and local knowledge, mitigating negative transfer effects.\n\nWeaknesses:\n1.\tThe method requires sending prompts to a central server, which may still introduce communication overhead despite avoiding full model aggregation.\n2.\tThe SVD decomposition of global prompts adds computational overhead at both the server and client side.\n3.\tSince activations and prompts are exchanged, potential privacy risks (e.g., gradient leakage, prompt inversion attacks) could arise.", "other_comments_or_suggestions": "See Strengths And Weaknesses", "questions_for_authors": "See Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a8TgavY2tm", "reviewer": "Reviewer_tY2b", "summary": "FedPHA is a novel FPL framework designed to address heterogeneous client adaptation in federated learning. Traditional FPL methods enforce uniform prompt lengths, which limits their adaptability to clients with diverse data distributions. To overcome this limitation, FedPHA proposes a dual-layer prompt architecture consisting of a fixed-length global prompt for efficient aggregation and variable-length local prompts to preserve client-specific knowledge. To mitigate conflicts between global and local knowledge, FedPHA incorporates svd-based projection to filter conflicting information and bidirectional alignment to maintain balance between generalization and personalization. Through these innovations, FedPHA provides a scalable, robust, and effective solution for federated learning across diverse, non-IID clients, ensuring both efficient knowledge sharing and local adaptability. ## update after rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer tY2b:\n\nWe sincerely thank the reviewer for the thoughtful and detailed evaluation of our work. We are especially grateful for the recognition of our contributions in addressing client heterogeneity through the dual-layer prompt architecture, the introduction of SVD-based projection and bidirectional alignment, and the strong experimental validations. Below, we address each of the reviewer’s key concerns:\n\n### Weakness\n\n**Q1: Distinction from Existing G-L Prompt Methods**\n\nA1: Thank you for raising this important point. As mentioned in Lines 55–99 of the *Introduction* and Lines 141–153 of the *Related Work* section, we have cited and briefly discussed other existing works. Here, we provide a more detailed explanation of how our approach differs fundamentally from previous Global-Local (G-L) prompt-based methods in terms of architecture design:\n\n- **FedOTP:** Each client uses a global prompt and a local prompt. However, due to the requirement of the **unbalanced optimal transport framework**, FedOTP enforces that **the global and local prompts must be of equal length**, which restricts flexibility and client adaptability.\n- **FedPGP:** Each client uses a global prompt and **two local adapters**. In this design, the local prompt is constructed by **adding a local adapter to the global prompt**, creating a strong dependency between them. This additive formulation forces the local prompt to inherit characteristics from the global prompt. Consequently, if the global prompt captures features that are misaligned with a client’s local data distribution, the local prompt may be **forced to adapt to irrelevant or even conflicting patterns**, reducing the effectiveness of personalization.\n- **FedPHA (Ours):** Each client uses a fixed-length global prompt and a local prompt of varying lengths. Unlike existing G-L prompt methods that often assume a **uniform prompt length across clients**, FedPHA explicitly supports **client-specific variable-length local prompts**, enabling better alignment with heterogeneous computational capabilities and data distributions across clients. **Moreover, by decoupling the global and local prompts, FedPHA effectively mitigates the negative transfer from global knowledge to client-specific representations**, enhancing the robustness of personalization.\n\n**Q2: Baseline Implementation Details and Fair Comparison**\n\nA2: We appreciate the reviewer’s request for greater transparency regarding the baseline configurations. To ensure a fair and controlled comparison, we implemented **FedOTP** and **FedPGP** following the **same experimental protocol** described in **Section 4.1 (Implementation Details)** and **Appendix B.2 (Details of Implementation)**. Below, we provide a summary of both the shared and method-specific settings:\n\n- **Shared Training Setup:** All methods were evaluated using a **frozen CLIP backbone** (default: ViT-B/16), with **local training epochs** set to E = 1 and **communication rounds** to  R = 50 (reduced to R = 25 for CIFAR-10/100). Training was conducted using **SGD** with a learning rate of 0.001 and a batch size of 32. The default **prompt length** was 16, with a 512-dimensional embedding.\n- **FedOTP-Specific Configuration:**\n  - Optimal Transport type: `COT` (Unbalanced Optimal Transport)\n  - Sinkhorn distance parameters: `THRESH = 1e-3`, `EPS = 0.1`\n  - OT maximum iterations: `MAX_ITER = 100`\n- **FedPGP-Specific Configuration:**\n  - Bottleneck dimension: `BOTTLENECK = 4`\n  - Additional loss parameters: `mu = 1`, `temp = 0.5`\n\nAll other parameters, such as context initialization (`CTX_INIT = False`), class-specific context (`CSC = False`), prompt precision (`PREC = \"fp16\"`), and class token position (`\"end\"`), were kept consistent across both baselines to match our implementation setup. These configurations ensure that the comparisons are fair and focused solely on the architectural and algorithmic differences.\n\n**Q3: Ablation of Key Hyperparameters**\n\nA3: We thank the reviewer for this valuable suggestion. We have already conducted ablation studies on several key hyperparameters and presented the results in the corresponding sections of the paper:\n\n- **Prompt Length:** The ablation on prompt length has been thoroughly analyzed in **Section 4.3 (Effectiveness of Prompt Length Heterogeneity)**, where we demonstrate that supporting variable-length prompts across clients leads to better performance under heterogeneous settings.\n- **Communication Rounds (R):** We provide an analysis of the effect of communication rounds in **Appendix C.1 (Convergence Analysis)**, showing how FedPHA converges efficiently and consistently across datasets.\n- **Hyperparameters ρ and α:** The ablation studies for the ratio ρ in Eq.(9) and the coefficient α in Eq.(12) are discussed in our response to reviewer **oKVN**, where we detail their influence on model performance and stability.", "claims_and_evidence": "YES", "methods_and_evaluation_criteria": "Yes, the chosen evaluation metrics align with the standard practices in the field.", "theoretical_claims": "Authors provide detailed definitions and analysis in Sec 3, including svd-based projection and bidirectional alignment.", "experimental_designs_or_analyses": "Experimental results are comprehensive, with evaluations conducted across various datasets and comparisons against multiple baseline methods.", "relation_to_broader_scientific_literature": "Authors introduce a novel approach to address heterogeneous client adaptation in FL and mitigate conflicts between global and local knowledge in FPL.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Pros:\nP1: FedPHA addresses client heterogeneity in FPL by introducing a dual-layer prompt architecture, combining a fixed-length global prompt for efficient aggregation with variable-length local prompts to adapt to diverse client data, ensuring better personalization while maintaining federated learning efficiency.\nP2: This design enhances personalization without sacrificing global knowledge sharing, enabling clients to retain local characteristics while benefiting from shared representations. This structure ensures better adaptation to non-IID data distributions, making FedPHA more effective in heterogeneous federated learning settings.\nP3: FedPHA provides a scalable, adaptable, and robust solution for FPL on non-IID datasets, effectively accommodating diverse client distributions and improving overall model performance compared to existing methods.\n\nCons: \nQ1:Many other methods also employ a Global-Local (G-L) prompt architecture for federated learning. Could the authors provide a more detailed explanation of how their approach fundamentally differs from existing G-L prompt methods?\nQ2:Could the authors provide more details on how the baselines were configured such as FedOTP and FedPGP, including their hyperparameter settings and any modifications made for fair comparison?\nQ3: The paper lacks ablation experiments to analyze the impact of its key hyperparameters. Could the authors provide additional experiments in this regard?", "other_comments_or_suggestions": "No", "questions_for_authors": "Please refer to Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D68jA1QDAq", "reviewer": "Reviewer_oKVN", "summary": "The authors introduce Federated Prompt Learning for Heterogeneous Client Adaptation (FedPHA), a novel approach to adapting pre-trained Vision-Language Models (VLMs) within federated learning. The primary motivation is to tackle the persistent heterogeneity challenge by integrating a uniform global prompt for efficient aggregation with diverse local prompts for personalization. Additionally, FedPHA aims to mitigate conflicts between global and local knowledge, enabling clients to maintain their unique characteristics while leveraging shared information. To achieve this, Singular Value Decomposition (SVD)-based projection and bidirectional alignment are introduced, enhancing model generalization while preserving local optimization. Experimental results confirm FedPHA’s superiority over state-of-the-art methods, striking a balance between personalization and global knowledge in heterogeneous federated learning settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oKVN: \n\nThank you for your thoughtful review and for raising key concerns regarding our work. We aim to address your concerns in our detailed responses below, hoping to provide clarity and demonstrate the effectiveness of our proposed approach.\n\n### Weakness\n\n**W1: Theoretical Explanation of SVD-Based Projection**\n\nA1: Thank you for highlighting this point. We agree that a clearer explanation of the SVD-based projection mechanism is essential, especially given its role in handling heterogeneous client data. In **Section 3.3**, we now provide a more detailed account of its purpose, mathematical formulation, and intuitive significance, which we summarize below.\n\nWhile the dual-layer prompt architecture (global + local) provides personalization flexibility, an **explicit mechanism is required to regulate the interaction between global and local prompts**, particularly in highly heterogeneous settings where simple alignment or orthogonality may fail. Our **SVD-based projection** addresses this by filtering out potentially **conflicting directions** in the global prompt space.\n\nFormally, we apply **Singular Value Decomposition (SVD)** to the current global prompt matrix, decomposing it into orthonormal components. We then use the **trailing singular vectors (i.e., low-variance directions)** to define a null space that captures information **less dominant in the global prompt**, and potentially misaligned with client-specific distributions. The local prompt is projected onto this subspace, effectively **removing dominant global components** that could interfere with personalization.\n\nTo mitigate the risk of over-suppression (i.e., losing important local information), we complement this with a **bidirectional alignment strategy** (Section 3.4). Specifically:\n\n- A **pull loss** ensures that the original and projected local prompts remain close in the feature space, preserving essential local semantics.\n- A **push loss** introduces a lower-bound distance between the local and global prompts, ensuring that local prompts remain sufficiently distinct and do not collapse toward overly generic representations.\n\nTogether, these mechanisms strike a balance between **global knowledge integration** and **local specialization**, making the projection process both effective and robust in non-IID federated settings. We wish this richer explanation will improve the clarity and motivation of our design.\n\n**W2: Sensitivity Analysis of Hyperparameters ρ and α** \n\nA2: Thank you for raising this important concern. We agree that analyzing the sensitivity of key hyperparameters is crucial to validating the robustness and adaptability of FedPHA across different federated settings. We conducted additional ablation experiments to investigate the impact of the following two hyperparameters:\n\n- ρ (in Eq. (9)): controls the number of low-rank components used for projecting local prompts into the null space of the global prompt.\n- α (in Eq. (12)): defines the margin in the push loss, ensuring local prompts remain sufficiently distinct from the global prompt.\n\nWe varied ρ∈ \\{0.5, 0.6, 0.7, 0.8, 0.9\\} to confirm the Effect of the Ratio in SVD-Based Projection. A larger ρ keeps more dominant directions from the global prompt, while a smaller ρ filters out more components. We observed that:\n\n- Performance is stable when ρ∈[0.7,0.9], suggesting that filtering out only the weaker global components is sufficient.\n- Extremely low ρ may lead to over-filtering and loss of useful shared information.\n\n**Table. Effect of projection ratio ρ on accuracy**\n\n| ρ | Caltech101 (%) | Office31 (%) | CIFAR-100 (%) |\n|-|-|-|-|\n| 0.5 | 97.88 | 93.21 | 74.91 |\n| 0.6 | 98.42 | 93.96 | 75.26 |\n| 0.7 | 98.86 | 94.02 | **76.37** |\n| 0.8 (default) | **99.05** | 94.74 | 75.63 |\n| 0.9 | 98.93 | **94.91** | 75.32 |\n\nWe tested α∈ \\{0.5, 1.0, 1.5, 2.0\\} to confirm the Effect of the Margin Parameter α in the Push Loss, and found:\n\n- Performance is robust when α∈ [1.0, 1.5].\n- Too small a margin (e.g., 0.5) causes local prompts to collapse toward global prompts, reducing personalization.\n- Too large a margin may hinder knowledge sharing, slightly degrading performance.\n\n**Table. Effect of push margin α on accuracy**\n| α | Caltech101 (%) | Office31 (%) | CIFAR-100 (%) |\n|-|-|-|-|\n| 0.5 | 98.11 | 93.07 | 74.02 |\n| 1.0 (default) | **99.05** | 94.74 | **75.63** |\n| 1.5 | 98.79 | **95.16** | 75.21 |\n| 2.0 | 98.32 | 93.85 | 74.85 |\n\nNote: All experiments use ViT-B/16 backbone and the same federated training setup as described in Section 4.1. The default setting in our main experiments is ρ=0.8, α=1.0.\n\n**W3: Consistency in Terminology and Presentation**\n\nA3: Thank you for catching this inconsistency. We have carefully reviewed and revised all dataset names and references throughout the paper to ensure consistency, especially in Table 1, Figure 3, and the surrounding text. These updates improve the overall clarity and readability of the manuscript.", "claims_and_evidence": "Yes, the claims are well-supported by empirical evidence, including extensive experiments and ablation studies. The results consistently demonstrate FedPHA’s superiority over baselines, validating its effectiveness in handling heterogeneity in federated learning.", "methods_and_evaluation_criteria": "Yes, the chosen datasets cover both single-domain and multi-domain scenarios, effectively simulating heterogeneous federated learning environments. The evaluation metrics and comparison with state-of-the-art methods align well with the problem setting.", "theoretical_claims": "The paper primarily focuses on experimental validation and no major theoretical inconsistencies were identified.", "experimental_designs_or_analyses": "Yes, I reviewed the experimental setup, including dataset partitioning, evaluation metrics, and ablation studies. The methodology is sound, and the comparisons with baselines are fair.", "relation_to_broader_scientific_literature": "The paper builds on existing work in federated learning and prompt learning by addressing heterogeneity using Singular Value Decomposition (SVD)-based projection and bidirectional alignment. It extends prior federated prompt learning approaches by enabling varying local prompt lengths, improving adaptability to non-IID data distributions.", "essential_references_not_discussed": "The paper provides a comprehensive review of related work, citing essential studies in federated learning, prompt learning, and vision-language models. No significant omissions were identified.", "other_strengths_and_weaknesses": "Strengths\n(1) The paper effectively combines federated learning and prompt learning, introducing a dual-layer prompt structure that balances global generalization and local personalization. The integration of Singular Value Decomposition (SVD)-based projection and bidirectional alignment is an innovative approach to mitigating conflicts between global and local knowledge.\n(2) Heterogeneous federated learning is a critical challenge, and this work provides a scalable and adaptable solution. The ability to support varying prompt lengths per client enhances personalization, making it applicable to real-world federated settings with non-IID data distributions.\n(3) The experiments cover multiple benchmark datasets, including single-domain, multi-domain, and cross-domain settings, demonstrating the robustness of FedPHA. Comparisons with state-of-the-art methods further validate its effectiveness. \n(4) The paper is well-written, with clear problem formulation, methodology descriptions, and result interpretations. Figures and tables are informative, enhancing readability.\n\nWeaknesses\n(1) The explanation of the SVD-based projection is unclear. Could the authors further elaborate on why and how it effectively mitigates conflicts between global and local prompts? A more detailed theoretical discussion would be helpful.\n(2) The choice of ratio (ρ) in Eq.(9) and alpha (α) in Eq.(12) plays a crucial role in model performance. A deeper sensitivity analysis of these hyperparameters could provide valuable insights into the stability and adaptability of FedPHA across different federated settings.\n(3) There are some inconsistent expressions in the paper, such as the dataset names in Table 1 and Figure 3, which are not uniform. The authors may consider carefully reviewing the wording for consistency to improve clarity and readability.", "other_comments_or_suggestions": "No further comments or suggestions.", "questions_for_authors": "I hope the authors can address the concerns I raised in the Weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c9hJqnZHK0", "reviewer": "Reviewer_TNUk", "summary": "This paper proposes a method called FedPHA (Federated Prompt Learning for Heterogeneous Client Adaptation) to enhance federated prompt learning in diverse client environments. It addresses two key challenges: the limitation of uniform prompt lengths in existing methods and the conflict between global and local knowledge during aggregation. FedPHA introduces a federated prompt heterogeneous architecture, combining a fixed-length global prompt for efficient aggregation with variable-length local prompts to capture client-specific characteristics. To further mitigate conflicts, FedPHA incorporates SVD-based projection to filter out conflicting information and bidirectional alignment to maintain local distinctiveness while benefiting from global knowledge. The approach is communication-efficient, requiring no major modifications to pre-trained models, and significantly improves performance in federated learning settings.\n\n## update after rebuttal\nThe paper is well-motivated and clearly structured, presenting a novel and effective solution to the practical Federated Learning problem. After carefully reading the rebuttal, my concerns have been sufficiently addressed. Therefore, I maintain my positive rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TNUk:\n\nWe sincerely thank the reviewer for the constructive and encouraging feedback. We are especially grateful for your positive recognition of our contributions to federated prompt learning, the novelty of the proposed architecture, and the overall experimental design. Below, we address the specific concerns raised in the Weaknesses section:\n\n### Weakness\n\n**W1: Correction of Padding Description**\n\nA1: Thank you for pointing out this discrepancy. You are correct that the current implementation applies **zero-padding at the end** of the prompt sequence, while the paper previously stated it was added to the middle. We apologize for this inconsistency. We have now **Corrected the description** in Section 3.2 to accurately reflect the implementation: zero-padding is appended to the end of the prompt sequence when needed.\n\n**W2: Broadening the Evaluation Scope with Additional Datasets**\n\nA2: Thank you for the thoughtful suggestion. We agree that evaluating the effectiveness of prompt length heterogeneity on additional datasets strengthens the generality of our findings.\n\nTo this end, we have extended our analysis in Section 4.3 to include the **OfficeHome** dataset, which comprises four domains: *Art, Clipart, Product,* and *Real-World*. For this experiment, we allowed each domain to select its own prompt length from eight possible values:[4, 8, 12, 16, 20, 24, 28, 32]. We then evaluated the global classification accuracy under each prompt length configuration.\n\nWe evaluated global classification accuracy under various prompt length configurations. As shown in the table below, **heterogeneous prompt length settings consistently outperformed uniform ones**. For example, the best-performing configuration on Office31 was [28, 12, 16], achieving **95.45%** accuracy, while the uniform setting [16, 16, 16] reached only 94.72%. Similarly, for OfficeHome, the best result was obtained with [12, 32, 12, 16] at **89.67%**, outperforming uniform settings such as [16, 16, 16, 16] (89.17%) and [32, 32, 32, 32] (88.41%). These consistent trends across datasets further support our claim that **prompt length heterogeneity is beneficial in federated learning scenarios with non-IID data**.\n\n**Table: Accuracy under different prompt length combinations on multi-domain datasets**\n\n| Prompt Lengths [A, W, D] | Office31 (%) | Prompt Lengths [A, C, P, R] | OfficeHome (%) |\n|-|-|-|-|\n| [4, 4, 4] | 95.13 | [4, 4, 4, 4] | 88.73 |\n| [16, 16, 16] | 94.72 |[16, 16, 16, 16] | 89.17 |\n| [32, 32, 32] | 94.05 |[32, 32, 32, 32] | 88.41 |\n| [28, 12, 16] (best) | **95.45** |[12, 32, 12, 16] (best) | **89.67** |\n| [20, 12, 28] | 95.19 | [12, 12, 28, 16] | 89.61 |\n| [24, 12, 32] | 95.15 | [12, 16, 20, 16] | 89.56 |\n\nDue to rebuttal space constraints, we will provide a more detailed and visual analysis (similar to Figure 4 and Figure 5) in the final version of the paper.\n\n**W3: Comparison with Fixed-Length FedPHA**\n\nA3: We thank the reviewer for this valuable suggestion. To evaluate the robustness of FedPHA under prompt length variation, we conducted a direct comparison between two settings: **FedPHA with a fixed local prompt length** (uniformly set to 16 tokens) and **FedPHA with randomly assigned prompt lengths** (ranging from 4 to 32) across clients.\n\nAs shown in the table below, the fixed-length version slightly outperforms the variable-length one on both CIFAR-10 and CIFAR-100. This difference may be attributed to the greater **optimization stability** and **aggregation consistency** achieved when all clients share the same prompt structure. In contrast, randomly assigned prompt lengths may **not always align with the specific needs or data distributions of individual clients**, inevitably resulting in minor performance drops.\n\nHowever, this setting also reflects a more realistic federated environment, where clients may differ significantly in computational capacity or data complexity. Notably, **FedPHA is the only method capable of operating under heterogeneous prompt configurations**, offering unique flexibility and scalability in such scenarios. In more diverse and non-IID datasets (e.g., Office31, OfficeHome), we observe that **randomized or adaptive prompt lengths lead to improved performance**, highlighting the benefits of prompt flexibility in cross-domain and heterogeneous environments.\n\n**Table: Comparison with the SOTA methods using random and fixed prompt lengths on CIFAR-10 and CIFAR-100 across 100 clients.** All methods except FedPHA (last row) use a fixed prompt length of 16.\n\n| Methods|CIFAR-10|CIFAR-100|\n|-|-|-|\n| CLIP|87.88|64.89|\n| PromptFL|91.70|72.58|\n| Prompt+Prox|91.83|72.08|\n| FedPGP|92.10|74.81|\n| FedOTP|93.43|75.07|\n| **FedPHA (fixed length)**|**94.11**|**75.92**|\n| **FedPHA (random length)**|93.80|75.63|", "claims_and_evidence": "Yes, the claims in the submission are supported by experimental results.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate for addressing client heterogeneity.", "theoretical_claims": "In this paper, the SVD-based projection mechanism has a solid theoretical foundation, and no significant issues were identified in the proofs.", "experimental_designs_or_analyses": "The experimental design is well-structured and valid, evaluating the proposed methods on diverse heterogeneous datasets with non-IID settings. Comparisons with five baseline methods and clearly defined hyperparameters ensure a robust assessment.", "relation_to_broader_scientific_literature": "Existing federated prompt learning methods cannot support different local prompt lengths due to aggregation requirements and structural constraints. In contrast, FedPHA introduces a federated prompt heterogeneous architecture, combining a fixed-length global prompt for efficient aggregation with variable-length local prompts to capture client-specific characteristics.", "essential_references_not_discussed": "No essential references appear to be missing.", "other_strengths_and_weaknesses": "Strengths:\n1. This approach tackles client heterogeneity in federated learning, improving adaptability to diverse data distributions. FedPHA is communication-efficient and scalable, requiring no extensive fine-tuning for real-world applications.\n2. The motivation is clear and compelling. The federated prompt heterogeneous architecture enables aggregation while accommodating individual client needs. The SVD-based projection mechanism resolves conflicts between global and local knowledge, preserving essential local information. The bidirectional alignment function ensures alignment and preserves the unique characteristics of both global and local representations.\n3. The experiment is comprehensive, covering a wide range of scenarios to thoroughly evaluate the proposed methods. \n\nWeaknesses:\n1. In Section 3.2, according to the submitted code, zero padding is applied at the end of the sequence, while the paper incorrectly describes it as being added to the middle. This inconsistency between the implementation and the description should be corrected.\n2. In Section 4.3, the experiments are only conducted on the Office31 dataset. It would be beneficial to include more datasets in this section to provide a more comprehensive evaluation.\n3. In Table 3, a comparison with a FedPHA model using a fixed local prompt length should be added. This comparison would help demonstrate the robustness of using random prompt lengths in FedPHA.", "other_comments_or_suggestions": "The method proposed in this paper enables federated prompt learning to adapt to varying local prompt lengths. I hope future work can explore this aspect further.", "questions_for_authors": "This paper proposes an innovative FPL approach. While the methodology is comprehensive, there are several concerns regarding the implementation details, additional dataset evaluation, and prompt lengths. Please address these concerns in the rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers highlighted this paper's novel setting, good writing, and convincing experiments. The authors also provided reasonable answers to the reviewers' doubts about the details of the proposed dataset and method. Therefore, the decision is to recommend Acceptance.
Decision: Accept (poster)