FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
Authors: Weihao Bo
Yanpeng Sun
Yu Wang
Xinyu Zhang
Zechao Li
Venue: NeurIPS 2025 poster ()
Keywords: Federal learning; Vision language Model; Prompt Learning;
Primary area: deep_learning
TL;DR: FedMGP introduces a multi-group text-visual prompt paradigm for federated learning that effectively balances personalization and generalization , achieving state-of-the-art performance with minimal communication parameters.

Abstract: In this paper, we introduce FedMGP, a new paradigm for personalized federated prompt learning in vision-language models (VLMs). Existing federated prompt learning (FPL) methods often rely on a single, text-only prompt representation, which leads to client-specific overfitting and unstable aggregation under heterogeneous data distributions. Toward this end, FedMGP equips each client with multiple groups of paired textual and visual prompts, enabling the model to capture diverse, fine-grained semantic and instance-level cues. A diversity loss is introduced to drive each prompt group to specialize in distinct and complementary semantic aspects, ensuring that the groups collectively cover a broader range of local characteristics.During communication, FedMGP employs a dynamic prompt aggregation strategy based on similarity-guided probabilistic sampling: each client computes the cosine similarity between its prompt groups and the global prompts from the previous round, then samples s groups via a softmax-weighted distribution. This soft selection mechanism preferentially aggregates semantically aligned knowledge while still enabling exploration of underrepresented patterns—effectively balancing the preservation of common knowledge with client-specific features. Notably, FedMGP maintains parameter efficiency by redistributing a fixed prompt capacity across multiple groups, achieving state-of-the-art performance with the lowest communication parameters (5.1k) among all federated prompt learning methods. Theoretical analysis shows that our dynamic aggregation strategy promotes robust global representation learning by reinforcing shared semantics while suppressing client-specific noise. Extensive experiments demonstrate that FedMGP consistently outperforms prior approaches in both personalization and domain generalization across diverse federated vision-language benchmarks.The code will be released on https://github.com/weihao-bo/FedMGP.git.

FedMGP: Personalized Federated Learning with Multi-Group Text- Visual Prompts
Weihao Bo Yanpeng Sun?; Yu Wang', Xinyu Zhang', Zechao Li'
*Nanjing University of Science and Technology
National University of Singapore
Baidu VIS University of Auckland
Abstract
In this paper, we introduce FedMGP, a new paradigm for personalized federated prompt learning in vision-language models (VLMs). Existing federated prompt learning (FPL) methods often rely on single, text-only prompt representation, which leads client-specific overfitting and unstable aggregation under heteroge- neous data distributions. Toward this end, FedMGP equips each client with mul- tiple groups of paired textual and visual prompts, enabling the model to capture diverse, fine-graing uineggranedd semantic and instance-level cues. A diversity loss is intro- duced to drive each prompt group to specialize in distinct and complementary se- mantic aspects, ensuring that the groups collectively cover broader range of local characteristics. During communication, FedMGP employs dynamic prompt ag- gregation strategy based on similarity- computes the cosine similarity between its prompt groups and the global prompts from the previous round, then samples groups via softmax- softmax-weighted distribu- tion. This soft selection mechanism preferentially aggregates semantically aligned knowledge while still enabling exploration of underrepresented patternseffectively balancing the preservation common knowledge with client-specific features. Notably, FedMGP maintains parameter efficiency by redistributing a fixed prompt capacity across multiple groups, achieving state-of-the-art performance with the lowest communication parameters (5.1k) among all federated prompt learning methods. Theoretical analysis shows that our dynamic aggregation strategy pro- motes robust global representation learning by reinforcing shared semantics while suppressing speccfc noise. Extensive experiments demonstrate that Fed- MGP consistently outperforms prior approaches in both personalization and do- main generalization across diverse federated vision-language henchmarks. The code will be released https: sttps://githube http://///eeaaoo/////aeeee/ giit.
probabilistic sampling: each client
Introduction
Large-scale vision vision-language models (VLMs) have demonstrated impressive performance across B2). As these models are increasingly deployed
wide range of multimodal tasks [I]. 51, IT. HE in privacy-sensitive and decentralized environmentsincludig healthcare, mobile devices, and indus- trial systemsthere is growing need adapt them privately without direct access raw data [5]]. In such settings, data remains local, and client distributions are often highly heterogeneous [IG BJ. To fully utilize local data, personalized federated learning (PFL) 112 SD] has emerged as an effective framework for adapting shared models across clients with non-identical data, while preserving pri- vacy. In parallel, prompt-based tuning has shown great promise for parameter-efficient adaptation of frozen VLMs. The integration of these two ideas has led to the rise of federated prompt learning (FPL)a lightweight and scalable approach to adapting VLMs in federated settings[[]) [X]
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite its potential, existing FPL methods face key limitations. Most approaches rely solely on textual prompts, which encode static class-level semantics. While efficient, these prompts lack the expressiveness to capture personalized visual cues specific to each client, limiting their ability to handle diverse or complex inputs. Furthermore, many methods adopt a local-global prompt frame- work [27]. 9, (37], in which each client maintains a local prompt and contributes a single global prompt for aggregation. This framework introduces two critical problems: (1) A single prompt per client is often insufficient to capture the diversity of local dataespecially when multiple semantic concepts or visual styles coexist within a client. (2) Aggregating one prompt per client leads to biased global representations, as the shared prompt tends to overfit to dominant local patterns while overlooking less frequent but informative ones from other clients. Together, these issues undermine both local personalization and cross client generalization, particularly under severe data heterogeneity.
To overcome these limitations, we propose Personalized Federated Learning via Multi-Gron Group Text- Visual Prompt (FedMGP), new paradigm for personalized federated adaptation of vision-language models. Each client FedMGP maintains multiple paired groups of textual and visual prompts, where each group captures distinct semantic and instance-level characteristics of the local data. To ensure prompt groups specialize in different aspects, we introduce a diversity loss that encourages representational separation within each client. For server aggregation, we develop a dynamic prompt selection strategy based on the similarity between local prompt groups and the global prompt from the previous round, ensuring that semantically aligned groups more likely to be selected, while still allowing exploration of less dominant patterns. This balanced approach reinforces common cross-client patterns while suppressing client--pecific noise.
FedMGP both parameter-efficient and communication-aware with the lowest communication parameters (5. among all federated prompt learning methods, it achieves state-of- -fte-of-the-rttt perfor- mance while distributing a fixed prompt capacity across multiple groups. Empirical results across various heterogeneous data settings, including pathological non-IID, Dirichlet distribution, and do- main generalization, demonstrate that FedMGP successfully balances personalization accuracy on local client data with generalization capability to unseen domains.
2
Related Work
2.1 Prompt Learning for Vision-Language Models
Vision-language models (VLMs) like CLIP [E have demonstrated strong zero-shot capabilities through contrastive learning on massive image-fext pairs [56, 55. 3y, III A8]. To efficiently adapt these models to downstream tasks, prompt learning introduces a small set of learnable parame- ters while keeping the original model weights frozen [60]. 30. 58]. Various prompt learning ap- proaches have been proposed, including enriching text representations through class-related descrip- tions [53) B5], additional descriptive sentences [4 57
[ell, external knowledge [22]], and visual annotations [4 E5I. As highlighted in our introduction, recent methods have begun addressing the critical balance between fitting to seen classes and maintaining generalization capabilities to unseen classes. For instance, CoCoOp [59] introduces visual cues while preserving general knowledge, and ProGrad [|50] proposes prompt alignment gradi- ents to maintain the model's 's inherent knowledge. However, these methods predominantly operate in centralized settings with direct access to all training data, overlooking privacy concerns and the chal- lenges of heterogeneous data distributions across multiple clientscritcaal limitations that necessitate new frameworks for grivacy-preservii distributed adaptation of VLMs [29].
prompts to capture fine-grained
2.2 Federated Prompt Learning
Federated Prompt Learning (FPL) [TA IT. 5L 3 combines prompt learning with federated learn- ing [34] B, &L 52,, 28] to enable privacy- selvaccy-preservng adaptation of vision-language models across dis- tributed environments. PromptFL [[7]] pioneered this approach by integrating prompt learning into federated frameworks with theoretical convergence guarantees. To address client heterogeneity, sev- eral researchers [27 57, [G]] developed local-global paradigms where clients maintain personalized prompts while contributing to shared global prompts. This approach improves local performance but often compromises generalization under non-III data distributions. Recent work [9] attempted to balance personalization and generalization through additional constraints. Despite progress, ex-

---

isting FPL methods have two key limitations. First, they rely solely on textual prompts, missing crucial visual cues needed for robust multimodal adaptation. Second, they lack effective prompt learning strategies and aggregation mechanisms tailored specifically for federated settings that can simultaneously maintain personalization while enhancing cross-client generalization.
3
Method
In this paper, we introduces FedMGP, a novel approach designed to address data heterogeneity and model stability challenges in federated learning. We first present the fundamentals of federated prompt learning (Section B.I),, including the core concepts of prompt learning and its application in federated settings. Then, we elaborate on two key mechanisms of FedMGP: the multimodal prompt co-learning mechanism (Section ()..)), which enhances representation capabilities through the synergistic interaction between text and visual prompts, and the dynamic prompt aggregation strategy (Section B.2.2), which effectively balances global knowledge sharing with local feature preservation.
3.1 Preliminary: Federated Prompt Learning
Prompt learning is parameter-efficient strategy for adapting large pre-trained Vision-Language Models (VLMs), such as CLIP [[E]], to diverse downstream tasks. It introduces small set of learnable parameters called "prompts" while keeping the VLM's encoder weights frozen. These learnable prompt vectors are combined with class name embeddings to create class-specific textual prompts that effectively adapt the model downstream tasks.
VLM typically consists of an image encoder f()) and a text encoder g()). The core workflow involves: (1) processing input image through the image encoder to obtain visual features, (2) pro- cessing text prompts through the text encoder to obtain textual features, and (3) computing similarity scores between these features determine class probabilities. The key prediction formula is:
p k() exp(sim(f(#s)
(f))sm(((().
where 4a = [[pe.cc } represents the text input formed by concatenating the learnable text prompt p with the embedding of class name Og- Here, sim(::) represents cosine similarity, K is the number of classes, and is temperature scaling factor.
Federated Prompt Learning (FPL) extends this approach distributed settings where multiple clients collaborate without sharing their raw data. The federated training process follows a cyclic pattern: (1) The server distributes the current global prompt to selected clients; (2) Clients perform local updates using their private data; (3) Updated local prompts are sent back to the server; (4) The server aggregates these local prompts to form an improved global prompt. This process repeats for multiple communication rounds, gradually refining the global prompt work well across all clients.
Despite its privacy-preserving benefits, standard FPI. faces significant challenges with heteroge- neous client data distributions. Client-specific optimization may lead to overfitting to local patterns, while naive aggregation methods like FedAvg [[4]] often struggle to preserve client-specific knowl- edge while extracting common patterns. Our proposed FedMGP framework specifically addresses these limitations through multi-group prompt architecture and dynamic prompt aggregation strat- egy.
3.2 FedMGP:Feedertee Learning via Multi-Group Text-Visual Prompt
To address the fundamental limitations of existing federated prompt learning methods, particularly their reliance on single text-only prompts and vulnerability to client-specific overfitting, we propose FedMGP. (The complete pseudocode can be found appendix A.) As illustrated in figure II, our framework introduces novel multi-group mechanism that enhances both prompt diversity and ro- bustness through complementary prompt groups. For each client in the federation of N clients, we define set of prompts P {p..1...., .P,,,, Po.1..... .o....... Pe,G), ). where Pt.j represents the j-th text
3

---

client
client
P.
Pu
Fu
Pr
Pu
Pinovla
div
clientt
Pg
logits
BETINTT apgrrggette
Server
client
Pui
div
1. Jecne Pr
Po
P.
P
Pu
prompt
pronpe
peceept
Aggropazzd peccppt
camppi
Figure I: Overview of FedMGP: The left portion shows the server distributing global prompts to clients; the middle portion illustrates the multi-gras group text-visual prompt co-learning mechanism within each client; and the right portion demonstrates the dynamic prompt aggregation strategy across communication rounds.
prompt and Peú represents the j-th visual prompt, with G being the number of prompt groups. We use P to denote the global aggregated prompts at the server and T denote the communication round.
This design offers two significant advantages. First, integrating visual and textual modalities en- riches contextual representation, capturing instance-specifie information more comprehensively than static class names. Second, distributing knowledge across multiple specialized prompt units en- hances aggregation robustnesseven if certain prompt groups overfit to local distributions, others may capture generalizable patterns, significantly improving model adaptability under heterogeneous data distributions without increasing the total parameter count.
3.2.1 Multimodal Prompt Co-learning Mechanism
For any local client, the multi-gn multi-group group prompt learning process operates follows. During training, for each group j. the text prompt Pt.j concatenated with the class embedding с₁ to form t&.j = (P.j.ck), which is fed into a text encoder 9(.). Simultaneously, the image is combined with the visual prompt Pa.j to form b = (r.Pr.h] which is passed through the image encoder f()). The predictive probability for class lt is computed based the similarity between the corresponding text and visual features:
= 2 Σk 1= exp(sim([ g(4,))))((.)))))
(2)
The classification loss is defined as the average cross-entropy across all G prompt groups:
1 LCE =
log p(p = x:; Pt.j.
G
To ensure that different prompt groups capture diverse semantic perspectives, we introduce a diver- sity loss that minimizes the cosine similarity between group- wise features within the same modality:
K Σ
Las
Σ (1 - (itccoo(ota.)..ttt//. + Σ 101-cosioe (o1-cos(f())) 3913
(4)
Amm ji
This encourages each group to specialize in different aspects of the input, thereby reducing redun- dancy and enhancing representational richness. The overall training objective combines both losses:
L Lce A Laiv
(5)
At inference time, we leverage prompt groups by computing predictions independently for each group and averaging the resulting logits:

---

G G
p(ù = h x1)
p k x; P.....Poo.))
(6)
This group-wise ensemble enhances robustness by aggregating complementary semantic views, im- proving prediction stability across heterogeneous inputs while incurring minimal overhead.
3.2.2 Dynamic Prompt Aggregation Strategy
To address the aggregation instability issues prevalent in traditional federated learning, FedMGP employs a novel dynamic prompt aggregation strategy, as illustrated in the right portion of Figure This approach is based on a fundamental insight: each prompt can be conceptually decomposed into information that common across clients (global knowledge) and information that unique to specific client (local knowledge). Therefore, we propose this dynamic aggregation mechanism that adaptively balances the preservation of global knowledge with exploration of client-specific fea- tures. In Appendix F, provide theoretical analysis demonstrating the superiority of our dynamic aggregation strategy over both full prompt aggregation methods([T7, [all and explicit global-local paradigms [2Z, 9, [7]], offering formal justification for our approach.
In this section, we use P without subscripts to refer to the entire set of prompts, while Pj (Pt.j.Pe.ji refers to the j-th group within that set. The global aggregated prompts are denoted by P. where P consists of top-s selected prompt groups.
In each communication round, our strategy dynamically selects a subset of top-s prompts from each client for aggregation, where 8 S G and G is the total number of prompt groups. When = G. our method reduces to standard FedAvg. By selecting only the most relevant prompts, we focus the aggregation on shared knowledge while preserving client specificity. The key steps of our dynamic aggregation strategy are as follows:
For communication round T, we first compute the cosine similarity between each client's local prompts and the global prompts from the previous round. This process performed separately for text and visual prompts, but follows the same procedure. For each local prompt group PJ" and its corresponding global prompt pT- (where i E {1,2..... .) indexes the top-s selected groups):
5
pf PT IPP
sim sim PT-1)
To avoid overly deterministic selection that might lead to prompt homogenization, we convert these similarity scores into selection probabilities using a softmax function with temperature parameter
prob(P!)) exp(sim( (PT (T,TT//1/)
exp(sim(P?) (PT, Pr 1)/-)
Based on these probabilities, sample prompt groups from each client. For the first commu- nication round (T 1), when no previous global prompts exist, we employ random selection to establish initial diversity, described in Appendix A.
After selecting the top-s most relevant prompt groups from each client, the server aggregates these prompts across all participating clients to form the updated global prompts for round T. For the i-th selected prompt group:
pT = Σ
The
PT 6,e
Edecr We E
CEC
where Or represents the set of clients participating in round T, n. denotes the number of samples at client and P represents the i-th selected prompt group from client
5

---

This dynamic prompt aggregation strategy offers several key advantages. First, by favoring prompts with higher similarity to previous global prompts, we effectively filter out client-specific idiosyn- crasies that might arise from local data distribution peculiarities. Second, the dynamic nature of our selection process prevents premature convergence to a static set of prompts, allowing the model to continually explore the prompt space and adapt to evolving patterns in the data. Third, this approach naturally balances the preservation of common knowledge with the exploration of diverse prompt configurations, leading to more robust federated learning.
After aggregation, the server distributes the updated global prompts back to the clients for the next round, continuing this process for multiple communication rounds to gradually refine the global prompts to work well across all clients.
Experiment
In this section, we conduct comprehensive experiments validate the dual capabilities of FedMGP: (I) maintaining strong personalization for individual clients while achieving robust cross-client gen- cralization, and (2) demonstrating superior performance across various heterogeneous data distri- butions. Our evaluation spans multiple scenarios including non-lID data partitions and Dirichlet distributions with varying concentration parameters, demonstrating FedMGP's effectiveness ad- dressing the fundamental challenges of federated learning with prompt-based multimodal adapta- tion.
4.1 Experimental Setup
Datasets and Data Heterogeneity.To thoroughly evaluate FedMGP's dual capabilities of person- alization and generalization across heterogeneous data distributions, we design experiments with three distinct scenarios. First, following [9.
[6]]. we select nine diverse datasets to assess base-
to-novel class generalization: Caltech101 [US for general object classification; OxfordPets [G8], Flowers 102 [[6]], Food101 [3], Stanford Cars P and FGVC Aircraft [(3]] for fine-grained classi-
fication; DTD [ for texture classification; UCF101 4 for action recognition; and SUN397 [54]] for scene recognition. We create pathological non-III setting by equally splitting each dataset into base and novel classes, then assigning non- non-overlapping base classes to different clients. Each client's model is trained on local classes and evaluated three test sets: local classes (personaliza- tion), base classes seen by other clients (cross-client knowledge transfer), and novel classes unseen during training (generalization to new Sancepts).Secommce to evaluate personalization under label dis- tribution shift, we employ CIFAR-10 and CIFAR-100 [26]), partitioning data among clients using Dirichlet distribution Dir(a: with varying concentration parameters. This creates realistic hetero- geneity where clients possess varying class proportions, allowing us to examine how effectively FedMGP's multi
multi-group group prompt mechanism adapts to imbalanced class distributions. Third,, to as- sess performance under both feature and label distribution shifts, we test FedMGP on multi-domann datasets: DomainNet [B9] with six distinct visual domains and Eifice-Caltechl [[']]] with four do- mains. This evaluates how effectively our text-visual prompt co-lcarning bridges domain gaps while maintaining local specialization. Comprehensive dataset details are provided in Appendix C.I.
Implementation Details. To ensure fair comparison with existing methods, we es- tablish a unified experimental framework by re-implementing all baseline approaches using their official code repositories under identical settings. Specifically, we adopt ViT-B/16 [I as the backbone for all meth- ods. For base-to-novel generalization ex- periments, we set communication rounds T 10 with 100% client participation rate, local epochs E = 2, and use 16-shot sam- ples per class. For CIFAR-10 and CIFAR-
Table 2: Results on CIFARIO and CIFAR100 with la-
bel shift with Dir partition(a = 0.5) into 100 clients.
Methods
CIFARIO
CIFAR 100
PromptFL
91.36
72.04
FedOTP
94.73
75.15
FedTPG [1
92.44
74.39
FedPGP [9]
92.41
74.11
PromptFolio [6>]]
93.33
74.14
FedMGP
95.48
75.39
100 experiments, we simulate a realistic federated environment with Dir(a = 0.5) distribution across 100 clients, with 10% client participation rate per round, utilizing the full training dataset. All models are trained using stochastic gradient descent (SGD) with an initial learning rate of 0.001

---

Table 1: Accuracy comparison (%) on clients' local accuracy and generalization.
(a) Average over 9 datasets.
(b) OxfordPets.
Methods
Local
Base
Novel
CM
Methods
Local
Base
Novel
CM
PrompeFI.
7
71.19
71.70
71.46
71.31
PromptFI. [
89.77
90.01
97.20
91.62
FedOTP
92.53
16.84
31.66
57.10
FedOTP [27
100.00
26.68
57.16
68.19
FedTPG
E11
71.62
71.91
68.32
70.66
FedTPG [E]
94.24
94.31
96.64
94.85
FedPGP [
84.32
72.45
68.97
77.42
FedPGP [8]
96.20
95.01
96.89
96.07
PromptFolio
96.02
39.75
51.02
70.29
PromptFolio [3
99.90
66.23
83.38
86.86
FedMGP
93.17
68.49
72.99
81.85
FedMGP
97.15
93.83
97.04
96.28
(c)
Flowers
(d) DTD.
Methods
Local
Base
Novel
Methods
Local
Base
Novel
CM
PromptFL [
70.33
71.79
75.39
71.94
PromptFL
55.32
57.06
44.32
$2.60
FedOTP [AM
99.73
13.06
21.51
57.99
FedOTP [16
96.44
20.06
41.23
61.71
FedTPG [+]
79.43
78.92
73.26
77.71
FedTPG
56.90
59.26
40.46
$2.49
FedPGP
91.83
80.22
68.46
82.85
FedPGP
78.47
67.22
50.93
68.21
PromptFolio [+
99.82
27.36
39.34
66.05
PromptFolio [G7]]
97.18
26.53
37.39
64.11
FedMGP
98.41
70.06
74.71
85,36
FedMGP
92.87
53.60
55.62
73.73
(e)
Caltech101.
(f)
Food101.
Methods
Local
Base
Novel
CM
Methods
Local
Base
Novel
CM
PromptFL. 7
94.16
95.35
94.98
94.66
PromptFL.
89.75
89.79
90.86
90.04
FedOTP [20
99.96
28.28
62.26
69.43
FedOTP [44
95.44
19.16
45.89
61.24
FedTPG EI]
96.17
97.16
91.92
95.32
FedTPG [I
90.36
90.42
91.78
90.73
FedPGP [+]
96.91
97.35
94.37
96.37
FedPGP [
90.51
90.48
91.12
90.65
PromptFolio [77]
99.79
73.69
81.10
88.50
PromptFolio [60]
97.24
57.40
67.64
79.67
FedMGP
99.47
96.02
93.61
97.13
FedMGP
95.08
88.47
89.53
92.04
(g) UCF101.
(h) SUN397.
Methods
Local
Base
Novel
CM
Methods
Local
Base
Novel
CM
PromptFL. [
77.08
76.94
70.36
75.29
PromptFL [1
76.25
76.20
75.68
76.09
FedOTP [24
92.39
16.33
19.07
54.99
FedOTP [27]
93.40
11.38
19.11
53.83
FedTPG [+
76.22
75.96
72.09
75.10
FedTPG [+I]]
73.72
73.71
75.17
74.08
FedPGP [_]
82.61
71.78
68.45
76.34
FedPGP [
89.43
66.51
67.43
78.20
PromptFolio [5
96.15
31,94
42.00
66.22
PromptFolio [6
95.18
32.89
44.47
66.50
FedMGP
92.69
68.38
72.86
81.62
FedMGP
91.83
68.51
72.20
81.07
(i)
Stanford Cars.
(i) FGVC Aircraft.
Methods
Local
Base
Novel
CM
Methods
Local
Base
Novel
CM
PromptFL [
62.98
63.14
69.87
64.66
PromptFL [MX]]]]]]
25.03
25.03
24.48
24.89
FedOTP [27
91.06
9.32
10.62
50.49
FedOTP
64.34
7.27
8.12
36.01
FedTPG [[]]]
65.50
65.47
69.10
66.37
FedTPG [[1]]
12.00
12.00
4.50
9.27
FedPGP
85.37
57.63
60.19
72.13
FedPGP [
47.59
25.89
22.89
35.94
PromptFolio [37]]
96.44
29.43
46.77
66.28
PromptFolio [32]]
82.50
12.29
17.09
48.40
FedMGP
92.61
56.48
71.19
77.80
FedMGP
78.46
21.03
30.15
51.62
and single-step learning rate scheduler. All other implementation specifics, including additional hy- perparameter settings. optimization strategies, and evaluation protocols, are detailed in the appendix to ensure reproducibility. For more details, please refer to Appendix C.2.
Baselines. We compare FedMGP against state-of-the-art Federated Prompt Learning (FPL) meth- ods, including PromptFL [ILZ], FedOTP [27]., FedTPG [4]]], FedPGP [H], and PromptFolio [37]] These baselines represent the full spectrum of existing FPL paradigms: from standard aggregation approaches to local-global frameworks and constrained local-global architectures. This compre- hensive comparison allows us to evaluate how effectively FedMGP addresses the critical balance between personalization and generalization that many existing methods struggle to achieve, particu- larly under severe data heterogeneity.
4.2 Performance Evaluation
Analysis of Base-to-Novel Generalization Results. To comprehensively assess both personaliza- tion and generalization capabilities, we introduce Combined Metric (CM) that balances local adaptation and cross-domain transfer. Following the approach in [17] for local accuracy evalua- tion and [60] for harmonized accuracy calculations, CM is computed as CM = (Local + HM)/2, where HM is the Harmonic Mean defined as HM 2 x Base Novel/ + Novel). This metric
7

---

Caltech101
DTD
Flowers 102
Combined Metric (CM) (%)
100
100
100
80
80
80
60
60
60
40
40
40
20
20
8
16
2
8
16
12
4
8
16
Number of Shots
Number of Shots
Number of Shots
PromptFL
FedOPT
FedTPG
FedPGP
PromptFolio
FedMGP (Ours)
Figure 2: Few shot experiment from 1 to 16 shots
effectively quantifies model's ability to simultaneously achieve personalization (measured by local accuracy) and generalization (measured by harmonized performance on base and novel classes). As shown in Table II, FedMGP achieves the highest CM score (81. (1.85%) averaged over all nine datasets, demonstrating superior overall performance while maintaining excellent balance between personal- ization (93.17% local classes) and generalization (68.49% on base classes and 72.99% on novel classes). In contrast, methods like FedOTP and PromptFolio achieve exceptional local accuracy (92.53% and 96.02% respectively) but struggle with generalization to base classes (16.84% and 39.75%), indicating severe overfitting to local distributions. FedPGP, though more balanced, still falls short of FedMGP in comprehensive performance. These results confirm our analysis that exist- ing approaches either excel at personalization at the expense of generalization or achieve moderate performance on both fronts without fully resolving the tension between these competing objectives.
Table 3: Parameter analysis of FedMGP and other state-of-the-art methods. Method
Table 4: Ablation study prompt leangth(I)
Trained
Communication
CM
Setting
Local
Base
Novel
CM
PromptFL
8.2k
8.2k
80.27
FedOTP 2A
16.4k
8.2k
63.74
FedMGP (I=4)
97.18
72.49
72.17
84.75
FedTPG [0
4208.1k
4208.1k
82.37
FedMGP (I=8)
98.05
64.00
64.91
81.25
FedPGP [
24.8k
16.4k
86,94
FedMGP (I=16)
97.62
57.47
61.56
78.53
PromptFolio [6Y]
16.4k
8.2k
77.10
FedMGP (I=2)
96.92
73.23
74.65
85.43
FedMGP
12.8k
5.1k
88.34
Performance on Label Distribution Shift. We evaluate FedMGP's effectiveness in handling realis- tic federated learning scenarios with 100 clients following a Dirichlet distribution (a 0.5), which creates substantial heterogeneity class distributions. As shown in Table D. FedMGP consistently outperforms all baseline methods on both CIFAR-10 and CIFAR-100 datasets. The multi-group prompt mechanism effectively captures diverse client data patterns through text-visual prompt co- learning and similarity- based selection, enabling robust performance even under severe label imbal- ance. Notably, while other methods struggle with the increased complexity of CIFAR- 100, FedMGP maintains its relative advantage, demonstrating strong scalability in federated learning with numer- ous clients and classes.
Few-Shot Analysis. Figure demonstrates FedMGP's effectiveness across few-shot settings (1- 16 shots per class). While FedMGP exhibits limitations in extreme 1-shot scenarios, it quickly surpasses competing methods with 2+ shots. This performance pattern aligns with our theoretical framework: in extremely limited data regimes, the multi-group mechanism struggles to effectively decompose knowledge into common and client-specific componentsa decomposition that is funda- mental to our approach as described in Section B.2.2 Specifically, with insufficient samples, prompt groups cannot effectively disentangle specialized representations nor establish robust text-visual cor- relations across client distributions. As sample size increases, FedMGP's dynamic prompt selection strategy activates its full potential, enabling superior cross-client knowledge transfer while preserv- ing client-specific information. Detailed discussions on FedMGP's limitations and future research directions can be found in Appendix B.
8

---

Table 5: Ablation study on Prompt Groups(m)
Table 6: Ablation study on Top-s.
Setting
Local
Base
Novel
CM
Setting
Local
Base
Novel
CM
FedMGP (m=4)
96.60
73.28
73.99
85.12
FedMGP (Top-s=1)
97.88 69.17
74.10
84.72
FedMGP (m=3)
89.95
77.68
74.48
83.00
FedMGP (Topk-s=3)
92.93 76.88
74.85
84.39
FedMGP (m==)
82.88
82.15
74.05
80.38
FedMGP (Topk-s=4)
86.77
79.44
74.89
81.93
FedMGP (m==1)
78.85
79.68
70.67
76.88
FedMGP (Topk-s=2)
96.92
73.23
74.65
85.43
FedMGP (m==5)
96.92
73.23
74.65
85.43
Parameter Efficiency. Table E highlights FedMGP's remarkable communication efficiency (5.1k baranieersssigniffiaanyyy lower than all competitors while achieving superior performance. This val- idates our core design: rather than increasing parameter count, FedMGP strategically distributes fixed capacity across multiple specialized prompt groups, more effectively capturing diverse client data characteristies with minimal communication overhead. Additional evaluation like domain eval- uation results are presented in Appendix D.
4.3 Ablation Study
To thoroughly understand FedMGP's design choices, we conduct extensive ablation studies examin- ing key components including prompt length, number prompt groups, top-s selection size, vision- text modality contributions, and diversity loss. For comprehensive evaluation and efficiency, all results are reported as the average performance across Caltech101, Flowers102. and DTD datasets, providing insights into FedMGP's optimal configuration.
Impact of prompt length. Table reveals that increasing prompt length beyond (=2 causes con- sistent performance degradation. In heterogencous federated environments, compact prompts excel by capturing essential semantic patterns without overfitting to client-specific details, enabling more effective knowledge sharing across diverse client distributions.
Table 8: Ablation study on Laiv-
Table 7: Ablation study the impact of vision and text prompt.
Setting
Local
Base
Novel
CM
Setting
Local
Base
Novel
CM
FedMGP (w/o Lav)
94.53 72.97
72.48
83.63
FedMGP (Vision Only)
75.94
76.48
72.92
75.30
FedMGP (Cav=2)
95.78
74.88
74.98
85.35
FedMGP (Text Only)
95.23
73.60
73.80
84.46
FedMGP (Lav=5)
96.35
73.50
74.31
85.13
FedMGP (Vision Text)
96.92
73.23
74.65
85.43
FedMGP (Ca.=10)
95.78
73.09
74.35
84.75
FedMGP (Ca.=1)
96.92
73.23
74.65
85.43
Effect of Group Number. Table E shows that multiple prompt groups are crucial for FedMGP's effectiveness, with performance declining as group count decreases. Our results indicate that 5 groups achieves optimal performance, with additional groups likely offering diminishing returns relative to the increased parameter count. This validates our multi-group design which effectively balances personalization and generalization without rigid global-local separation.
Selection Strategy Analysis. Table B demonstrates how our dynamic prompt aggregation strategy trade-off. With smaller selection size (Top-s=1), the model preserves client specificity but limits knowledge sharing, while larger selection size (Top- s=4) improves generalization but significantly compromises personalization. Top-s=2 emerges as the optimal balance point, effectively addressing the aggregation instability issues.
navigates the critical personalizztioonggeeeril
Effect of Vision and Text Components. Table Ø confirms the necessity of incorporating both vision and text components FedMGP. Removing either modality leads to noticeable performance degradation, highlighting the complementary roles they play. While textual prompts capture high- level semantic categories, visual prompts provide fine-grained, instance-specife cues. Their joint contribution enables FedMGP to better represent diverse client data and facilitates more effective cross-client knowledge transfer. This finding supports the design choice of our multi-group text- visual prompt co-learning framework.
Impact diversity Loss. Table demonstrates the critical importance of diversity loss FedMGP with its removal causing significant performance drop (CM decreases by 1.8%). This component

---

ensures effective separation between prompt groupsa fundamental mechanism we analyze in detail in Appendix Remarkably, performance remains stable across different weight values (1-10), con- firming its insensitivity to hyperparameter settingsa significant advantage in federated environments with heterogeneous data distributions. Appendix E. contains additional ablation studies on tempera- ture parameters, diversity loss formulations, and other design choices.
4.4
Visual Analysis
CCM
1.60
LAN
1.54
-100
531
+M
6
Clich
6M4
3.75
0.40
1.80
1.01
C20
Figure 3: Intra-client prompt similarity visual- ization. (a) Text prompt similarity matrix show- ing moderate inter-grow group diversity. (b) Visual prompt similarity matrix showing higher inter- group diversity.
Figure 4: Inter-client prompt similarity visu- alization. (a) Text prompt similarity matrix showing high correlations (0.9-1.0). (b) Visual prompt similarity matrix showing moderate di- versity (0.7-0.9).
To validate our diversity loss mechanism, we analyze internal prompt similarity patterns within a representative client after FedMGP training Caltech101. Figure B presents similarity matrices for text and visual prompt groups, revealing distinct specialization. Text prompts show moderate inter-group correlations (0.5-0.8), maintaining shared linguistic patterns, while visual prompts ex- hibit significantly lower correlations (often near zero or negative), achieving superior diversification. This confirms that visual prompts capture more fine-grained, instance-specifie features than text prompts. The diversity loss successfully encourages each prompt group to specialize in distinct patterns, enabling comprehensive local data coverage while supporting both personalization and cross-client generalization.
To further validate our dynamic aggregation mechanism, examine inter-client prompt similar- ity patterns. Figure E shows that text prompts maintain consistently high correlations (0.9-1.0) across clients, preserving common semantic knowledge while avoiding the complete homogeniza- tion in PromptFL. [[Z]]. Visual prompts show moderate correlations (0.7-0.9), striking an optimal balance between knowledge transfer and client-specifie adaptation. Unlike FedOPT's [27]] global- local paradigm that often results in excessive divergence, our approach maintains sufficient similarity for knowledge sharing while preserving diversity for personalized learning. This confirms that Fed- MGP's dynamic aggregation effectively prevents over-homogenization and excessive divergence, achieving superior performance across heterogeneous client distributions.
Conclusion
This paper presents FedMGP, a novel federated learning paradigm that addresses the fundamental trade-off between personalization and generalization in existing federated prompt learning meth- ods through multi-group text-visual prompt co-learning. The key innovations of FedMGP include: (I) leveraging multiple text-visual prompt pairs overcome the limited expressiveness of single prompts, with each prompt group focusing on different semantic features; (2) introducing diversity loss ensure representation separation between prompt groups, enhancing the model's expressive power: (3) designing a similarity- based dynamic prompt selection strategy that effectively balances shared knowledge and client-specific features. Extensive experiments demonstrate that FedMGP achieves superior balance between personalization and generalization capabilities across various heterogeneous data environments while maintaining minimal communication parameters. future work, we will explore alternative regularization constraints and integrate -ategory-specific linguis- tic information to further enhance diverse representations across prompt groups, while investigating more sophisticated text-visual prompt collaboration mechanisms to improve cross-modal alignment in federated settings.
Acknowledge This work was supported by National Natural Science Foundation of China (Grant No. 62425603) and Basic Research Program of Jiangsu Province (Grant No. BK2024001
10

---

References
[1] Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In Advances in Neural Information Processing Systems, pages 80396-80413, 2024.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations.
[3] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning scale: System design. Proceedings of machine learning and sys- tems, 1:374-360 388, 2019.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101- diseriminative components with random forests. In Proceedings of the European Conference on Computer Vision, pages 446-461, 2014.
[5] Zhongyi Cai, Ye Shi, Wei Huang, and Jingya Wang. Fed-co _22]: Cooperation of online and offline models for severe data heterogeneity federated learning. In Advances in Neural Information Processing Systems, pages 21343-21367, 2023.
[6] Yuan Cao. Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. In Advances in Neural Information Processing Systems, pages 25237-25250, 2022.
[7] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[8] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Fedavg with fine tuning: Local updates lead to representation learning. In Advances in Neural Information Processing Systems, pages 10572-10588, 2022.
[9] Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. Harmonizing generalization and person- alization in federated prompt learning. In Proceedings of the International Conference on Machine Learning, pages 9646-960 96669,96,116 2024.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. image is worth 16x16 words: Transformers for image recognition scale. In Pro- ceedings of the International Conference on Learning Representations, pages 11960-11973, 2020.
[11] Penghui Du, Yu Wang, Yifan Sun, Luting Wang, Yue Liao, Gang Zhang, Errui Ding, Yan Wang, Jingdong Wang. and Si Liu. Lami-detr: Open-vocabulary detection with language model instruction. In European Conference Computer Vision, pages 312-328. Springer, 2024.
[12] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. arXiv preprint arXiv:2002.079448 2020.
[13] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few train- ing examples: An incremental bayesian approach tested on 101 object categories. In Proceed- ings of the IEEE Conference on Computer Vision and Puttern Recognition Workshop, pages 178-178, 2004.
[14] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 20/2 IEEE conference on Computer Vision and Pattern Recognition, pages 2066-2073 IEEE, 2012.

---

[15] Mingzhe Guo, Zhipeng Zhang, Liping Jing, Haibin Ling, and Heng Fan. Divert more atten- tion lision-language object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 46(12))8600-86) 86188888 8, 2024.
[16] Tao Guo, Song Guo, and Junxiao Wang. Pfedprompt: Learning personalized prompt for vision- language models in federated learning. In Proceedings of the ACM Web Conference 2023, pages 1364-1374, 2023.
[17] Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let feder- ated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing, 23(5):517--5194, 2023.
[18] Wei Huang, Ye Shi, Zhongyi Cai, and Taiji Suzuki. Understanding convergence and gener- alization in federated learning through feature learning theory. In The Twelfth International Conference on Learning Representations, 2023.
[19] Xin Jiang, Hao Tang, Junyao Gao, Xiaoyu Du, Shengfeng He, and Zechao Li. Delving into multimodal prompting for fine-grained visual classification. In Proceedings of the AAAl con- ference on artificial intelligence, pages 2570-2578, 2024.
[20] Xin Jiang, Hao Tang, and Zechao Li. Global meets local: Dual activation hashing network for large-scale fine-gratd fine-grained image retrieval. IEEE Transactions on Knowledge and Data Engineer- ing, 36(1 62799, 2024.
[21] Xin Jiang, Hao Tang, Rui Yan, Jinhui Tang, and Zechao Li. Dvf: Advancing robust and accurate fine-grained image retrieval with retrieval guidelines. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2379-2388, 2024.
[22] Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tuning for generalizable vision-language models. In Proceedings of the IEEE International Conference on Computer Vision, pages 35670-15680, 2023.
[23] Muhammad Uzair Khattak, Muhammad Ferjad Nacem, Muzammal Nascer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision-language models. arXiv preprint arXiv:2401.02416 2024.
[24] Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In International Conference on Machine Learning, pages 17615-17659. PMLR, 2023.
[25] Jonathan Krause, Michael Stark. Jia Deng. and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] Hongxia Li. Wei Huang, Jingya Wang, and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12151-12161, 2024.
[28] Qinbin Li, Bingsheng He, and Dawn Song. contr-ccoe federated learning. In Proceed- ings of the IEEE conference on Computer Vision and Pottern Recognition, pages 10713-10722, 2021.
[29] Zengxiang Li, Zhaoxiang Hou, Hui Liu, Tongzhi Li, Chengyi Yang, Ying Wang, Chao Shi, Longfei Xie, Weishan Zhang, Liang Xu, et al. Federated learning in large model era: Vision- language model for smart city safety operation management. In Companion Proceedings of the ACM Web Conference 2024, pages 1578-1585, 2024.
[30] Haodong Lu, Xinyu Zhang, Kristen Moore, Jason Xue, Lina Yao, Anton van den Hengel, and Dong Gong. Continual learning on clip via incremental prompt tuning with intrinsic textual anchors. 2025.
12

---

[31] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7086-7096, 2022.
[32] Jun Luo, Chen Chen, and Shandong Wu. Mixture of experts made personalized: Federated prompt learning for vision-language models. arXiv preprint arXiv:24 10. 101 14, 2024.
[33] Subhransu Maji. Esa Rahtu, Juho Kannala. Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.5551, 2013.
[34] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-effitee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[35] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In Proceedings of the International Conference on Learning Representations, 2023.
[36] Maria-Eles Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, 2008
[37] Bikang Pan, Wei Huang, and Ye Shi. Federated leariing from vision- lisounlaagguage foundation models: Theoretical analysis and method. In Advances in Neural Information Processing Systems, pages 30590-30623, 2024.
[38] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3498 3505, 2012.
[39] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 1406-1415 2019.
[40] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? gen- erating customized prompts for zero-shot image classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 15691-15701, 2023.
[41] Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Lin. Federated text-driven prompt generation for vision-language models. In The Twelfth International Conference on Learning Representations, 2024.
[42] Hongyu Qu, Rui Yan, Xiangbo Shu, Hailiang Gao, Peng Huang, and Guo-Sen Xie. Mvp- shot: Multi-velocity ioogressive-alignmert framework for few-shot action recognition. JEEE Transactions on Multimedia, 2025.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[44] Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEE International Conference on Computer Vision, pages 15746-15757, 2023.
[45] Cheng Shi and Sibei Yang. Logoprompt: Synthetic text images can good visual prompts for lasion-language models. In Proceediiggs of the IEEE International Conference on Computer Vision, pages 2932-2941, 2023.
[46] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vims. Proceedings of the IEFE International Conference Computer Vision, pages 11987-11997, 2023.
13

---

[47] K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[48] Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. Singular value fine-tuning: Few-shot segmen- tation requires few-parameters fine-tuning. In Advances in Neural Information Processing Systems, pages 37484-37496. Curran Associates, Inc., 2022.
[49] Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang. and Zechao Li. Vrp-sam: Sam with visual reference prompt. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 23565-23574, 2024.
[50] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, pages 21394-21405, 2020.
[51] Linh Tran, Wei Sun, Stacy Patterson, and Ana Milanova. Privacy-preserving personalized fed- erated prompt learning for multimodal large language models. In The Thirleenth International Conference on Leurning Representations.
[52] Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning non- iid data with reinforcement learning. In IEEE INFOCOM 1020-IEEE Conference on computer communications, pages 1698- 1707. IEEE, 2020.
[53] Yubin Wang. Xinyang Jiang, De Cheng, Dongsheng Li, and Cairong Zhao. Learning hierarchi- cal prompt with structured linguistic knowledge for vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5749-5757, 2024.
[54] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
[55] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang. Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[56) Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Pro- ceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 18123- 18133, 2022.
[57] Renrui Zhang. Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt. generate, then cache: Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 15211-15222, 2023.
[58] Xinyu Zhang. Jiahui Chen, Junkun Yuan, Qiang Chen, Jian Wang, Xiaodi Wang. Shumin Han, Xiaokang Chen, Jimin Pi, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. CAE v2: Context autoencoder with CLIP latent alignment. Transactions on Machine Learning Research, 2023.
[59] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[60] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-l models. International Journal of Computer Vision, 130(9))2337-2348, 2022.
[61] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE International Conference on Computer Vision, pages 15659-15669, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We empirically verify the claims and contributions made the abstract and introduction through extensive experiments across diverse federated vision-language bench- marks. Our theoretical analysis further supports the design motivations, including prompt diversity, probabilistic sampling, and cross cross-modal pairing.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include discussion of the limitations of our work in the appendix, high- lighting potential areas for improvement and directions for future research.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide the full set of assumptions and complete proofs for our theoretical results the appendix, ensuring correctness and clarity of the analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed descriptions of all experimental settings, including dataset splits, model architecture, training hyperparameters, and evaluation metrics. These details are sufficient to reproduce the main results and support the core claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release our code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis..om cc/
for details.
While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
issicnoolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The training and test details are presented in section 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: This paper does not report error bars statistical significance measures. The results are presented as aggregate performance metrics without confidence intervals or vari- ance analysis.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of eπrоrs is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were conducted on either NVIDIA 4090-D GPU (24 GB) or NVIDIA A100 GPU (40 GB), with each training run completing within 24 hours for all datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't it make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed and adhered the code of ethics throughout our research and writing process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All publicly available assets (models, code, and data) used in this work have been properly credited, and their respective licenses and terms of use have heen explicitly mentioned and adhered to.
Guidelines:
The answer NA means that the paper does not existing assets.
19

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license. copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets in the submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
The paper does not Large Language Models (LLMs) important, original, or non-standard component of the core methods.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non sta-staddard components.
Please refer to our LLM for what should or should not be described.
21

---

Supplementary organization:
Fed ALGORITHM
23
Limitations and Broader Impacts
24
Experimental Details
24
Dataset Setup
24
Experimental Setup
25
Additional Experimental Results
26
Domain Gennrallzation for NomainNet
26
Domain Gennralization for Offece CallecCh
................................
26
Stability Prompt Group Selection
27
Additional ablation study
27
Effect of Temperature Parameter Prompt Selection
28
Inference- Time Prompt Group Weighting Strategies
................................
28
Diversity Loss Formulation Variants
28
Dynamic Aggregation Strategy
29
Theoretical Analysis
29
Assumptions and Notation Definitions
30
Formalization of Three Aggregation Strategies
................................
31
Stosel-Noise Decomposition and Performance Metrics
32
Dynamic Aggregation Superiority Theorem and Detailed Proo
33
22

---

A FedMGP ALGORITHM
Algorithm FEDMGP:
Federated Learning via Multi-Group Text-Visual Visual Prompt Co-Learning
Inputs:
Communication rounds T, local epochs R. number of clients N, local datasets
{Dele=1 image encoder f(-). text encoder 9(.), number prompt groups G, top-s size for aggregation, temperature diversity loss weight X, learning rate Outputs: Personalized multi-group prompts {PeJa=r where
P. Pp....... .P.... .....--..1 ..............
1: Server Executes:
2: Initialize global prompts po (pr. Pr.. Co..Po......... P.
3: for each client c = I,... N do
4: Distribute copies: Pa nn
5: end for
6: for each communication round = 1,..................... .,....TTnn maz do
7: Server selects a subset of clients Cr.
8: for each client e E CT in parallel do
9: p. CLIENTUPDATE(C, Pe. Des R, f.g. G,T,A,n)
10:
end for
Dynamic prompt aggregation stage
11:
for each client e E Cr do
if then
13:
Select 8 prompt groups randomly from P. denoted as pl e,sclected
14:
else
15:
Compute similarity scores between client prompts PT and global prompts PT-1
using Eq. (7)
16:
Convert similarities to probabilities using Eq. (8)
17:
Probabilistically select prompt groups from P based on these probabilities, de-
noted as p2 e.sciected end if c,selected to server.
18:
19:
Send PT
20:
end for
21: Server aggregates collected prompts form pr using Eq. (9)
22:
for each client e E Cr do
23:
Update the selected prompt groups in P. with corresponding prompts from pr end for
24:
25:
end for
26: return Final personalized prompts (Po]
1:
procedure CLIENTUPDATEIC, Pe, R.g,
2: Let local prompts P. = PD.. Pu.l. P..c]
3:
for each epoch 1,..., do Sample mini-batch (z, y) ~ De
4:
5:
for each prompt group j 1, ... G do
6:
Form visual input = fx, Po.j and compute f(;y)
7:
for each class & do
8:
Form text input (x, = {pr,j, )x) and compute g(4u.))
9:
Compute logits using Eq. (2)
10:
end for end for
12:
Compute classification loss LcE using Eq. (3) Compute diversity loss Laiv using Eq. (4)
14:
Total loss L - LeE + X Laiv Update prompt parameters Pa using gradient descent with learning rate n)
16:
end for
return Updated prompts P. end procedure
18:
23

---

B Limitations and Broader Impacts
As shown in Table while FedMGP consistently outperforms existing methods across most set- tings, it shows limitations in extremely data-scarce scenarios (1-2 shots). This stems from our multi- group prompt mechanism, which requires sufficient data effectively disentangle different seman- tic aspects. With minimal samples, prompt groups cannot specialize properly, leading to unstable training. The text-visual co-learning mechanism further compounds this challenge, as establishing robust moosl correlations requires visual diversity absent in I-shot settings. Additionally, our dynamic aggregation strategy becomes less reliable when prompt representations are unstable due to data scarcity. Simpler methods like PromptFolio occasionally perform better in these extreme low-shot scenarios precisely because they avoid the complexity that makes FedMGP powerful in data-rich environments.
To address these limitations, several future directions emerge: (1) developing adaptive mechanisms to dynamically adjust the number of prompt groups based on available data, reducing groups when data is searce; (2) initializing prompt groups with knowledge from related tasks to provide stronger starting points for specialization; and (3) incorporating meta-leaming techniques to improve learn- ing efficiency from limited examples. While FedMGP contributes positively to privacy-preserving adaptation of vision-language models in decentralized environments, acknowledge that, like all federated learning systems, it may remain vulnerable to various attacks. As these technologics advance toward deployment in sensitive domains, continued research must address both technical limitations and broader societal implications.
Table 9: Performance Comparison of Different Methods on 1-16 shots
Dataset
Method
1-shot 2-shots
4-shots
8-shots
16-shots
PromptFL
95.23
95.30
95.62
96.20
94.66
FedOPT
92.81
92.95
89.27
81.37
69.43
FedTPG
95.74
94.88
96.00
96.02
95.32
Caltech101
FedPGP
96.02
96.13
95.54
95.87
96.37
PromptFolio
95.84
95.39
93.90
90.39
88.50
FedMGP (Ours)
96.03
95.48
96.48
97.14
97.07
PromptFL
53.78
52.69
55,08
60.58
52.60
FedOPT
68.73
63.93
65.68
65.51
61.71
FedTPG
59.30
64.28
66.22
65.85
52.49
DTD
FedPGP
58.15
63.60
62.00
63.31
68.21
PromptFolio
64.86
63.92
65.11
62.42
64.11
FedMGP (Ours)
62.00
68.32
69.58
69.12
73.92
PromptFL
78.20
72.37
73.27
69.71
71.94
FedOPT
68.62
68.42
66.32
60.54
57.99
FedTPG
74.59
73.71
78.05
78.85
77.71
Flowers102
FedPGP
73.49
73.16
76.09
84.39
82.85
PromptFolio
81.24
79.51
79.35
70.99
66.05
FedMGP (Ours)
73.75
78.07
83.80
84.53
85.16
C Experimental Details
C.1
Dataset Setup
Our evaluation leverages nine diverse visual classification datasets, spanning fine- recogni- tion, texture analysis, general object classification, and domain adaptation tasks. Table II provides comprehensive details about these datasets, including classes, sample sizes, domains, and training protocols.
For our base-to-novel generalization experiments (Oxford-Pets to Food101), we employ few-shot training paradigm, where each client provided with only samples per class (16-shot) for the main experiments. These datasets are partitioned by splitting classes cqually into base and novel

---

categories, with non- von-overlapping base classes distributed across clients to establish the pathological non-IlI setting described in Section 4.
For label distribution shift experiments, we utilize CIFARIO and CIFARTO0 with Dirichlet distribu- tion partitioning (a 0.5) across 100 clients, using the full training set. This creates realistic client heterogeneity with varying class proportions.
For domain adaptation scenarios, we leverage Office-Caltec110 with its four domains (Amazon, Cal- tech, DSLR, and WebCam) and DomainNet with six domains (Clipart, Infograph, Painting, Quick- draw, Real, and Sketch). Each domain is split into 5 clients under Dirichlet distribution (a 0.3), resulting in a total of 20 and 30 clients respectively. This setup introduces natural feature shifts across domains and moderate label skew within each domain.
Table 10: Statistical details of datasets used in experiments.
Dataset
Classes
Train
Test
Domains Training Protocol Task
OxfordPets
68
37
2,944
3,669
Few-shot (16-shot)
Pets recognition
Flowers 102
66
102
4,093
2,463
Few-shot (16-shot)
Flowers recognition
DTD [+]
47
2,820
1,692
I
Few-shot (16-shot)
Texture recognition
Caltech101
100
4.128
2,465
Few-shot (16-shot)
Object recognition
Food101
101
50,500
30,300
Few-shot (16-shot)
Food recognition
Stanford Cars
[A]
196
6,509
8,041
Few-shot (16-shot)
Cars recognition
FGVC Aircraft
[6%
100
3,334
3,333
Few-shot (16-shot)
Aircraft recognition
UCF101
97
101
7,639
3,783
Few-shot (16-shot)
Action recognition
SUN397 64
397
15,880
19,850
Few-shot (16-shot)
Scene recognition
CIFARIO 10 [26]
10
50,000
10,000
Full dataset
Image classification
CIFAR 100
[6
100
50,000
10,000
Full dataset
Image classification
DomainNet 500
10
18,278
4,573
6
Full dataset
Domain adaptation
|0]
10
2,025
508
4
Full dataset
Domain adaptation
C.2 Experimental Setup
We employ SGD optimizer with learning rate r = 0.001 and single-step learning rate scheduler across all experiments. All implementations are based on PyTorch and experiments were conducted on NVIDIA RTX 4090 (24GB) or A100 (40GB) GPUs. Across all experiments, we use ViT-B/16 pretrained on ImageNet the backbone. Images resized 224 224 using bicubic interpola- tion with standard data augmentation (random resized crop, random flip, and normalization), For FedMGP, we set both text and visual prompt lengths 2, use prompt groups for each modality, and initialize with the text "a photo of a". All models are trained with mixed precision (fp16) for computational efficiency.
The following sections detail the specific configurations for different experimental scenarios.
Base-to-Novel Class Generalization. For the five fine-grained classification datasets, we partition each dataset equally into base and novel classes, then distribute non-overlapping base classes to each of the 10 clients. We employ few-shot (16-shot by default) training paradigm with batch size 8. The federated learning process proceeds for 10 communication rounds with 100% client participation and 2 local epochs per round. Each client trains on their local classes, and we evaluate performance on: (1) local classes (personalization), (2) base classes (classes seen by other clients), and (3) novel classes (unseen during training). The Combined Metric (CM) is computed as CM (Local + HM)/2, where HM is the harmonic mean of Base and Novel accuracies.
Label Distribution Shift. For CIFAR- 10 and CIFAR-100, we partition the full training set among 100 clients following Dirichlet distribution with concentration parameter a = 0.5. Communication proceeds for 100 rounds with 10% client participation per round and 2 local epochs per round. We use batch size for training and 300 for testing. This creates realistic heterogeneous environment with varying class proportions across clients.
25

---

Domain Adaptation. For 1ahice-Cattech0 and DomainNet, we leverage their inherent domain structure (4 domains for Pafice-Caltech0O and 6 domains for DomainNet). Each domain assigned 5 clients, resulting in total of 20 clients for Office-Caltech1 and 30 clients for DomainNet. This setup introduces both feature shift and label skew. The federated learning process runs for 25 rounds with 25% client participation per round and 1 local epoch per round. We evaluate each client's performance on all domains to assess cross-domain generalization.
D Additional Experimental Results
D.1 Domain Generalization for DomainNet
Table 11: Results on DomainNet with feature shift and label shift with Dir(a 0.3) partition into clients/domain
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Average
PromptFL 7
25,80 x2
16.05 10.48
14.72 15.39
14.79 6.29
FedOPT [20]]
25.25,. 10.00
28.07 23.55
=7.00
35.56
28.45
33.64
35.42
FedTPG
17.16
23.5
13.58
16.25
17.13
9.16
16.14
FedPGP
[
12.01 10..21
10.49
23.40
11.39
=7.02
21.77 10.70
14.29
20.50
10.13 112
13.35
PromptFolio
[7
41.80
42.38
29.69
2e.33
34.70
28.99
35.72
35.55 12..25
FedMGP
48.48 47.76 30.36
35.19 11.70
33.02
36.74 47
38.59 12.00
The values in Table I represent the maximum and minimum accuracies among the five clients within each domain under the Dirichlet distribution, illustrating the performance variation mul- tiple clients sharing the same domain characteristics. The domain adaptation experiments on Do- mainNet demonstrate FedMGP's superior performance in handling domain shifts and label distri- bution heterogeneity. As shown in Table III, FedMGP achieves average accuracy of 38.59%, significantly outperforming the closest competitors PromptFolio (35.55%) and FedOPT (35.42%) FedMGP exhibits particularly strong performance domains with high visual abstraction, such as Clipart (48.4 48%) and Infograph (47.76%). substantially outperforming other methods. This demon- strates that multi-group text-visual prompt co-learning mechanism adapt to different visual representations across DomainNer's diverse artistic styles. The performance stability across diverse domains, evidenced by comparatively lower standard deviations, confirms our theoretical analysis that the multi-group architecture effectively decomposes knowledge into common and client-specific components. The visual prompts capture domain-specific artistic fea- tures while text prompts provide cross-domain semantic connections, enabling FedMGP to main- tain both domain adaptability and semantic consistency. This approach effectively addresses the core challenge of domain generalization in federated learning by simultaneously preserving domain- specific knowledge while enabling cross- coos--domiin knowledge transfer across DomainNet's 's six distinct visual domains.
effectively capture and
D.2 Domain Generalization for Office-Caltech10
Table 12: Results on Office-Caltech10 with feature shift and label shift with Dir(a 0.3) partition into 5 clients/domain
Amazon
Caltech
DSLR
Webcam
Average
PromptFL [
16.88 9.23 25.55
144.0
8.33
25.59 226.85
15.01 218.11
FedOPT
[
28.20
23.67 35.22
30.71
29.45
FedTPG [1
6.94
9.10
16.33
28.1
225.78
15.
12
FedPGP [X]
8.50
19.33
11.33
24.89
114.04
16,01
414.49
PromptFolio [37]
32.48 13.33
36.21
20.33
11.59
22.ss
15.15_mm 114.36
FedMGP
31.32
38.28
41.33
44.73
38.92
For the Office-Caltech10 dataset, as shown in Table 12 FedMGP demonstrates even more substan- tial improvements, with average accuracy of 38.92% compared to the second-best performer
26

---

FedOPT (29.45%) Unlike DomainNet's artistic style variations. Office- Cafice-alltech0O presents chal- lenges related to imaging conditions and equipment specifications. The advantage particularly pronounced on specialized equipment captures like DSLR (41 (41.33%) and Webcam (44.73%) where FedMGP outperforms other methods by large margins. These results validate the effectiveness of our approach in handling technical domain shifts beyond artistic variations. Most baseline methods exhibit substantial performance variations across domains, indicating their vulnerability to domain- specific overfitting in squipment-baset scenarios. In contrast, FedMGP maintains more consistent performance, demonstrating its robustness to both artistic and technical domain shifts. This confirms that integrating visual and textual modalities enriches contextual representation, capturing instance- specific information more comprehensively than text-only approaches across different types of do- main variations. The performance on
confirms that FedMGP's multi-group archi- tecture effectively distributes knowledge across specialized prompt units rather than concentrating it in single structure, enabling robust cross-domain generalization while preserving domain-specifer adaptation capabilities for both artistic and technical domain characteristics.
D.3 Stability of Prompt Group Selection
Table 13: Selection frequency of each prompt group across training rounds on OxfordPets dataset.
Values show the number of clients (out of 20 total) selecting each group, with percentages in paren- theses.
Round
gg
g_
t
g4
Lg
v_gl
v_g2
v_g3
v_g4
y_g
3(15%) 2(10%) 6(30%) 5(25%) 4(20%) 3(15%) 2(10%) 6(30%) 5(25%) 4(20%)
2
6(30%) 2(10%) 6(30%) 3(15%) 3(15%) 6(30%) 2(10%)
3(15%) 3(15%)
3
7(35%) 2(10%) 5(25%) 3(15%) 3(15%) 7(35%) 2(10%) 5(25%) 3(15%) 3(15%)
6(30%) 3(15%) 5(25%) 3(15%) 3(15%) 7(35%) 3(15%) 4(20%) 2(10%) 4(20%)
6(30%) 3(15%) 3(15%) 3(15%) 5(25%) 7(35%) 3(15%) 3(15%) 2(10%) 5(25%)
6(30%) 4(20%) 3(15%) 2(10%) 5(25%) 7(35%) 4(20%) 3(15%) 1(5%) 5(25%)
7
7(35%) 4(20%) 4(20%) 1(5%) 4(20%) 7(35%) 4(20%) 4(20%) 1(5%) 4(20%)
6(30%) 4(20%) 4(20%) 2(10%) 4(20%) 7(35%) 4(20%) 5(25%) 1(5%) 3(15%)
9
4(20%) 5(25%) 6(30%) 1(5%) 4(20%) 5(25%) 5(25%) 7(35%) 0(0%) 3(15%)
5(25%) 5(25%) 5(25%) 2(10%) 3(15%) 4(20%) 7(35%) 6(30%) 1(5%) 2(10%)
Table D presents the selection frequency of cach prompt group across training rounds on the Ox- fordPets dataset, demonstrating the stability of prompt group assignments in FedMGP. The results reveal that while prompt groups exhibit dynamic selection patterns, their roles remain relatively sta- ble throughout training. Notably, certain groups consistently receive higher selection frequencies (e.g., text group and visual group 1 maintain 30-35% selection after round 3), indicating their specialization in capturing shared global knowledge that benefits multiple clients. Conversely, other groups show lower but persistent selection rates (e.g., text group 4 ranges from 5-15%), suggesting their focus on client-specific local features. This pattern validates our dynamic aggregation mecha- nism's 's design principle: rather than forcing uniform participation, the similarity-guided probabilistic sampling naturally guides prompt groups to specialize in complementary aspectssome evolving to capture common patterns through frequent selection, while others preserve personalized knowledge through selective aggregation. The temperature parameter 7 in our selection process plays a crucial role maintaining this dynamic balance, preventing any prompt group from being permanently ex- cluded (as evidenced by the absence of consistently zero selections) while still allowing meaningful specialization. This ensures that FedMGP retains both strong generalization capabilities through shared knowledge and effective personalization through client client-specific features, achieving the opti- mal trade-off demonstrated our main experimental results.
E
Additional ablation study
In this section, we present additional ablation studies to further analyze the effectiveness of different components and design choices in FedMGP. These experiments provide deeper insights into the model behavior and validate the design decisions discussed in the main paper.
27

---

E.1
Effect of Temperature Parameter in Prompt Selection
Table 14: Effect of Temperature (r) on FedMGP Performance
Setting
Local
Base
Novel
CM
FedMGP (r=0.1)
84.84
95.92
72.54
83.72
FedMGP (7=0.5)
85.45
96.46
73,47
84.43
FedMGP (7=0.8)
84.40
95.31
72.94
83.52
FedMGP (r=2.0)
85.17
96.63
72.61
84.04
FedMGP (7=1.0)
96.92
73.23
74.65
85.43
The temperature parameter in our dynamic prompt selection strategy plays a critical role in bal- ancing exploration and exploitation during federated learning. As shown in Table 11, the optimal performance is achieved r 1.0 with a Combined Metric (CM) of 85.43%, significantly out- performing both lower temperatures (r = 0.1, 0.5) and higher temperatures (r = 2.0). Lower temperatures lead to more deterministic selection based on prompt similarity, resulting stronger base class performance (96.46% at 0.5) but weaker local personalization. Conversely, higher temperatures introduce more randomness, allowing for greater exploration but potentially disrupting the convergence of shared knowledge. This confirms our theoretical framework in Section 3.2.2
E.2 Inference-Time Prompt Group Weighting Strategies
Table 15:
Effect of Different Inference Strategies on FedMGP Performance
Setting
Local
Base
Novel
CM
FedMGP (Max logits)
81.96 89.48
74.79
81.72
FedMGP (Feature avg)
85.09 95.56
74.32
84.35
FedMGP (Group 0)
79.65
85.80
73.08
79.29
FedMGP (Group 1)
78.19 83.65
72.68
77.99
FedMGP (Group 2)
77.92 83.69 72.66
77.85
FedMGP (Group 3)
80.70 96.94
61.52
77.99
FedMGP (Group 4)
80.52
90.76
69.37
79.58
FedMGP (Average)
96.92 73.23 74.65 85.43
The effectiveness of different inference-time strategies for combining predictions from multiple prompt groups is examined in Table IS Simple logit averaging across all groups yields the best overall performance (MM=85.43%) significantly outperforming alternative strategies such as maxi- mum logit selection (CM=81. 1.72%) and feature-level averaging 3.M....3%% Notably, relying on any single prompt group (groups 0-4) substantially degrades performance, with the best individual group achieving only CM-79.58% This confirms hypothesis presented in Section 3.2.1 that the multi-group architecture enables different prompt groups to specialize in complementary aspects of the input data. The superior performance of ensemble averaging demonstrates that each prompt group contributes unique and valuable semantic perspectives, collectively enhancing model robust- ness. Group 3 exhibits the highest base class accuracy (96.94) (96.94%) but poor novel class performance (61.52%), indicating its specialization in capturing shared patterns across clients rather than gen- eralizable featuresprecisely the type of specialization our diversity loss was designed to encourage. These results validate our core design principle of distributing knowledge across multiple specialized prompt units rather than concentrating in single monolithic structure.
E.3 Diversity Loss Formulation Variants
The choice of diversity loss function significantly impacts FedMGP's ability to learn specialized prompt representations. As shown in Table IIG the L.-based diversity formulation achieves the best overall performance (CM=85.43%) outperforming both cosine similarity (CM=83.9669) and 1.2- based approaches (CM-83.83%) The LI formulation leads to substantially better local accuracy
28

---

Table 16: Effect of Diversity Loss Type FedMGP Performance
Setting
Local
Base
Novel
CM
FedMGP (COS)
84.72
95.68
73.61
83.96
FedMGP (L2)
84.57
94.76
73.98
83.83
FedMGP (L1)
96.92
73.23
74.65
85.43
(96.92%) compared cosine (84.72%) and L2 (84.57%), while maintaining comparable perfor- mance on novel classes. This performance pattern aligns with our analysis in Section [.2.III where we emphasized the importance of encouraging prompt groups to capture diverse semantic perspec- tives. The LI norm's 's sparsity- indduiing property appears to create cleaner separation between prompt groups, allowing each to specialize more effectively in different aspects of the data distribution. Co- sine similarity, while effective at enforcing orthogonality, appears less suited to the federated setting where capturing complementary rather than strictly orthogonal features is beneficial. These results validate our diversity loss design as a key component of FedMGP's architecture, enabling effective knowledge distribution across prompt groups and contributing to the model's strong performance balance between personalization and generalization.
E,4
Dynamic Aggregation Strategy
Table 17: Comparison of Different Aggregation Strategies (averaged over 5 datasets)
Setting
Local
Base
Novel
CM
CAM
86.13
85,94
83.57
85.43
FAM
97.37
77.47
78.71
87.73
DAM
96.65
79.20
80.86
88.34
To validate the effectiveness of our dynamic aggregation mechanism, we compare three aggrega- tion strategies shown in Table I7. Complete Aggregation Mechanism (CAM) aggregates all prompt groups across clients each communication round, resulting identical parameters across clients (similarit==1.0), While this ensures strong base class performance (85 885,94), it sacrifices local personalization (86.13%) by forcing uniform representations. Fixed Aggregation Mechanism (FAM) maintains certain prompt groups without aggregation, achieving the highest local accuracy (97.37%) but severely compromising generalization on base (77.47%) and novel classes (78.71%) due to insufficient cross-client knowledge transfer.
Our Dynamic Aggregation Mechanism (DAM) strikes an optimal balance, achieving the best Com- bined Metric (88.34 (88.34%) by selectively aggregating the most similar prompt groups between clients each round. This similarity-guided probabilistic sampling reduces the weight of client-specific biased features while preserving personalization. The temperature parameter ensures every prompt group has opportunities for aggregation, enabling FedMGP to learn parameters with high inter-client similarity (promoting generalization) while maintaining diversity within each client's 's prompt groups (enabling personalization). This explains why FedMGP achieves strong local accuracy (96.65% comparable to FAM while maintaining substantially better performance on base (79.20%) and novel classes (80.86%) than FAM, demonstrating superior generalization capability through dynamic cross-client knowledge transfer.
F Theoretical Analysis
In this section, we present comprehensive theoretical analysis that establishes the formal guaran- tees for FedMGP's effectiveness heterogeneous federated learning environments. We demonstrate that our dynamic aggregation strategy consistently outperforms both full aggregation (represented by PromptFL. (IZZ] and fixed aggregation (represented by FedOTP (EI) approaches, particularly un- der IDD data distributions. We begin by establishing the foundational assumptions and notations
29

---

that frame our analysis (Section E.D. including how prompts can be decomposed into global, local, and noise components. We then formalize the three competing aggregation strategies (Section EiJ. highlighting their distinct characteristics in balancing global knowledge with client client-specific features. Next, we introduce a signal-noise decomposition framework (Section (E3) that enables quantitative comparison between different strategies through their signal-to-noise ratios. Finally, we present and prove our main theoretical result (Section [e)); the dynamic aggregation superiority theorem, which establishes that FedMGP's approach achieves strictly better signal-to-noise ratios than alternative strategies, directly translating to improved classification performance practice.
F.1 Assumptions and Notation Delinitions
To establish rigorous theoretical framework, we first define the notation and key assumptions that underpin our analysis. The notation used in this section and their meanings are as follows:
N: Total number of clients;
G: Number of prompt groups per client;
Number of prompt groups selected for aggregation in each round, where 1 S 8 S G;
Cr S (1...,,N) Set of clients participating in aggregation at round T, with n = |Crl:
c Cr: Client index;
j e {1...., G}:
Prompt group index;
J.c E Rd. The j-th prompt group of client round T;
pT E ER.: The j-th global prompt group at the server after round T'' aggregation:
Sh S (I...., Set of prompt group indices selected from client in round T for aggregation, with |
affc: Selection score for the j-th group from client round T, computed based on similarity to global prompts;
T > 0: Temperature parameter controlling selection score smoothness;
sim(2, y): Similarity measure (e.g., cosine similarity);
cc e RR:: Unit vector representing global task-related features shared across all clients;
Me E Idd Unit vector representing local task-related features specific to client c;
L: Total number of noise feature dimensions in the latent space;
& E R4: The I-th unit vector representing task-irrelevant noise features;
ST. E R: Coefficient quantifying the contribution of global features prompt PT
Thc E R: Coefficient quantifying the contribution of client-specific features to prompt PT
j,cr
oT
j.e.d R: Coefficient quantifying the contribution of the 1-th noise feature prompt PT
Xa TR: Metric quantifying the degree of data heterogeneity for client
Our analysis based on the following assumptions, which are grounded in feature learning theory and previous work on federated learning:
Assumption (Feature Space Decomposition). According to feature learning theory [37 0 the latent feature space can be decomposed into three orthogonal subspaces:
1. Global task-related features represented by a unit vector S (shared across all clients)
2. Local task-related features represented by unit vectors {ue] (client-spp sseeefft-peecfic)
3. Task-irrelevant noise features represented by unit vectors (Ei (6))/=, (noise)
These three subspaces are mutually orthogonal, i.e., (fe".pe) 0, (u.. O, and (MerEi) O
for all E [I.....N] N} and l (1....,L), Here, L represents the dimensionality of the noise subspace, which can be significantly larger than the dimensionality of task-relevant subspuces. This decomposition allows us to separately analyze the impact of each component on the aggregation process and quantify the information content in prompts.
30

---

Assumption 2 (Prompt Representation), Each prompt group j of client at round T can be repre- sented as a linear combination of features from the three orthogonal subspaces:
3/c = 53 + t + Σ GapE
(10)
where:
37, represents the coefficient for global features, indicating how much the prompt captures knowledge shared across all clients
The represents the coefficient for local features, indicating how much the prompt captures client- client-pecciicc knowledge
I.e. represents the coefficient for the 1-th noise feature dimension
Since y² Hel and & are unit vectors defined in Assumption M. this representation directly quan- tifies the strength of each component the prompt. This allows us to analyze how each prompt captures common knowledge versus client-specifer knowledge versus irrelevant noise [I8, 23
Assumption (Data Heterogeneity). The degree of data heterogeneity between clients is defined by the metric:
Xc Σ Pes Pc
(11)
This simplification valid because Pe is unit vector, so || =. This metric measures how similar client c's local features are to those of other clients. When Xe approaches N, it indicates that client c's features are highly aligned with other clients, suggesting an IID (Independent and Identically Distributed) data scenario. Conversely, when Xe is close to 1 (its minimum value, rep- resenting alignment only with üseeff) indicates that client c's features have limited overlap with other clients, suggesting highly non-IID data distribution [3A [2]]. This metric allows to relate the performance of different aggregation strategies to the level of data heterogeneity and provides a quantitative basis for analyzing the effectiveness of our approach in various federation settings.
F.2 Formalization Three Aggregation Strategies
We now formally define three different prompt aggregation strategies that represent the spectrum of approaches in federated prompt learning. Each strategy has distinct characteristics in how it handles the balance between preserving global knowledge and managing client-specific variations.
Full Aggregation (PromptFL) The full aggregation strategy, employed in PromptFL [M ag- gregates all prompt groups from all participating clients. This represents the most straightforward application of federated averaging [34]]] to prompt learning:
PJ - PT
(12)
Je
RECT
While this approach maximizes knowledge sharing, it may suffer from interference between client- specific features when data distributions are heterogeneous.
Fixed Aggregation (FedOTP) The fixed aggregation strategy, inspired by approaches like Fe- dOTP [H]], only aggregates a predetermined subset of prompt groups (typically the first groups), setting all others to zero:
1 EeEr het
PT
j = 1.
P
(13)
0
j == + G.
This static partition-based approach attempts to balance shared knowledge with client specificity, but lacks adaptivity evolving knowledge patterns across communication rounds.
31

---

Dynamic Aggregation (FedMGP) Our proposed dynamic aggregation strategy selects prompt groups based on their similarity to the global prompts from the previous round. First, it computes selection scores:
exp(sim(( (//) j.ct
T
(14)
Σjz=1 exp (spps PJ yeer )/T)
where P5-1 is the j-th global prompt from the previous round. Then, for each client e, select the top-s groups with the highest selection scores to form S? and aggregate only the selected groups:
1 Σ I(ee)) P. ---
(15)
n CECr
where 1(j E 8t is the indicator function denoting whether group j is selected from client c in round T. This adaptive approach balances knowledge sharing and client specificity in data-driven manner, potentially offering advantages over fixed strategies.
F.3 Signal-Noise Decomposition and Performance Metries
To analyze the effectiveness of different aggregation strategies, introduce signal-noise decom- position framework that allows us to quantitatively compare their performance. This approach en- ables us examine how effectively each strategy preserves important information while suppressing noise.
From Assumption A we have the representation of each prompt as:
PT
B + 1.
Σ Te E
(16)
For individual prompts, we can define their total signal and noise components. The signal compo- nents include both global and local task-related information:
Signal (8) 2 + (nf.e)
(17)
while the noise component represents the irrelevant information:
Noisej.c
(18)
However, when evaluating aggregated global prompts in federated learning, we are primarily in- terested in how well they preserve global knowledge. From this perspective, even client-specific features Tee can be considered as interference when aggregated across heterogeneous clients. Therefore, for evaluating global prompts, we define:
Global Signal"' =89)22
(19)
Global Noise (Client-specific noise) + (Task-irrelevant noise)
(20)
The key performance metric we use evaluate the quality of aggregated prompts the signal-to- noise ratio (SNR):
Global Signal?
(3 o7
SNR, Global Noise T
(21)
where 37 is the coefficient of the global feature f in the aggregated prompt PT. and oJ quantifies the total noise power including both client- seent--pecific variations and task- irrelevant noise.
This metric is directly related to the generalization performance of the model: a higher SNR indicates better preservation of global features and more effective suppression of noise, which translates to improved classification performance and lower test error G [8]
32

---

F44 Dynamic Aggregation Superiority Theorem and Detailed Proof
We now present our main theoretical result, which establishes the superiority of FedMGP's dynamic aggregation strategy over both full aggregation and fixed aggregation strategies. Based on equa- tion (2II), higher signal-to-noise ratio leads lower classification error. The following theorem and proof demonstrate that:
SNR SNRixed SNRd
Theorem F.1 (Dynamic Aggregation Superiority). Under Assumptions M of selected prompt groups E G1, we have:
and G for any number
SNRmn
SNRaxxd
<
SNRdyn
Proof. The proof consists of three parts: first analyzing the SNR of full aggregation, then comparing with fixed aggregation, and finally establishing the superiority of dynamic aggregation.
(1) Analysis of Full Aggregation SNRfall- From equation (l22) and decomposition (IG), we can express the global and noise coefficients for the full aggregation strategy:
Σ CEC
n
For the noise term, we must consider both the pure noise components 07,0, and the client-specific features 7/.0/ which act interference when aggregated across heterogeneous clients. The total noise power after aggregation is:
full
1 + nt
Σ (
n2 CECr
CEC
The first term represents the traditional noise components, while the second term accounts for client- specific features that do not align globally. This is a more complete characterization of noise in federated settings.
For the signal-to-noiss ratio of group j. we have:
(3fullyz
E.ec, B3,0) NO E.Ec, (o no Eoec,
SNRFuu(F)
full
key observation that full aggregation can actually enhance SNR through constructive signal accumulation. When client signals are positively correlated (as is typically the case for global knowledge), the numerator grows quadratically with n, while the noise terms in the denominator grow linearly if they are uncorrelated across clients. This is the fundamental principle behind why federated learning works.
The overall SNR of full aggregation determined by the worst-performing group:
SNR min SNRaau JSG
(2) Analysis of Fixed Aggregation SNRaxed- From equation for j < (groups that aggregated), we have:
gfoxed = afulll
=
fixed
therefore SNR (j) = ()) for these groups.
For > (groups that are not aggregated), we have PT according to equation (L), which means these groups do not contribute to the model's 's predictions. We exclude these groups from the SNR calculation since they do not affect model performance.
The overall SNR of fixed aggregation determined by the worst-performing group among the ag- gregated ones:
SNRaxed = min SNR (j) 2 min (NRRan()) = SNR
JSG
33

---

This shows that fixed aggregation guarantees SNR at least as good as full aggregation, since it excludes potentially noisy groups that might degrade the overall performance. The inequality is strict when at least one group j has a lower SNR than all groups j S
(3) Proving SNRdyn SNR For dynamic aggregation, select prompt groups based their similarity to the global prompts from the previous round, as defined in equation (IE)) To be precise about our selection mechanism: for each client c, we compute similarity scores between each of its prompt groups and the corresponding global prompts, then select the top-s groups with highest similarity. This deterministic selection can be expressed as:
5e = Li
G} af.e is among the top-s highest for client c}
The selection score a3 in equation (LE) serves as a normalized measure of similarity between local and global prompts, with lower temperature r making the scores more concentrated on the highest similarities.
Let us define the global and noise coefficients for dynamic aggregation:
Σ
I(j St )83
cECT
iije SO) dyn Σ
E + (1T.0)
CЕCт
The key insight is that our selection mechanism preferentially selects groups with higher global signal S7. and lower noise (both of.cd, and 75.0). This is because groups with higher similarity to global prompts tend have higher global signal components and lower noise components.
Let us define Nj = =E I[j E ST) as the number of clients that select group j for aggregation. This value depends the "popularity" of group j across clients. For particular group j:
j represents important global knowledge (high 3 across clients), then many clients will select it, resulting in large N,. 2. If captures primarily client-specific knowledge noise, fewer clients will select it, resulting in a small Nj.
For groups that are selected by at least one client (i.e., N2 > 0), we can rewrite:
gdyn
N
1
Σ
N₂
3od
TI
Nj
J.c
CECroEST
dyn
N n
1
Σ
Eiof (0 + (3,,0)2
N see
n² Øj
N
cECr:jesT
where 3 the average STe among clients that selected group j. and Øj sel power among clients that selected group j.
is the average noise
The key our dynamic selection advantage is that 33 B, and o, < o where B, and o, are the averages across all clients. This is because our selection mechanism favors high-signal, low-noise prompt groups.
For quantitative analysis, we can use a parameter 63 > 1 to capture this selection advantage:
3
mel N 83 B, and
Φ I _ -,
ael
0j
where 3, = EcEcr 37. = 8fulll and ,, = 1 Execr
(Etar0 +(TTo)) -n.g &full
This leads to:
34

---

gdyn 2
N, dj
85
full )
71
Nj
N
dyn
full
n
65
The SNR for dynamically aggregated group j is therefore:
sfulll 2
(2 dyn
N2 Ny
(5full) Ny.
SNRdyn(J)
SNNRau(
full
of
For groups S that would be selected by the fixed strategy, have SNR (since SNRaxed = minzg. (NR(an(k)). Therefore:
SNRrxod
N₂ 83
N₂ 6 SNRaxed
SNRdyn(j)
SNR
n
n
Under reasonable assumptions about our selection mechanism, we can establish that for important groups (those containing significant global knowledge);
1. N2 will be high, many clients will select these groups 2. 6y will be significantly greater than 1, as the selection process effectively identifies high-signal, low-noise components
N5 > even when N, < n, because the cubic term 83
For these important groups, the factor provides powerful amplification of the selection advantage. This is particularly true for groups that represent core global knowledge, which will have the highest ô, values.
Taking the minimum over all selected groups, we have:
SNRayn min SNRayn()) > SNRaxed J:N,20
This inequality is strict for the following reason: Our dynamic selection mechanism ensures that each client selects its best groups in terms of similarity to global knowledge. This means that the dynamic strategy will: 1, Select any globally important groups that the fixed strategy would select 2. Replace any poor-quality groups that the fixed strategy would select with better alternatives 3. Achieve higher 5, values for the selected groups through its adaptive selection process
In the extreme case where fixed selection is optimal, dynamic selection would converge to the same selection pattern, matching its performance. However, in practice, especially with heterogeneous data, dynamic selection will identify better groups than a predetermined fixed selection, leading to strictly better performance.
In conclusion. SNRtul & SNRaxed SNRayns establishing that FedMGP's dynamic aggregation strategy is strictly superior to both full aggregation and fixed aggregation strategies in terms of signal- to-noise ratio. This theoretical advantage directly translates to improved classification performance and lower test error in practical applications, particularly under heterogeneous data distributions.
35


Reviews:
[{"review_id": "LvDN8K0wOo", "reviewer": "Reviewer_8EoJ", "summary": "This paper presents FedMGP, a personalized federated prompt learning framework for VLMs. Their proposed approach is designed to involve both viusal and text prompt, which is different from single textual prompts and static local-global aggregation. In the proposed system, multi-group text-visual prompt co-learning at each client to capture diverse semantic and instance-specific features, where a diversity loss to encourage representation separation across prompt groups. A similarity based dynamic prompt aggregation strategy is proposed to balance personalization and generalization. The authors compared to PromptFL, FedOPT, FedTPG, FedPGP and PromptFolio across datasets including CIFAR, Flowers, DTD, Caltech101, etc under heterogeneity settings.", "questions": "Questions:\n1. In many group-based FL algorithms, group membership tends to drift across rounds. Could the authors clarify whether prompt group assignments remain stable across training rounds? Is there any tracking or visualization of how group semantics evolve over time?\n2. Fairness and bias: Do the multi-group prompts risk encoding biased semantic clusters, especially if clients have class-imbalanced datasets?\n3. The diversity loss encourages group-wise differentiation, but it remains unclear what kinds of semantics each group learns. Could the authors provide qualitative examples or visualizations?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The paper proposed multi-group text-visual prompt design, enabling richer client-specific representation compared to single-text prompt baselines. The softmax-based prompt group selection balances global knowledge sharing and local personalization under data heterogeneity\n- The authors evaluated across various datasets under non-IID settings; consistently achieves state-of-the-art results in both personalization and generalization.  Validates the impact of prompt length, group count, aggregation strategy, modality contributions, and diversity loss.\nWeakness:\n- The paper does not report error bars or statistical significance tests, which limits the strength of its claims regarding performance robustness under data variability.\n- The paper claims that different prompt groups specialize in distinct semantics via diversity loss, but provides less interpretability and explainability.\n- Although the method is motivated by privacy-sensitive federated settings, the paper does not assess whether the shared prompt vectors, especially visual prompts could leak private information or be vulnerable to reconstruction attacks.", "was_revised": true, "final_justification": "Final justification", "rebuttal": "**Comment:**\n\nWe appreciate your valuable comments. We were wondering if our responses have addressed your concerns. Please let us know if you have additional questions. Thank you!\n\n---\n\n**Q1:The paper does not report error bars or statistical significance tests, which limits the strength of its claims regarding performance robustness under data variability.The paper does not report error bars or statistical significance tests, which limits the strength of its claims regarding performance robustness under data variability.**\n\n**A1**: To rigorously evaluate performance robustness, we have conducted main experiments across the five datasets using **three distinct random seeds (0, 1, 2)** . The updated table presents the results, confirming that FedMGP not only achieves the **highest performance** but also demonstrates **superior stability** with the lowest standard deviation. Crucially, this leading performance and robustness are achieved with the **lowest communication overhead (2.6k parameters)** among all methods, highlighting the exceptional efficiency and reliability of our approach.\n\n| **Method** | **OxfordPets** | **Flowers102** | **DTD** | **Caltech101** | **Food101** |\n|-------------|----------------|----------------|--------------|----------------|--------------|\n| PromptFL    | 94.39±1.45     | 80.74±0.09     | 64.53±1.98   | 95.25±1.13     | 88.91±0.47   |\n| FedOTP      | 63.83±2.38     | 60.03±0.66     | 62.11±2.50   | 73.59±4.31     | 59.68±1.69   |\n| FedTPG      | 95.22±1.03     | 79.12±1.74     | 61.21±1.68   | 96.11±0.41     | 90.61±0.22   |\n| FedPGP      | 95.77±0.45     | 82.63±0.76     | 67.42±2.47   | 96.18±0.29     | 90.45±0.25   |\n| PromptFolio | 77.58±8.54     | 70.11±1.71     | 62.17±1.60   | 89.19±0.45     | 79.82±1.32   |\n| **FedMGP** | **95.99±0.11** | **84.87±0.96** | **72.42±0.50** | **96.98±0.21** | **91.03±0.74** |\n\n---\n\n**Q2: The paper claims that different prompt groups specialize in distinct semantics via diversity loss, but provides less interpretability and explainability.**\n\n**A2:**  Thank you for this suggestion. We fully recognize the importance of interpretability and have addressed it in our paper as follows:\n\n- As shown in Figure 1 of the appendix, our diversity loss effectively enforces distinct representations. The **remarkably low inter-group correlation** among visual prompts provides direct evidence that each group learns to capture unique and complementary visual features.\n- In appendix F, **signal-noise decomposition theory** mathematically proves that the multi-group mechanism enhances the model's **signal-to-noise ratio** by promoting representational diversity . This framework provides a rigorous foundation for why our design choice is critical for balancing personalization and generalization.\n\n---\n\n**Q3: Although the method is motivated by privacy-sensitive federated settings, the paper does not assess whether the shared prompt vectors, especially visual prompts could leak private information or be vulnerable to reconstruction attacks.**\n\n**A3**: We appreciate the reviewer’s concern. Our framework adheres to the **FedAvg paradigm[1]**, which has been extensively proven to offer strong privacy protection because:\n\n- Our method exchanges only updated text and visual prompt parameters, **without sharing gradients, prediction logits, or data embeddings**, eliminating the key information required for model inversion, and preventing adversaries from obtaining labels or embeddings to build shadow models.\n- With only approximately **2.6K parameters** per round, PromptFL[2] has shown that low-parameter communication is more privacy-preserving than full model uploads; this also drastically reduces the attack surface and minimizes the risk of membership inference and attribute inference.\n- Furthermore, our method leverages diversity loss and  dynamic selection to prevent overfitting of single prompts and enhance protection of privacy-sensitive features.\n\n[1] McMahan, Brendan, et al. \"Communication-efficient learning of deep networks from decentralized data.\" *Artificial intelligence and statistics*. PMLR, 2017.\n\n[2]Guo, Tao, et al. \"Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model.\" *IEEE Transactions on Mobile Computing* 23.5 (2023): 5179-5194.\n\n---\n\n**Q4:In many group-based FL algorithms, group membership tends to drift across rounds. Could the authors clarify whether prompt group assignments remain stable across training rounds? Is there any tracking or visualization of how group semantics evolve over time?**\n\n**A4**: Thank you for your suggestion! Our prompt group assignments remain highly stable throughout training.  We quantitatively demonstrate this by **tracking each group’s selection frequency per round.**\n\n- As shown in the table below, which demonstrates the selection frequency of each group during training on the OxfordPets dataset. Our **dynamic aggregation mechanism** is designed to guide the specialization of prompts; some evolve to capture **shared global knowledge** by being selected frequently, while others focus on **client-specific local features** through less frequent selection.\n- The **temperature parameter (τ)** in our selection process is crucial for maintaining this dynamic balance, preventing any prompt group from being permanently excluded and thereby ensuring the model retains both **generalization and personalization** capabilities.\n| round | t_g1 | t_g2 | t_g3 | t_g4 | t_g5 | v_g1 | v_g2 | v_g3 | v_g4 | v_g5 |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 1 | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) |\n| 2 | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) |\n| 3 | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) |\n| 4 | 6(30%) | 3(15%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 3(15%) | 4(20%) | 2(10%) | 4(20%) |\n| 5 | 6(30%) | 3(15%) | 3(15%) | 3(15%) | 5(25%) | 7(35%) | 3(15%) | 3(15%) | 2(10%) | 5(25%) |\n| 6 | 6(30%) | 4(20%) | 3(15%) | 2(10%) | 5(25%) | 7(35%) | 4(20%) | 3(15%) | 1(5%) | 5(25%) |\n| 7 | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) |\n| 8 | 6(30%) | 4(20%) | 4(20%) | 2(10%) | 4(20%) | 7(35%) | 4(20%) | 5(25%) | 1(5%) | 3(15%) |\n| 9 | 4(20%) | 5(25%) | 6(30%) | 1(5%) | 4(20%) | 5(25%) | 5(25%) | 7(35%) | 0(0%) | 3(15%) |\n| 10 | 5(25%) | 5(25%) | 5(25%) | 2(10%) | 3(15%) | 4(20%) | 7(35%) | 6(30%) | 1(5%) | 2(10%) |\n\n---\n\n**Q5:Fairness and bias: Do the multi-group prompts risk encoding biased semantic clusters, especially if clients have class-imbalanced datasets?**\n\n**A5**: Thank you for your valuable Question. **Our FedMGP method demonstrates superior generalization and carries no risk of encoding biased semantic clusters.** In the main paper (Table 2, Dirichlet α=0.5), we show exceptional performance under severe class imbalance . In the cross-domain experiments in Appendix D, we apply an even stricter Dirichlet distribution (α=0.3) on each domain; **even under simultaneous feature and label shifts, FedMGP maintains robust performance**, directly validating the effectiveness of our dynamic aggregation mechanism.\n\n---\n\n**Q6：The diversity loss encourages group-wise differentiation, but it remains unclear what kinds of semantics each group learns. Could the authors provide qualitative examples or visualizations?**  \n\n**A6**: Thank you for the suggestion. As demonstrated in the Q2 experiments, **prompt groups that are selected more frequently during training tend to aggregate broader, global semantics, while those with lower selection frequencies preserve more localized, client-specific semantics.** These complementary behaviors balance generalization and personalization. Although we cannot include visualizations in this submission, we will add this discussion with qualitative examples and visualizations in the final version."}, {"review_id": "fhRZXTbRmq", "reviewer": "Reviewer_t5MR", "summary": "This paper proposes FedMGP, a personalized federated learning framework based on multi-group text-visual prompts, designed to address client overfitting and aggregation instability in vision-language models (VLMs) under heterogeneous data. The core contributions include:\n1.Multi-group text-visual prompt pairs: each client maintains multiple paired text and visual prompts to enhance fine-grained semantic representation.\n2.Diversity loss enforces complementary feature learning across different prompt groups.\n3.Dynamic prompt aggregation utilizes similarity-based probabilistic sampling to balance global shared knowledge and client-specific information.", "questions": "Please refer to the weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The proposed method effectively addresses the expressiveness limitations of unimodal prompts. The dynamic aggregation strategy (similarity-weighted sampling) provides a compelling solution to the classic trade-off between personalization and generalization.\n2. The proposed method is comprehensively evaluated across five datasets, Dirichlet-partitioned heterogeneous settings (CIFAR-10/100), and multi-domain scenarios (DomainNet), demonstrating strong generalizability.\n3.FedMGP significantly reduces communication overhead compared to baselines, making it well-suited for resource-constrained federated environments.\n\nWeaknesses:\n1. The use of a fixed number of prompt groups (e.g., G=5) across all clients fails to consider data heterogeneity. Clients with simpler data (e.g., only 1–2 classes) may require fewer groups, while more complex clients (e.g., multi-domain or long-tail data) may benefit from more.\n2.  The dynamic aggregation process relies on the cosine similarity between local prompts and the global prompts from the previous round. In the extreme case where a local prompt directly adopts the previous global prompt, the similarity reaches its maximum, resulting in the highest selection probability. However, this similarity-based sampling strategy may not be optimal.\n3. At inference time, the model averages predictions across all prompt groups equally, ignoring confidence differences between groups. This uniform treatment could reduce prediction reliability, especially when some groups are more semantically aligned with the input than others.", "was_revised": true, "final_justification": "The response to my questions and discussion addressed my questions. I will raise my score.", "rebuttal": "**Comment:**\n\nThanks a lot for your time and feedback. We have to say that the reviewer asks valuable questions and provides thoughtful clues. We appreciate your inspiring reviews. And we are happy to address the concerns.\n\n---\n\n**Q1: The use of a fixed number of prompt groups (e.g., G=5) across all clients fails to consider data heterogeneity. Clients with simpler data (e.g., only 1–2 classes) may require fewer groups, while more complex clients (e.g., multi-domain or long-tail data) may benefit from more.**\n\n**A1:** Thank you for this insightful question. \n\n- The referred setting corresponds to the Dirichlet distribution experiments in **Table 2 and Appendix D (α=0.5 / α=0.3)**, a widely adopted benchmark for measuring data heterogeneity[1][2]. Even under severe class imbalance, FedMGP delivers outstanding performance, demonstrating the robustness and generalization of our dynamic aggregation mechanism across diverse client distributions. \n- Moreover, according to your valuable suggestion. We have supplemented **an ablation study on the number of prompt groups (G) under this imbalanced setting**. The results are as follows. These findings show that G=5 strikes the optima**l balance between performance gains and parameter efficiency**, further enabled by our dynamic selection strategy that **adapts to each client’s data distribution**. Overall, FedMGP exhibits high efficiency, strong robustness, and superior generalization in handling data imbalance and heterogeneity.\n\n| **Group** | **cifar10** | **cifar100** |\n| --------- | ----------- | ------------ |\n| 2         | 93.36       | 72.27        |\n| 3         | 94.29       | 73.62        |\n| 4         | 94.35       | 74.57        |\n| 5         | **95.48**   | **75.39**    |\n\n[1] Li, Hongxia, et al. \"Global and local prompts cooperation via optimal transport for federated learning.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[2] Cui, Tianyu, et al. \"Harmonizing Generalization and Personalization in Federated Prompt Learning.\" International Conference on Machine Learning. PMLR, 2024.\n\n---\n\n**Q2：The dynamic aggregation process relies on the cosine similarity between local prompts and the global prompts from the previous round. In the extreme case where a local prompt directly adopts the previous global prompt, the similarity reaches its maximum, resulting in the highest selection probability. However, this similarity-based sampling strategy may not be optimal.**\n\n**A2:** Thank you for you question. Our Method  have considered this scenario. \n\n- Our **dynamic aggregation** strategy incorporates a **temperature parameter (τ)** , which critically controls the selection probability distribution. As detailed in our ablation study in **Appendix E.1** , this parameter allows us to fine-tune the balance between **exploitation** (deterministically selecting high-similarity prompts with a low τ) and **exploration** (enabling the selection of diverse, lower-similarity prompts with a higher τ).\n- This **probabilistic sampling** , governed by the temperature, ensures a diverse set of prompts is aggregated in each round. **As demonstrated in the selection statistics table below** , even prompt groups with lower initial similarity are consistently given a chance to be selected across different communication rounds, effectively preventing premature convergence to a homogenized global prompt and fostering the continuous discovery of **diverse semantic features** .\n- Ultimately, this mechanism creates an intelligent trade-off inherent to personalized federated learning. Prompts frequently selected across clients tend to encapsulate **generalizable knowledge** , strengthening the core model, while less frequently selected prompts often hold unique, **client-specific features** crucial for **personalization** . Our method dynamically balances these two competing objectives, achieving a robust equilibrium between cross-client generalization and local adaptation.\n| round | t_g1 | t_g2 | t_g3 | t_g4 | t_g5 | v_g1 | v_g2 | v_g3 | v_g4 | v_g5 |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 1 | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) |\n| 2 | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) |\n| 3 | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) |\n| 4 | 6(30%) | 3(15%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 3(15%) | 4(20%) | 2(10%) | 4(20%) |\n| 5 | 6(30%) | 3(15%) | 3(15%) | 3(15%) | 5(25%) | 7(35%) | 3(15%) | 3(15%) | 2(10%) | 5(25%) |\n| 6 | 6(30%) | 4(20%) | 3(15%) | 2(10%) | 5(25%) | 7(35%) | 4(20%) | 3(15%) | 1(5%) | 5(25%) |\n| 7 | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) |\n| 8 | 6(30%) | 4(20%) | 4(20%) | 2(10%) | 4(20%) | 7(35%) | 4(20%) | 5(25%) | 1(5%) | 3(15%) |\n| 9 | 4(20%) | 5(25%) | 6(30%) | 1(5%) | 4(20%) | 5(25%) | 5(25%) | 7(35%) | 0(0%) | 3(15%) |\n| 10 | 5(25%) | 5(25%) | 5(25%) | 2(10%) | 3(15%) | 4(20%) | 7(35%) | 6(30%) | 1(5%) | 2(10%) |\n\n---\n\n**Q3: At inference time, the model averages predictions across all prompt groups equally, ignoring confidence differences between groups. This uniform treatment could reduce prediction reliability, especially when some groups are more semantically aligned with the input than others.**\n\n**A3:** We appreciate your valuable comments. We wish to clarify that our inference strategy directly incorporates the confidence of each group, as you suggested. For each prompt group, the **final logits** , computed from the similarity between its guided text and image features, directly **represent that group's confidence distribution over the classes for a given input**. Therefore, averaging these logits is a principled way to ensemble the predictions, where each group's contribution is naturally weighted by its confidence. As rigorously demonstrated in our **ablation study in Table 6** , this method significantly outperforms alternatives like selecting the single most confident group (Max logits) or averaging the prompt-guided features. This result powerfully validates our integrated design: the multi-group architecture enables diverse specializations—some capturing broad, generalizable knowledge while others master **client-specific, personalized nuances** . Our inference strategy then optimally synthesizes these varied perspectives, allowing the model to draw upon shared knowledge for general tasks while accurately reflecting personal characteristics when needed, thus achieving a superior balance between **generalization and personalization** ."}, {"review_id": "dF43kdITHT", "reviewer": "Reviewer_XV8h", "summary": "This paper introduces FedMGP, a novel personalized federated learning framework for Vision-Language Models (VLMs) that aims to address the limitations of existing federated prompt learning (FPL) methods, particularly client-specific overfitting and unstable aggregation under heterogeneous data distributions.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe concept of using multi-group text-visual prompts is an interesting direction for personalized federated learning in VLMs\n\n2.\tThe paper claims competitive performance across various heterogeneous data settings, including non-IID and Dirichlet distributions, and domain generalization.\n\nLimitations \n\n1.\tLack discussion on related group prompt based federated learning, [1] is crucial as it not only proposes learning group-specific prompts to capture heterogeneous client features but also provides a theoretical understanding of the effectiveness of such group-based methods. A thorough discussion against such foundational work is essential.\n\n2.\tThe proposed dynamic prompt aggregation strategy, which relies on similarity-guided probabilistic sampling, introduces a potential for imbalance in the global aggregation process. The paper does not adequately address how it resolves the critical issue where certain client-side prompt groups might be consistently less similar to the global prompts, leading to them being \"hardly selected\" for aggregation. \n\n3.\tEquation 9 appears to imply a simple averaging of prompts from different clients. This raises a concern regarding potential bias in the global model. If some clients possess substantially more training data than others, a straightforward averaging scheme could disproportionately weigh the contributions of larger clients.\n\n4.\tSince only select prompts most similar to global prompts for aggregation, will this lead to a fixed prompts being aggregated while others always keep local? How is the frequency that each group prompt being aggregated globally?\n\n[1] Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning", "was_revised": true, "final_justification": "The author's response resolved my concern, I will maintain my positive score", "rebuttal": "**Comment:**\n\nThank you for your valuable feedback! Below are address all raised concerns of the paper.\n\n---\n\n**Q1：Lack discussion on related group prompt based federated learning, [1] is crucial as it not only proposes learning group-specific prompts to capture heterogeneous client features but also provides a theoretical understanding of the effectiveness of such group-based methods. A thorough discussion against such foundational work is essential.**\n\n**A1:** Thank you for the insightful suggestion! We will include **a detailed comparison with [1]** in the final paper. Both methods leverage the group-based mechanism to **balance global generalization and local personalization**. However, the key distinctions are:\n\n- SGPT[1] maintains **a global prompt group pool** on the server for different data domains; FedMGP keeps multiple prompt groups **locally on each client** to enhance diversity .\n- In Inference time，SGPT requires explicit selection of a prompt group **based on input features**; FedMGP employs  **dynamic aggregation**, performing  **average of group confidences** at inference without additional selection .\n- SGPT incurs approximately **100k** parameters for communication and training; FedMGP reduces this to **12.8k** training parameters and **2.6k** communication parameters\n\n[1] Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning.\n\n---\n\n**Q2：The proposed dynamic prompt aggregation strategy, which relies on similarity-guided probabilistic sampling, introduces a potential for imbalance in the global aggregation process. The paper does not adequately address how it resolves the critical issue where certain client-side prompt groups might be consistently less similar to the global prompts, leading to them being \"hardly selected\" for aggregation.**\n\n**A2**: Thank you for you question. Our Method  have considered this scenario. \n\n- Our **dynamic aggregation** strategy incorporates a **temperature parameter (τ)** , which critically controls the selection probability distribution. As detailed in our ablation study in **Appendix E.1** , this parameter allows us to fine-tune the balance between **exploitation** (deterministically selecting high-similarity prompts with a low τ) and **exploration** (enabling the selection of diverse, lower-similarity prompts with a higher τ).\n- This **probabilistic sampling** , governed by the temperature, ensures a diverse set of prompts is aggregated in each round. **As demonstrated in the selection statistics table below** , even prompt groups with lower initial similarity are consistently given a chance to be selected across different communication rounds, effectively preventing premature convergence to a homogenized global prompt and fostering the continuous discovery of **diverse semantic features** .\n- Ultimately, this mechanism creates an intelligent trade-off inherent to personalized federated learning. Prompts frequently selected across clients tend to encapsulate **generalizable knowledge** , strengthening the core model, while less frequently selected prompts often hold unique, **client-specific features** crucial for **personalization** . Our method dynamically balances these two competing objectives, achieving a robust equilibrium between cross-client generalization and local adaptation.\n\n| round | t_g1 | t_g2 | t_g3 | t_g4 | t_g5 | v_g1 | v_g2 | v_g3 | v_g4 | v_g5 |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 1 | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) | 3(15%) | 2(10%) | 6(30%) | 5(25%) | 4(20%) |\n| 2 | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) | 6(30%) | 2(10%) | 6(30%) | 3(15%) | 3(15%) |\n| 3 | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 2(10%) | 5(25%) | 3(15%) | 3(15%) |\n| 4 | 6(30%) | 3(15%) | 5(25%) | 3(15%) | 3(15%) | 7(35%) | 3(15%) | 4(20%) | 2(10%) | 4(20%) |\n| 5 | 6(30%) | 3(15%) | 3(15%) | 3(15%) | 5(25%) | 7(35%) | 3(15%) | 3(15%) | 2(10%) | 5(25%) |\n| 6 | 6(30%) | 4(20%) | 3(15%) | 2(10%) | 5(25%) | 7(35%) | 4(20%) | 3(15%) | 1(5%) | 5(25%) |\n| 7 | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) | 7(35%) | 4(20%) | 4(20%) | 1(5%) | 4(20%) |\n| 8 | 6(30%) | 4(20%) | 4(20%) | 2(10%) | 4(20%) | 7(35%) | 4(20%) | 5(25%) | 1(5%) | 3(15%) |\n| 9 | 4(20%) | 5(25%) | 6(30%) | 1(5%) | 4(20%) | 5(25%) | 5(25%) | 7(35%) | 0(0%) | 3(15%) |\n| 10 | 5(25%) | 5(25%) | 5(25%) | 2(10%) | 3(15%) | 4(20%) | 7(35%) | 6(30%) | 1(5%) | 2(10%) |\n\n---\n\n**Q3: Equation 9 appears to imply a simple averaging of prompts from different clients. This raises a concern regarding potential bias in the global model. If some clients possess substantially more training data than others, a straightforward averaging scheme could disproportionately weigh the contributions of larger clients.**\n\n**A3**: Thank you for your valuable comment. Consistent with FedAvg,  we **weight the aggregation by each client’s data volume**  to prevent bias arising from unequal dataset sizes.  The revised aggregation formula for Equation 9 is: $$\\tilde{P}^T_i = \\sum_{c \\in C_T} \\frac{n_c}{\\sum_{c' \\in C_T} n_{c'}} \\,P^T_{i,c},$$ where   $n_c$ denotes the number of samples at client c.\n  \nWe will incorporate this clarification in the final paper.\n\n---\n\n**Q4: Since only select prompts most similar to global prompts for aggregation, will this lead to a fixed prompts being aggregated while others always keep local? How is the frequency that each group prompt being aggregated globally?**\n\n**A4**: Thank you for this important question. Our method does not result in the same groups being consistently aggregated while others remain perpetually local. We provide aggregation frequency statistics for each group during training on OxfordPets in the table below. As mentioned in Q2, we introduce **a temperature parameter** to mitigate such extreme scenarios . Furthermore, our **dynamic aggregation mechanism** inherently requires different groups to maintain varying aggregation frequencies: prompt groups that are selected more frequently during training tend to aggregate **broader, global semantics**, while those with lower selection frequencies preserve more **localized, client-specific semantics** . These complementary behaviors balance generalization and personalization. Although we cannot include visualizations in this submission, we will add this discussion with qualitative examples and visualizations in the final version.\n\n| t_g1  | t_g2  | t_g3  | t_g4  | t_g5 | v_g1  | v_g2  | v_g3  | v_g4 | v_g5  |\n| ----- | ----- | ----- | ----- | ---- | ----- | ----- | ----- | ---- | ----- |\n| 28.0% | 17.0% | 23.5% | 12.5% | 19%  | 30.0% | 18.0% | 24.5% | 9.5% | 18.0% |"}, {"review_id": "321I2AAgFy", "reviewer": "Reviewer_KEQK", "summary": "This paper proposes FedMGP, a personalized federated prompt learning approach for VLMs. FedMGP leverages multiple prompt groups with diversity loss to capture diverse and instance-level features. The method also employs a similarity-based probabilistic strategy to select the aggregated prompt group.", "questions": "The questions align with the weaknesses mentioned above. **Additional clarification on the generalization mechanism and fairer baseline comparisons would be valuable.**", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength: \n- The idea of using multiple groups of prompts to capture diverse prompt representations is novel and well-motivated for federated prompt learning.\n- The method is supported by comprehensive experiments from various perspectives and provides theoretical analysis for different aggregation strategies.\n\nWeakness:\n- The main weakness lies in the choice of baselines. Since this paper uses the CM metric to evaluate the overall generalization performance, the authors should conduct further experiments comparing against prompt learning methods that specifically focus on generalization ability in federated learning, such as FedCoCoOp and FedMaple. Please refer to the baseline methods shown in Table 2 of FedMVP ([https://arxiv.org/pdf/2504.20860](https://arxiv.org/pdf/2504.20860)).\n- The paper proposes a multi-group prompt training and aggregation strategy, but it is unclear why FedMGP can better generalize to unknown datasets compared to existing methods. A more detailed explanation would strengthen the contribution.\n- The experimental results for FedOTP and PromptFolio appear unusual. These methods achieve significantly high performance locally while showing substantial performance drops on base and new datasets. To my knowledge, FedOTP and PromptFolio are not specifically designed for generalization scenarios, which may not provide a fair comparison to validate the effectiveness of FedMGP.", "was_revised": true, "final_justification": "Thank you for your comprehensive rebuttal. The final rebuttal clearly addresses my concerns regarding the generalization between prompt diversity and aggregation strategies. I hope this important discussion will be placed in a prominent position in final version and reorganized in a more accessible format for readers to understand. Based on these clarifications, I will raise my score.", "rebuttal": "**Comment:**\n\nWe appreciate your valuable comments. We were wondering if our responses have addressed your concerns. Please let us know if you have additional questions. Thank you!\n\n---\n\n**Q1: The main weakness lies in the choice of baselines. Since this paper uses the CM metric to evaluate the overall generalization performance, the authors should conduct further experiments comparing against prompt learning methods that specifically focus on generalization ability in federated learning, such as FedCoCoOp and FedMaple. Please refer to the baseline methods shown in Table 2 of FedMVP.**\n\n**A1:** Thank you for your suggestion. We have added the experimental results of FedCoCoOp and FedMaple as follows. FedCoCoOp and FedMaple indeed demonstrate excellent performance in pure generalization capabilities. However, both methods **exhibit significant deficiencies in client personalization abilities.** In the CM comprehensive metric that balances personalization and generalization, FedMGP significantly outperforms these baseline methods. Moreover, FedMGP's **training parameters (12.8k)** are substantially lower than **FedCoCoOp(35.4K**) and **FedMaple(3.55M).  This will result in high communication costs in federal learning.**\n\n| dataset        | method    | Local | Base  | Novel | CM    | Trained parameter |\n| -------------- | --------- | ----- | ----- | ----- | ----- | ----------------- |\n| dtd            | FedCoCoOp | 61.06 | 64.00 | 43.96 | 56.59 | /                 |\n| dtd            | FedMaple  | 70.65 | 71.99 | 56.28 | 66.91 | /                 |\n| food101        | FedCoCoOp | 89.68 | 89.72 | 90.88 | 89.99 | /                 |\n| food101        | FedMaple  | 89.69 | 89.69 | 90.71 | 89.94 | /                 |\n| oxford_pets    | FedCoCoOp | 95.09 | 95.06 | 95.41 | 95.16 | /                 |\n| oxford_pets    | FedMaple  | 95.24 | 95.00 | 97.04 | 95.62 | /                 |\n| oxford_flowers | FedCoCoOp | 77.40 | 77.87 | 72.41 | 76.22 | /                 |\n| oxford_flowers | FedMaple  | 86.03 | 86.32 | 73.05 | 82.58 | /                 |\n| caltech101     | FedCoCoOp | 96.14 | 97.35 | 94.76 | 96.09 | /                 |\n| caltech101     | FedMaple  | 96.05 | 97.42 | 94.00 | 95.86 | /                 |\n| Average        | FedCoCoOp | 84.55 | 85.22 | 79.39 | 83.23 | 35.4k             |\n| Average        | FedMaple  | 88.44 | 88.94 | 84.95 | 87.60 | 3.55 M            |\n| Average        | FedMGP    | 96.65 | 79.20 | 80.86 | **88.34** | **12.8K**             |\n\n---\n\n**Q2: The paper proposes a multi-group prompt training and aggregation strategy, but it is unclear why FedMGP can better generalize to unknown datasets compared to existing methods. A more detailed explanation would strengthen the contribution.**\n\n**A2:** Thank you for this suggestion. We fully recognize the importance of interpretability and have addressed it in our paper as follows:\n\n- As shown in Figure 1 of the appendix, our diversity loss effectively enforces distinct representations. The **remarkably low inter-group correlation** among visual prompts provides direct evidence that each group learns to capture unique and complementary visual features.\n- In appendix F, **signal-noise decomposition theory** mathematically proves that the multi-group mechanism enhances the model's **signal-to-noise ratio** by promoting representational diversity . This framework provides a rigorous foundation for why our design choice is critical for balancing personalization and generalization.\n\n---\n\n**Q3: The experimental results for FedOTP and PromptFolio appear unusual. These methods achieve significantly high performance locally while showing substantial performance drops on base and new datasets. To my knowledge, FedOTP and PromptFolio are not specifically designed for generalization scenarios, which may not provide a fair comparison to validate the effectiveness of FedMGP.**\n\n**A3:** We appreciate the reviewer's concerns. However, in the seminal federal learning paper[1]. highlighted the core motivations of federated learning:\n\n- Decoupling model training from raw data access, safeguarding user privacy while avoiding risks of centralized data collection;\n- Achieving high-quality models with fast convergence and low communication overhead via iterative model averaging under non-IID, unbalanced, and massively distributed data.\n\nIn pathological IID and Dirichlet distribution scenarios, focusing solely on local performance is equivalent to **conducting fully-supervised training on each client.** This not only fails to achieve **cross-client or cross-domain generalization,** but also undermines federated learning’s original intent of obtaining **global data generalization performance** under conditions of reduced communication costs and privacy preservation. Therefore, generalization should be a necessary consideration for every federated learning method. As representative works in federated prompt learning, FedOTP and PromptFolio exhibit outstanding local performance but suffer significant performance drops on both global seen classes and unseen classes in same data distribution, which highlights the critical importance of evaluating generalization and provides a fair benchmark for validating FedMGP’s effectiveness in broader scenarios.\n\n[1] McMahan, Brendan, et al. \"Communication-efficient learning of deep networks from decentralized data.\" *Artificial intelligence and statistics*. PMLR, 2017.\n\n---\n\n**Q4: Conducting experiments on all CLIP datasets would be beneficial. The experimental settings can reference those used in CoOp.**\n\n**A4:** Thank you for your valuable suggestion. We have supplemented experiments on StanfordCars, SUN397, UCF101, and FGVC-Aircraft, demonstrating the superiority of our method on these benchmarks.The experimental results are shown in the table below.\n\nWe are unable to test EuroSAT and ImageNet for the following reasons.:\n\n- EuroSAT consists of only 10 classes. After splitting into seen and unseen sets, the seen subset contains just 5 classes, making it infeasible to simulate pathological non-IID settings (which require one class per client, i.e., 10 clients) under the standard federated prompt learning protocol [1–3].\n- ImageNet is prohibitively large. Due to time and resource constraints, we were unable to conduct ImageNet experiments within the current timeline. Moreover, previous federated prompt learning studies [1–3] did not evaluate on ImageNet, placing it beyond the scope of directly comparable baselines.\n\nWe believe this selection maintains experimental rigor while fully aligning with established federated prompt learning standards.\n\n| Dataset       | Method      | Local     | Base  | Novel     | CM      |\n| ------------- | ----------- | --------- | ----- | --------- | --------- |\n| ucf101        | FedPGP      | 82.61     | 71.78 | 68.45     | 76.34     |\n| ucf101        | FedTPG      | 76.22     | 75.96 | 72.09     | 75.10     |\n| ucf101        | PromptFolio | 96.15     | 31.94 | 42.00     | 66.22     |\n| ucf101        | FedOPT      | 92.39     | 16.33 | 19.07     | 54.99     |\n| ucf101        | FedMGP      | 92.69     | 68.38 | 72.86     | 81.62     |\n| ucf101        | PromptFL    | 77.08     | 76.94 | 70.36     | 75.29     |\n| sun397        | FedPGP      | 89.43     | 66.51 | 67.43     | 78.20     |\n| sun397        | FedTPG      | 73.72     | 73.71 | 75.17     | 74.08     |\n| sun397        | PromptFolio | 95.18     | 32.89 | 44.47     | 66.50     |\n| sun397        | FedOPT      | 93.40     | 11.38 | 19.11     | 53.83     |\n| sun397        | FedMGP      | 91.83     | 68.51 | 72.20     | 81.07     |\n| sun397        | PromptFL    | 76.25     | 76.20 | 75.68     | 76.09     |\n| stanford_cars | FedPGP      | 85.37     | 57.63 | 60.19     | 72.13     |\n| stanford_cars | FedTPG      | 65.50     | 65.47 | 69.10     | 66.37     |\n| stanford_cars | PromptFolio | 96.44     | 29.43 | 46.77     | 66.28     |\n| stanford_cars | FedOPT      | 91.06     | 9.32  | 10.62     | 50.49     |\n| stanford_cars | FedMGP      | 92.61     | 56.48 | 71.19     | 77.80     |\n| stanford_cars | PromptFL    | 62.98     | 63.14 | 69.87     | 64.66     |\n| fgvc_aircraft | PromptFolio | 82.50     | 12.29 | 17.09     | 48.40     |\n| fgvc_aircraft | FedOPT      | 64.34     | 7.27  | 8.12      | 36.01     |\n| fgvc_aircraft | FedPGP      | 47.59     | 25.89 | 22.89     | 35.94     |\n| fgvc_aircraft | FedTPG      | 12.00     | 12.00 | 4.50      | 9.27      |\n| fgvc_aircraft | PromptFL    | 25.03     | 25.03 | 24.48     | 24.89     |\n| fgvc_aircraft | FedMGP      | 78.46     | 21.03 | 30.15     | 51.62     |\n| Average       | PromptFolio | 92.57     | 26.64 | 37.58     | 61.87     |\n| Average       | FedOPT      | 85.30     | 11.08 | 14.23     | 48.88     |\n| Average       | FedPGP      | 76.25     | 55.45 | 54.74     | 65.67     |\n| Average       | FedTPG      | 56.86     | 56.79 | 55.22     | 56.42     |\n| Average       | PromptFL    | 60.34     | 60.33 | 60.10     | 60.27     |\n| **Average**   | **FedMGP**  | **88.90** | 53.60 | **61.60** | **73.11** |\n\n[1]Guo, Tao, et al. \"Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model.\" *IEEE Transactions on Mobile Computing* 23.5 (2023): 5179-5194.\n\n[2] Li, Hongxia, et al. \"Global and local prompts cooperation via optimal transport for federated learning.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[3]Pan, Bikang, Wei Huang, and Ye Shi. \"Federated learning from vision-language foundation models: Theoretical analysis and method.\" *Advances in Neural Information Processing Systems* 37 (2024): 30590-30623.\n\n---\n\n**Q5: Other Paper Formatting Concerns**\n\n**A5:** Thank you very much for your detailed suggestions on paper formatting. We will address these issues in the final version of the paper."}]

Author rebuttal:

Meta-review: This study presents a personalized Federated Learning framework developed to alleviate client overfitting and aggregation instability in VLMs by utilizing a multi-group text-visual prompt design, a diversity loss and dynamic prompt aggregation based on similarity probabilistic sampling to balance global and client-specific knowledge. All reviewers recognised that the proposed approach is meaningful and novel, and it is sufficiently evaluated over a decent number of benchmarks (under non-IID settings). At the same time, they identified a number of technical issues which I believe that the authors have sufficiently addressed during the rebuttal (as also recognized by the majority of the reviewers). Since (almost all of) the concerns are resolved, I'm happy to recommend accept for this paper.
Decision: Accept (poster)