NeurIPS 2025

FedMGP: Personalized Federated Learning with
Multi-Group Text-Visual Prompts
Weihao Bo1,Yanpeng Sun2∗, Yu Wang3, Xinyu Zhang4, Zechao Li1
1Nanjing University of Science and Technology
2National University of Singapore
3Baidu VIS
4University of Auckland
Abstract
In this paper, we introduce FedMGP, a new paradigm for personalized federated
prompt learning in vision-language models (VLMs). Existing federated prompt
learning (FPL) methods often rely on a single, text-only prompt representation,
which leads to client-speciﬁc overﬁtting and unstable aggregation under heteroge-
neous data distributions. Toward this end, FedMGP equips each client with mul-
tiple groups of paired textual and visual prompts, enabling the model to capture
diverse, ﬁne-grained semantic and instance-level cues. A diversity loss is intro-
duced to drive each prompt group to specialize in distinct and complementary se-
mantic aspects, ensuring that the groups collectively cover a broader range of local
characteristics. During communication, FedMGP employs a dynamic prompt ag-
gregation strategy based on similarity-guided probabilistic sampling: each client
computes the cosine similarity between its prompt groups and the global prompts
from the previous round, then samples s groups via a softmax-weighted distribu-
tion. This soft selection mechanism preferentially aggregates semantically aligned
knowledge while still enabling exploration of underrepresented patternseffectively
balancing the preservation of common knowledge with client-speciﬁc features.
Notably, FedMGP maintains parameter efﬁciency by redistributing a ﬁxed prompt
capacity across multiple groups, achieving state-of-the-art performance with the
lowest communication parameters (5.1k) among all federated prompt learning
methods. Theoretical analysis shows that our dynamic aggregation strategy pro-
motes robust global representation learning by reinforcing shared semantics while
suppressing client-speciﬁc noise. Extensive experiments dem...