FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models
Authors: Xinting Liao
Weiming Liu
Jiaming Qian
Pengyang Zhou
Jiahe Xu
Wenjie Wang
Chaochao Chen
Xiaolin Zheng
Tat-Seng Chua
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning; Out-of-distribution
Primary area: social_aspects->privacy
TL;DR: Enhancing the out-of-distribution roboustness of federated prompt learning on vision language models.

Abstract: Federated prompt learning (FPL) for vision-language models is a powerful approach to collaboratively adapt models across distributed clients while preserving data privacy. However, existing FPL approaches suffer from a trade-off between performance and robustness, particularly in out-of-distribution (OOD) shifts, limiting their reliability in real-world scenarios. The inherent in-distribution (ID) data heterogeneity among different clients makes it more challenging to maintain this trade-off. To fill this gap, we introduce a Federated OOD-aware Context Optimization (FOCoOp) framework, which captures diverse distributions among clients using ID global prompts, local prompts, and OOD prompts. Specifically, FOCoOp leverages three sets of prompts to create both class-level and distribution-level separations, which adapt to OOD shifts through bi-level distributionally robust optimization. Additionally, FOCoOp improves the discrimination consistency among clients, i.e., calibrating global prompts, seemly OOD prompts, and OOD prompts by Semi-unbalanced optimal transport. The extensive experiments on real-world datasets demonstrate that FOCoOp effectively captures decentralized heterogeneous distributions and enhances robustness of different OOD shifts. The project is available at GitHub.

FOCoOp:
Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models
Xinting Liao 1
Weiming Liu "1 Jiaming Qian'
Pengyang Zhou' Jiahe Xu' Xiaolin Zheng' Tat-Seng Chua
Wenjie Wang? 2 Chaochao Chen
Abstract
downstream tasks (Liu et al., 2024). To meet the necessity of privacy preservation, federated prompt learning (FPL) methods are recently emerging, which utilize transferable knowledge of VLMs by collaboratively tuning prompt con- texts with decentralized data (Guo al., 2023b; Li et al., 2024). It reduces the overwhelming computation and com- munication burdens, while bringing effective personalized transferring under privacy regularization. However, the existing FPL methods mainly suffer from a significant trade- off between performance and out-of diitr---iittibttion (OOD) robustness, i.e., enhancing the accuracy comes at the cost of failing to generalize covariate-shift data or detect semantic- shift data (Lafon al., 2025; Kumar al., 2022). As shown in Fig. 1, though the classification accuracies of existing FPL methods (e.g.. FedOTP (Li et al., 2024), and Prompt- Folio (Pan et al., 2024)) benefit from maintaining global generalization and local personalization, they almost fail to harness OOD robustness, c.g., detecting OOD samples in downstream tasks. Because OOD samples are completely unseen the pretraining distribution of the CLIP or the fine-tuning distribution of clients in FPL (Liao et al., 2024b; Yu et al., 2023).
Federated prompt learning (FPL) for vision- language models is powerful approach to collab- oratively adapt models across distributed clients while preserving data privacy. However, exist- ing FPL approaches suffer from a trade-off be- tween performance and robustness, particularly in out-of-distribution (OOD) shifts, limiting their reliability in real-world scenarios. The inherent in-distribution (ID) data heterogeneity among dif- ferent clients makes it more challenging to main- tain this trade-off. To fill this gap, we introduce a Federated oOD-aware Context Optimization (FOCoOp) framework, which captures diverse dis- tributions among clients using ID global prompts, local prompts, and OOD prompts. Specifically, FOCoOp leverages three sets of prompts to create both class-level and distribution-lewt separations, which adapt to OOD shifts through bi-level dis- tributionally robust optimization. Additionally, FOCoOp improves the discrimination consistency among clients, i.e., calibrating global prompts, seemly OOD prompts, and OOD prompts by Semi-unbalanced optimal transport. The exten- sive experiments on real-world datasets demon- strate that F FOCoOp effectively captures decen- tralized heterogeneous distributions and enhances robustness of different OOD shifts. The project available at GitHub.
This brings the problem of enhancing OOD robustness for federated prompt learning on pretrained VLMs.
Applying federated OOD-aware prompt learning on VLMs suffers from two aspects of challenges. i.e., CHI: How to maintain the class-level and distribution-level separations for distinguishing client data samples? and CH2: How to enhance the consistency of OOD robustmess among all clients? The first challenge arises when applying prompt learning in federated scenarios, leading to inferior class discrimination and reduced distinction between different distributions. Because each client is not only limited to ex- ploit the whole in-distribution (ID) data distribution, but also unavailable for modeling the OOD shifts of data distribu- tion (Liao et al., 2024b). On one thing, it is inevitable for the client model to be overfitting for prompt learning with local data, degrading generalization for unseen ID data in other clients, and covariate-shift data (Cui et al., 2024). On an- other thing. it brings a mismatch between OOD prompts and real-world OOD distributions, impacting client detection
1. Introduction
In recent days, pretrained rision-language models (VLMs), e.g., CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), are widely studied for their benefits of unifying multi- modal data and learning transferrable representation across
Equal contribution 'Zhejiang University Universsty of Sci- ence and Technology of China 'National University of Singapore. Correspondence to: Xiaolin Zheng <xlzheng @ zju.edu.en>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FOCoOp: Enhancing Out-of-Distributinn Robustness in Federated Prompt Learning for Vision-Language Models
[a) Semantic-shift Detection
(b) Covariate-shift Generalization
[c] FPR95 vs Top-1 CACC
OOCoOp
PrompiFolio
FOCoUp
PromptFFlio FOCoOp
20
50
4 FeoOTP
50
FedPGP FadGallL
FedGall.oP FedFGP 4
FdOOTP
FedGalloP
Top- CACC
Top 1op-1 CACC
FPR95
PromptFolic
50
FedPGP
PrompiiF
FedLoCoOp
+
PromptFL
PeoLoCoOp
Feo
Fedl
60
FedE APT
PromptFL
FedLAPT
6D
100
80
100
80
Top-1 ACC
Top-1 ACC
FPR95
Figure 1: The performance and OOD robustness of FPL. methods. The FPR95 reflects detection capability, while ACC and CACC are generalization performance on ID data and covariate-shift data, respectively.
capability, as in Fig. 1. The second challenge stems from personalization of heterogeneous data, where variations in domains and class representations hinder achieving perfor- mance consistency. The inherent heterogeneity of client data prevents the direct application of centralized OOD methods, limiting their effectiveness in improving the OOD robust- ness of FPL. In Fig. I, the adaptions of FedAvg (McMahan al., 2017) with centralized OOD prompt tuning methods fail to effectively handle heterogeneous data, where only FedGalLoP (Lafon et al., 2025) maintains its performance. The crucial reason is that each client maintains the local OOD robustness on heterogeneous data, which is incon- sistent among clients in FPL. Even clients in FedGalLop are still confused about identifying data unseen at local but presented in other clients (Yu et al., 2023).
OOD prompts for each client, reducing misclassification of data in other clients, and detecting semantically novel OOD data from a global perspective.
To conclude, our main contributions come from four as- pects: (1) we are the first to propose a federated owDre prompt learning framework, i.c., FOCoOp, which maintains performance without sacrificing OOD robustness. (2) We design the BOS module, which leverages bi-level distri- butionally robust optimization to enhance class class-matching between images and prompts while ensuring clear separa- tion between ID and OOD data. (3) We develop the GOC module, which utilizes semi-unbalanced optimal transport mapping to calibrate OOD prompts and global prompts in consistency. (4) In experiments, we validate the effec- tiveness of FOCoOp on extensive real-world datasets with different tasks, achieving consistently competitive results.
In this work, promote the OOD robustness of federated prompt learning VLMs, and propose a Federated OOD- aware Context Optimization framework, i.e., FOCoOp. Specifically, FOCoOp integrates the Bi-level OOD Sepa- rations (BOS) and Global-view OOD Consistency (GOC) modules to simultaneously learn global ID prompts for gen- eralization, local prompts for personalization, and OOD prompts for detection. To address CH1 in each client, BOS learns three types of prompts to align local ID images with bi-level distributionally robust optimization, which per- turbs global prompts and OOD prompts to explore wider semantic matching with limited image data. The perturba- tions on global prompts improve class-level separation by penalizing the mismatching scores with ID image data, thus avoiding interference with the original class discrimination among decentralized data. Similarly, BOS perturbs OOD prompts to minimize the impact of overall semantic match- ing between ID image data and OOD prompts, bringing more distinctive distribution-
2. Related Work
2.1. Federated Learning on VLMs
Federated learning (FL) models decentralized data in each client and aggregate client models for global model in server (McMahan et al.. 2017). Personalized federated learning emphasizes tailoring to personalized performance through regularization (Li et al., 2020; Karimireddy et al., 2020), contrastive learning (Li et al., 2021), model decou- pling (Chen & Chao, 2021; Dong et al., 2022), hyperbolic modeling (Liao et al., 2023; Liu et al., 2025a), and so on. However, conventional FL methods require the collabora- tion of client models with full parameter sets on the server, which becomes impractical as model size is growing by scaling law (Li et al., 2024; Cui et al., 2024). Recent works have explored parameter-efficient fine-tuning methods to reduce the communication and computation costs. For ex- ample, PFPT (Weng et al., 2024) uses visual prompts to tackle class-imbalance problem in pretrained vision models, and FL.RA (Wang et al., 2024b) utilizes Low-rank adapters with varying ranks to fine-tune language models. Similarly, we further consider using federated prompt learning to col- laboratively adapt client data by tunable textual prompts
separation. For tackling CH2 in server, GOC aggregates global prompts from par- ticipating clients, and aligns them with OOD prompts via Semi-unbalanced optimal transport mapping. Based on the mapping, GOC selects seemly OOD prompts that are mostly close to ID global prompts, to calibrate aggregated global prompts, and sends the most distant OOD prompts back to clients. This approach mitigates ambiguous identification of
2

---

FOCoOp: Enhancing Out-of-Distributi Robustness in Federated Prompt Learning for Vision-Language Models
rather than entire VLMs, reducing communication and com- putation cost (Guo al., 2023b;a). Recently, a series of work focuses on balancing personalization and generaliza- tion. e.g.. CLIP2FL (Shi et al., 2024). DiPrompt (Bai et al., 2024a), FedOTP (L.i et al., 2024), FedTPG (Qiu et al., 2023) and pFedPG (Yang et al., 2023). FedFolio (Pan et al., 2024) further provides theoretical insights into trade-off's between generalization and personalization. Although these methods improve federated prompt learning for heterogeneous data, these works overlook the OOD robustness issues.
VLMs to detect OOD data that is highly correlated with ID data. With the constraints of data privacy and heterogeneity of FPL, it is further demanding to efficiently and consis- tently apply prompt tuning on pretrained VLMs to adapt decentralized data.
3. Methodology
3.1. Problem Setting
Federated Prompt Learning Formulation. As shown in Fig. 2, FPL methods collaboratively adapt pretrained VLMs with decentralized data among server and K clients by fine-tuning prompts, i.c.. N tunable context embed- dings t = fer.... ......... The server is responsible for the generalization of prompts in a global view. And each client k focuses on fine-tuning prompts for frozen VLM using local heterogeneous data D = { 3k 3) where Øk.y denotes j-th input sample and Uk. is the asso- ciated label. For ID image Zx..j belonging to class c, i.e. yk.j = the image encoder I extracts its visual represen- tation, hxs. = Iø (xx.))) Correspondingly, we compute the textual embeddings for context prompts based on text en- coder T, i.e., e, = To (t., ne.), where n. is the class name embedding of yı and all prompts are randomly initialized.
2.2. OOD Robustness in Federated Learning
It is long-term issue but rarely a topic for improving 00D robustness in federated scenarios (Hendrycks & Gimpel, 2016; Li & Wang. 2024; Huang et al., 2024). Recent FL methods aim to enhance generalization by preserving in- variant relationships between data and labels (Jiang & Lin, 2022; Tan et al., 2023; Tang et al., 2022), smoothing lo- cal loss landscapes (Qu et al., 2022), and capturing robust representations handle heterogeneous distributions and adapt to unseen clients (Yuan et al., 2021; Nguyen et al., 2022a; Guo et al., 2023c; Liu et al., 2021). FedGOG (Zhou et al., 2025a) further improves OOD generalization in de- centralized graph data (Zhou et al., 2025b). Meanwhile, FOSTER (Yu et al., 2023) learns a class-conditional gen- erator to synthesize virtual external-class OOD samples and facilitate OOD detection in FL for the first time. And FOOGD (Liao et al., 2024b) captures global distribution with score matching model to tackle OOD generalization and detection simultaneously. Nevertheless, these models are not scalable for large pretrained VLMs and struggle to directly adapt to FPL while maintaining OOD robustness.
Then FPL minimizes similarity between image data and prompt corresponding to its label, which is cosine distances, he,,e= i.e., so (o(x.,.. t) [Zolzx.) (llTT((t.m)))) = llhe, ||||=||| And we denote S(x, t) = S(h, e) exp (so(tt))))// as similar- ity score. After model converges, prompts for aggregation in each client & are sent to the server and aggregated by
class, e.g., to >x k=l Σ, [Pe. ti,0-
Da.
Objective of FOCoOp. FOCoOp is a FPL framework that captures heterogeneous client distributions and en- hances OOD robustness by using three sets of prompts, i.e., (1) ID global prompts T = (ta) of C classes that captures shared ID global distribution, (2) OOD prompts Te = {t;) U that are trained to mismatch with ID data, and (3) ID local prompts TV = (E/ that captures het- erogeneous distribution in each client k. In detail, the local prompts T. adapt local data realize personalization of client k. The global prompts T enhance the generalization of covariate-shiff and heterogeneity in global view. And the OOD prompts Tod capture the mismatching relationship between ID visual data and OOD textual prompts. The over- all objective is to maintain performance as well as enhance OOD robustness:
2.3. OOD Robustness on Pretrained VLMs
The pretrained VLMs contain large-scale model parameters and provide transferrable representation for OOD gener- alization and zero-shot capabilities (Radford et al., 2021; Jia et al., 2021; Liu et al., 2025b). However, VLMs rely heavily pretrained textual-imen immae matching distribution, causing the degradation of generalization and detection ca- pabilities once the textual prompts are diverse and incor- rect (Zhou et al., 2022; Mayilvahanan et al., 2023; Yang et al., 2024; Shu et al., 2023). CoOp (Zhou et al., 2022) pro- poses to learn the representation vector of prompt context words during adapting pretrained VLMs, enhancing the generalization on distribution shifts. To enhance robust- ness, OOD generalization and detection are further explored. For instance, CDC (Zhang et al., 2024) employs causal analysis (Wang et al., 2024a; 2025a; Qi et al., 2024) to identify task-irrelevant knowledge interference. Similarly, CLIPN (Wang et al., 2023) finetunes VLMs to generate neg- ative prompts that assess the probability of an OOD concept. Moreover, ID-Liie (Bai et al., 2024b) extends pretrained
K
Σ (DD "",T",T") + (DO0D TT)")
min
(I)
where (-) is the in-distribution generalization loss on local testing ID data DIP and covariate-shift data DJDC (0()) is the duttt--distiibuttin detection loss for OOD data DOOD.
3

---

FOCoOp: Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
ID-/ Prompt
ID-g Prompt
OOD Prompt
ID Image
OOD Image
Learnable
Frozen
Server
Client k
Bi-Level OOD Separations
Global-view OOD Consistency
Prompt Learning
E1
Sa(h, m
Sy ht E
ID-I +<n.
:
SUOT
Text
5
e1)
Seemly
il-g
Encoder T
OOD
ID
OOD
OOD
+
Probability
To k
eu
S > 2
Tap-M Calibration
Bottom-U
Selection
+
Image Extraction
S(,
+
+<n
+<n,>
>
(T9 T
ID
Similarity
Image Similarities
+<1
+m
Image
Image Encoder I
h
7
OOD Image
+<
+
+<nØ
61
6, Prompts
Cc
82
ID-g
Seemly OOD OOD
ID
OOD Prompts
Figure 2: Framework of FOCoOp. For each client, FOCoOp uses bi-level OOD separations module to fine-tune three sets of prompts adapting to pretrained VLM. While in the server, FOCoOp leverages the global-view OOD consistency module to enhance the discrimination among ID global prompts and OOD prompts.
3.2. Client: Bi-Level OOD Separations
[CLASS NAMEJ". Following (Jiang et al., 2024), we select the top U negative class labels (n., based on their distances, i.e., ((du)(((d)u_))))
For the sake of limited access to training data, it mainly includes two aspects OOD robustness, i.e., OOD gener- alization and OOD detection, in federated prompt learning for VLMs (Liao al., 2024b; Bai al., 2023). Regarding OOD generalization, FOCoOp needs to maintain the class- matching between ID prompts and intriguing IDD data that are heterogeneous, covariate-shifted, and presented in untrained clients. In terms of OOD detection, F FOCoOp should identify semantic shifts, rather than mistakenly categorizing unseen samples from other clients as outliers. To tackle the above issues, we first introduce the initialization and modeling procedure of prompt tuning for three sets of prompts. Then devise bi-level distributionally robust optimization for widely capturing intriguing relationships between different prompts and image data, i.e., (1) class-matching between im- age and prompts of the same class, and (2) distribution-level separation between local image data and OOD prompts.
Local Prompt Fine-tuning. After initialization, we com- pute the similarity scores, and leverage them as prediction probability (Li et al., 2024; Zhou et al., 2022). To realize the class-level separation, we encourage the similarity score of image and ID prompts associated with its label to be high, while hindering remaining similarity scores to be low. The prediction loss can be formulated as:
e. EaLD [- log (p(y c|x)]
(3)
where plu c/a) E S((ett))+ 8 with p)t! + ptg.
S(w.tc)
proportional ID prompts fusion to
Similarly, we encourage distribution-len separation, by op- timizing the loss for prediction probability of all ID prompts that are larger than OOD prompts, i.e.,
Prompt Context Initialization. For three sets of prompts, the textual inputs are formulated with class labels t, {t, nac), where n. the word embeddings corresponding to the e-th class name. The global prompts 19 and local prompts t' share the same ID) class set, while OOD prompts t' are initialized with candidate class names sampled from a lexical database, i.c., WordNet (Fellbaum, 1998), which is widely used in OOD robustness (Jiang et al., 2024; Zhang al., 2025). Specifically, we calculate the negative cosine similarities between textual embeddings e, for ID classes {ye)_n and ē. for OOD candidate classes {ya 1 i.c.,
e ERED [- log p(yn 1(a:),
(4)
Σ S(a,te) where p(yn = 1|x) = De 8(a.)))+ (2.1g)
Σ s(x
IS (l ((m.g)) In this case, obtain the over- all objective of constructing class-level and distribution- level separation as below,
C(e, 0, t', t,, (t) BeED logp(y c(x)p(yn 1|x)]]
(5)
Bi-level Prompts distributionally robust optimization.
Though we can achieve the OOD robustness via tuning prompts based on Eq. (5), the separations are intriguing to realize, since the data heterogeneous and limited in local tuning. To realize the wider distribution exploration for fine-tuning IDD local data, we further introduce bi-level
d, = Percentile, (1-cos (ē.. ed)) Yue[u),
(2)
where is the fixed context embedding of photo the

---

FOCoOp: Enhancing Out-of-Distribatonn Robustness in Federated Prompt Learning for Vision-Language Models
distributionally robust optimization. Specifically, we per- turb the global prompts and OOD prompts to the worst-case, whose distribution is mostly divergent from clean prompt distribution by given discrepancy constraint. Note that global prompts enhance generalization without compromis- ing local personalization. We aim to explore perturbed global prompts within broad space defined by an optimal transport (OT) divergence from the original global prompts. Unlike KL-ddivergence, capturing categorical distribution without considering geometry in feature space, OT diver- gence preserves the geometry of latent feature spaces, which is vital to text-image feature matching in VLMs. While OOD prompts are designed to enhance the detection of open-world semantic shifts unseen during training, they often result in outliers that are geometrically distant from the clean prompt distribution, yet difficult to separate or remove (Wang, 2025). Therefore, OOD prompts are con- strained using an unbalanced optimal transport (Liao et al., 2024a; Liu et al., 2024) divergence, simultaneously captur- ing geometric uncertainty and non- non-geemetric contamination. That is, we seek the worst-case of prompt distributions to mathh image with textual prompts in a noint-to-uncertaing set way, i.e.,
By constructing the class-level and distribution-level sep- arations with bi-level distributionally robust optimization, the local prompt tuning explores wider perception for the generalization via global prompts as well as detection via OOD prompts. The algorithm is presented in Algorithm 2.
3.3. Server: Global-view OOD Consistency
Although we can maintain the robustness-awe separations in each client, the data heterogeneity prevents the prompts from optimizing to a consistent optimum for the global dis- tribution. In other words, in each client, global prompts, local prompts, and OOD prompts are fine-tuned by the reference of local data distribution, causing them to be indis- tinguishable in the server. For example, the OOD prompts in one client match unseen data from other clients with high similarity scores, hurting the generalization in a global view. The seemly OOD prompts are actually ID global prompts to be optimized, bringing the necessity to calibrate the global prompts and OOD prompts in the server. To resolve this ambiguity while preserving generalization performance, we introduce semi-unbalanced optimal transport (SemiUOT) alignment. Specifically, we align the OOD prompts learned among all clients with the global prompts to avoid inconsis- tent OOD robustness.
_DRRO_
= inf sup Biap (xx 0, 19,)) (%,
0 PEP,
Prompt Aggregation. The server collects global prompts [T and OOD prompts tuned in clients {TT) k= For global prompts of clients, we aggregate them in terms of the corresponding class to = Σ En IPe.ol
P E D: Dor P. Po) < m)} Q= 10 e D: Duor Q Qo) S n12),
|
s.1.
|Da.el th as con-
where Po and Qo are estimated distributions of global prompts T and OOD prompts T", respectively, n. and 122 are discrepancy constrains, Dor and Duor are optimal transport distance and unbalanced optimal transport distance defined in Appendix C.1 and C.2, respectively.
ventional federated methods do (McMahan et al., 2017). However, to is limited in generalization, since they cap- ture different client data distributions. Moreover, the OOD prompts capture the misalignment between local ID data and prompt context, mistakenly identifying the ID data from other clients outlier. To enhance the discrimination of global prompts T fe and OOD prompts T we seek out the seemly OOD prompts by aligning the distribu- tions of global and OOD prompts using semi-unbalanced optimal transport. We first concatenate (Te]k= into TRu- and apply SemiUOT with T The marginal probability of global prompts is constrained to be uniform to avoid bias, while the marginal probability of OOD prompts is loosely constrained to explore wider OOD space. The objective is:
Theorem 3.1. Suppose that the optimal dual variables A. and T are strictly positive, bi-level separation loss C(x, 0,t', t9,t") in Eq. (5) is continuous and differentiable, Bi-level distributionally robust optimization in Eq. (6) can be solved via:
8 = arg min EA 60
sup [L(F") - TI((t" t9)) tanp
8 arg min Ev
sup (L(t") - T2c (t+, {t))
t0
min JsemiUOT (C, TT) + TKL(T (T 1kulla) #20
ERDRO arg min ta log Ejo,t
expl
f(0)
(8)
T2H
s.t. T1C b. T E XUKU Tej 2 0 AyE[KU],
(7)
where j is the regularization of KL divergence, f(0) C(x, 0,t', (²) Tic(Et, - T2c (t", t"), [(t(-)) is short for L(x, 0, t',,' ta (o) optimizing } and c(-,-) is the cost function of optimal transport, iee, computing L2-norm distance.
where the a and b are probability weights initialized equally as a 1ku = b_cc, C E RC de- notes the cost matrix and it can be calculated via KL(T 1|(a) Σ>Ku Tez log Emal Foi - >K F. + a.] denotes the KL. de Divergence between two probability masses T 1 E KU
Coj = ||t - t VjE[KU],Ce[]]-
The proof details are provided in Appendix C.3.
5

---

FOCoOp: Enhancing Out-of-Distribation Robustness in Federated Prompt Learning for Vision-Language Models
and a E RRKU, Next, we optimize Eq. (8) via Frank-Wolfe Algorithm (Clarkson, 2010; Jaggi, 2013), which seeks the optimization direction and update it by gradient descent.
taining performance and OOD robustness, we simu- late heterogeneous distribution following both Dirichlet and Pathlogical settings (McMahan et al., 2017; Li et al., 2020) on CIFAR-100 (Krizhevsky et al., 2009) and Tinylm- ageNet (Le & Yang, 2015) as conventional work does (Liao et al., 2024b). We test the generalization based on CIFAR- 100-C (Hendryeks & Dietterich, 2018) and TinylmageNet- C (Le & Yang, 2015). Meanwhile, we compute on iNatu- ralist (Van Horn et al., 2018), iSUN (Xiao et al., 2010), Places (Zhou et al., 2017), and Texture (Cimpoi et al., 2014b) for evaluating the OOD detection capability in the testing phase, following existing CLIP-based meth- ods (Wang et al., 2023; Miyai et al.., 2024). To widely evalu- ate OOD generalization and detection, we follow previous work of federated prompt learning (Cui et al., 2024; Guo et al., 2023b;a), to study (1) heterogeneous label shift gen- eralization on Food101 (Bossard et al., 2014), DTD (Cim- poi et al., 2014a) 111 (Fei-Fei et al., 2004), Flow- ers (Nilsback & Zisserman, 2008), and OxfordPet (Parkhi et al., 2012) to predict the accuracy of personalization fol- lowing pathological heterogeneity, and (2) feature shift do- main generalization on DomainNet (Peng et al., 2019), and Office-Caltech 10 (Gong al., 2012), by leave-one-domain out validation strategy (Nguyen et al., 2022b). Specifically, for N - 1 domains of one dataset, we train each client with distinct domain data, and test its model generalization on the whole target data of remaining one domain.
Theorem 3.2. Semi- Unbalanced Optimal Transport Op- timization of Eq. (8) can be solved by Frank- Wolfe Algo- rithm ( (Clurkson, 2010; Jaggi, 2013), which iteratively up- dates π E' B(TT* =²), with step size 3 _
following Armijo condition (Armijo, 1966) and optimal di- rection& s¹ satisfying:
if c' arg min JemmiUor (T'))
0,
otherwise
6) O,
c=0
with initialization soj
otherwise
We provide the details in Appendix C.4. The optimal mapping matrix nO is used to figure out the seemly OOD prompts and enhance the OOD prompts in a global consis- tency.
Prompt Calibration. Finally, we seek seemly OOD prompts T and mapping मे based on the top-M OOD prompts whose probabilities satisfy Top (T' lku). Then we transport it to the semantic space of global prompts, and update the global prompts via exponential moving weight updating.
T = aT + (1- a)
(10)
Comparison Methods. We categorize the comparison methods into two types, i.e., (1) Existing federated prompt learning methods: PromptFI. (Guo et al., 2023b), Fe- dOTP (Li et al., 2024), FedPGP (Cui et al., 2024), Prompt- Folio (Pan et al., 2024), (2) Adapting existing centralized OOD robustness methods for federated scenarios: Fed- LAPT (Zhang et al., 2025), FedGalLoP (Lafon et al., 2025), FedLoCoOp (Miyai et al., 2024). The method implementa- tion details are illustrated in Appendix D.
While the remaining OOD prompts are filtered to keep top U OOD prompts with less signigicant alignment potential,
ie., Topu (-T T-' 1ku). The final OOD prompts for global updating are reformulated as
T = TØ = Tox T Ikk)]]
(11)
Since the updated OOD prompts are mostly distant from ID global prompts, we enhance the discrimination between the ID distribution and OOD distribution.
Implementation Details and Evaluation Metries. We con- duct experiments on ViT-B/16 (Dosovitskiy, 2020) CLIP models. To study the heterogeneity generalization on 100//00yylmmmeeeee datasets, we simulate both cross- device and cross-silo scenarios. That is, we set local train- ing epoch E = 2, communication round T = 25, and the number of clients K 10 for fully participation. While in cross-device setting, we choose local training epochs E 2, communication rounds T = 100. and K = 100 for 10% participation. To obtain fair comparisons, all comparison methods are tuned for converging using their best hyper- parameters, and we report the average of the results from three random seeds. We set the learnable prompt vectors with length as 16, embedding size as 512, class token posi- tion as 'end', and random initialization. We choose I prompt per class for both local and global ID prompts, and 100 OOD prompts in total. We report the average Top-1 I accuracies
In each communication round, the server sends the updated global prompts T and OOD prompts T as the consistent initial points for local adapting. We finally have commu- nication of federated OOD-aware prompt learning, whose overall procedure is in Algorithm 1. By jointly utilizing Bi-level OOD robustness separation in local client modeling based on Algorithm 2, and doobll-viww OOD robustness con- sistency in Algorithm 3, FOCoOp simultaneously resolves the trade-off' between performance and OOD robustness.
4. Experiments
4.1. Experimental Setup
Datasets. We study the OOD robustness of federated prompt learning on fifteen datasets. In terms of main-
6

---

FOCoOp: Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
Table I: Main results of federated prompt learning on CIFAR-100 and TinylmageNer.
Heterogeneity
Pathological Non-overlap (K 10)
Pathological Overlap (K 100)
Datasets
CIFAR- 100
TinylmageNet
CIFAR-100
TinyImageNe
Methods
ACC
CACC
FPR95
AUROC
ACC
CACC FPR95
AUROC|
ACC
CACC
FPR95
AUROCI ACC CACC FPR9S AUROC
PrompiFL
69.35
65.14
84.51
68.28
65.58
59.37
76.75
69.82
72.86
68.98
82.07
70.92
70.76
65.36
72.51
73.38
FedOTP
90.68
88.73
38.22
87.56
82.40
78.68
57.12
79.18
89.76
78.01
51.05
85.89
74.48
71.29
66.61
73.09
FedPGP
85.78
82.35
51.57
84.68
81.85
76.29
50.86
83.05
72.06
67.74
83.25
71.83
68.91
63.76
72.50
73.47
PromptFolio
91.82
89.60
44.26
88.06
88.03
83.39
42.84
87.34
81.99
75.86
73.53
75.71
78.16
73.27
61.08
79.66
Fed
64.13
60.23
77.59
68.76
58.08
52.05
72.96
72.07
72.93
67.98
79.76
70.54
69.05
63.22
67.92
75.06
FedGalLoP
91.19
88.63
41.45
89.64
87.18
82.55
45.54
86.53
91.37
81.70
57.78
87.42
82.92
78.78
53.38
83.71
FedLAPT
60.35
56.73
82.44
67.51
60.67
55.79
73,46
71.05
60.42
57.05
83.42
68.77
60.76
56.17
74.06
70.49
FOCoOp
93.85
91.47
19.50
95.42
88.35
83.56
21.73
96.56
94.10
82.23
24.00
92.82
83.53
79.20
38.05
89.85
-w/o-BÓS
91.07
89.06
27.31
93.08
84,39
78. 16
31.09
92.60
91.35
77.45
32.85
92.30
79.83
72.25
46,30
86,30
-w/o-GOC
88.04
85.09
37.01
87.47
85.69
80.29
28.48
93.10
89.10
74.05
41.85
88.08
72.20
66.33
50.13
86.07
Table 2: Main results of federated prompt learning on CIFAR-100 with different Dirichlet distributions.
Heterogeneity
0.1 a
a
0.5
a 5.0
Methods
ACC
CACC
FPR95
AUROC
ACC
CACC
FPR95
AUROC
ACC
CACC
FPR95
AUROC
PromptFI.
71.22
67.55
76.58
72.20
75.65
71.52
82.13
69.65
74.92
71.37
79.52
74.25
FedOTP
76.81
73.50
61.88
79.14
68.43
65.67
73.78
73.45
66.20
63.16
77.73
71.15
FedPGP
76.77
72.55
74.81
74.45
72.95
69.25
83.65
71.37
73.01
69.15
82.57
72.65
Folip
80.07
76.89
65.30
77.95
75.98
71.98
78.61
71.44
74.19
70.60
79.64
72.74
FedLooCOOp
67.87
63.70
76.81
70.40
74.44
70.35
73.28
72.56
74.87
70.98
74.82
73.72
FodGalLoP
80.53
77.61
60.72
82.66
75.87
72.85
68.72
79.66
74.32
71.14
72.72
79.13
FedLAPT
61.20
57.54
80.28
69.97
59.41
56.33
81.97
66.73
60.03
56.29
80.13
68.42
FOCoOp
82.42
78.52
46.56
86.98
77.71
73.59
54.26
83.40
77.66
73.59
$1.02
83.22
-w/o-BOS
79.18
76.04
54.30
82.34
74.39
70.55
58.40
81.27
75.09
71.76
54.92
81.64
-wo--GO
78.66
75.99
53.97
82.56
74.78
70.88
57.83
81.55
75.04
71.50
55.20
81.85
for generalization of ID (ACC1) and ID-C (CACC1). We compute maximum concept matching (MCM) (Ming et al., 2022) as OOD detection score, which is based on similar- ity between textual features and image features. Based on MCM, we report the standard metrics used for OOD detec- tion, i.e., AUROC (+) and FPR95 (L) (Yang et al., 2024).
FedLAPT perform worse in both generalization and detec- tion, meaning that they cannot robustly capture semantic- matching between image data and contextual prompts with few of samples. FedGalLoP can achieve OOD robustness, even performing best on OxfordPets. However, it still lacks of discrimination between ID and OOD data, due to locally identifying OOD samples without a global consistency of discrimination. Thirdly, FOCoOp maintains the perfor- mance and OOD robustness. FOCoOp outperforms other baselines on different heterogeneities and different participa- tion settings. Specifically, FOCoOp detects OOD effectively without sacrificing the prediction performance, validating that class-level separation and distribution-level separation are supposed to be considered at the same time. With the benefits of global-view discrimination between ID global prompts and OOD prompts, the bi-level separations become more consistent and powerful among clients, achieving the best results.
4.2. Performance Evaluation
Maintaining performance and OOD robustness on het- erogeneous data. We study the capability of Maintaining performance and OOD robustness on heterogeneous data on CIFAR-100 100, TinylmageNet, and additional five datasets on Tab. 1, Tab. 2, and Tab. 3. Without specification, we use brightness covariate shift as ID-C and texture as OOD data. Since DTD and Texture are identical, and Texture is commonly used for generalization in FPL methods, we eval- uate its detection performance by using iSUN as the OOD dataset. Firstly, existing FPL methods are not OOD- aware. PromptFL does not perform well because it does not adapt to heterogeneity. The FPL methods considering het- erogeneity, e.g., PromptFolio, achieve better results for both generalization ID data and ID-C data, but fail significantly in detecting OOD data. This indicates that these meth- ods can maintain the class-level separation among clients while ignoring the distribution- separation between ID and OOD data. Secondly, existing OOD-aware methods are hindered by heterogeneous data. FedLoCoOp and
Ablation Studies. We design two variants of FOCoOp by removing bi-level OOD separations (-w/o-BOS), and global- view OOD consistency (-w/o-GOC), respectively, to verify the effects of different modules. Though both variants suf- fer from performance degradation compared with FOCoOp, their detection capability remains competitive. This means that both BOS and GOC play a crucial role in distinguishing semantic-shift data, either considering local distribution- level separation or enhancing prompt discrimination in a
7

---

FOCoOp: Enhancing Out-of-Distribation Robustness in Federated Prompt Learning for Vision-Language Models
Table 3: Other datasets comparison (%) on the Pathological heterogeneity
Comporiean Mothods CIFA, Tinylnagehe.
setting (K = 10).
CF46.39
Thy repalel
70
Fo1d101
DTD
Caltech101
Flowers 102
OxfordPets
ACC
FPR95
ACC
FPR95
ACC
FPR95
ACC
FPR95
ACC
FPR95
P
11.87
83.16
$7.11
83.47
72.83
44.18
24.03
82.25
46.23
59 88
FedOTP
51.88
73.05
91.60
25.07
66.46
50.43
84.38
40.12
88.52
37.34 34
FedPGP
42.91
70.13
76.02
13.60
67.95
54.79
74.11
53.60
62.44
50.38
Fedl Op
PromptFolio
49.94
14.25
87.97
64.39
90.37
56.25
20.86
39.21
69.67
74.72
43.53
43.79
77.73
46.86
41.42
69.25
88.03
35,89
21.83
85.42
Prongeht
Romplibbl
Retesad
Feetee
rocol
FedGalLoP
94.58
12.11
91.15
17.57
86.48
31.09
96.58
18.07
99.09
1,70
FedLAPT
9.09
87.53
39.26
19.36
53.84
54.13
7.07
33.10
74.92
FOCoCp
95.52
5.19
95.23
13.29
87.89
14.22
98.50
1,86
98.74
5.52
Figure 3: The average generalization results
-w/o-BOS
72.55
27.02
93.29
22.61
79.44
22.14
98.17
1.94
97.66
7.47
-w/o-GOC
78.40
19.59
94.51
20.13
79.41
24.27
98.09
2.59
98.16
6.14
on data with different covariate-shife
Table 4: Domain generalization on DomainNet (left) and Office (right). The column notations are short for domain names.
Method
C
P
Q
R
S
Avg (DN)
Ca
D
W
Avg (O)
PromptFL
96.28
74.84
95.81
60.28
96.77
96.36
86.72
96.21
94.64
99.20
97.03
96.77
FedOTP
91.03
61.52
86.98
53.04
91.16
89.73
78.91
94.64
93.15
98.93
96.46
95.80
FedPGP
93.67
75.07
93.62
58.09
95.44
95.48
85.23
95.17
95.36
99.73
97.45
96.93
PromptFolio
95.24
75.64
94.78
59.02
95.58
95.41
85.95
96.73
94.29
98.66
97.59
96.82
FedLoCoOp
95.34
72.31
92.78
60.11
96.07
96.12
85,46
96,47
94.15
93.60
95.33
94.89
FedGalLoP
95.62
75.40
94.78
65.08
96.23
96.71
87.30
97.30
96.33
99.73
98.58
97.99
FedLAPT
92.36
66.54
89.02
48.38
94.07
92.07
80.41
77.28
84.61
86.40
86.86
83.79
FOCoOp
96.44
76.59
96.72
62.99
97.16
96.22
87.68
98.21
96.54
99.71
98.20
98.16
15UM
Tless [6] AUROE Comparson
(4) 19995 Comparison
Similariyy between Image Features ID/00D Prompts
Claa
Cam2-
Cls
Clss
resspp
Pomper
Figure 4: Detection comparison on CIFAR REAR-110 R-000.
Chs
PromodF
FP
Fed
ReILAPT
Ped011 IP
Promothalo
LoP
FOCoOp
oco
[k] CIFAR1 00 Comparison
TryttrageNee Comparison
93.0
Figure 6: Similarity heatmap of FOCoOp.
63.0
73.0
Peformance/
20.2
prompt matrix shows the highest similarities, suggesting intra-class alignment and clear class separation. Meanwhile, the similarities of OOD prompts are notably lower than those of ID prompts, further indicating clear distribution separation.
00.2
63.0
50.2
40.0
Number of 2hots
Number of shats
Figure 5: The effect of different number of shots.
Domain Generalization. We study the domain generaliza- tion on DomainNet and Office in Tab. 4. FOCoOp achieves state-of-the-art domain generalization, yet FedGalLoP and PromptFolio also perform competitively, indicating that all methods effectively leverage the transferability of pretrained VLMs for Peature-shift distributions. Compared with label- shifts, heterogeneity impact slightly on all methods, making PromptFL performs well in domain generalization. Among the methods evaluated, FedLAPT exhibits the lowest gen- eralization performance, suggesting that FedLAPT less effective in leveraging aggosiie features.
global view. GOC has a more significant per- formance drop, illustrating that simply applying bi-level distribution robustness optimization is inferior in heteroge- neous data. This also verified the importance of maintaining consistency of ID global prompts and OOD prompts.
Visualization. In Fig. 6, we model FOCoOp on Cifar10 (10 ID prompts and OOD prompts, respectively). and sample 100 images per class to compute the average of similari- ties between images and prompts. The diagonal of the ID
8

---

FOCoOp: Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
CIFAR-I00
b), coefficient of updating global prompts [0,0.2, 0.4, 0.6, 0.8, 1.0) in Fig. 7(c-d), BDRO hyperpa- rameters 71 10.1, 3, 4,5, 10] in Fig. 7(e), and 72 10.1,1,244 4, 6, 8, 10) in Fig. 7(f) on ACC and FPR95 across CIFAR-100 and TinylmageNer datasets. We substitute ViT- B/16 with ResNet50 (He et al., 2016) to verify in Appendix Tab. We can find that: (1) FOCoOp outperforms in all cases, and we select 8 shots per class for prompt learning as it approximates convergence. (2) Increasing p initially improves accuracy but degrades performance beyond op- timal threshold p = 2, which reflects the same behavior in FPR95. (3) a influences the calibration of ID global prompts with seemly OOD prompts, where different datasets have different turning points. (4) The effects of т and 72 are different, i... Ti can find the trade-off points where the op- timal transport regularization is the best for classification, while T2 increasingly encourages uncertainty exploration to have better detection as the value grows. (5) The results validate that F FOCoOp maintains strong generalization and detection capabilities even with smaller models.
ACC (%))
90
FPR95 (%6
TinylmageNNet
CIFAR-100
TinylmageNet
20
0
0.1
0.2 0.4 0.6
0.8
0
0.1
0.2
0.3
0.6 0.8
(a) Accuracy effect of
(b) FPR95 effect of
92.5
FPRDO (%)
25
ACC (%6
100
CIFAR-100 TinylmageNet
90.0
TinylmageRee
20
87.5
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8 1.0
(c) Accuracy effect of a
(d) FPR95 effect of
92.5
ACC (%6
CIFAR-100 ThyymmggeNet
1%%1
90.0
CIFAR-100
20
HnylmageNet
87.5
0.1
1
2
3
4
5
10
0.1
1
2
8
10
Ti
T2
(e) Accuracy effect
m
(f) FPR95 effect of 72
Figure 7: Hyperparameter sensitivity studies.
5. Conclusion
In this work, propose FOCoOp, Federated OOD-aware Context Optimization framework to enhance robustness and performance in Federated Prompt Learning for VLMs. FOCoOp integrates bi-level OOD separations to improve class-matching and distribution separations, and global-view OOD consistency to align and calibrate global and OOD prompts via cemi-unbalanced optimal transport. Extensive experiments across fifteen datasets demonstrate that FO- CoOp effectively handles heterogeneous distributions, im- proves OOD detection, and achieves -tate-of-the-- art per- formance without compromising generalization, making it a promising solution for federated vision- Lasgua-an model learning.
Method capability on other datasets. comprehensively evaluate the OOD robustness FPL methods, we present the average ID-C covariate shift generalization in Fig. 3 and OOD semantic shift detection in Fig. 4 and Appendix Fig. 9. Additionally, we provide detailed numerical re- sults in Tables 11-14 Appendix. FOCo0p demonstrates the strongest generalization and detection performance, con- sistently achieving the highest accuracy and 001) robust- ness across different covariate-shifts and semantic-shiff on CIFAR-100 and TinyImageNet. FedGalLoP performs well in ID-C generalization, while suffering from detection due to lacking consistent discrimination among clients. FedOTP and FedPGP strike a balance between generalization and robustness. In contrast, FedLAPP and PromptFL. exhibit significant performance degradation, hindering their ability to consistently align image and contextual prompts while compromising OOD robustness.
Acknowledgement
This research was supported by Zhejiang Provincial Natural Science Foundation of China under Grant No. LZYQ25F020002, and the National Natural Science Foundation of China No. 62172362.
Table 5: Pathological Nve-laer (10 clients, K=10)
Dataset
CIFAR-100
TinyImageNet FPR95
Method(3) (5)
ACC
CACC
FPR95
AUROC
ACC
CACC
AUROC
Impact Statement
PromptFolio
51,96
47.47
50.14
86.45
44.70
30.61
57.57
83:03
FedOTP
55.34
50.98
61.38
74.92
43.61
28.98
72.67
73.86
This paper provides new insight into maintaining perfor- mance well as enhancing OOD robustness for federated
FedLoCoOp
17.03
12.09
93.36
$2.66
9.44
4.96
93.63
$6.40
POCoOp
55.92
28.83
95.54
87.75
prompt learning on pretrained vision-language models. This aligns with the development of trustworthy machine learn- ing in privacy and robustness. We propose FOCoOp are comprehensively studied with extensive real-world datasets, validating its effectiveness in both performance and robust- ness.
Sensitivity Analysis of Hyperparameters. We present comprehensive hyperparameter sensitivity analysis, which examines the effect of shot numbers in {1,2, 8, 16)
in Fig. 5. effect of ID global and local prompt fu- sion p (0,0.1,0.22 0.2, 0.3, 0.4, 0.5, 0.6, 0.8) in Fig. 7(a-
9

---

FOCoOp:
Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
References
A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 9646-9661 PMLR, 21-27 Jul 2024. URI.httss: /ttpss//prooceeeeeee ngs. ml press/ v235 cui24c. html.
Armijo, L. Minimization of functions having lipschitz con- tinuous first partial derivatives. Pacific Journal of mathe- matics, 16( 16(1):1-3, 1966.
Bai, H., Canal, G., Du, X., Kwon, J., Nowak, R. D., and Li, Y. Feed two birds with scone: Exploiting wild data for both out-of-distribution generalization and detection. In International Conference on Machine Learning, pp. 1454-1471. PMLR, 2023.
Dong, X., Zhang, S. Q. Li, A., and Kung, H. Spherefed: Hyperspherical federated learning. In ECCV, pp. 165- 184. Springer, 2022.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Bai, S., Zhang, J., Guo, S., Li, S., Guo, J., Hou, J., Han, T., and Lu, X. Diprompt: Disentangled prompt tuning for multiple latent domain generalization in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27284 27293, 2024a.
Fei-Fei, L., Fergus, R., and Perona, Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, pp. 178-178. IEEE, 2004.
Bai, Y., Han, Z., Cao, B., Jiang, X., Hu, Q., and Zhang, C. Id-like prompt learning for few-shot out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17480- 17489, 2024b.
Fellbaum, C. Wordnet: An electronic lexical database. MIT Press google schola, 2:678-686, 1998.
Feng. C.-M., Li, B., Xu, X., Liu, Y., Fu, H., and Zuo, W. Learning federated visual prompt in null space for mri reconstruction. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8064-8073, 2023.
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101- mining discriminative components with random forests. In Computer vision-ECCV 2014: 13th European con- ference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 446-461. Springer, 2014.
Gong, B., Shi, Y., Sha, F., and Grauman, Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference computer vision and pattern recognition, pp. 2066-2066777 IEEE, 2012.
Bui, T. A., Le. T., Tran, Q. Zhao, H., and Phung, D. A unified wasserstein distributional robustness framework for adversarial training. arXiv preprint 22Xiv:2202.13437, 2022.
Guo, T., Guo. S., and Wang, J. Pfedprompt: Learning per- sonalized prompt for lasion-language models in federated learning. In Proceedings of the ACM Web Conference 2023, pp. 1364-1374, 2023a.
Chen, H.-Y. and Chao, W.-L. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations 2021.
Guo, T., Guo, S., Wang. J., Tang, X., and Xu, W. Promptil: Let federated participants cooperatively learn prompts instead of fedele-federratd learning in age of foundation model. JEEE Transactions on Mobile Computing, 2023b.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEFE conference computer vision and pattern recognition, pp. 3606-3613, 2014a.
Guo, Y., Guo, K., Cao, X., Wu, T., and Chang, Out- of-distribution generalization of federated learning via implicit invariant relationships. In International Confer- ence Machine Learning, pp. 11905-11933. PMLR. 2023c.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014b.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Clarkson, K. L. Coresets, sparse greedy approximation, and the frank-wolfe algorithm. ACM Transactions on Algorithms (TALG), 6(4):1-30, 2010.
Cui, T., Li, H., Wang. J., and Shi, Y. Harmonizing gener- alization and personalization in federated prompt learn- ing. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller,
Hendrycks, D. and Dietterich, T. G. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint arXiv: / 2018.
10

---

FOCoOp: Enhancing Out-of-Distribation Robustness in Federated Prompt Learning for Vision-Language Models
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference Learning Rep- resentations, 2016.
learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12151- 12161, 2024.
Li, Q.. He, B., and Song, D. Model-contrastive federated learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10713- 10722, 2021.
Huang. W., Ye, M., Shi, Z., Wan, G., Li, H., Du, B., and Yang. Q. Federated learning for generalization, robust- ness, fairness: A survey and benchmark. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 250, 2020.
Jaggi, M. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on ma- chine learning, pp. 427-435. PMLR, 2013.
Li, X. and Wang, J. Position paper: Assessing robustness, privacy, and fairness in federated learning integrated with foundation models. arXiv preprint arXiv:2402.01857 2024.
Jia, C., Yang, Y., Xia, Y. Chen, Y.TT., Parekh, Z., Pham, H.,, Le, Q.. Sung, Y.H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021.
Liao, X., Liu, W., Chen, C., Zhou, P., Zhu, H., Tan, Y., Wang, J., and Qi, Y. Hyperfed: hyperbolic prototypes exploration with consistent aggregation for non-iid data in federated learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI '23, 2023. ISBN 978-1-956792-03-44
Jiang, L. and Lin, T. Test-time robust personalization for federated learning. In The Eleventh International Confer- ence on Learning Representations, 2022.
Jiang, X., Liu, F., Fang, 7., Chen, H., Liu, T., Zheng, F., and Han, B. Negative label guided ood detection with pretrained vision-language models. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Liao, X., Liu, W., Chen, C., Zhou, P., Yu, F., Zhu, H., Yao, B., Wang, T.., Zheng, X., and Tan, Y. Rethinking the representation in federated unsupervised learning with non-iid data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patler Recognition, pp. 22841- 22850, 2024a.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averag- ing for federated learning. In International conference on machine learning, pp. 5132-5143. PMLR, 2020.
Liao, X., Liu, W., Zhou, P., Yu, F., Xu, Wang. J., Wang, W., Chen, C., and Zheng, X. Foogd: Federated collab- oration for both nut-of-distribution generalization and detection. arXiv preprint arXiv:2410.11397, 2024b.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10055 2022.
Liu, Q., Chen, C., Qin, J., Dou, Q., and Heng, P.-A. Feddg: Federated domain generalization mentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1013- 1023, 2021.
medical image seg-
Lafon, M. Ramzi, E., Rambour, C., Audebert, N., and Thome, N. Gallop: Learning global and local prompts for vision-language models. In European Conference on Springer, 2025.
Liu, W., Zheng, X., Chen, C., Xu, J., Liao, X., Wang, F,, Tan, Y., and Ong, Y.-S. Reducing item discrepancy via differentially private robust embedding alignment for srivacy-presevving cross domain recommendation. In Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 32455-32470. PMLR, 21-27 27 Jul 2024.
Computer Vision, pp.
Le, Y. and Yang. X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Liu, Y., Qi, L., Mao, X., Liu, W., Wang, F,, Xu, X., Zhang, X., Dou, W., Zhou, X., and Beheshti, A. Hyperbolic vari- ational graph auto-encoder for next poi recommendation.
Li, H., Huang, W., Wang. J., and Shi, Y. Global and local prompts cooperation via optimal transport for federated

---

FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models
In Proceedings of the ACM on Web Conference 2025, pp. 3267-3275, 2025a.
Qi, L., Liu, Y., Liu, W., Pei, S., Xu, X., Zhang. X., Wang, Y., and Dou, W. Counterfactual user sequence synthe- sis augmented with continuous time dynamic preference modeling for sequential poi recommendation. In Proceed- ings of the ThirttyThine International Joint Conference on Artificial Intelligence (IJCAI), pp. 2306-2314, 2024.
Liu, Y., Zhai, S., Du, M., Chen, Y., Cao, T., Gao, H., Wang, C., Li, X., Wang, K., Fang, J., et al. Guardreasoner- vl: Safeguarding vlms via reinforeed reasoning. arXiv preprint arXiv:2505. 1049, 2025b.
Qiu, C.,, Li, X., Mummadi, C. K., Ganesh, M. R., Li, Z., Peng, L.,, and Lin, W.-Y. Text-driven prompt generation for vision- -sioonlanguage models in federated learning. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS, 2023.
Mayilvahanan, P., Wiedemer, T., Rusak, E., Bethge, M., and Brendel, W. Does clip's generalization performance mainly stem from high train-test similarity? arXiv preprint ar Xiv:2231000956,, 2023.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. Communication-effiieen learning of deep networks from decentralized data. Artificial intelli- gence and statisties, pp. 1273-1282. PMLR, 2017.
Qu, Z., Li, X., Duan, R., Liu, Y., Tang, B., and Lu, Z. Gen- eralized federated learning via sharpness aware minimiza- tion. In International Conference on Machine Learning, pp. 18250-18280 PMLR, 2022.
Ming, Y., Cai, Z., Gu, J., Sun, Y., Li, W., and Li, Y. Delving into out-of-distribution detection with lasounlanguage representations. Advances in neural information process- ing systems, 35:35087-35100 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Miyai, A., Yu, Q., Irie, G., and Aizawa, K. Locoop: Few- detection via prompt learning. Advances in Neural Information Processing Systems, 36, 2024.
shot
Shi, J., Zheng, S.. Yin, X., Lu, Y., Xie. Y., and Qu, Y. Clip-guided federated learning on heterogeneity and long- tailed data. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 38, pp. 14955-14963, 2024.
Nguyen, A. T., Torr, P., and Lim, S. N. Fedsr: A simple and effective domain generalization method for federated learning. Advances in Neural Information Processing Systems, 35:38831-3882 2022a.
Shu, Y., Guo, x., Wu, J., Wang, X., Wang, J., and Long, M. Clipood: Generalizing clip to out-of-disttibutions. In International Conference on Machine Learning, pp. 31716-31731. PMLR, 2023.
Nguyen, A. T., Torr, P., and Lim, S. N. Fedsr: A simple and effective domain generalization method for federated learning. Advances Neural Information Processing Systems, 35:38831-38844 2022b.
Sinha, A., Namkoong, H., Volpi. R., and Duchi, J. Cer- tifying some distributional robustness with principled adversarial training. arXiv preprint 17Xvv:171011057 2017.
Nilsback, M.-E, and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Tan, Y., Chen, C., Zhuang, W., Dong, X., Lyu, L., and Long, G. Is heterogeneity notorious? taming heterogeneity to handle test-time shift in federated learning. Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Pan, B., Huang, W., and Shi, Y. Federated learning from vision-language foundation models: Theoretical analysis and method. arXiv preprint arXiv:2409. 19610, 2024.
Tang, X., Guo, S., and Zhang, Exploiting personalized invariance for better out-of-distribution generalization in federated learning, arXiv preprint arXiv:2211.11245 2022.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415,, 2019.
12

---

FOCoOp: Enhancing Out-of-Distribation Robustness in Federated Prompt Learning for Vision-Language Models
Wang, F., Chen, C., Liu, W., Fan, T., Liao, X., Tan, Y., Qi, L., and Zheng, X. Ce-refr: Robust counterfactual regression for consensus-enabled treatment effect estimation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- 3013-3023, 2024a.
Yu, F,, Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint
2015.
Yu, S., Hong, J., Wang, H., Wang. Z., and Zhou, J. Turning the curse of heterogeneity in federated learning into blessing for out-of-distributtion detection. In 2023 Inter- national Conference on Learning Representations, 2023.
Wang, F., Qi, I.., Liu, W., Yu, B., Chen, J., and Xu, Y. Inter- and intra- similarity preserved counterfactual incentive effect estimation for recommendation systems. ACM Trans. Inf. Syst, 2025a. doi: 10. 145/3722104.
Yuan, H., Morningstar, W. R., Ning, L., and Singhal, K. What do we mean by generalization in federated learning? In International Conference Learning Representations, 2021.
Wang, H., Li, Y.. Yao, H., and Li, X. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1802-1812, 2023.
Zhang, Y., Li, J., Liu, L., and Qiang, W. Rethinking mis- alignment in vision-language model adaptation from causal perspective. In Advances in Neural Information Processing Systems, volume 37, pp. 39224-39224 2024.
Wang, Z. Outlier-robust distributionally robust optimization via unbalanced optimal transport. In Annual Conference on Neural Information Processing Systems, 2025.
Zhang, Y., Zhu, W., He, C., and Zhang. L. Lapt: Label-
Wang, Z., Shen, Z., He, Y., Sun, G., Wang, H., Lyu. L., and Li, A. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. Advances in Neural Information Processing Systems, volume 37, pp. 22513-22533 Curran Associates, Inc., 2024b.
driven automated prompt tuning for ood detection with Aision-language models. In European Conference Computer Vision, pp. 271-288 Springer, 2025.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6): 1452-144452144 2017.
Wang, Z., Shen, Y., Zavlanos, M. M., and Johansson, K. H. Outlier-robust distributionally robust optimization via un- balanced optimal transport. In The Tight Annual Conference on Neural Information Processing Systems, 2025b.
Zhou, K., Yang. J., Loy, C. C., and Liu, Learning to prompt for vision-language models. International Jour- nal of Computer Vision, 130(9):2337-2345) 2022.
Weng, P.-Y., Hoang, M., Nguyen, L.. Thai, M. T. Weng, L., and Hoang, N. Probabilistic federated prompt-tuning with non-iid and imbalanced data. Advances in Neural Information Processing Systems, 37:81 77:81133388958. 2024.
Zhou, P., Chen, C., Liu, W., Liao, X., Shen, W., Xu, J., Fu, Z., Wang, J., Wen, W., and Zheng, X. Fedgog: Federated graph duittf-distribution generalization with diffusion data exploration and latent embedding decorrelation. In Proceedings of the AAAI Conference Artificial Intelli- 2025a.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 1485-3492. IEFF, 2010.
gence, volume 39, pp.
Zhou, P., Chen, C., Liu, W., Liao, X., Yu, F., Fu, Z., Lou, X., Wen, W., Zheng, X., and Wang. J. Fedgf: Enhancing structural knowledge via graph factorization for federated graph learning. In Proceedings of the Eighteenth ACM In- ternational Conference on Web Search and Data Mining, pp. 448-456, 2025b.
Xu, P., Ehinger, K. A., Zhang, Y., Finkelstein, A., Kulka- mi, S. R., and Xiao, J. Turkergaze: Crowdsoureing saliency with webcam based eye tracking. arXiv preprint arXiv: 154.065504. 2015.
Yang, F.-E., Wang, C.-Y., and Wang. Y.C. F. Effi- cient model personalization in federated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19169919168, 2023.
Yang, J., Zhou, K., Li, Y., and Liu, Z. Generalized out-of- distribution detection: A survey. International Journal of Computer Vision, 132(12))5635-566 2024.
13

---

FOCoOp: Enhancing Out-of-Distribttoon Robustness in Federated Prompt Learning for Vision-Language Models
The supplemental materials consist of five sections: (A) related work details, (B) algorithms, (C) theoretical analysis of optimization, (D) datasets and implementation, and (R) additional experimental results.
A. Related Work
A.1. Federated Learning on VLMs
Federated learning (FL) models decentralized data in each client and aggregate client models for a global model at server (McMahan et al., 2017). Personalized federated learning emphasizes tailoring to diverse client data to preserve personalized performance through techniques such as regularization (Li et al., 2020; Karimireddy et al., 2020), contrastive learning (Li et al., 2021), and the decoupling of model parameters (Chen & Chao, 2021; Dong al., 2022), among others. However, conventional personalized federated learning methods requires the collaboration of client models with full parameter sets on the server, which becomes impractical model size grows due to scaling laws (Li et al., 2024; Cui et al., 2024). Federated prompt learning collaboratively adapts client data by tunable prompts rather than entire VLMs, which not only utilizes the generalization ability of pre-trained VLMs like CLIP to learn transferable representations, but also preserves data privacy through federated learning (Li et al., 2024; Guo et al., 2023a). PromptFL (Guo et al., 2023b) learns a unified prompt for all clients to enable federated learning. CLIP2FL (Shi et al., 2024) bridges server and client communication using CLIP, handling heterogeneous and long-tailed data. FedPR (Feng et al., 2023) designs federated visual prompts for domain-specific tasks like MRI reconstruction. FedOTP (Li et al., 2024) balances global alignment and personalization with an optimal transport optimization- baaedd approach. FedTPG (Qiu et al., 2023) and pFedPG (Yang et al., 2023) enhance generalization through prompt generation techniques. PFedPrompt (Guo et al., 2023a) adapts prompts for personalized federated learning. FedPGP (Cui et al., 2024) balances generalization and personalization via low-rank decomposition and CLIP guidance. FedFolio (Pan et al., 2024) further provides theoretical insights into trade-offs between generalization and personalization. While these methods improve federated prompt learning, these works overlook the OOD robustness issues, suffering from the trade-off between performance and talking OOD shifts.
A.2. OOD Robustness in Federated Learning
OOD robustness indicates the capability of model to discriminate distribution shifts, e.g., performance generalization for covariate shifts and outlier detection for semantic shifts (Hendrycks & Gimpel. 2016; Li & Wang. 2024; Huang et al., 2024). which is long-term issue but seldom studied in federated scenarios. Recent FL methods aim to enhance generalization by preserving invariant relationships between data and labels (Jiang & Lin, 2022; Tan et al., 2023; Tang et al., 2022), smoothing local loss landscapes (Qu et al., 2022), and capturing robust representations to handle heterogeneous distributions and adapt to unseen clients (Yuan et al., 2021; Nguyen et al., 2022a; Guo et al., 2023c; Liu et al., 2021). Meanwhile, FOSTER et al., 2023) learns a class conditioal generator to synthesize virtual external-class OoD samples and facilitate OOD detection in FL. for the first time. And FOOGD (Liao et al., 2024b) captures global distribution with score matching model, and simultaneously tackles OOD generalization and detection based on score function values. Nevertheless, these models are not scalable for large pretrained VLMs and fail to adapt the federated prompt learning to enhance OOD robustness for VLMs.
FOINER (Yu
A.3. OOD Robustness on Pretrained VLMs
The pretrained VLMs contain large-scale model parameters and provide transferrable representation for OOD generalization and zero-shot capabilities (Radford et al., 2021; Jia et al., 2021). However, VLMs rely heavily on pretrained textual-image matching distribution, causing the degradation of generalization and detection capabilities once the textual prompts are diverse and incorrect (Zhou et al., 2022; Mayilvahanan et al., 2023; Yang et al., 2024). CoOp (Zhou et al., 2022) proposes to learn the representation vector of prompt context words during adapting pretrained VLMs, enhancing the generalization on distribution shifts. Motivated by this, CLIPN (Wang et al., 2023) fineunes VLMs to generate negative prompts that access the probability of an OOD concept. Moreover, ID-Like (Bai et al., 2024b) extends pretrained VLMS to detect OOD data that are highly correlated with ID data. With the constraints of private data and data heterogeneity of FPL, it is further demanding to efficiently and consistently apply prompt tuning on pretrained VLMs to adapt decentralized data. This problem is still unresolved in existing work, since they cannot adapt decentralized data and detect OOD data in global view.
14

---

FOCoOp:
Enhancing Out-of-Distributinn Robustness in Federated Prompt Learning for Vision-Language Models
B. Algorithm
The overall algorithm of FOCoOp is in Algorithm The steps 1-12 are main procedure of communication server and clients with prompts. In step 9, clients execute local training improve prompt learning and build bi-level OOD separations, where the details are illustrated in steps 14-24. And the server will calibrate global prompts and OOD prompts in global view in step 1,, which enhances discriminations. Additionally, global-view OOD consistency as follows. By jointly utilizing Bi-level OOD robustness separation in local client modeling based on Algorithm 2, and global-view OOD robustness consistency in Algorithm 3, FOCoOp resolves the trade-off between performance and OOD robustness, at the same time.
provide the crucial algorithms for bi-level OOD separations and
Algorithm Training procedure of FOCoOp
Input: Batch size B, communication rounds T, number of clients K, local steps E, dataset D = UkE(K] Ds Output: context prompts, i.e., Tý. ( and Tj
1: Server executes():
2: Initialize prompts To (T k.0 and To with random distribution
for t = ...,. T
do
4: for each client
to K in parallel do 5: if ===00 0 then
6: Send (Th to client k
7: end if
8: Send prompts T. and Tç to client
9: TT + Client executes(k, {T.. })
end for
Calibrate prompts (T 1 to achieve global OOD consisteney by Algorithm 3
12: end for
13: return [T)
14: Client executes(k, :T
15: Assign prompts from server to local model (Te, Te. <- (T,, T?)
16: for each local epoch e = 1, 2..,
....., E do
for batch of samples (x vi.B) E D; do
Obtain the latent viusal representation for image data h Io(x
19:
Obtain the latent textual representations for prompts e, (tc, n.) with t. = p)tp p)t++ ptg, f. ē. (o nu)
20:
Compute the similarity scores Sh,, e) between visual representations h and textual representations 1C
and (ew Mi
21:
Optimize prompts to build bi-level OOD separations by Algorithm 2
22:
end for
23:
end for return 0 server
C. Theoretical Analysis
Definition C.1 (Optimal Transport Distance). The optimal transport distance is the distribution divergence between two probability masses P and Po, i.e.,
Don (P, Po) (Pm(P. Po) / c(L, t)dT
inf
(12)
s.t. = a. = b,
with (P,, ,, Po) is the couplings between P and Po. म and ny are marginals of π with assumption a 1 = bT 1, and c(t, t) short for the non-negative metric cost of samples P and t ~ Po-
Definition C.2 (Unbalanced Optimal Transport Distance). The optimal transport distance the distribution divergence
15

---

FOCoOp:
Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models
Algorithm 2 Training procedure of Bi-level Distributional Robustness Optimization
Input:
Batch size B,local steps E, dataset D
Ds
Output: and
Robust model parameters
Worst
case
global
prompts
and
OOD prompts, i.e.
T
To
1: Sample 21.8 Dr.
2: Initialize random noise for global prompts eg ~ ~N(( Ve {I,...,c]
3: Perturb global prompts Eg to + c
4: for Exploration step 1 N do
5: Ie? Vez aarg ming. - s {L((9) c(( 19)}] 7 [|]]
6: eg + eg + lrge, Ve {1, (L...... C}
7: end for
8: Estimate robust global prompts to via ig t 18 + +
9: Initialize random noise for global prompts Eg ~
10: Perturb OOD prompts &. t Eu
for Exploration step m
M do
12: Oeg Ve: arg ming. Qu supio { , {ン)}]
13: C + C + Iroe. Vu E f1....,U U}
14: end for
15: Estimate robust OOD prompts E + t +
16: Update parameters of 0' by computing the gradient of Eq. (30)
Algorithm Training procedure of Semi unbalanced optimal transport based prompt calibration
Input: prompt {TE, Tg, from participating client k Output:
Robust model parameters 0, global-view consistent global prompts and OOD prompts, i.e. T
T9
and
1: Concatenate OOD prompts from clients TRu + [T
1= 2: Aggregate global prompts from clients by class Eo >
De. t Vee(o] and obtain T in server k- En De,e
Seek the ummi-unbalnnced optimal transport ग between OOD prompts Tku and T
4: Estimate robust global prompts to via Eg. (8)
5: Compute the matching probabilities π" 1x to figure out top-M as seemly-OOD prompts
6: Update global prompts T with seemly-OOD prompts by Eq. (10)
7: Update OOD prompts TH by Eq. (11) where top-U prompts satisfying negative matching probabilities - 1kG
between two probability masses Q and Qu i.e.
/ c(i, ((i, t)dy + m Dxr((1ll))) + (rDl1l))))0)),
inf Duor(Q,Qo) 10
(13)
where 7(, Qo) is the couplings between Q and Qo Yi and 72 are marginals 7, μi and μl are regularization coefficient, c(f, t) is short for the neg-negative metric cost of samples and Qo. In terms of (Wang al., 2025b), considering
Ni = 0, Eq. (13) can be rewritten special case:
{ ((t)))dr (DKL(Q110))
inf
(14)
Theorem C.3. Suppose that the optimal dual variable of and T5 are strictly positive, bi-level separation loss L(x, O t! 19 1o) Eq. (5) concave and differentiable, Bi-level distribution robust optimization Eq. (6) can be
16

---

FOCoOp: Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
solved via:
i arg min Eo Po
sup {L(E)) - ( {0)} tnmP
to
t
arg min E t
Qo
sup {T((e) T2e (8", (8),
(15)
f(x, 0, to 8 TIc(t9, (o) T2C (t", t²)
EEDRO
T2fa log ,
exp
T2f
where yt is the regularization of KL divergence, L(t()) is short for C(x, 0, t', 19, 1o) optimizing ((-)
Proof. The OOD loss can be written as
C B b=1
L(an, 0, 0,
(16)
ExED |- log pvy c(x))((w 1[x)],
B
with ply = c/z) = E
S(w.t.) S(z,t.) - S(a)
and p(yD = 1|x) = Er S(x.te) S(z,12)
De S(w.t.)
Σ Σ4 S(,,C2)
Ees S(e.t))+ S(x,t))
The corresponding distribution robust optimization objective is rewritten as:
CBDRO(œ, t, t0, t,,t") = inf
0, 0,0, 4, 8)),
sup PEP RE?
P = E D: Dor P, Po) S m
(17)
s.t.
Q = eQ D: Duor (Q, Qo) 12
where Dor and Duor are optimal transport distance and unbalanced optimal transport distance defined in Appendix C.2 and C.I.
First, we expand Eq, (17) with Lagrangian multipliers:
E Eo [C 0, 80 80] TI (Dor(P, (0)0-m))-- - Qo) 112)
LADRO inf sup PEP,QEQ
722,7220
inf / (((0,19)77(19, (0) -m11
inf sup PEP,OEQ
E Biapiin [L 0, 8 [0]]
72
inf (f () ))v(Q.Qu)
-7
7/2
(18)
Then we optimize for the worst-case distribution of global prompts t that are independent with ood prompts to, meaning that we can treat Q as constant. By denoting C(x,, 0, f', ta, (²) as L(E"), if L(t9) is upper semi-continuous, we can obtain the dual form of Eq. (17) with regarding to (Bui et al., 2022; Sinha al., 2017), which can be formulated as:
inf sup Ei [L(*)] T. Dor Po) m
1220 PEP
inf sup Binn [(t9)] Ti P c(E", ml
n20 Per
(19)
inf 2020
Tilli + E
sup E Ein (L(89) TI c/6 c((9,
PEP
= =EaSP
sup {L(E)) - Tic(19, (o(1,,19)}
iomp
17

---

FOCoOp: Enhancing Out-of-Distributinn Robustness in Federated Prompt Learning for Vision-Language Models
where the last equation holds by relaxing T1 2 0 (Sinha et al., 2017). Thus we can obtain the worst case of global prompts via
arg min Po a
sup iamp
{L(E") - Tic(t, {9)}
(20)
t9
Similarly, we fix P optimize Q by denoting ((xr, 0,0,, io (0) L((9), below:
inf sup F. (e.=Q((((9)) - (Dvor(Q,Q0) -
1220 QEQ
E [ - Tel le (t)) + HDxx (Q1120)
inf 7272 + sup QeQ
220
Q (to)
(21)
inf T2 +
sup @EQr((PPP))
E. Beo Bite [((") - 720 (t",t%) T2222
7:20
Qo (to)
sup {L(E0) T2C (80, - T2 ( iun
inf
T2 + sup QeQ
E
22
Q0 (8o)
Let 12 Q(2))
with the probability constraint [R(2]] = 1, s(Po) sup {T(()) T2e (t0, 10) the sencond term of
(6) Eq. (21) can rewritten as:
sup (I(P²) - T2e ((), - T2 log (t)
J = Fjon
Qo (io)
iun
Q (to)
sup {L(E)) T2C (1,,1)} - T2/4 (80)
@ (to)
Qo
log
(22)
Q0 (fo)
Qo (to)
Qo (8o)
jnn
[2.f(t)) Topglogg1]
8.1. =1
The Lagrangian expansion
inf 20,
EvenQ (t)(tt)) - T2wDloog2 (E[2) 1),
(23)
where v is the multiplier. Dy deviating.
aJ = f(8o) - T2 Q + == 0.
(24)
0N
The optimal 2² is realized
2² = =exp f(i9) - 1).
(25)
T2f4
Then we take the optimal 2 back to Eq. (23), we can achieve:
[(to) -v
J
inf 220,220,0200
T2 inpo
exp(
-1) +
T2f1
(26)
f(t²)
T2f P2pe
1)BimPo
expl
+ v.
T2
T2f1
Since v² holds when its gradient equals 0, i.e.
0J = exp (-1") T2f
f(t))
-1
E
Bxx
exp
+ +1
(27)
0v
Tzfl
18

---

FOCoOp:
Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
exp [(E))
We have solution //" = T2f log P(w)
72 and finally:
f(t²)
min J max 72 log Po(()) 72p20 tm
exp
(28)
T2f4
Finally, it achieves supremum of Q P:
inf sup Ega (E--Q(((9)) (Qo Qo) - 1/2
m220 QEQ
(29)
f(t²)
inf Tell + sup T2 log P expl n2
QEC
T2f
Remind that the model parameters are independent with T1 72 and μr, we can treat them as hyperparameters with positive values optimizing Eq. (17), and finally update model with prompts by the gradient loss
C(x, 0, 20, (0) 0,,,
cit( (²) T2e (8), 19)
BDRO T2f log Eig f
exp(
(30)
T2/4
Similarly, obtain the worst case OOD prompts via
j
arg min Qo
-20 sup (I(f)) T20 (t, (t,,
(31)
t
Worth to mention that, the values of ?) and 72 are constants in the optimization, which is directly controlled via and respectively.
Theorem C.4. Semi- Unbalanced Optimal Transport Optimization of Eq. (8) can be solved by Frank- Wolfe Algorithm (Clark- son, 2010; Jaggi, 2013), which ueraatiely updates π i+1 = T 3( 8¹) with step size 3 742 following Armijo condition (Armijo, 1966) and optimal directions' satisfying
if d² arg min V JsseiiUrT (Tt))
(32)
0,
otherwise.
(6). 0,
c= 0
with initialization so
otherwise.
Proof. Now we expand semi-unbalanced optimal transport below:
(C, + XKL(T T kkulla)
min JSemiUOT
#
8.1.
TIE = b, Fej 2 0
KU
KU
ΣKU Tej
KU
Σ
Tej Caf A
Σ
Rej
log
Σ m=1
Tej + ac
(33)
ac
Tej bi Tej
{JE[KU]-
Then optimize the problem via Frank- Wolfe algorithm (Clarkson, 2010; Jaggi, 2013), i.c.
> DKe Tem
arg min l
ΣΣ
(
ΣΣ
C.j
log
Ge
(34)
s.t.
Rej
bj
0.
19

---

FOCoOp: Enhancing Robustness in Federated Prompt Learning for Vision-Language Models
The solution be assigned, i.e.,
KU Em=
Tem ae
arg min
Coj + T log
(35)
Notably, Eq. (35) is a linear function, where the minimum holds on the minimal values of VJ (m%)) Since Tej 2 0,
VJ (mTE)) [Ce + log 2m=1 Femm de is the monotonically increasing function whose minimal value is determined via
columne-wise, thus we can achieve:
bj: 80
if c' c arg min V SemitOOT (T'))
(36)
0,
otherwise.
Then update π via
n ++| = -1' + _Bss,
(37)
where गे is the value of in last updating iteration and μ is the step-size which can be updated with Armijo linear search methods (Armijo, 1966), e.g., B 1/(1+2). In the beginning, we can assign s0 and π will penalize the wrong guess and adjust the optimal direction iteratively.
69.
c= 0
our initial point,
0.
otherwise
D. Datasets and Implementation Details
Datasets. We study the OOD robustness of federated prompt learning on fifteen datasets: CIFAR-100 (Krizhevsky et al., 2009) and TinylmageNet (Le & Yang, 2015) for both generalization and detection, Food101 (Bossard et al., 2014), DTD (Cimpoi et al., 2014a), Caltech101 (Fei-Fei et al., 2004), Flowers (Nilshack & Zisserman, 2008), and OxfordPet (Parkhi et al., 2012) for label shift generalization, DomainNet (Peng et al., 2019), Office-Caltechl0 (Gong et al., 2012), and PACS (Li et al., 2017) for feature-shift (domain) generalization, CIFAR- 100-C, TinylmageNet-C (Hendrycks & Dietterich, 2018) for covariate-shiff generalization, as well as Places365 (Zhou et al., 2017). Texture (Cimpoi et al., 2014b), iSUN (Xu et al., 2015), LSUN-C and LSUN-R (Yu et al., 2015) for detection. We provide the summary of data in Tab. 6. In terms of maintaining performance and OOD robustness, we simulate heterogeneous distribution following both Dirichlet and Pathlogical settings (MeMahan al., 2017; Li et al., 2020) CIFAR-100 (Krizhevsky al., 2009) and TinylmageNet (Le & Yang, 2015) as conventional work does (Liao et al., 2024b). We test the generaliazation based CIFAR- (Hendrycks & Dietterich, 2018) and TinylmageNet-C (Le & Yang, 2015). Meanwhile, we study on iNaturalist (Van Horn et al., 201 8), iSUN (Xiao et al., 2010), Place (Zhou et al., 2017), and Textures (Cimpoi et al., 2014b), following existing CLIP-based OOD detection methods (Wang et al., 2023; Miyai et al., 2024). To widely evaluate OOD generalization and detection, we follow previous work of federated prompt learning (Cui et al., 2024; Guo al., 2023b;a), to study (1) heterogeneous label shift generalization on Food101 (Bossard al., 2014), DTD (Cimpoi et al., 2014a), Caltech101 (Fei-Fei et al., 2004), Flowers (Nilsback & Zisserman, 2008), and OxfordPet (Parkhi al., 2012) predict the accuracy of personalization following pathological heterogeneity, and (2) feature shift domain generalization on DomainNet (Peng et al., 2019), and Office-Caltech 10 (Gong et al., 2012), by lave--oee-domanne
validation strategy (Nguyen et al., 2022b). Specifically, for N 1 domains of one dataset, we train each client with distinct domain data, and test its model generalization on the whole target data of remaining one domain.
Comparison Methods. We categorize the comparison methods into two types, i.e., (1) Existing federated prompt learning methods for VLMs generalization: pFedprompt (Guo al., 2023a), PromptFL (Guo et al., 2023b), FedOTP (Li et al., 2024), FedPGP (Cui et al., 2024), PromptFolio (Pan al., 2024), and (2) Adapting existing centralized OOD robustness methods for federated scenarios: FedGalLoP (Lafon et al., 2025), FedLoCoOp (Miyai et al., 2024), and FedLAPT (Zhang et al., 2025). We select baselines from the state-of-the-arr (SOTA) personalized federated lcarning (PFL) methods and centralized prompt-based based OOD detection methods. For centralized prompt-based OOD baselines, we choose GalLop and LAPT due to their strong performance in both generalization and OOD detection. We also include LoCoOp, as Gall.p is proposed as an improvement over
PromptFL (Guo et al., 2023b) replaces the federated model training with the federated prompt training, accelerating both the local training and the global aggregation.
20

---

FOCoOp:
Enhancing Out-of-Distributinn Robustness in Federated Prompt Learning for Vision-Language Models
Dataset
Classes
Train
Test
Domains
Task
CIFARIO0 (Krizhevsky al., 2009)
100
50,000
10,000
Generalization and Delection
TinylmageNer (Le & Yang, 2015)
200
100,000
10,000
Food101 (Bossard et al., 2014)
101
50,500
30,300
DTD (Cimpui et al., 2014a)
47
2,820
1,692
Caltech101 (Fei-Fei al., 2004)
100
4,128
2,465
Label Shift Generalization
Flowers (Nilsback Zisserman, 2008)
102
4.093
2,463
OxfordPet (Parkhi et al., 2012)
37
2,944
3,669
DomainNet (Peng al., 2019)
10
18,278
4,573
Office-Caltech10 (Gong et al., 2012)
10
2,025
508
4
Feature Shift (Domain) Generalization
CIFAR-100-C (Hendryeks & Dietterich, 2018)
100
50,000
10,000
Covariatc-Shift Generalization
Tiny Ieayeee-geetCCCCC (Hendrycks & Dietterich, 2018)
200
100,000
10,000
Places365 (Zhou et al., 2017)
434
18,000,000
36,000
Texture (Cimpoi et al., 2014h)
47
2,820
1,692
iSUN (Xu al., 2015)
813
50,000
12,000
Detection
LSUN-C (Yu et al., 2015)
10 50,000
10,000
LSUN-R (Yu et al., 2015)
10
50,000
10,000
Table Statistical details of datasets used in experiments.
FedOTP (Li et al., 2024) provides each client with a global prompt and local prompt and utilizes unbalanced Optimal Transport to align local visual features with these prompts.
FedPGP (Cui et al., 2024) uses low-rank decomposition to adapt global prompts to heterogeneous local distributions and integrate an extra contrastive loss, considering both personalization and generalization.
PromptFolio (Pan et al., 2024) analyzes via feature learning theory and combines global and local prompts into a prompt portfolio to balance generalization and personalization.
FedGalLoP is federated version of GalLoP (Lafon et al., 2025). GalloP learns multiple diverse prompts leveraging both global and local visual features, enforcing prompt diversity using the "prompt dropout" technique.
FedLoCoOp is a federated version of LoCoOp (Miyai et al., 2024). LoCoOp uses local regularization to minimize ID-irrelevant nuisances in CLIP features, improving the separation between ID and OOD classes
FedLAPT is a federated version of LAPT (Zhang al., 2025). LAPT reduces the need for manual prompt engineering by automatically generating distribution-awen aware prompts.
Implementation Details and Evaluation Metrics. We conduct experiments on ViT-B/16 (Dosovitskiy, 2020) CLIP models. To study the heterogeneity generalization on CIFAR- BIFAA-100/Tinylmaeeht datasets, we simulate both cross-device and cross-silo scenarios. That is, we local training epoch E 2, communication round T 25, and the number of clients K 10 for fully participation. While in cross-device setting, we choose local training epochs F. 2, communication rounds 7 = 100. and K 100 for 10% participation. To obtain fair comparisons, all comparison methods are tuned for converging using their best syper-parametery. and we report the average of the results from three random seeds. We set the learnable prompt vectors with length as 16, emhedding size as 512, class token position as 'end', and random initialization. We choose prompt per class for both local and global ID prompts, and 100 OOD prompts in total. We report the average Top-1 accuracies for generalization of ID (ACC1) and ID-C (CACCt). We compute maximum concept matching (MCM) (Ming al., 2022) as OOD detection score, which based on similarity between textual features and image features. Based MCM, we report the standard metrics used for OOD detection, i.e., AUROC (1) and FPR95 (L) (Yang al., 2024).
E. Additional Experimental Results
In this section, we report the results on CIFAR- 100 and TinyImageNet under different Dirichlet distributions (Tables 6-7), which are consistent with Tables 1-3 and further verify OOD robustness. Table 4 has been split into Tables 8 and 9 for
21

---

FOCoOp: Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
Table 7: Main results of federated prompt learning on CIFAR-IOM with different Dirichlet distributions (K = 10).
a
0.1
a 0.5
a
5.0
Methads
ACC
CACC
FPR95
AUROC
ACC
CACC
FPR95
AUROC
ACC
CACC
FPR95
AUROC
PromptFL
71.22
67.55
76.58
72.20
75.65
71.52
82.13
69.65
74.92
71.37
79.52
74.25
FedOTP
76.81
73.50
61.88
79.14
68.43
65.67
73.78
73.45
66.20
63.16
77.73
71.15
FedPGP
76.77
72.55
74.81
74.45
72.95
69.25
83.65
71.37
73.01
69.15
82.57
72.65
ProiptFolio
80.07
76.89
65.30
77.95
75.98
71.98
78.61
71.44
74.19
70.60
79.64
72.74
FedLoCoOp
67.87
63.70
76.81
70.40
74.44
70.35
73.28
72.56
74.87
70.98
74.82
73.72
FedGalLoP
80.53
77.61
60.72
82.66
75.87
72.85
68.72
79.66
74.32
71.14
72.72
79.13
FedLAPT
61.20
$7.54
80.28
69.97
59.41
56.33
81.97
66.73
60.03
56.29
80.13
68.42
FOCcOp
82.42
78.52
46.56
86.98
77.71
73.59
54.26
83,40
77.66
73.59
51.02
83.22
-w/o-BOS
79.18
76.04
54.30
82.34
74.39
70.55
58.40
81.27
75.09
71.76
54.92
81,64
-woo-GOC
78.66
75.99
53.97
82.56
74.78
70.88
57.83
81.55
75.04
71.50
55.20
81,85
Table 8: Main results federated prompt learning on IinylmageNet with different Dirichlet distributions (K 10).
0.1
a
0.5
a
5.0
Methods
ACC
CACC
FPR95
AUROC
ACC
CACC
FPR9S
AUROC
ACC
CACC
FPR95
AUROC
PromptFL
70.29
63.41
69.38
73.09
73.22
65.57
68.91
75.39
73.09
66.27
71.04
74.32
FedOTP
70.36
64.49
71.82
69.75
63.32
57.87
78.70
63.59
60.94
55.44
81.14
62.17
FedPGP
74.10
67.45
66.65
75.01
71.97
64.77
68.68
73.83
70.92
64.82
70.73
74.00
PromptFolio
78.09
71.78
61.24
78.78
74.08
68.83
75.23
71.73
65.91
70.23
74.48
FedLoCoOp
65.97
58.72
70.47
72.24
71.92
63.64
67.97
75.24
72.51
64.13
65.78
75.13
FedGalL.oP
79.08
73.00
58.37
80.60
74.95
68.70
64.20
79.01
72.63
66.91
65.48
77.96
FodLAPT
59.82
54.97
75.34
69.74
59.66
54.72
74.98
70.72
59.88
54,58
75.93
69.34
FOCoOp 81.58
74.74
45.44
85.16
76.31
48.92
84.42
74.41
68.80
49.86
83.17
-w/o-BOS
79.49
72.45
51.44
82.38
74.96
68.33
53.29
81.12
73.81
66.80
54.73
80.45
-wo--GO
78.63
71.59
54.22
80.99
73.53
66.76
55.37
80.48
72.98
66.13
56.33
79.92
clarity. Tables 10-11 provide numerical results for Figure (OOD detection on various OUT datasets), while Tables 12-13 are numerical results corresponding to Figure 5 (generalization on different ID-C datasets). Figures 8-9 are also enlarged for better readability.
22

---

FOCoOp: Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
Table 9: Domain generalization on DomainNet.
Method Domain
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
average
PromptFL
96.28
74.84
95.81
60.28
96.77
96.36
86.72
FedOTP
91.03
61.52
86.98
53.04
91.16
89.73
78.91
FedPGP
93.67
75.07
93.62
58.09
95.44
95.48
85.23
PromptFolio
95.24
75.64
94.78
59.02
95.58
95.41
85.95
FedLoCoOp
95.34
72.31
92.78
60.11
96.07
96.12
85.46
FedGalLoP
95.62
75.40
94.78
65.08
96.23
96.71
87.30
FedLAPT
92.36
66.54
89.02
48.38
94.07
92.07
80.41
FOCoOp
96.44
76.59
96.72
62.99
97.16
96.22
87.68
Table 10: Domain generalization Office.
Method \ Domain
Amazon
Caltech
DSLR
WebCam
Avg
PromptFL
96.21
94.64
99.20
97.03
96.77
FedOTP
94.64
93.15
98.93
96.46
95.80
FedPGP
95.17
95.36
99.73
97.45
96.93
PromptFolio
96.73
94.29
98.66
97.59
96.82
FedLoCoOp
96.47
94.15
93.60
95.33
94.89
FedGalLoP
97.30
96.33
99.73
98.58
97.99
FedLAPT
77.28
84.61
86.40
86.86
83.79
FOCoOp
98.21
96.54
99.71
98.20
98.16
Table 11: Detection results on CIFAR- 100 non-overlap pathological heterogeneity.
Dataset
INaturalist
Texture
iSUN
Places
Method
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
PromptFL
78.23
64.51
84.51
68.28
65.57
78.55
90.24
53.50
FedOTP
43.20
85.26
38.22
87.56
35.61
89.44
42.92
87.22
FedPGP
49.01
82.21
51.57
84.68
44.62
86.78
63.78
77.91
PromptFolio
40.23
87.02
44.26
88.06
35.39
89.58
54.57
82.73
FedLoCoOp
94.26
51.63
77.59
68.76
61.92
82.80
87.01
60.45
FedGalLoP
34.41
89.29
41.45
89.64
36.54
90.15
51.93
83.22
FedLAPT
80.67
55.75
82.44
67.51
64.67
78.53
86.48
55.97
FOCoOp
18.02
94.75
15.97
96.47
23.71
93.71
17.23
96.03
23

---

FOCoOp: Enhancing
Robustness in Federated Prompt Learning for Vision-Language Models
INaturalist
Texture
iSUN
Places
(a) FPR95 Comparison
(b) AUROC Comparison
100
80
80
FPR95 (%)
60
AUROC (%)
60
40
20
20
PromptfL
FedOTP
FedPGP
PromptFolia
FedLoCoOp
FedGalLoP
FedLAPT
FocoOp
PromptFL
FedOTP
FedPGP
PromptFolia
FedLoCooP
FedGalLoP
FedLAPT
FOCoOp
Figure 8: Detection Comparison on CIFAR-100.
INaturalist
Texture
ISUN
Places
(a) FPR95 Comparison
(b) AUROC Comparison
100
80
80
FPR95 (%)
AUROC (%)
60
40
40
20
0
PromptfL
FedOTP TP
FedPGP
PromptFolio
FedLoCoOp
FedGalLop
FedLAPT
FocoOp
PromptFL
FedOTP TP
FedPGP
PromptFollo
FedLoCoOp
FedGalLoP
FedLAPT
FOCoOp
Figure 9: Detection comparison on TinylmageNet.
Table 12: Detection results on TinylmageNet non-overlap pathological heterogeneity.
Dataset
INaturalist
Texture
iSUN
Places
Method
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
PromptFL.
59.64
74.22
76.75
69.82
47.25
87.45
81.94
63.56
FedOTP
49.36
78.48
57.12
79.18
43.66
84.84
54.07
81.52
FedPGP
45.76
82.13
50.86
83.05
35.76
89.84
59.63
79.00
PromptFolio
35.44
87.78
42.84
87.34
28.58
91.98
45.34
85.68
FedLoCoOp
71.62
68.96
72.96
72.07
52.17
86.49
83.06
64.96
FedGalLoP
32.01
88.96
45.54
86.53
28.97
92.04
46.66
85.23
FedLAPT
70.46
63.41
73.46
70.15
37.75
90.71
72.89
67.58
FOCoOp
31.84
91.98
21.98
95.75
22.61
93.80
24.82
94.99
24

---

FOCoOp: Enhancing Out-of- Diiti---ittrrbttionn Robustness in Federated Prompt Learning for Vision-Language Models
CIFAR-100
FPR95 (%)
30
ACC (%)
90
TinylmageNet
CIFAR-100
85
TinylmageNet
20
0
0.1 0.2 0.3 0.4 0.6 0.8
0.1 0. 2 0.3 0.4 0.6 0.8
p
p
(a) Accuracy effect of
(b) FPR95 effect of
92.5
25
ACC (%)
100
FPR95 (%)
CIFAR-100 TinylmageNet
90.0
Tit
20
87.5
0.2
0.4
0.6
0.8
1.0
0.2 0.4
0.8 1.0 a
0.6
a
(d) FPR95 effect of
(c) Accuracy effect of
92.5
10F
ACC (%)
CIFAR 100
FPR95 (%)
25
TinylmageNet
90.0
TinylmageNet
87.5
20
0.1
1
2
3
4
10
0.1
1
2
3
4
5
10
T1
T1
(f) FPR95 effect of
(e) Accuracy effect of
25
92
FPR95 (%)
ACC (%)
CIFAR-100
90
CIFAR-100
TinylmageNet
20
TinylmageNet
88
0.1
1
2
4
6
8
10
0.1
1
2
4
6
8
10
T2
T2
gg)
Accuracy effect T2
(h) FPR95 effect of T2
Figure 10: Hyperparameter sensitivity studies.
25

---

FOCoOp: Enhancing Dut-of-Distribuute Robustness in Federated Prompt Learning for Vision-Language Models
6k
0842
4615
77.05
$3.00
7955
24.36
74.36
40.84
02.25
20:50
33.91
55.99
06
05.38
23:00
64
49.44
27.25
50.20
18:26
65.01
42.90
75.00
30.87
56.2
32.9
66.45
22.78
75.83
50.5
$3
41.31
01 16
57.36
71.15
01
5,50
14
2325
40 00
52 33.00
30.00 10.36
36:00
75550
5254
63.50
41:6
77.25
3221
86.30
11
86209
Table ID-C generalization for FL. methods trained on CIFAR- 100.
22.19
47.25
50.50
3514
00.43
35.69
74.74
19.1
23 24
23 24
49. 40
81.03
12
72 79
72 70
40 00
81.16
43.25
16
74.55
50.5
82.74
11.
21.8
71.5
45.8
7338
3341
1675
44.75
20:09
54.76
11:33
32
34
34
51.00
62.5 25
35
23
17
58.55
08-47
PooopFall
bype
FallGP
VeILAFT
26

---

FOCoOp: Enhancing Dut-of-Distribuute Robustness in Federated Prompt Learning for Vision-Language Models
46
66.24
61.92
70.30
69. 19.70
40.00
71.67
48.38 33.34
70.83
66.04
74,95
73.47
44.00
Zoom
30 99.97
63.58
58.8
66.72
67.26
35.03
34. 34.40
57.67
52.53
60.71
61.30
29.65
Gausiion
50.23
70.96
17.55
76.02
74.33
46,50
Elastic
53.80 80
74.84
72.04
79 79.69
50.14
77.88
JPEG
Blar
46.96
70.10
64.84 84
73.69
72.74
43.86
Pheeltte
49.59
71.99
07.84 87.84
73.99
75.23
46. 16
Table 14: ID-C generalization for FL methods trained on TinylmageNee.
32, 16
51 31.98
49.55
57.17
60.11
26.28
Impolee
Frost
50.6
71:45
07. 88
76.1 76.14
75 38
46 46
28 64
48.76
93
$1.68
$1.29 29
2703
45.0 $5,66
68.64
6.3 17
72.57
71.72
42.15
51.35
77.73
09.
76.55
48
Glies Blur
28 30
52.51
45.41
55.07
$3.60 60
23.51
47.41
68.35
64.56
73.06
72.08
4353
59.35 38
68
76. 29
83.40 13.40
82.55 55
53.79
Fedl Fooii
ID-C Type
PrompiFL.
FelOTP
FeiRGP
PedGalLaP
ReILAPT
rUcoup
27


Reviews:
[{"review_id": "QouR3O9odf", "reviewer": "Reviewer_cE28", "summary": "The paper focuses to devise an out-of-distribution (OOD) enabled federated prompt learning method based on CLIP model. In addition to global and local prompts, the papers proposes to learn OOD prompts to enable OOD detection at each client in the federated learning framework. Similarity scores in the CLIP embedding space are used to optimize all the three prompts, and further encourage to make the prediction probability of local and global prompts to be larger than OOD prompts on the client data. One of the main contributions of the paper is to introduce Bi-level Prompts Distribution Robust Optimization (BDRO), that perturbs both global and OOD prompts to the worst case at each client using optimal transport discrepancy to enable wider distribution exploration. Besides, the paper also provides an understanding that OOD prompts of one client could observe data from other client as OOD. To eliminate such discrepancy, the paper extracts relevant global prompt information from such OOD prompts and extract strictly OOD prompts that are consistent across all the clients using semi-unbalanced optimal transport. Experiments conducted on CIFAR-100, TinyImageNet and their corrupted variants show better clean accuracy and generalization with significantly improved OOD scores compared to existing federated learning approaches. Similar improvements are also noticed across datasets like Food101, DTD, Caltech101, Flowers102 and OxfordPets. Domain generalization on datasets like DomainNet and Office show competetive results.\n\n## update after rebuttal\nI thank the authors for providing sufficient justification and explanation addressing my questions. I encourage the authors to reflect this rebuttal in their revised version and make it easy for the readers to comprehend the method. Given the current state of the paper and based on the author response, I am inclined to accept the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1Reasons for using optimal transport (OT).**\n\n**OT is a powerful tool for comparing distributions, and is used for different goals in BOS and GOC.**\n\n**In BOS, OT is used to constrain uncertainty set in distributionally robust optimization (DRO).** Unlike KL-divergence, capturing categorical distribution without considering geometry in feature space, OT divergence preserves the geometry of latent feature spaces, which is vital to text-image feature matching in VLMs. As noted in lines 211.left-188.right, global prompts enhance robustness in ID data in all clients and covariate-shift data near ID data, making OT as a suitable constraint. However, we adopt unbalanced OT (UOT) to constrain the uncertainty set of OOD prompts. Because it combines the strengths of OT and KL-divergence, capturing both geometric and non-geometric semantic shifts.\n**In GOC, it uses semi-UOT to improve discrimination between global prompts and OOD prompts.** By coupling global prompts and all OOD prompts, we can select seemly OOD prompts based on the close distance, and distribute mostly distant (strict) OOD prompts to all clients. \n\n**2 The reason for creating worst-case prompts and how to realize wider distribution exploration.**\n\nWe obtain the worst-case prompts via DRO, as defined in Eq. (6). Specifically, we construct an uncertainty set $P$ for global prompts, which is centered around the original distribution of global prompts $P_0$​, and bounded by an OT divergence threshold $\\eta_1​$. Within this set, we identify and optimize the perturbed prompts that yield the worst performance, as formulated in Eq. (6). The distribution of worst-case prompts is denoted as $\\hat{P}$.\nThe same procedure with UOT-divergence is applied to OOD prompts, where original distribution $Q_0$, uncertainty set $Q$, and worst-case prompts $\\hat{\\boldsymbol{t}}^o \\sim \\hat{Q}$ are corresponding notations. \n\nSince we match images with textual prompts that yield the worst performance, the matching process is no longer point-to-point. Instead, it becomes a point-to-uncertainty-set matching, enabling more robust alignment in a wider semantic space. \n\n\n\n\n\n**3 In Eq.(7), $\\mathfrak{c}(\\hat{t}^g,t^g)$ is the cost function of optimal transport, we implement it by computing L2-norm distance.**\n\n**4 Explanation on computing Eq.(8) to get seemly and strict OOD prompts.**\nAs in line 270.left, we compute all costs between aggregated global prompts and OOD prompts from all clients, and we regularize the assignments for OOD prompts with soft KL regularization while the assignments for global prompts are balanced regularization. Then we solve the mapping $\\pi$ by theorem 3.2, where we use ${\\pi^*}^\\top 1_{\\text{KU}}$ as distance. Based on it, we can rank OOD prompts, and the seemly OOD prompts are close to global prompts, but strict ones are mostly distant. The whole procedure can be found in lines 266.left-265.right.\n\n**5 We will add explanations for server block in Fig. 2.** Specifically, in global-view OOD consistency, we use Semi-UOT to couple aggregated global prompts and all client OOD prompts. Then we take  ${\\pi^*}^\\top 1_{\\text{KU}}$ as probability/distance, to select seemly OOD prompts and strict OOD prompts for the next communication round. Based on mapping, we can re-assign global prompts with seemly OOD prompts by Eq.(10), and remain strict OOD prompts by Eq.(11). Then server sends the final global prompts and OOD prompts to all clients for consistency.\n\n**6 We sincerely appreciate the reviewer’s constructive suggestions, and will improve the clarity of our revised version.**", "claims_and_evidence": "The claims on improving the performance with bi-level distribution robust optimization of both global and OOD prompts and also with  global-view OOD consistency across clients are sufficiently backed up by the empirical evidence shown in Tables 1, 2 and 3.", "methods_and_evaluation_criteria": "Evalution Criteria: To my knowledge, authors made an exhaustive evaluation with various benchmark datasets like CIFAR-100, TinyImageNet, Food101, DTD, Caltech101, Flowers102 and OxfordPets, DomainNet and Office.\n\nMethods: Both the bi-level distribution robust optimization and global-view OOD consistency are heavily developed based on the optimal transport. Despite the empirical improvements, it is not clear to realize the why optimal transport is right fit in this setting. In particular, authors intend to create worst case global and OOD prompts via bi-level distribution robust optimization. The rationale to create worst case prompts and the choice to make use of optimal transport to achieve it is not clear and justified. The authors connect it to wider distribution exploration, but it is hard to realize how such exploration is possible with the proposed setup. Moreover, in equation (7), the \\hat{t}^g gets sampled from \\hat{P} and \\hat{t}^o gets sampled from \\hat{Q}. How do these \\hat{P} and \\hat{Q} are defined, as these are not clear even after looking into Algorithm 2 in appendix. Main paper should contain insights about them. There is a function that operates on (\\hat{t}^g, t^g), there is no definition given to this function. Similarly, the equation 8 related to global-view OOD consistency, there is no clear explanation on computing or obtaining seemly OOD prompts and strict OOD prompts using optimal transport. Overall, the current explanation of the method needs a good refinement. A detailed explanation supporting the equations would provide a good understanding to the readers.", "theoretical_claims": "There are two theorems presented in the paper. I have not verified their correctness.", "experimental_designs_or_analyses": "The authors follow the existing literature experimental protocol setting to conduct their experiments.", "relation_to_broader_scientific_literature": "There is a list of literature focusing on federated prompt learning methods using CLIP model. This work complements this literature by introducing OOD detection into the framework that could be leveraged by all the participating clients.", "essential_references_not_discussed": "To my knowledge, I see that authors have cited and discussed the relevant literature.", "other_strengths_and_weaknesses": "Strengths: \n1. The paper show significant improvements for OOD detection while improving accuracy on federated prompt learning setting across multiple benchmarks. \n2. Ablation studies provide better understanding of the method. \n3. I also appreciate the authors for providing the Algorithms and details on their experimental setup.\n\nWeakness: \n1. In Figure 2, the server block is hard to infer based on the illustration. Explanation in the caption could help to understand.\n2. The explanation of the method section lacks clarity and justification, and requires refinement to improve readability and understanding the concepts presented in the section. Please see my comments in the method block above.", "other_comments_or_suggestions": "None", "questions_for_authors": "What is the rationale to create worst case prompts?\nWhat makes the optimal transport the right choice to make use of for the problem?\nHow do the \\hat{P} and \\hat{Q} are defined? \nWhat is the function definiton that operates on (\\hat{t}^g, t^g) in equation 7?\n\nBased on my comments on the method section, please incorporate the suggestions and improve the readability. The results shown in the paper are promising, but it should be complemented with a well explained method section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "an1S7oY30o", "reviewer": "Reviewer_zfft", "summary": "Federated Prompt Learning (FPL) allows models to adapt across clients while maintaining data privacy. However, current methods face challenges balancing performance and robustness, especially when encountering out-of-distribution (OOD) data shifts, limiting their real-world reliability. This is mainly due to data heterogeneity among different clients. To overcome these challenges, FOCoOp introduces a strategy utilizing three types of prompts to create clear separations at both class and distribution levels. Experiments demonstrate that FOCoOp effectively improves robustness and adaptability to OOD scenarios in heterogeneous federated learning settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1 Inconsistency in maintaining local OOD robustness in FPL.**\n\n**FOCoOp aims to improve the OOD robustness of FPL concerning the global distribution, which covers all clients' training data.** The consistency we focus on lies in detecting semantic shifts beyond all client data and ensuring generalization under covariate shift and client heterogeneity within the global distribution. \n\n**However, this consistency cannot be directly achieved by prompt tuning for local robustness.**\nWhile prompts can transfer the knowledge of foundation models, they are adapted to heterogeneous local data during training, **leading to inconsistent matching patterns across clients.** \nThis makes prompts wrongly identify any data different from local distribution as OUT, degrading both generalization and detection.\n**Results in Tab. 1-3 reflect this limitation.**\nPromptFL and FedLoCoOp lack consistency mechanisms and underperform in all metrics.\nSimilarly, FPL methods like FedOTP and PromptFolio aim to balance generalization and personalization, yet still fail in detection. \n\n**2 Reason for OOD prompts in client match unseen data from other clients.**\n\n**This is a general case caused by heterogeneous data distributions across clients.** In federated settings, clients often have imbalanced or missing classes, e.g., client 1 may only have data of dog and cat classes, while client 2 has dog, cat, and fish.\nDuring training, OOD prompts are optimized to avoid resembling locally seen data. However, lacking the reference of global distribution, OOD prompts may mistakenly match with samples locally unseen but present in other clients. For instance, in client 1, OOD prompts may falsely match \"fish\" images as OUT, but fish images are in client 2.\n\n**3 We realize the goal of OOD prompts, i.e., capturing misalignment between local ID data and the OOD prompt context, by explicitly optimizing Eq. (5).** Specifically, we penalize higher similarities between OOD prompts and local ID data(Eq. 3), and ensure the total similarity of OOD prompts is lower than that of ID prompts(Eq. 4). **This design notably improves OOD detection across all evaluated tasks in Sec 4.2.** For example, it consistently improves the detection of different semantic shift datasets in Fig.4. \nHowever, since each client only observes a subset of the global distribution, the learned OOD prompts may mistakenly match ID samples unseen in this client but valid from other clients as OUT. Therefore, the GOC module is used to enhance consistent OOD robustness across clients.\n\n**4 Experiments on exploring OOD data.**\nWe clarify that **only in-distribution (ID) data is used during training**, which aligns with federated settings where OOD data is typically unavailable. **Our method does not explicitly explore OOD data, instead, it uses a DRO-based objective to explore prompt space**, allowing global (OOD) prompts to better match (mismatch) ID samples.\n\n**All OOD data (both ID-C and OUT) are used only for evaluation(main paper Table 5), not for training prompts.**\n\nNext, we evaluate OOD robustness on various OOD datasets, i.e., (1) Tables 1–2 show FOCoOp’s strong generalization to ID-C datasets (e.g., CIFAR100-C, TinyImageNet-C) and its improved detection of OUT data (e.g., Texture), (2) Extra evaluation of various OUT datasets also verified that FOCoOp can consistently enhance OOD robustness. These confirm the effects of exploring prompts in FOCoOp.\n\n**5 We can verify the clear separations for both class and distribution levels by similarity matrix.** In **[link](https://anonymous.4open.science/r/FOCoOp/prompt_separation.png)**, we model FOCoOp on Cifar10 (10 ID prompts and OOD prompts), and sample 100 images per class to compute the average of similarities between images and prompts. The diagonal of the ID prompt matrix shows the highest similarities, suggesting intra-class alignment and clear class separation. Meanwhile, the similarities of OOD prompts are notably lower than those of ID prompts, further indicating clear distribution separation.\n\n**6 We extend our evaluation to ResNet50 on CIFAR100 and TinyImageNet. The results validate that FOCoOp maintains strong generalization and detection capabilities even with smaller models.**\n\nPathological Non-overlap (10 clients, K=10)\n|Dataset|||CIFAR100|||TinyImageNet||||\n|-|-|-|-|-|-|-|-|-|-|\n|Method(%)|ACC|CACC|FPR95|AUCROC|ACC|CACC|FPR95|AUCROC|\n|PromptFolio|51.96|47.47|50.14|86.45|44.70|30.64|57.57|83.03|\n|FedOTP|55.34|50.98|61.38|74.92|43.61|28.98|72.67|73.86|\n|FedLoCoOp|17.03|12.09|93.36|52.66|9.44|4.96|93.63|56.40|\n|FOCoOp|60.94|55.92|28.83|95.54|49.96|35.89|53.51|87.75|\n\n**7 Privacy analysis. Prompts encode only class-level statistics rather than instance-specific information, making it non-trivial to infer original individual image data stored on clients.** The privacy of class-level statistics can be strengthened by applying differential privacy on prompts before sending to server.\n\n8 We will carefully fix the typos mentioned.", "claims_and_evidence": "1. In lines 68-72, the authors mention that \"The crucial reason is that each client maintains local OOD robustness on heterogeneous data distribution, which is inconsistent among FPL.\" It is unclear why the maintenance of local OOD robustness on heterogeneous data distributions would be considered inconsistent in FPL, especially given that prompts are able to handle other downstream tasks, like different data distributions. Could the authors elaborate on this inconsistency with a more detailed explanation or provide empirical results that clarify this aspect?", "methods_and_evaluation_criteria": "1. In lines 234-235, the authors mention that \"The OOD prompts in client match the unseen data from other clients with high similarity scores, hurting the generalization in global view.\" Could the authors clarify why these OOD prompts would match unseen data from other clients? Is this a general case or a specific situation? A more detailed explanation would help in understanding the reasoning behind this claim.\n\n2. In lines 250-254, the authors mention that \"The OOD prompts capture the misalignment between local ID data and prompt context, mistakenly identifying the ID data from other clients as outlier.\" Could the authors clarify why the OOD prompts would capture this misalignment between local ID data and the prompt context? A more detailed explanation or simple results to support this claim would be appreciated.", "theoretical_claims": "Theorems 3.1 and 3.2.", "experimental_designs_or_analyses": "1. Could the authors provide any experimental results demonstrating that FOcoOp is effective in exploring OOD data?\n\n2. Are there any experiments that show how the proposed three types of prompts create clear separations at both the class and distribution levels?\n\n3. Do the proposed methods work with ResNet CLIP models? Given that ViT-B/16 may be difficult for clients to deploy, it would be helpful to understand if the methods are also effective with more accessible models like ResNet-50.", "relation_to_broader_scientific_literature": "The paper proposes FOCoOp, a Federated OOD-aware Context Optimization framework to enhance robustness and performance in Federated Prompt Learning (FPL) for VLMs.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. What are the privacy considerations regarding the client prompts, particularly when they are transmitted to the server? Could the authors address how this aspect is handled or discuss it with any related works?", "other_comments_or_suggestions": "A typo: in line 81-82 in Section 2.1, two CLIP2FL methods.", "questions_for_authors": "Please refer to the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i2N33JZw9V", "reviewer": "Reviewer_NsKV", "summary": "In this paper, the authors provide a Federated OOD-aware Context optimization framework named as FOCoOp. FOCoOp optimizes three sets of prompts for generalization, personalization, and detection, respectively. In the client local training, the authors devise a BOS module to maintain the class-level and distribution separations for discriminate client data. And a GOC module tackles challenges of enhancing the consistency of OOD robustness among clients.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1 We will carefully refine current version and correct typos.**\n\n**2 The supplemental materials consist of five sections: (A) related work details, (B) algorithms, (C) theoretical analysis of optimization, (D) datasets and implementation, and (E) additional experimental results.**\n**In Section E**, we report the results on CIFAR-100 and TinyImageNet under different Dirichlet distributions (Tables 6–7), which are consistent with Tables 1–3 and further verify OOD robustness. Table 4 has been split into Tables 8 and 9 for clarity.\n Tables 10–11 provide numerical results for Figure 4 (OOD detection on various OUT datasets), while Tables 12–13 are numerical results corresponding to Figure 5 (generalization on different ID-C datasets). Figures 8–9 are also enlarged for better readability. We will add them in a future version.\n\n**3 Clarification on data usage in FPL**\n\n**We do not use OOD data when optimizing prompts. Please refer to Answer 4 in Review zfft for further details.**\nAs indicated in Eq. (1), the final evaluation targets minimizing classification errors on ID and ID-C test sets, and minimizing the failure to detect OUT data. We will clarify this more explicitly in the future. A summary of training and evaluation settings is provided below:\n\n**Training datasets:**\n\nCIFAR-100, TinyImageNet, Food101, DTD, Caltech101, Flowers, and OxfordPets, each split into train/test sets and simulated as heterogeneous clients using Pathological or Dirichlet partitioning.\n\nDomainNet and Office-Caltech10 use an N-1 domain split strategy.\n\n\n**Evaluation settings:**\n\nID ACC: Accuracy on the test sets of the above training datasets.\n\n\nID-C ACC: Accuracy on CIFAR-100-C, TinyImageNet-C, and the held-out domain in DomainNet and Office-Caltech10.\n\n\nOOD FPR95 / AUROC: Evaluated on Places365, Texture, iSUN, LSUN-C, and LSUN-R.\n\n\n**4 Baseline Selection.**\nWe select baselines from the state-of-the-art (SOTA) personalized federated learning (PFL) methods and centralized prompt-based OOD detection methods. For centralized prompt-based OOD baselines, we choose GalLop and LAPT due to their strong performance in both generalization and OOD detection.\nWe also include LoCoOp, as GalLop is proposed as an improvement over it.\n\n**5 Our computation cost is comparable to existing FPL methods such as PromptFolio and FedOTP, with the global and local prompt computation cost being $\\mathcal{O}(KEBCd)$. The introduction of OOD prompts adds an extra but manageable overhead of $\\mathcal{O}(KEB(C+U)d)$, where $U$ is the number of OOD prompts.** Since $U$ is a controllable parameter, incorporating OOD prompts is a reasonable choice for significantly improving OOD detection performance.\nAll notations are defined in our main paper, i.e., client numbers $K$, local epochs $E$, averaged batch size $B$, class number $C$, and prompt embedding $d$.\nFurthermore, we measured the average computation time for each communication round. An extra set of OOD prompts only increases computation cost by 4.76%, and 1.61% for CIFAR100 and TinyImageNet. The most computational burdens stem from encoding features rather than prompt learning. This indicates that employing three sets of prompts remains a practical and efficient design. FOCoOp containing BOS and GOC causes slightly larger but still controllable computation, which increases cost by 1.58s(9.06%) and 1.64s(6.18%) for CIFAR100 and TinyImageNet, respectively.\n| Method| CIFAR100(s) | Increase | TinyImagenet (s) | Increase |\n|-|-|-|-|-|\n| PromptFolio| 17.4765| -| 26.5969 | - |\n| FOCoOp-w/o-dro&uot | 18.3078 | +0.8313 (+4.76%) | 27.0261           | +0.4292 (+1.61%)    |\n| FOCoOp  | 19.0597 | +1.5832 (+9.06%)  | 28.2402| +1.6433 (+6.18%)    |\n\n\n**6 We will carefully refine our presentation and correct typos.**", "claims_and_evidence": "The claims are supported by clear and convincing evidence. For example, Federated prompt learning methods are inferior in OOD robustness, which is evidently reflected in Fig. 1.", "methods_and_evaluation_criteria": "The proposed method and evaluation make sense. Specifically, FOCoOp aims to maintain the performance and enhance OOD robustness, both of which are evaluated valid in various empirical setups. The authors also provide ablation studies for the effect of BOS and GOC modules.", "theoretical_claims": "The proofs of theoretic claims are correct.", "experimental_designs_or_analyses": "The experimental design and analysis are well-structured and conclusive. Firstly, the authors select a diverse range of datasets for various tasks. Secondly, they evaluate FOCoOp and its representative baselines under different levels of data heterogeneity and varying client participation. Lastly, the analyses align with the claimed contributions of enhancing OOD robustness, supported by comprehensive experimental discussions.", "relation_to_broader_scientific_literature": "The proposed FOCoOp contributes maintaining the performance and enhancing OOD robustness on federated prompt learning for vision-language models. It is quite novel and vital for the development of utilizing foundation models in a privacy-preserving and robustness-aware approach.", "essential_references_not_discussed": "The essential references are discussed.", "other_strengths_and_weaknesses": "Strengths: \n1.The paper is well organized and the main idea is easy to follow.\n2.The problem of enhancing OOD robustness in federated prompt learning is vital and novel for the development of foundation models. This motivates border studies of this topic in the future.\n3.The proposed methods are sound and effective in addressing challenges related to the improved discrimination of in-distribution classes, handling distribution shifts in clients, and ensuring consistent OOD robustness under client heterogeneities.\n4.The experiments and analyses are extensive and investigable for contribution of model designs. \n5.The designed crucial modules provide motivative insights. For example, the alignment between global prompts and OOD prompts are coupled with semi-unbalanced optimal transport, and further enhance the discrimination of two sets of prompts based on coupling results. \n\n\nWeakness:\n1.Though the good organization of main paper, the presentation should be refined, e.g., clarifying typos.\n2.Some details need more discussions: a) the contents in supplementary materials, b) the data used for federated prompt learning and testing, and c) the choice of baseline methods.\n3.The need to optimize multiple prompt sets (Global, Local, OOD) results in higher computational costs compared to traditional FPL methods.", "other_comments_or_suggestions": "The authors should clarify some typos, i.e., \na) T^g_c in line 262, \nb) robustness3 in line 416, \nc) \\pho rather p in line 742, \nd) two rather three in line 328,\ne) last \\epsilon in line 787.", "questions_for_authors": "Q1: Are the OOD shifts data used for the optimization of prompts? If so, could the authors explain how these OOD shifts data are trained in the process. If not, please explain the objective of Eq.(1) for clarity.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4sImJbxA0H", "reviewer": "Reviewer_p5vg", "summary": "This work focuses on enhancing Out-of-distribution (OOD) robustness for federated prompt learning on pretrained vision-language models. The authors propose a federated OOD-aware Context Optimization framework, i.e., FOCoOp, which contains two main modules, i.e., BOS and GOC. BOS not only enhances class-level matching between image and prompts corresponding the class, but also maintains distribution-level separation between ID and OOD data. GOC further enhances the consistency in OOD generalization and detection among all clients. The authors evaluate FOCoOp in extensive experiments, validating its effectiveness in generalization and robustness in OOD shifts.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1 We are willing to release our code and have made the project available at [https://anonymous.4open.science/r/FOCoOp/](https://anonymous.4open.science/r/FOCoOp/).  It will be included in the final version of the paper.**\n\n**2 $\\mathfrak{c}()$ in Eq.(7) is the cost function, which can be computed via L2-norm distance.**\n\n**3 d+ and d- indicate the positive and negative distance set, respectively.**\nSpecifically, $d^+$ means we compute the set by satisfying $\\eta$ percentile of positive cosine distance, while $d^-$ means the set satisfying $\\eta$ percentile of negative distance. \n\n**4 We will carefully revise our presentation and fix the typos.**\n\n**5. Clarification on the Role and Aggregation of Prompts.**\nAs described in lines 78–81, the three sets of prompts serve distinct roles:\n\n**Global prompts** capture the shared ID global distribution across all clients.\n\n\n**Local prompts** capture each client's ID personalized but heterogeneous distribution.\n\n\n**OOD prompts** are trained to mismatch with ID data, preventing them from mistakenly aligning with samples that are unseen locally but present in other clients.\n\n\n**During training, each client sends both global prompts and OOD prompts to the server, while leaving local prompts at clients.**\nThe server aggregates global prompts to learn a consistent alignment with the global distribution, and selects the most dissimilar OOD prompts to represent a consistent misalignment across clients.\nSince OOD prompts are shared and distribution-agnostic, they generalize well to open-world OOD datasets, leading to superior performance in OOD detection tasks.\nWe will elaborate on these design choices more clearly in the final version.\n\n\n**6 Explanation on FOCoOp’s greater impact on OOD detection.**\nWe appreciate the reviewer’s observation. **FOCoOp is the first method that is designed to enhance OOD robustness while maintaining strong performance on standard FPL tasks.** This is indeed reflected in the results: while improvements on ID and ID-C generalization are moderate (as existing FPL methods already achieve strong performance), FOCoOp demonstrates substantial gains in OOD detection.\nThis discrepancy arises from our explicit design: FOCoOp introduces OOD prompts and uses a DRO objective to explore a broader prompt space. While this helps generalization to some extent, the major benefit lies in detection. Specifically, we distinguish between seemly OOD prompts (which match with ID samples that are unseen locally but seen globally) and strict OOD prompts (which are mismatched with all ID data), enabling more accurate identification of OOD data.\nWe will clarify this distinction and its impact in future revisions.\n\n\n**7 Clarification on aggregating prompts.**\nIn lines 157.left-158.left, we present a **general formulation of PFL** and refer to **prompts matched with local data on each client as local prompts**. As described in lines 246–247, we also explicitly introduce the method of **FOCoOp, which first aggregates global prompts across all clients**, and calibrates global prompts later by Eq. (10). We will clarify this distinction more clearly in the next version.", "claims_and_evidence": "Yes, the claims in the paper are verified with convincing empirical results, e.g., the motivation example in Fig. 1, and subsequent experiments in section 4.", "methods_and_evaluation_criteria": "Yes, both the proposed method and evaluation criteria are well-aligned with the problem at hand. The OOD robustness of federated prompt learning is a crucial challenge in real-world scenarios, and the proposed FOCoOp framework effectively addresses multiple types of OOD shifts simultaneously. Moreover, the evaluation criteria are directly relevant to the contributions of FOCoOp in enhancing OOD robustness, ensuring that the assessments meaningfully reflect the framework’s effectiveness.", "theoretical_claims": "Yes, the theoretical claims and proofs are correct and reasonable.", "experimental_designs_or_analyses": "Yes, the experiment designs and analysis are solid and comprehensive for verifying the claims and method designs. The study includes rigorous evaluations on plenty of real-world datasets, ensuring both performance and OOD robustness. The detailed analysis, such as alation studies and hyper-parameter sensitivity studies are sufficient.", "relation_to_broader_scientific_literature": "This work builds on prior research in federated learning and vision-language models, aiming to enhancing OOD robustness. Unlike previous studies, FOCoOp optimizes prompts to enhance both class-level and distribution-level separations per client while maintaining consistency at the server, improving both performance and OOD robustness.", "essential_references_not_discussed": "The references are sufficient, no extra essential reference is needed to include.", "other_strengths_and_weaknesses": "Strengths: \nS1. The novelty of considering OOD robustness for federated prompt learning is vital and potentially necessary for the era of foundation models. And the authors make contributions to it with both methodology and evaluations.\nS2. The authors provide the correct and sound theoretical inductions for the proposed methods, i.e., the optimizations of Eq.(6) and Eq.(8).\nS3. The proposed bi-level prompts distribution robust optimization is quite interesting and insightful. The authors constrain the perturbation of global prompts within optimal transport cost for generalization, while perturbing OOD prompts within unbalanced optimal transport cost for the awareness of open-world unseen outliers.\nS4. The experiments are comprehensive in comparing both federated prompt learning methods and federated learning with Clip-based OOD detection methods. \n\n\nWeakness:\nW1. There are some unclear notations, e.g., (1) $c()$ function in Eq.(7), (2) $d^+$ and $d^-$ in Eq.(2) and right part of lines 239-252, and (3) typos in step 13 of Algorithm 2, the last $\\epsilon_u^o$ is supposed to be $\\boldsymbol{o}_{\\epsilon_u^o}$.\nW2. The authors introduce three set of prompts, ID global prompts, local prompts, and OOD prompts, but their relationships could be further explained for readability.\nW3. Based on the empirical results, it is interesting to find that FOCoOp seems to have more contributions on OOD detection tasks while insignificant improvements on ID and ID-C generalization. The authors could provide more explanation with regarding to it.", "other_comments_or_suggestions": "See weakness", "questions_for_authors": "Can the authors explain the confusion, i.e., the local prompts are aggregated in lines 158-157, while global prompts are aggregated in lines 246-147?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Overall, the reviewing panel is very positive with this paper. We all find this paper acceptable.

The rebuttal has also resolved multiple concerns effectively, reinforcing its acceptability.

I also have a set of questions for the authors at the beginning of the reviewer-author discussion phase. My main concerns are that the authors have missed two relevant works on federated prompt-tuning and federated LoRA.

The work on federated prompt-tuning could potentially be used for the same setting while the federated LoRA work proposes an orthogonal approach to federated fine-tuning (though not specifically designed for OOD).

[1] https://arxiv.org/abs/2409.05976 (NeurIPS-24) 

[2] https://arxiv.org/abs/2502.19752 (NeurIPS-24)

However, with clarification from authors, the proposed approach has an added benefit/advantage of being able to tune text prompts to visual data while the above use vision-prompt for visual data; or text prompt for text data. 

Given this, I consider those as concurrent approaches. The authors are encouraged to cite and position their work with the above in the camera-ready paper as they have done comprehensively in their response to me. 

--

In summary, this is a solid work. I recommend acceptance.
Decision: Accept (poster)