Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning
Authors: Zhuang Qi
Yu Pan
Lei Meng
Sijin Zhou
Han Yu
Xiaoxiao Li
Xiangxu Meng
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning;Federated Prompt Learning;One-shot Federated Learning
Primary area: general_machine_learning

Abstract: Federated Prompt Learning (FPL) enables communication-efficient adaptation by tuning lightweight prompts on top of frozen pre-trained models. Existing FPL methods typically rely on global information, which is only available after the second training round, to facilitate collaboration among client models. Therefore, they are inherently dependent on multi-round communication to fully exhibit their strengths. Moreover, existing one-shot federated learning methods typically focus on fitting seen tasks, but lack cross-task generalization. To bridge this gap, we propose the global prompt refinement with non-interfering attention masking (GPR-NIAM) method for one-shot FPL. The core idea is to design a masking mechanism that restricts excessive interaction between the original text embeddings and the learnable prompt embeddings. GPR-NIAM achieves this through the collaboration of two key modules. Firstly, the attention isolation module suppresses attention from the learnable prompt tokens to the original text tokens, and reweights the reverse attention which preserves generalization across tasks. Secondly, the cross-silo collaborative refinement module integrates decentralized visual knowledge into a unified base and calibrates the global prompt through multi-source cross-modal knowledge alignment, further mitigating the inconsistency caused by data heterogeneity. Extensive experiments conducted on ten benchmark datasets under two tasks show that GPR-NIAM outperforms eight state-of-the-art methods in both class-level and domain-level generalization.

Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning
Zhuang Qi', Yu Pan', Lei Meng".,, Sijin Zhou², Han Yu², Xiaoxiao Li4.5, Xiangxu Meng' School of Software, Shandong University, China AIM Lab, Faculty of Engineering, Monash University, Clayton, VIC, Australia
Coollege of Computing and Data Science, Nanyang Technological University, Singapore "Department of Electrical and Computer Engincering, University of British Columbia, Canada Vector Institute, Canada z_ z_qi@mail... sdu. edu. cn, 202100300334tmail. sdu. edu en, Imeng@sdu. edu. cn, s sjzhou1995@gmail. com, han yuEntu. edu. sg, xiaoxiao.. 1i Cece ubc. ca, mxx0sdu edu. cn
Abstract
Federated Prompt Learning (FPL) enables communication-effieee adaptation by tuning lightweight prompts on top of frozen pre-trained models. Existing FPL methods typically rely global information, which is only available after the second training round, to facilitate collaboration among client models. Therefore, they inherently dependent multi-round communication fully exhibit their strengths. Moreover, existing one-shot federated learning methods typically focus on fitting seen tasks. but lack cross-task generalization. To bridge this gap. we propose the Global Prompt Refinement with Non- Interfering Attention Masking (GPR-NIAM) method for one-shot FPL. The idea design masking mech- anism that restricts excessive interaction between the original text embeddings and the learnable prompt embeddings. GPR-NIAM achieves this through the col- laboration of two key modules. Firstly, the attention isolation module suppresses attention from the learnable prompt tokens the original text tokens, and reweights the reverse attention which preserves generalization across tasks. Secondly, the cross-silo collaborative refinement module integrates decentralized visual knowl- edge into unified base and calibrates the global prompt through multi-source cross-modal knowledge alignment, further mitigating the inconsistency caused by data heterogeneity. Extensive experiments conducted on ten benchmark datasets under two tasks show that GPR-NIAM outperforms eight state-of-the-at methods both class-level and domain-level generalization.
Introduction
Federated learning (FL) enables collaborative modeling across data sources in a knowledge manner, which has gained traction across multiple domains 2 4 5 It learns global
by aggregating model weights or gradients instead of moving data to a central entity 6 g 9]
Recently, Federated Prompt Learning (FPL) has emerged support parameter-efficient fine-tuning in this setting. It not only leverages pre-trained models (e.g., CLIP (10) more effectively, but also reduces communication overhead between FL clients and the FL server by transmitting only learnable prompts [T] 12 I3 [4]. However, heterogeneous data distributions across clients often degrade model aggregation performance and slow down convergence [55] 16]. This because significant differences in the update directions across local models can cause conflicts during aggregation
CCoresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

7, 75 Tros
1,
Tros
7, Tp
Transformen Layer
marr Lower
Non-interfaring Masking
Trandormma
Transfomme Poneefomeert
74
120s
Figure 1: An overview of the proposed SPR-NIAM method. It supports three interaction types: full attention (0), hard masking (-oo), and reweighting (A). Hard masking blocks learnable tokens Tp from affecting original text tokens Tr, while X- A-weighted links enable partial influence from special tokens TEOs. The upper triangular mask follows causal attention for autoregressive decoding.
21. As result, existing FPL methods still require multi-round communications between clients and the server to reconcile inconsistencies arising from client-specific training.
17 18 19
20
To deal with data heterogeneity, early FPL solutions (e.g., PromptFoilo 17 and DP-FPL [22
typically focused on incorporating global prompt information constrain the update direction of local prompts in order to enhance the consistency and stability of federated optimization [17 23 22 24
However, since global prompt information typically available only after the second communication round, these methods inherently rely on multi-round communications. This poses challenge in real-world scenarios with limited communication resources. To this end, one-shot federated learning (OSFL), which requires only a single round of client-server interaction, offers promising direction for further exploration 25 26 27
28 29.. OSFL significantly reduces the communication overhead by eliminating iterative exchanges, making attractive for real- world deployments with strict latency However, directly applying existing OSFL methods to FPL settings is not suitable, as they are typically optimized for seen tasks while neglecting the preservation of generalization to unseen tasks.
or bandwidth constraints (30 31 32
To bridge this important gap. we propose the Global Prompt Refinement with Non- Non-Interfering Attention Masking (GPR-NIAM) method. It prevents the modification of the original token embeddings through masking mechanism, thereby preserving transferable knowledge across tasks. As shown Figure [ the interaction between the learnable prompt embeddings and the original text token embeddings is unidirectional, which ensures that the latter remain unchanged. Specifically, AMMMNIAM consists of main modules. Firstly, the Attention Isolation module selectively regulates the flow of prompt information through novel masking strategy. It suppresses attention from the learnable prompt tokens the original text tokens across encoder layers, thereby effectively preventing alterations to the textual input token embeddings. In addition, it re-weights the reverse attention to encourage the model to extract task-relevant features more effectively from the original texts. Secondly, the cross-silo collaborative refinement (CSCR) module leverages visual representations extracted from multiple clients and utilizes multi-source visual supervision to refine the global prompt in centralized manner. This further mitigates the negative impact of data heterogeneity.
We conduct extensive experiments on 10 datasets under two tasks: 1) moseetolbasennovl generaliza- tion and 2) daae--one-ommin--e generalization, and carry out performance comparisons, ablation studies, in-depth analysis and case studies. The results demonstrate that GPR-NI AMM achieves superior overall performance by maintaining strong balance between fitting seen tasks and generalizing to unseen ones, significantly outperforming eight competitive state-of-the- approaches.
2 Related Work
2.1 Federated Prompt Learning
FPL aims to integrate FL with prompt learning by freezing the backbone model within the FL framework and introducing set of learnable prompt parameters [77 23 22
24
B3, 34 36).
This enables efficient personalization for on-device data while significantly reducing communication overhead, as only prompt parameters are shared between clients and the server. To mitigate the challenges posed by data heterogeneity across clients, popular FPL methods (e.g., FedOTP (24). FedPGP (23), PromptFolio [17], DP-FPI. (22) mostly adopt a shared prompt paradigm. In this design, a global prompt is constructed on the server and combined with client-specific information to

---

improve training stability and generalization under non-IlD distributions. However, these methods are essentially designed for multi-round communications and require access to global feedback, which limits their effectiveness in one-shot FL. scenarios.
2.2 One-Shot Federated Learning
OSFL is communication--efiiiiee variant of federated learning, where model training is completed within single round of client-server interaction [25
42 Recent studies on OSFL mainly focus on generative and ensemble- based approaches. The former employs generative models to synthesize proxy data or representations on the server. For example, FedCVAE
26
37
27
28 38
41 uses variational autoencoder to compress local knowledge. FedDISC [2 and FedDEO [8
leverage diffusion models to generate synthetic training data. The latter is dedicated to refining the
fusion of model predictions from different clients. For example, FedOV 277 introduces an open-set recognition-ban based voting mechanism to address label skew in OSFL; IntactOFL. [26] learns weighted expert-bass based combinations to enhance overall performance. However, these methods are only designed for conventional tasks, aiming to fit the training objective while overlooking the preservation of cross-task generalization.
3 Preliminaries
Suppose there are K clients denoted by C {C1, Cr....,,Crr and each client Ck E C holds private dataset DDA The goal of traditional FL. is to collaboratively train global model f by [D.
is the weight of client minimizing the objective: miny Ek=1 Ps: [(J, Dx), where pk Σラー [2j
In FPL. each client & optimizes a set of learnable prompts 6ū using its private dataset Dr. while keeping the backbone model f frozen: ming, C(f: 6k; Dx). To support class-level prediction, the prompt for class e constructed by concatenating the learnable prompt and fixed class token. Specifically, let Pe RN. denote the class token for class and 6Ą E Np denote the learnable prompt from client k, where Ne and N₂ are the lengths of the fixed and learnable prompts, d, is the dimension of hidden embedding,
k. and C(.) is a task task-specific loss function
43
45 46).
respectively. The full prompt can be formed as ((() = [6x: Pc]. The text feature for class is
computed by feeding the prompt into text encoder T(-): f = T(P.A)) Given an input image x, the image feature is extracted by the frozen image encoder f: f, f(x). The similarity score between the image and class-c textual features is calculated as P.. Cosine(f; (f; fz). and the prediction exp(ne/T)
probability is computed via softmax over similarities: p(0 = x) Σソ=ı exp(p,/T) The training
objective is the cross-entropy loss between the similarity-based prediction and the ground-truth label: fos = {(p ey). where p = IPi.....PPE ey is the one-hot label vector, and é denotes the cross-entropy loss function. After local training, client Cs uploads its learned prompt ôμ to the server. The server then aggregates the local prompts 1 construct a global prompt dy. typically by weighted averaging: 5, = > Pk- 5k-
The Proposed -MAM Method
In this section, we propose the non-interfering attention masking method for one-shot federated prompt learning, called GPR-NIAM. Its core idea preserve the transferable knowledge by reducing the influence of learnable prompt tokens on the input text tokens, as illustrated in Figure
4.1 The Attention Isolation (AttnIso) Module
Prompt tuning typically focuses on fitting local training tasks. However, data heterogeneity often leads to divergent optimization across clients and weakens its cross-task generalization. To address this, the Attnlso module designs attention masks to selectively block the information flow from learnable prompt tokens to original text tokens (e.g. "a photo of a [CLASS]"). We denote the text attention matrix as M E is the length of input textual feature tokens. The additional 1 corresponds to the [EOS] token. In addition, we apply customized attention control for the [FOS] token to ensure proper aggregation of contextual information from both the prompt and the input tokens.
where N, is the number of learnable prompts, N,
3

---

Ciients
Server
Cllent
Images
Image Features
Prototype Fgs. (6) Calculation
Visual Protatypes
Simiirity
Learnoble
Image Encoder
Update Prompts
(4)
Refine Promots Eq.
Clasification (3)
Similarity Comparison Eq (7)
Prompts
Attention Isolation
Asgrogated Prompts
Attention Isolation
(Q):(1)(2)
Egg: (1)(2
12
72
Text
photo
Encoder
Encoder
['Cat,... "Lion']
I'Cat', 'Lion]]
Athention
Masking
Text Features
Masking
Text Features
Figure 2: Illustration of the GPR-NIAM framework. The Attention Isolation module preserves the original text embeddings to reduce the forgetting of generalized knowledge. The global prompt is refined using visual prototypes from all clients to mitigate data heterogeneity.
To achieve the above goal, the text attention matrix M can be formulated as follows,
if [1, A.j Aj E [N + 1, N2 + No] or E [N - 1, N₁ + M,] Aj E [1,
-00
(Hard Masking)
M
or j N₂ N₂ +1
(I)
if N2 No + HAAEEE (N, N, N.] (Reweighting)
otherwise
(Full Attention)
where Mid denotes the i-th row and j-th column element of M, and A denotes logical AND. Moreover, integrate the attention mask M into the text encoder T, we replace the original computation with the following:
No (0,00 ,M) Concat(/1, he....,..pp.. h.....,,.))..... PP411-.-. -.ih 441 141 ((-1) 10
g008
QiK? M) Vi, i= i=.....,
where h; softmax
vds
where y()) E (l) and (1+1) NN are the input and output of the (1+ 1)-th encoder layer. F/+ 1/+ and 3 represent the output embeddings the i-th prompt token, j-th text token, and the [EOS] token, respectively. Q: Kg. Vi E NoN,+NN++))) are the query, key, and value matrices for the i-th attention head, d: d//H denotes the hidden dimension of each attention head. (oncat(h1,.....k))
NNN,,++++)xxHd denotes the concatenated outputs of all heads, and WO e Hd is the output projection matrix mapping the result back to d, Following the masked encoding process, we obtain the final output of the text encoder as: ..
y()) ip....., PL _Np 3.rr ,t....., Afterwards, st. is projected into shared embedding space via a pre-trained head H.(.). On the visual side, the input image x is encoded by frozen vision encoder f(-) to obtain the visual representation. Overall, it focuses minimizing the objective:
[ log [
exp(sim (so)m(s())))) L.
mins, 6x Σ k=
Dul [D]
E (xxx))Da
-log
(3)
DS exp(sii (Jo()), sl
=(c) where f denotes client k's [EOS] embedding for class c, sim(-,) .) denotes cosine similarity function. The learnable prompts can be optimized by gradient descent in all clients:
exp(simm (o(() (5 log
8 8 1 [Byl E(.))BB Ván
4k [K], (4)
Σe exp sim (oo (l0)
where 7) denotes the learning rate, B& is batch data.
4.2 Cross-Silo Collaborative Refinement (CSCR) Module
To further alleviate the inconsistency caused by data heterogeneity, the CSCR module calibrates the global prompt through cekottye-guuide refinement across clients. To support this, each client C2

---

extracts class-wise visual prototypes from its local dataset Dz. For class the prototype is computed as randomly weighted average of the feature vectors from local samples in that class:
fo = Σen Ox fx(x);
where Σ
0x = ax ((,
(5)
XED;
XED:
where Ox denotes a random weight for each sample x in class e, drawn from a uniform distribution and normalized to ensure the weights sum to 1. Notably, we can randomly sample the weight vector
times to generate multiple diverse prototypes per class P = [
u.k w.ko
po.2
which enhances
representational diversity, which differs from existing methods 47
48
49
50
51
fok denotes i-th
visual prototypes of class c in client k. Afterwards, each client Cs sends P
= {P)
PN to the
k
server, which aggregates all received prototypes into global pool P:
K P - (P & [K], c CE [n]} km1
P
(6)
To calibrate the global prompt 5g. we leverage the global prototype pool P (Pl ....... PO}
and enforce the alignment between textual representations and visual prototypes via a class-level contrastive objective. Specifically, we design the following loss function:
ming, Cep log (sim (S, (((T(6,))))) log Σラ= exp (sim (S. He(T ())1)) (7)
where (.) is the sigmoid function, te denotes 'a photo of c-th classname'. We then follow the gradient descent method to obtain the optimal parameters:
6g + 6g - 7. E(SE)ER Vs, logo (sim (f. (T, be)) log ΣS= exp (sim (f. (6g ()))) (8)
where the global prompt 6g is initialized as the weighted average all cpient-specific prompts (i.e..
5g
DDel
5k), and is learning rate.
1= D
4.3 Training Procedure
In this section, we provide the detailed training procedure of GPR-NIAM in Algorithm [ two-stage optimization strategy, GPR-NIAM consists of local prompt tuning phase and global prompt refinement phase. Notably, can be easily integrated into multiple prompt learning methods such as CoOp and TCP. In the first stage, each client initializes its local prompt from the global prompt and updates it on private data with masked attention learning. Simultaneously, class-wise visual prototypes are obtained by performing randomly weighted aggregation over local feature representations. Subsequently. the global prompt is optimized to align with all class prototypes in this pool via cross-silo collaborative retraining. With the two-stage design, GPR-NIAM alleviates the challenges posed by heterogeneous client data and promotes the retention transferable knowledge for unseen tasks.
As
Moreover, the local prompt tuning stage optional. Firstly, this stage provides more meaningful initialization for the global prompt by adapting it to all clients data. Secondly, it generates class-wise visual prototypes to guide global refinement. While the framework can still perform global refinement without this phase, since it does not affect prototype modeling, omitting it might result in less effective initialization and slightly limit the overall refinement performance.
5
Experimental Evaluation
5.1
Experiment Settings
Datasets Following existing works 17
52 experiments are conducted on CIFARIO
Oxford-
Pets [54]. 101tech101 (55 DTD 56 FGVCAireraft [57], Flowers (58). StanfordCars (5 and
UCF101 60 to evaluate baooee///oove generalization, and on Office-Home 6 and NomainNet
62
to assess leave-one-domain-out generalization. Their statistics shown in Table 5
5

---

Algorithm GPR-NIAM
1: Initialize the global prompt parameter 5
2: // Local Prompt Tuning Stage
3: for each client Cs C in parallel do
4: Initialize local prompt parameter 55 = 5
5: for 1,...... F do
6: Sample mini-batch from local data DA
7: Update local prompt òv using masked attention by Eq. (4)
8: end for
9: Compute visual prototype features for each class using Eq. (5)
10: end for
// Global Prompt Refinement Stage
12: Aggregate all visual prototypes to form a global prototype pool d by Eq. (6)
13: for e' 1,..,..,, E' do
14: Sample mini-batch of prototypes ç' from P
Update global prompt 6g via prototype-guice alignment using Eq.
16: end for
report the Top-1 I Accuracy (Acc) results. For base-to- to-baseenovee generalization tasks, we also compute the harmonic mean (HM = 2-Acouue AcC of accuracy on base (Acobas) and novel classes (AcCnovet) to evaluate overall performance.
Evaluation Metrics
Following studies 47 49 63
Accwe Accaest
Implementation Details In all experiments, we set local training and glohal refinement epochs to 10, using the SGD optimizer with a learning rate of 0.001, weight decay of 0.001, and a batch size of
Table I:
Performance comparison between GPR-NIAM and baselines on 8 datasets with 3 = 0.5 in the buse-to-base/nove base/novel generalization cases.
Average 8 datasets
(b) CIFARI0
(c) OxfordPots
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
HM
65.79
74.32
69.70
95.30
95.50
95.39
CL.IP
81.53
95.33
87.89
PromptFL
72.57
72.80
72.64
PromptFL
95.14
96.26
95.69
PromptFL
91.31
93,70
92.48
70.52
73.63
95.68
94.74
95.20
89.00
95.49
92.13
FedTPG
70.46
73.09
71.69
FedTPG
95.38
96.56
95.96
FedTPG
91.66
95.13
93.36
72-46
74.60
73.41
95.68
96.52
90.01
95.39
92.62
PromptFaltoo
73.25
73.35
73.15
PromptFotio
96.04
96.16
96.06
PromptFolio
91.13
96.10
93.54
DP-FPL
71.90
71.75
69.88
DP-FPL
95.42
95.18
95.29
DP-FPL
91.97
94.80
93.36
FedDISC
56-43
54.84
53.87
FedDISC
94.30
92.82
93.55
FedDISC
83.83
72.07
77.50
FedELMY
76.62
48.23
60.03
FedELMY
95.32
40.00
79.66
FedELMY
89.28
76.47
GP% KIAMP
75.56
73.28
74.24
GPR BIAMP
95.66
97.00
96.32
GPA BIAM
91.18
95.89
93.47
PIAM
76.43
74.48
75.30
GPR-NIAN,
95.84
96.48
96.15
GPA-NIAN,
92.48
95.14
93.79
Caltech101
CVVCAircaft
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
82:27
94.05
87.77
CLIP
53.35
59.05
56.06
CLIP
26.51
31.21
28.67
PromptFL
88.27
93.88
90.99
PrompthL
60.06
53.86
56.79
PrompxFL
31.19
30.91
31.05
RPO
89.90
93.80
91.81
RPO
54.62
59.17
56.81
RPO
29.27
30.43
29.84
FedIPG
89.12
92.61
90.83
FedTPG
56.71
55,43
56.06
FedTPG
29.57
30.25
29.90
TCP
$8.20
95.33
91.62
61.68
55.48
58.41
30.77
32.71
31.71
PromptFotio
87.28
93.12
90.10
PromptFotio
61.11
54.58
57.66
PromptFotio
31.37
32.39
31.87
DP-FPL
87.92
93.46
90.60
DP-FPL
62.26
49.63
55.24
DP-FPL
28.85
28.15
28.49
FedDISC
44.75
85.22
58 68
FedDISC
71.52
38.28
49.87
FedDISC
20.87
13.20
16.17
FedEL
87.71
89.21
33.77
19.48
24.70
GPR-FIAN,
91.59
94.22
92.89
GPA-BIAMP
68.40
$3.50
60.04
GPR-MIAMP
30.95
29.41
30.16
GPA-BIAN,
92.65
94.73
93.68
GPR-BIAM
69.90
55.91
62.13
GPR-BIAM-
32.15
31.12
31.62
(g)
Flowers102
(h) StanfordCare
(6 UCFI0I
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
62.39
78.10
69.36
CLIP
56.87
69.40
62.51
CLIP
68.12
71.92
69.97
PromptFL
79.88
75.15
77.44
PromptFL
62.96
67.30
65.00
PromptFL
71.82
71.35
71.68
RPO
74.92
76.21
75.56
RPO
$8.64
68.73
63.29
RPO
72.19
70.49
71.33
FedTPG
72.30
77.26
74.70
FedTPG
58.67
69,78
63,74
FedTPG
70.34
67.74
69.01
TCP
77.25
78.73
77
TCP
60.19
71.18
65.22
TCP
75.93
71.52
73.65
PromptUalio
81.04
74.51
77.63
PromptFotio
61.22
69.91
65.27
PromptFolio
76,88
69.75
73,14
DR.
74.63
77.26
75.92
DP-FPI.
$9.07
66.05
62.36
DR-FPI.
75.14
69.49
72.20
FedDISC
24.95
32.21
28.11
FedDISC
42.77
49.92
46.07
FedDISC
68.49
55.03
61.03
FedELMY
89.79
55.15
68.33
FedELMY
67.09
47.97
55.94
FedELMY
79.94
41.41
$4.56
CPR ETAND
84.54
78.52
81.42
GPR BIANP
63.59
67.48
65.48
GPR BEANP
78.57
70.28
74.19
CPR-NIAMP
86.88
77.05
81.67
EPR-NIAM,
63.54
69.18
66.24
GPR-NIAM-
78.04
76.27
77.14

---

Table 2: Performance comparison Office-Horiete and DomainNet under the leave-one-domain-out setting, with each domain's 's data randomly split across clients
OfficeHHome
DomainNet
Method
Art
Clipurt
Product
RealWorld
Average
Clipurt
Infograph
Puinting
Quickdraw
Real
Sketch
Average
CLIP
74.57
62.33
80.17
79.38
74.08
92.87
77.02
85.35
43.06
98.61
94.22
81.85
PrompeFL.
74.24
66.20
82.47
79.77
75.17
93.87
80.82
87.76
49.47
98.72
94.82
84.24
RPO
74,86
64.33
84.31
76.06
78.32
FedTPG
74.04
61.55
83.32
80.01
75.48
95.14
80.93
88.25
52.89
98.97
95.40
TCP
75.31
66.20
82.17
80.97
76.41
94,13
79.80
88.05
45.85
98,80
95.34
83.66
PromptFolio
67.74
84.35
DP-IPL
75.03
65.91
82.71
79.82
76.12
95.03
78.41
86.74
54,03
98.83
94.85
84.64
FedDISC
72.27
65,40
81.77
79.11
74.63
93.82
80.42
86.17
43,74
98. 66
94.05
82.81
FodELMY
GPR-XTAM-
78.32
71:40
87.61
84.96
80.57
94,72
80.13
88.70
56.98
99.18
94. 88
85.76
-PAMM
79.39
72.06
88,03
85.49
81.24
95.03
80.82
87.92
56.32
99.08
95,60
85.79
32. The number of communication rounds 1. We simulate 50 clients for CIFAR-10 and 10 clients for other datasets in pase-to-bassenone generalization, while using 9 clients for Office- and 15 for DomainNet. To simulate non-IID data, we adopt Dirichlet distribution with parameter 3 = 0.5. Each client generates n E {5, 10, 20} visual prototypes per class, and the reweighting parameter À is selected from {0.2, 0.5, 0.7, 1.0]. More details and results can be found in the Appendix. And each client has one NVIDIA RTX 3090 with 24 GB GPU for training.
5.2 Results and Discussion
This section compares
variants (GPR-NIAMp and GPR-NIAMy, derived by in-
tegrating GPR-NIAM into PromptFL and TCP, respectively) with nine baselines: Zero-Shot CLIP [10 with hand-crafted text prompt template, e.g.. "a photo of class]" [64). PromptFL [65] RPO
166,, FedTPG [52], TCP (67). PromptFolio [17), DP-FPL [22), FedDISC (28]. FedELMY 42 For
fair comparison, all methods the same backbone: frozen CLIP with V16-B16 [10]. Following
prior work, the Icarnable prompt 10 x 512 parameter matrix, with 10 prompt tokens and 512-dimensional embedding.
Base-to-Base/Novel Generalization. We evaluate the generalization capability of GPR-NIAM by comparing its performance on base and novel classes, and the overall harmonic mean (HM) across all benchmarks. The results are presented in Table [ and key observations are summarized as follows:
GPR-NIAM attains second-best performance on base classes among all methods, showing particular competitiveness against FedMLEY while maintaining more balanced capabilities across all classes.
GPR-NIAM yields competitive performance novel classes. Notably, it ranks first on Flowers 102 and UCF101, underscoring its strong generalization capability unseen tasks. Moreover, FedDISC may fail on both base and novel classes due to inevitable gaps between synthetic and real samples.
Benefiting from the synergy of attention masking and cross-silo collaborative retraining, GPR-NIAM achieves the best HM scores across all benchmarks.
Leave-One-Dommain-On Generalization. [ the domain generalization results under leave-one-domain-ouu setting. The main findings summarized below:
GPR-NIAM consistently achieve the highest average accuracy across most domains. Notably, GPR-NIAMT ranks first in 9 out of 12 domains, highlighting its strength in cross-domain general- ization, especially in challenging cases like Infograph and Quickdraw.
By combining token-level and client-level perspectives, CPR-NIAM outperforms single-view meth- ods such as PromptFL. and FedTPG, which benefits from attention masking for preserving original semantics and unified global prompt for cross-domain knowledge integration.
5.3 Ablation Study
This section presents an ablation study to examine the effects of key components in GPR-NIAM, including the Attention Isolation (AttIIss) module and the Cross-Silo Collaborative Retraining (CSCR) module. The results are reported in Table 3

---

Table Ablation results averaged over datasets (base-to- boasse-o-baas/oovelll generalization) and across all domains
out generalization).
Base to Base/Novel Generalization
Caave-On-Domain-- Out Generalization
Method
Base
Novel
HM
Ofhce-Hoom
DomainNet
PromptHI.
64.06
71.48
66.98
75.17
84,24
AttnIsona
62. 19
72.27
67.04
77.26
84,87
Attnisoue
64,22
71.51
67.76
75.67
84.36
AttnIsogx.RE
64.11
72.66
67.87
77.83
85.01
+ CSCR
73.34
64,89
67.24
78.59
85.11
Attnison4 + CSCR
69.32
70.78
70.06
79.13
85.46
Attalssar CSCR
70.56
68.11
69.65
78.42
85.23
Attnlsonnsse CSCR
72.32
69.47
70.72
80.57
85.76
Different configurations of the AttnIso module exhibit distinct strengths. Hard masking (AttnIsohA) effectively suppresses token interference and enhances novel-class generalization, while reweight- ing (AttnIsORE) provides more stable improvement. Their combination leads to improved overall generalization, showing that the two strategies are complementary.
The CSCR module plays vital role in optimizing base-class performance by learning unified prompt that mitigates inter-client discrepancies, which can alleviate the negative impact of data heterogeneity. However, its contribution to novel-class generalization remains limited due to insufficient adaptability to unseen categories.
By integrating both modules, CPR-NIAM achieves an optimal balance between acquiring new knowledge and preserving generalization. Importantly, this also suggests promising direction for future work by combining complementary strategies rather than relying on a single one.
(a) Caltech101
(c) Office-Home
Base
$4.4
Class
Product
43.5
$4.4
0.
0.2
1.0
(b) Flowers102
CSpart
77.0
16.0
Figure 3: The impact of hyperparameters on performance is evaluated by varying from
0.2, 0.7, 1.0], the number of prototypes per class from {3, 5, 10) on three datasets.
5.4 Sensitivity Analysis
This section evaluates the robustness of GPR-NIAM across varying hyperparameter settings on the DTD, Flowers 102 and Office-Home datasets. Specifically, we investigate the effects of hyperparame- ters n (the number of prototypes generated per class on the client side) and À (the weighting factor in the AttnIso module), evaluated over {3, 5, 10) and {0, 0.2, 0.5, 0.7, 1.0). respectively. As depicted in Figure 3 -PR-NIAM consistently achieves higher overall HM scores than its baselines in all cases of the base-to-basennovel generalization task, showcasing its superior balance between retaining trans- ferable knowledge and learning task-specific knowledge. On the one hand, OPR-NIAM demonstrates consistent improvements over baselines on base classes, which benefits from the CSCR module's ca- pacity to alleviate data heterogeneity. On the other hand GPR-NIAM exhibits competitive performance on novel-class generalization, frequently outperforming other methods, which attributed to attention masking mechanism of the Attnlso module. Furthermore, in the Leave-One- One-Domain- Out Generalization task, similar conclusions are drawn, as GPR-NIAM outperforms the baseline across multiple domains.

---

Table 4: Performance comparison between GPR-NIAM and baselines on DTD and Flowers with local training epochs E {1, 5, 20) in the mase-to-base/novel generalization setting.
(Epochs=1)
(b) DTD (Epochs=5)
(Epochs=20)
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
53.35
59.05
56.06
CLIP
53.35
59.05
56.06
CLIP
53.35
59.05
56.06
PromptFL
50.34
51.69
51.01
PromptFL
55,55
56.76
56.15
PromptFL
63.31
51.08
56.54
RPO
52.77
59,42
55.90
RPO
53.24
59.78
56.32
RPO
53.93
60.26
56,92
FedTPG
40.27
36.47
38.28
FedTPG
49.88
47.22
48.51
FedTPG
56.25
56.28
56.26
TCP
53.24
53.38
53.31
TCP
57.63
56.40
57.01
TCP
63.07
59.54
61.25
PromptFolio
52.54
53.50
53.02
PromptFolio
58.68
56.28
57.45
PromptFolio
58.68
57.60
58.13
DP-FPL
43.75
41.54
42.61
DP-FPL
59.25
50.12
54.30
DP-FPL
64.12
47.82
54.78
GPR-NIAMP
65.97
51.20
$7.65
GPR-NIAMP
68.51
52.65
59.54
GPR-NIAN,
70.13
53.26
60.54
GPR-NIAN,
63.07
51.81
56.89
GPR-NIANT
55.07
59.98
GPR-NIAMT
69.67
54.83
(d) Flowers102 (Epochs=1)
(e) Flowers102 (Epochs=5)
(f) Flowers102 (Epochs:20)
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
CLJIP
62.39
78.10
69.36
CLIP
62.39
78.10
69.36
CLIP
62.39
78.10
69.36
PromptFT.
70.84
74.10
72.43
PromptFT.
78.42
76.00
77.19
PrompiFI.
80.46
73.68
76.92
RPO
69.97
76.21
72.95
RPO
71.72
76.84
74.19
RPO
74.05
77.05
75.52
FedTPG
68.51
77.05
72.53
FedTPG
73.76
77.05
75.37
FedTPG
74.34
75.67
TCP
75.51
78.10
76.78
TCP
76.09
79.15
77.59
TCP
79.88
78.31
79.09
PromptFolio
73.76
76.42
75.06
PromptFolio
79.59
75.57
77.53
PromptFolio
83.09
74.52
78.57
DP-FPL
69.38
78.31
73.58
DP-FPL
73.76
79.57
76.55
DP-FPL
75.80
75.78
75.79
GPR PPR-NIAMp
82,50
75.57
78.89
GPR G
84.83
78.31
81.44
GPR-NTAN,
85.13
77.47
81.12
GPR-NIAMT
87.11
77.83
82.20
GPR-NIAMT
86.58
78.94
82.59
GPR-NIAM
87.75
78.73
83.00
Cougar Body
Dollar Bill
Joshua Tree
Faces
Cannon
Bonsai
Kangaroo
Mandolin
Lotus
Nautilus
Input Image
Before
Refinement
After
Refinement
(a) Base Class
(b) Novel Class
Figure 4: Visualization of similarity heatmaps between text features and image patches before and after prompt refinement on Caltech101. In the similarity maps, red and yellow indicate higher similarity between image patches and the text features, while blue represents lower similarity.
5.5 Performance Comparison across Different Local Training Epoch Settings
To assess the robustness of the proposed CPR-NIAM under different local training efforts, we evaluate its performance with local epochs E 1,5, 20, data heterogeneity level of 0.5, and 10 participat- ing clients. As shown in Table both GPR-NIAM, and CPR-NIAM7 consistently outperform baselines across all datasets and settings. Notably, GPR-NIAMr achieves the highest harmonic mean (HM) in most cases, indicating better generalization across base and novel classes. On the Flowers 102 dataset, maintains strong performance with increasing local epochs, reflecting robustness against overfitting. Overall, both variants remain stable and effective despite changing local training dynamics.
5.6 Effectiveness of Global Prompt Refinement
This section aims to evaluate the effectiveness of the global prompt refinement method. Specifically, we visualize the similarity maps between text features and image patches before and after refinement, shown in Figure Intuitively, the similarity responses between visual targets and textual semantics

---

are often scattered and misaligned before refinement. For example, in the "Dollar Bill" and "Bonsai" samples, the high similarity responses in the second row primarily concentrate on background regions near the image boundaries. In contrast, the responses become significantly more concentrated on semantically relevant regions, such as the face on the bill the foliage of the bonsai after refinement. Notably, this advantage is also evident in novel classes. For instance, the refined prompts enable the model to better localize foreground objects while suppressing irrelevant background activations in "Kangaroo" and "Nautilus" samples. Moreover, it becomes tightly aligned with the spiral structure of the shell in the "Nautilus", indicating enhanced semantic grounding.
Flower102
UCF1OL
CTD
Flower102
Local
70
E490
Bisc
Novel
HM
50
Bise
Movel
Rowd
Figure 5: Comparison between Random and Lo- cal Prompt-Tuning for Initialization.
Figure 6: Failure case analysis. Most GPR-NIAM errors overlap with zero-shot CLIP.
5.7 Evaluation of Local Prompt Tuning as Initialization
This section evaluates the performance of random initialization and local prompt tuning (LPT) Flowers 10,, UCF101 and DTD datasets. As shown in Figure LPT yields superior performance compared to random initialization on all cases. This can be attributed to the local training that be more aligned with the target data, allowing the model to better capture domain domain-specific features. In particular, LPT outperforms the random initialization by 6% the DTD dataset for base classes, which highlights the method's ability to adapt more efficiently to the nuances of the texture.
5.8 Failure Case Analysis
This section reports the percentage of cases where the zero-shot CLIP model also failed in instances where the GPR-NIAM model made incorrect predictions (Figure Clearly, there is a high overlap in the failure cases of both models. This reflects that the imprecision of zero-shot generalized knowledge hinders the model's generalization, which also provides insights for future work to address the challenges posed by the lack of domain-speciffe adaptation in generalization models.
6 Conclusions and Future Work
In this paper, we balance the model capability to learn task-specific knowledge for downstream tasks with its capacity retain generalized knowledge. Specifically, propose GPR-NIAM for one-shot FPL, which introduces an attention masking mechanism to limit the interaction between the Icarnable prompt tokens and the original text tokens. Furthermore, learns unified prompt through global prompt refinement, thereby mitigating the data heterogeneity issue. Experimental results demonstrate that GPR-NIAM significantly improves the performance of global prompts across diverse visual tasks.
Despite the promising results of GPR-NIAM, there remains room for improvement. The communication overhead from prototype transmission may become bottleneck in large-scale federated systems with limited bandwidth, which we plan to address in future work. And both the multi-label
(68
669. multi-modal (70 71 72 73 problems and real-world medical imaging tasks 774
75 76
become considerably more challenging in federated settings. In addition, several issues discussed in Section 5.8 remain open and deserve further investigation.
Acknowledgments
This work is supported in part by the Key Research and Development Program of Shandong Province (Grant No. 2024TSGC0667); the Ministry of Education, Singapore, under its Academic Research Fund Tier I (RG101/24); the RIE2025 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) (Award F2301,0026), administered by A STAR,, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e- e-Sustainability CorpLab (ANGEL).
10

---

References
[1] Zhuang Qi, Weihao Hc, Xiangxu Meng, and Lei Meng. Attentive modeling and distillation for out-of- distribution generalization federated learning. 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6, IEEE, 2024.
[2] Jinyu Cai, Yunhe Zhan, Zhoumin Lu, Wenzhong Guo, and See- See-Kiong Ng. Towards effective federated graph anomaly detection via self-boosted knowledge distillation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5537-5546, 2024.
[3] Xianjie Guo, Kui Yu, Hao Wang, Lizhen Cui, Han Yu, and Xiaoxiao Li. Sample quality heterogeneity- aware federated causal discovery through adaptive variable space selection. In Proceedings of the Thirty-Thine International Joint Conference on Artificial Intelligence, pages 4071-4079, 2024.
[4] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 30-3),00-60, 2020.
[5] Lele Fu, Bowen Deng, Sheng Huang. Tianchi Liao, Shirui Pan, and Chuan Chen. Less more: Feder- ated graph learning with alleviating topology heterogeneity from a causal perspective. In Forty-second International Conference on Machine Learning, 2025.
[6] Jinyu Cai, Yunhe Zhang, Jicong Fan, and See-Kiong Ng. Lg-fgad: An effective federated graph anomaly detection framework. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 3760 3769, 2024.
[7] Xinghao Wu, Jianwei NIu, Xuefeng Liu, Tao Ren, Zhangmin Huang, and 7hetao Li. pfedgf: Enabling personalized federated learning via gradient fusion. In 2022 IEEE International Porallel and Distributed Processing Symposium (IPDPS), pages 639-649, 2022.
[8] Liping Yi, Han Yu, Chao Ren. Gang Wang, Xiaoxiao Li, et al. Federated model heterogeneous matryoshka representation learning. Advances in Neural Information Processing Systems, 37:66431-664555 2024.
[9] Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. pfedes: Model heterogeneous personalized federated learning with feature extractor sharing. arXiv preprint urXiv:2311.06877 2023.
[10] Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages $748-8763 PmLR, 2021.
[11] Zhuang Qi, Sijin Zhou, Lei Meng. Han Hu, Han Yu, and Xiangxu Meng. Federated deconfounding and debiasing learning for out-of-distributtin generalization. arXiv preprint MXXiv::500504977 2025.
[12] Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication-efficcee and privacy- Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IREE, 2023.
privacy-preserving preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International
[13] Chao Ren, Han Yu, et al. Advances and open challenges in federated foundation models. IEEE Communi- cations Surveys and Tatorials, 2025.
[14] Tao Fan, Hanlin Gu, et al. Ten challenging problems in federated foundation models. IEEE Transactions on Knowledge and Data Engineering, 2025.
[15] Fu-En Yang. Chien-Y Yi Wang, and Yu-Chiang Frank Wang. Elficient model personalization in federated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19159-19168, 2023.
[16] Xianjie Guo, Kui Yu, Lizhen Cui, Han Yu, and Xiaoxiao Li. Federated causally invariant feature learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 16978-16986, 2025.
[17] Bikang Pan et al. Federated learning from vision- languae foundation models: Theoretical analysis and method. Advances in Neural Information Processing Systems, 37:30590-30633 30623, 2024.
[18] Wenke Huang, Mang Yc, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. Federated learning for generalization, robustness, fairnsss: A survey and benchmark. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[19] Jiahua Dong, Lixu Wang, Zhen Fang, et al. Federated class- incremental learning. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2022.
11

---

[20] Lele Fu, Sheng Huang, Yanyi Lai, Chuanfu Zhang, Hong-Ning Dai. Zibin Zheng, and Chuan Chen. Federated domain-inddeenddnl prototype learning with alignments of representation and parameter spaces for feature shift. IEEE Trunsactions on Mobile Computing, 2025.
[21] Jiayuan Zhang. Xuefeng Liu, Yukang Zhang, Guogang Zhu, Jianwei Niu, and Shaojie Tang. Enabling collaborative test-time adaptation in dynamic environment via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4191-4202, 2024.
[22] Linh Tran, Wei Sun, Stacy Patterson, and Ana Milanova. prompt learning for multimodal large language models. In The International Conference on Learning Representations, pages 1-19, 2025.
personalized federated
[23] Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. Harmonizing generalization and personalization in federated prompt learning. arXiv preprint XVXiv22405.09771 2024.
[24] Hongxia Li, Wei Huang, Jingya Wang, and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12151-12161, 2024.
[25] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414-2144 2022.
[26] Hui Zeng, Minrui Xu, Tongqing Zhou, Xinyi Wu, Jiawen Kang, Zhiping Cai, and Dusit Niyato. One- Multimedia, pages 1070-11079) 2024.
federated learning. In Proceedings of the 32md ACM International Conference on
[27] Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing lahel skews in one-shot federated learning. In The Eleventh International Conference on Learning Representations, 2023.
[28] Mingzhao Yang, Shangchao Su, Bin Li, and Xiangyang Xue. Exploring onc-shot semi- semi-supervised federated learning with pre-trained diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16325-16333 2024.
[29] Xinghao Wu, Jianwei Niu, Xuefeng Liu, Guogang Zhu, Shaojie Tang, Wanyu Lin, and Jiannong Cao. The diversity bonus: Learning from dissimilar clients in personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2025.
[30] Dezhong Yao, Yucxin Shi, Tongtong Liu, and Zhiqiang Xu. Fedmho: Heterogencous onc-shot federated lcarning towards resource-constrainee edge devices. arXiv preprint arXiv:2502.085//, 2025.
[31] Yanbiao Ma, Wei Dai, Wenke Huang, and Jiayi Chen. Geometric knowledge guideed localized global distri- bution alignment for federated learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20958-20966 2025.
[32] Ming Hu, Yue Cao, Anran Li, Zhiming Li, Chengwei Liu, Tianlin Li, Mingsong Chen, and Yang Liu. Fedmut: Generalized federated learning via stochastic mutation. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 12528 12537, 2024.
[33] Shangehao Su, Mingzhao Yang, Bin Li, and Xiangyang Xue. Federated adaptive prompt tuning for multi-domain collaborative learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 15117-15125, 2024.
[34] Yongju Jia, Jiarui Ma, Xiangxian Li, Baiqiao Zhang, Xianhui Cao, Juan Liu, and Yulong Bian. Lim- empowered dynamic prompt routing for vision- languale models tuning under long-tailed distributions. arXiv preprint arXiv:2508. 15688, 2025.
[35] Tao Guo, Song Guo, and Junxiao Wang. Pfedprompt: Learning personalized prompt for vision-languse models in federated learning. In Proceedings of the ACM Weh Conference 2023, pages 1364-13374 2023.
[36] Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompt-tuning in bridging generalized and personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6087-6097, 2024.
[37] Yichen Li. Xiuying Wang, Wenchao Xu, Haozhao Wang. Yining Qi. Jiahua Dong, and Ruixuan Li. Feature distillation is the better choice for hodee--hetroogeneous federated learning. arXiv preprint arXiv:2507.10344 2025.
12

---

[38] Mingzhao Yang, Shangchao Su, Bin Li, and Xiangyang Xue. Feddeo: Description-enarnn enhance one-shot federated learning with diffusion models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6666-6675, 2024.
[39] Youssef Allouah, Akash Dhasade, Rachid Guerraoui, Nirupam Gupta, Anne-Marie Kermarrec, Rafael Pinot, Rafael Pires, and Rishi Sharma. Revisiting ensembling in one-shot federated learning. arXiv preprint Axiiv:4411.071880 2024.
[40] Yi-Han Chiang, Koudai Terai, Tsung-Wei Chiang. Hai Lin, Yusheng Ji, and John CS Lui. Optimal transport-based one-shot federated learning for artificial intelligence of things. IEEE Internet of Things Journal, 1 12))211662218)) 2023.
[41] Clare Elizabeth Heinbaugh, Emilio Luz-Ricea, and Huajie Shao. Data-free one-shot federated leam- ing under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.
[42] Naibo Wang. Yuchen Deng, Wenjie Feng. Shichen Fan, Jianwei Yin, and See-Kiong Ng. One-shot sequential foderated learning for non-iid data by enhancing local model diversity. In Proceedings of the 32nd ACM International Conference Multimedia, pages 5201-5210, 2024.
[43] Ming Hu, Priheng Zhou, Zhihao Yue, Zhiwei Ling, Yihao Huang, Anran Li, Yang Liu, Xiang Lian, and Mingsong Chen. Fedcross: Towards accurate federated learning via multi-model cross-aggregation. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 2137-2150. IERE, 2024.
[44] Zhuang Qi, Runhui Zhang, Lei Meng. Wei Wu, Yachong Zhang, and Xiangxu Meng. Global intervention and distillation for federated out-of-distribution generalization. arXiv preprint arXiv:2504. 00850, 2025.
[45] Yichen Li, Haozhao Wang. Yining Qi, Wei Liu, and Ruixuan Li. Re-fed+: A better replay strategy for federated incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[46] Jiayuan Zhang, Xuefeng Liu, Jianwei Niu, et al. Causality inspired federated learning for ood generalization. In Forty-second International Conference on Machine Learning.
[47] Zhuang Qi, Lei Meng, Zitan Chen, Han Hu, Hui Lin, and Xiangxu Meng. Cross-silo prototypical calibration for federated learning with non-iid data, In Proceedings of the 31st ACM International Conference on Multimedia, pages 3099-3107, 2023.
[48] Lei Meng, Zhuang Qi, Lei Wu, Xiaoyu Du, Zhaochuan Li, Lizhen Cui, and Xiangxu Meng. Improving global generalization and local personalization for federated learning. IEEE Transactions on Neural Networks and Learning Systems. 36(1):76-87, 2025.
[49] Zhuang Qi. Lei Meng. Weihao He, et al. Cross with multi view knowledge fusion for heterogenous federated learning. arXiv preprint arXiv:2405.20044 2024.
[50] Xiangyu Meng. Xiangxian Li, Lin Zhao, Yajuan Shen, Hui Sun, Xiaoming Cong, Xianhui Cao, Yunfeng Bi. Juan Liu, and Yulong Bian. Robust representation of domain incomplete tabular data via cross modal debiasing and prototype- reconstruction. Journal of King Saud University Computer and Information Sciences, 37(7):009, 2025.
[51] Zhuang Qi. Lei Meng, Zhaohcuan Li, Han Hu, and Xiangxu Meng. Cross-silo feature space alignment for federated learning on clients with imbalanced data, In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1 9, 2025.
[52] Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan- Yi Lin. Federated text-driven prompt generation for rision-languas models. In The Twelfth International Conference on Learning Representations, 2024.
[53] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[54] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[55] Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and puttern recugnition workshop, pages 178-178. IEEE, 2004.
[56] Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3606 3613, 2014.
13

---

[57] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-graines visual classification of aircraft. arXiv preprint 1rXiv:1306.5151, 2013.
[58] Maria- Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729.
IEEE, 2008.
[59] Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3d object representations for fine-grainel categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554 561, 2013.
[60] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from vidcos in the wild, arXiv preprint arXiv: 2012.
[61] Antonio Torralba and Alexci Efros. Unbiased look dataset bias. In CVPR 2011, pages 1521-1528. JEEE, 2011.
[62] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEF/CVF international conference computer vision. pages 1406-1415, 2019.
[63] Haoyuan Liang, Xinyu 7hang, Shilei Cao, Guowen Li, and Juepeng Zheng. Tea-feddg: Leveraging test-time adaptation to address federated domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18658-18666 1158-18666. 2025.
[64] Wenhao Li. Qiangchang Wang, Peng Zhao, and Yilong Yin. Knn transformer with pyramid prompts for few-shot learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1082-1091, 2024.
[65] Tao Guo, Song Guo, Junxiao Wang. Xueyang Tang. and Wenchao Xu. Promptil: Let federated participants cooperatively Icarn prompts instead of feders-federated learning in age of foundation model. IEEE Transactions on Mobile Computing, 23(5):5179-5199 2023.
[66] Dongjun Lee, Scokwon Song, Jihee Suh, Joonmycong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEEF/CVF international conference computer vision, pages 1401-1411, 2023.
[67] Hantao Yao, Rui Zhang, and Changsheng Xu. Tep: Textual-based class-aware prompt tuning for visual- language model. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 23438-23445 2024.
[68] Zhiqiang Kou, Si Qin, Hailin Wang. Mingkun Xie, et al. Label distribution learning with biased annotations by learning multi-label representation. 2025.
[69] Zhiqiang Kou, Jing Wang. Jiawei Tang, Yuheng Jia, Boyu Shi. Xin Geng. Exploiting multi-label corre- lation in label distribution learning. In Kate Larson, editor, Proceedings of the Thirty- -Tird International Joint Conference on Artificial Intelligence IJCAI-24, pages 4326 4334, International Joint Conferences on Artificial Intelligence Organization, 8 2024. Main Track.
[70] Lei Meng, Xiangxian Li, Xiaoshuo Yan, Hankai Ma, Zhuang Qi, Wei Wu, and Xiangxu Meng. Causal infer- ence over visual- semaaaic--aiiiggeeee graph for image classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19449-19457, 2025.
[71] Zheng Wang, Zhenwei Gao, Yang Yang, et al. Geometric matching for cross-modal retrieval. JEFE Transactions on Neural Networks and Learning Systems, 36(3):5509-5521 2025.
[72] Yuqing Wang, Zhuang Qi, Xiangxian Li, Jinxing Liu, Xiangxu Meng, and Lei Meng. Multi-channel attentive weighting of visual frames for multimodal video classification. In 2023 International Joint Conference on Neurul Networks (IJCN), pages 1-8, IEEE, 2023.
[73] Harry Cheng, Yangyang Guo, Tianyi Wang, al. Voice-face homogeneity tells deepfake. ACM Transac- tions on Multimedia Computing, Communications and Applications. 20(3):1-22, 2023.
[74] Wei Feng, Lie Ju, Lin Wang, Kaimin Song. Xin Zhao, and Zongyuan Ge. Unsupervised domain adaptation for medical image segmentation by selective entropy constraints and adaptive semantic alignment. In 2023.
Proceedings of the AAAI Conference Artificial Intelligence, volume 37. pages
[75] Wei Feng. Sijin Zhou, et al. Neighbor-guinn guided unbiased framework for generalized category discovery in medical image classification. IEEE Journal of Biomedical and Health Informatics, 2025.
[76] Chengrun Dang, Zhuang Qi. Tao Xu, Mingkai Gu, et al. Deep learning- whole slide image analysis in cancer pathology. Laboratory Investigation, page 104186, 2025.
14

---

A Appendix
A.1 Datasets
conduct our evaluation. we selected 10 diverse visual classification datasets as benchmarks. Table S provides a detailed overview, including the number of classes, the size of training and testing sets, the number domains, and the corresponding federated settings.
For datasets with multiple domains, adopt the commonly used Office-Home benchmark, which includes four domains: Art, Clipart, Product, and RealWorld. In addition, utilize DomainNet, large-scale dataset spanning six domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. We focus on training with selected subset of 20 classes from each datasct. Figure illlstrates representative raw instances from the two multi-domain datasets.
Table 5: Statistics and federated settings of the datasets used our experiments.
Test
Federated Settings
Dataset
Classes
Train
Domains
Base
Novel
Clients
Heterogeneity
CIFARIO
25,000
5,000
5,000
5/10/20
0.1/0.5
OxfordPets
37
1,785
1,782
1,887
5/10/20
0.1/0.5
Caltech101
100
3,310
1,416
1,178
5/10/20
0.1/0.5
DTD
47
1,440
864
828
5/10/20
0.1/0.5
FGVCAircraft
100
3,333
1,667
1,666
5/10/20
0.1/0.5
Flowers 102
102
2,807
343
475
5/10/20
0.1/0.5
StanfordCars
196
4,052
4,002
3,998
5/10/20
0.1/0.5
UCF101
101
3,926
1,895
1,888
5/10/20
0.1/0.5
DomainNet
20
30,777
9/15
Office-Hoome
65
15,588
15/25
Clipart
Art
infograph
Clipart
Painting
Quickdraw
Product
Real
Sketch
RealWorld
(a) DomainNet
(b) Office-Home
Figure 7: Illustrative examples of class instances across domains: "Ambulance" from DomainNet and "Bed" from Office-Home.
A.2 Experimental Results
A.2.1 Base-to-Base/Novel Generalization under High-Level Data Heterogeneity
To more comprehensively evaluate the effectiveness of the methods, we test all approaches on eight datasets under higher level of heterogeneity (3 = 0.1). focusing their performance in the base-to mase-to-based base/novel generalization setting. As presented in Table 6 our proposed method GPR-NIA obtains the highest average
15

---

Table 6: Performance comparison between GPR-NIAM and baselines 8 datasets with / 0.1 in the base-to-base/nont bawbbeeee//velleeee generalization setting. The best performance indicated in bold.
(a) Average over datasets
(b) CIFARIO
(c) OxfordPets
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
65.79
74.32
69.70
CLIP
95.30
95.50
95.39
CLIP
81.53
95.33
87.89
PromptFL
69.15
72.50
70.71
PromptFL
94.72
95.68
95.19
PromptFL
89.33
95,91
92.51
RPO
67.81
73.66
70.57
RPO
95.10
94.76
94.92
RPO
86.19
95.54
90.63
FedTPG
67.06
71.71
69.20
FedTPG
94.30
96.64
95.45
FedTPG
86.58
94.70
90.46
TCP
70.30
74.19
72.13
TCP
95.30
95.66
95.47
TCP
90.01
96,23
93.02
PromptFolio
69.49
73.44
71.33
PromptFolio
95.20
96.74
95.96
PromptFolio
90.90
95.17
92.99
DP-FPL
71.90
71.75
69.88
DP-FPL
94.24
95.42
94,82
DP-FPL
91.30
95.12
93.17
FedDISC
52.27
54.40
50.47
FedDISC
92.47
91.37
91.91
FedDISC
81,48
66.74
73.38
FedELMY
68.05
47.29
53.41
FedELMY
89.84
28.90
43.73
FedELMY
86.70
82.77
84.69
GPR-NIAMP
73.87
72.84
73.18
GPR-NIAMP
95.88
96.84
96.35
GPR-NIANP
90.57
95.23
92.84
GPR-NIAMT
73.62
74.48
73.97
GPR-NIAMT
96.03
96.24
96.13
GPR-NIANT
91.86
94.38
93.10
Caltech101
DTD
FGVCAireraft
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
82.27
94.05
87.77
CLIP
53.35
59.05
56.06
CLIP
26.51
31.21
28.67
PromptFL
87.07
93.03
89.95
PromptFL
56.36
52.53
54.38
PrompiFL
26.03
30.61
28.13
RPO
84.95
93.80
89.16
RPO
53.12
60.02
56.36
RPO
26.51
30.37
28.31
FedTPG
85.38
93.37
89.20
FedTPG
53.35
50.00
$1.62
FedTPG
21.17
26.35
23.48
TCP
88.41
94.16
91.19
TCP
58.10
57.60
57.85
TCP
28.37
32.23
30.18
PromptFolio
82.41
92.61
87,21
PromptFolio
53.58
55.55
54.55
PromptFolio
28.13
34.15
30.85
DP-FPL
87.57
93.54
90.46
DP-FPL
51.85
45.65
48.55
DP-FPL
26.63
29.35
27.92
FedDISC
22.37
81,39
35.09
FedDISC
65.23
41.23
49.87
FedDISC
18.49
17.61
18.03
FedELMY
85.59
90.40
87,93
FedELMY
40.27
11.95
18.43
FedELMY
23.45
5.46
8.86
GPR-NIAM.
91.24
93.88
92.54
GPR-NTAM
67.59
52.89
59.34
GPR-NIAN
29.69
30.81
30.23
GPR-NIANT
90.60
94.05
92.30
GPR-NIAMT
65.62
57.79
61.45
GPR-NIANT
29.39
29.96
29.67
(g)
Fllower1022
(h)
CannoodCCars
(i) UCF101
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
CLJIP
62.39
78.10
69.36
CLIIP
56.87
69.40
62.51
CLIIP
68.12
71.92
69.97
PromptFL
72.88
75.78
74.30
PromptFL
57.31
67.58
62.02
PromptFL
69.55
68.90
69.22
RPO
72.01
76.63
74.24
RPO
$7.64
66.73
61.85
RPO
67.01
71.45
69.16
FedTPG
70.84
77.26
73.91
FedTPG
56.89
69.55
62.59
FedTPG
68.02
65.83
66.91
TCP
73.61
77.94
75.71
TCP
57.79
69.28
63.01
TCP
70.88
70.43
70.65
PromptFolio
75.51
76.42
75.96
PrompiFolio
58.97
69.05
63.61
PrompiFolio
71.29
67.84
69.52
DP-FPL
68.97
71.43
70.08
DP-FPL
53,99
63.65
58.43
DP-FPL
73.61
69.80
71.66
FedDISC
25.44
32.47
28.52
FedDISC
42.59
51.23
46.51
FedDISC
70.12
53.23
60.51
FedELMY
81.63
57.05
67.16
FodELMY
60.26
49.94
54.62
FedELMY
76.72
51.90
61.92
GPR-NIAM,
79.30
77.26
78.26
GPR-NIAMP
60.81
66.98
63.75
GPR-NIAMP
75.88
68.85
72.19
GPR-WIAMy
80.17
79.15
79.65
GPR-NIAM?
60.14
68.65
64.12
GPR-NIANT
75.19
75.63
75.41
performance on the eight datasets. Specifically, GPR-NIAMP and GPR-NIA -IAMM achieve the top two performances on the base classes, ranking first and second. respectively. In terms of novel class generalization, GPR-NIAM, outperforms all baselines, demonstrating its superior capability to adapt to unseen categories. Furthermore. GPR-NIAMy also achieves the highest overall performance in terms of harmonic mean (HM), indicating strong balance between knowledge retention and generalization. These results highlight the robustness and adaptability of the proposed method under highly heterogencous federated settings.
A.2.2 Leave- Oww---nnein--OOOO-- Generalization with Increased Client Diversity
To validate the effectiveness of the proposed EPA-NIAM under more challenging scenarios, we conduct experi- ments in a Srave-one-domiin-on
generalization setting [] using the Office-Home and DomainNet datasets. In this setting. the data from each domain is randomly partitioned among 5 clients, and for each evaluation, one domain is held out entirely for testing while the remaining domains are used for federated training. As shown in T P and GPR-NIANI NIAMy consistently outperform existing baselines across both datasets. For the Office-Home, OPR-NIAMy achieves the highest average accuracy, with GPR-NIAMp closely following. Notably, MPR-NIAMP yields the best performance on the "RealWorld" and "Product" domains, while GPR-NIAM, leads on "Art" and "Clipart". For the DomainNet, both variants of GPR-NII PRM-NIAM also demonstrate strong cross- domain
16

---

Table 7: Performance comparison between GPR-NIAM and baselines the Office-Home and Do- mainNet datasets under the leave-one-domain-o0 generalization setting, where the data from each domain is randomly partitioned among 5 clients.
Offic-Homee
DomainNet
Method
Art
Clipart
Product
RealWorld
Average
Clipart
Infograph
Painting
Quickdraw
Real
Sketeh
Average
74.57
62.33
80.
79.38
74.08
77.02
98.61
94.22
$1.85
PrompitFL
73.71
64.49
75.35
80.09
52.68
99.00
95.63
RPO
73.83
62.13
79.95
79.36
73.81
93.66
77.31
84.65
44.46
98.83
93.87
82.13
FedTPG
74.90
65,54
82.85
80.37
75.91
94.35
79.76
87.40
48.05
98.92
95.89
84,06
TCP
75.77
65.10
83.10
80.99
76.24
93.92
79.72
87.67
46.82
98,96
95.00
83.68
PromptFolio
75.81
65,36
83.53
80.39
76.27
95.51
81.00
89.33
48.82
98.99
95.20
84.80
DP-FPL
74.82
63.55
82.85
78.79
75.00
94.77
80.20
89.00
48.35
98.78
94.82
84.32
FedDISC
71.19
62.48
81.31
78.35
73.48
94,23
80.25
83.85
44.21
98.
92.49
82.23
FedELMY
75.45
64,29
$2.36
77.58
74.92
94.15
75.48
86.89
47.39
98.72
95.13
82.96
78.16
72.76
89,09
80,92
80.60
58.68
99.10
94.91
85.
GPR-XIAMr
77.66
71.95
89.47
85,93
81.25
95.19
80.49
87.23
57,96
99.07
95.37
85.88
generalization, achieving top results in multiple domains. These results highlight the robustness of our method handling both domain shift and non-IID data across clients. In summary, the proposed GPR-NIAM exhibits superior generalization performance under ceree-onn-domiine conditions, demonstrating its effectiveness in federated scenarios with high domain and client-level variability.
A.2.3 Performance Comparison across Different Local Training Epoch Settings
Table Performance comparison between EPR-NIAM and baselines on StanfordCars and UCF101 with local training epochs 5,, 20} in the base-to-basennovel generalization setting.
(g) StanfordCars (Epochs=1)
(h) StanfordCars (Epochs=5)
(i) StanfordCars (Epochs=20)
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
56.87
69.40
62.51
CLIP
56.87
69.40
62.51
CLIP
56.87
69.40
62.51
PromptFL
58.87
68.40
63.27
PromptFL
60.39
67.35
63.68
PromptFL
59.74
67.83
63.52
RPO
56.39
69.03
62.07
RPO
57.12
69.70
62.79
RPO
57.42
70.06
63.11
FedTPG
$7.09
69.73
62.78
FedTPG
58.14
69.65
63,38
FedTPG
58.69
69.25
63.54
TCP
58.04
69.40
63.22
TCP
58.72
69.91
63.82
TCP
$9.29
68.68
63.64
PromptFolio
58.69
69.83
63.78
PromptFolio
60.04
69.78
64.54
PromptFolio
61.46
69.10
65.06
DP-FPL
49.30
62,85
55.25
DP-FPL
59.02
68.33
63,33
DP-FPL
59.24
64.35
61.69
CPR-NIAMP
60.61
67.58
63.91
GPR-NIAMP
62.09
67.05
64.48
GPR-NIAMP
64.46
66.38
65.41
GPR-NIAMY
61.54
67.40
64.33
GPR-NIAM?
60.99
69.45
64.94
GPR-NIANG
63.01
69.00
65.87
(j) UCF101 (Epochs=1)
(k) UCF101 (Epochs=5)
(l) UCF101 (Epochs-20)
Method
Base
Novel
HM
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
68.12
71.92
69.97
CLIP
68.12
71.92
69.97
CLIP
68.12
71.92
69.97
PromptFL
69.55
69.01
69.28
PromptFL
73.03
72.08
72.55
PromptFL
68.07
64.24
66.10
RPO
65.96
70.55
68.17
RPO
66.80
71.50
69.07
RPO
68.75
71.13
69.92
FedTPG
64.69
64.08
64.39
FedTPG
68.86
67.47
68.16
FedTPG
68.86
66.73
67.77
TCP
73.82
72.47
73.13
TCP
75.83
72.27
74.00
TCP
76.09
73.00
74.51
PromptFolio
73.72
69.80
71.71
PromptFolio
76.56
70.12
73.20
PromptFolio
78.15
68.37
72.94
DP-FPL
72.13
70.65
71.38
DP-FPL
74.08
71.76
72.91
DP-FPL
73.24
68.16
70.61
GPR-NIAMP
76.88
68.53
72.47
GPR-NIAMP
79.10
68.75
73.56
GPR-NIAKS
78.89
69.38
73.83
GPR-NIAMT
77.78
74.62
76.17
GPR-NTAMY
78.15
75.79
76.95
GPR-NTANT
78.04
76.58
77.31
To evaluate the effectiveness of the proposed GPR-NIAM under different levels of local training effort, this section investigates its performance across varying local training epochs. Specifically, we consider three settings with local training epochs E { 5, 20), under a data heterogeneity level (8 0.5) and with 10 clients participating in training. As shown in Table andd Table [ both GPR-NIAM, and GPR-NIAMy consistently outperform the baselines across all datasets and training epochs. Notably, GPR-NIAMT delivers the best overall performance in terms of harmonic mean (HM) in most settings, which demonstrates superior generalization across both base and novel classes. Flowers102 102 and UCF101, GPR-NIAM/ shows strong robustness even when the number of local epochs increases. This highlights its ability to prevent overfitting to local distributions. Across datasets, both variants of GPR-NIAN remain stable and effective even as the local training dynamics change, outperforming methods that are more sensitive to overfitting class imbalance under prolonged local updates.
17

---

A.2.4 Sensitivity Analysis Regarding the Number of Prompt Tokens.
Table 9: Comparison of different methods under two settings (5x512 and 20x512) Caltech 101, DTD, and UCF101 datasets. Bold numbers indicate the best performance.
Caltech101
DTD
UCF101
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
5x512
PromptFL
88.41
93.97
91.11
59.25
52.77
55.83
76.78
69.91
73.18
FedTPG
86.65
93.37
89.88
60.18
56.15
58.10
74.30
70.63
72.41
PromptFolio
89.47
93.46
91.42
61.02
56.28
57.62
74.40
70.65
72.48
GRP-NIAM P
89.90
93.37
91.60
60.96
55.95
59.12
77.98
70.39
73.99
20x512
PromptFL
88.91
94.31
91.53
62.50
52.53
57.08
76.83
69.12
72.77
FedTPG 88.84
92.86
90.81
58.10
54.83
56.41
76.30
73.41
74.41
PromptFolio
89.97
94.22
92.05
56.13
53.98
55.03
75.72
74.41
75.06
GRP-NIAM P
90.02
94.28
92.10
61.43
56.23
58.71
77.23
74.58
75.88
We have conducted additional sensitivity experiments regarding the number of prompt tokens. Specifically, we adjusted the prompt length 5x512 and 20x512. The results show that GPR-NIAM remains robust across different prompt lengths, while consistently maintaining advantage over all baselines in all case.
A.2.5 Evaluating Method Scalability with Varying Client Numbers
Table 10:
Performance comparison between GPR-NIAM and baselines on DTD and Flowers 102
datasets under varying number of clients K {20, 30} in the base-to-base/novel generalization.
(a) DTD (K=20)
(b) DTD (K=30)
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
53.35
59.05
56.06
CLIP
$3.35
59.05
56.06
PromptFL
54.16
54.22
54.19
PromptFL
$2.89
56.52
54.64
RPO
53.24
60.02
56.42
RPO
52.77
59.66
56.00
FedTPG
57.83
54.93
56.34
FedTPG
52.19
58.09
54.98
TCP
58.21
55.93
57.04
TCP
56.82
58.42
57.60
PromptFolio
58.18
52.91
55.41
PromptFolio
55.43
52.41
53.88
DP-PPL
56.13
50.72
53.29
DP-PPL
57.06
51.20
53.97
GPR-NIAMP
63.31
52.29
57.27
CPR-NIAM
60.64
51.93
55.95
GPR-NIAMy
62.38
53.86
57.81
CPR-NIAMT
60.41
56.52
58.40
(d) Flowers102
(K=20)
(e) Flowers102
(K=30)
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
62.39
78.10
69.36
CLIP
62.39
78.10
69.36
PromptFI.
74.34
74.10
74.22
PromptF1.
72.01
74.73
73.34
RPO
71.72
76.84
74.19
RPO
71.13
76.42
73.68
FedTPG
74.63
78.31
76.43
FedIPG
74.92
78.31
76.58
TCP
75.21
77.68
76.43
TCP
73.46
78.73
76.01
PromptFolio
75.51
78.31
76.88
PromptFolio
73.76
78.31
75.97
DP-FPL
74.34
74.10
74.22
DP-FPL
69.09
75.57
72.19
GPR-NIAMP
82.79
77.05
79.82
CPR-NIAMP
81.63
78.31
79.93
GPR-NI NIA-NIMMy
83.96
77.05
80.36
BPR-NIAMy
82.50
78.94
80.68
This section aims to evaluate the scalability of the proposed GPR-NIAM with respect to the number of participating clients. We consider settings with K [20,30) 30) under heterogeneity level | 0.5 and fixed local training epochs As shown Table 10 and TT both PPR-NIAMP and GPR-NIAM, consistently outperform all baselines across datasets and clinnt configurations. GPR-NIAM, achieves the best overall performance on most datasets and demonstrates strong generalization to novel classes. For example, on Flowers 102 and UCF101, it achieves the highest harmonic mean (HM) under both client settings. Meanwhile, GPR-NIAMP maintains strong
18

---

Table 11: Performance comparison between GPR-NIAM and baselines StanfordCars and UCFI01
datasets under varying number of clients K E {20, 30} in the base-to- lase-to-base/novel generalization.
(g)
StanfordCars (K=20)
(h) StanfordCars (K=30)
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
56.87
69.40
62.51
CLIP
56.87
69.40
62.51
PromptFL
59.82
67.03
63.22
PromptFL
54.42
66,33
59.79
RPO
57.22
69.65
62.83
RPO
57.24
69.60
62.82
FedTPG
59.24
66.73
62.76
FedTPG
59.42
69.08
63.88
TCP
58.77
67.78
62.95
TCP
59.32
68,21
63.45
PromptFolio
60.91
65.78
63.25
PromptFolio
59.59
65.25
62.29
DP-FPL
57.74
65.65
61.44
DP-FPL
58.37
67.75
62.71
GPR-NIAMP
60.04
68.23
63.87
PM
60.09
67.40
63.54
GPR-NIAM7
61.02
69.40
64.94
CPR-NIAMr
60.14
69.38
64.43
(j) UCF101 (K=20)
(k) UCF101 (K=30)
Method
Base
Novel
HM
Method
Base
Novel
HM
CLIP
68.1 12
71.92
69.97
CLIP
68.12
71.92
69.97
PromptFL
75.67
68.75
72.04
PromptFL
71.34
66.04
68.59
RPO
66.80
71.34
69.00
RPO
66.75
71.39
68.99
FedTPG
74.88
74.78
74.83
FedTPG
76.09
74.62
75.35
TCP
74.08
73.83
73.96
TCP
73.50
75.58
74.53
PromptFolio
74.45
71.76
73.08
PromptFolio
77.09
74.04
75.54
DP-FPI.
75.51
69.75
72.52
DP-FPI.
72.71
68.96
70.79
GPR-NIAG NIAMP
76.41
69.01
72.52
GPR-NIAM:
76.62
72.50
74.50
GPR-NIAMY
77.04
76.21
76.62
GPR-NIAM7
77.62
76.05
76.83
performance on base classes. These results show that GPR-NIAM maintains high performance and stability even as the number of clients increases, demonstrating its scalability and effectiveness in more challenging cases.
A.2.6 Evaluation of Module Contributions
Table 12:
Ablation study on the effectiveness of NIAM and CSCR modules under the base-to- base/novel generalization. Results are reported on four datasets with PromptFL the baseline.
(a) DTD
(b)
Flowers102
Method
Base
Novel
HM
Method
Base
Novel
HM
PrompiFL
55.55
56.76
56.15
PrompiFL
78.42
76.00
77.19
+NIAM
55.78
54.73
55.25
+NIAM
76.13
77.78
76.94
+NIAM+CSCR
68.51
$2.65
59.54
+NIAM+CSCR
84.83
78.31
81.44
(c) StanfordCars
(d)
UCF101
Method
Base
Novel
HM
Method
Base
Novel
HM
PromptFL
60.39
67.35
63.68
PromptFL
73.03
72.08
72.55
+NIAM
55.17
68.13
60.96
+NIAM
67.96
73.02
70.39
+NIAM+CSCR
62.09
67.05
64.48
+NIAM+CSCR
79.10
68.75
73.56
We evaluate the contributions of the NIAM and CSCR modules by gradually adding them to the PromptFI. baseline. Experiments conducted four datasets using 10 clients, with heterogeneity level 0.5 and 5 local training epochs. As shown Table T2 NIAM improves performance on novel classes by preserving generalized knowledge. CSCR enhances base class performance through visual supervision. Together, they deliver the best overall performance in terms of harmonic mean (HM), highlighting their complementary effects.
A.2.7 Sensitive Analysis of Textual Prompt Template
The model's 's performance is not highly sensitive to the choice of textual prompt templates. Following prior works such as CLIP and PromptFL, use "a photo of [CLASS]" as the default template. Since our method mainly optimizes the learnable prompt tokens (Tp) while keeping the text tokens (Tr) fixed, the initial template primarily serves as starting point for semantic alignment and has limited impact on the final performance. We also conducted small-scale experiment comparing different templates (e.g.. "a picture of a [CLASS]", "an image showing [CLASS]"), and the variation in Top-1 accuracy was less than 1 %6, indicating that the GPR-NIAM framework is robust the choice of templates.
19

---

Table 13: Prompt template comparison on Caltech101, DTD, and UCFI01. Best numbers in bold.
Ca
DTD
UCF101
Template
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
A photo of a [CLASS]
91.59
94.22
92.89
68.40
53.50
60.04
78.57
70.28
74.19
A picture of [CLASS]
91.43
94.34
92.86
67.37
52.83
59.22
78.61
69.54
73.79
An image showing [CLASS]
91.52
93.88
92.68
68.57
54.12
60.49
78.77
70.14
74.20
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This study outlines the research problem and its contributions in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize other setrings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the conclusion and future work sections.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed in the paper.
The authors encouraged create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-ter toxtt system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how
they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
20

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not involve thcoretical assumptions.
Guidelines:
The answer means that the paper does not include theoretical results.
All theorems, formulas, and proofs in the paper should be numbered and cross- roos-reefreeeed.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper provides detailed description of the experimental setup, including the range of parameter tuning, and the code will be uploaded as supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contribution the contribution is specific model empirical evaluation, it may necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce the model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
model (e.g.. large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reprodueibility. In the case of closed-soure source models, it may be that access to the model is limited some way (c.g., g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer: [Yes]
Justification:
The code will be uploaded supplementary material.
21

---

Guidelines:
The answer means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines htttps://mips.cc/pubbt
for more details.
While we encourage the release of cnde and data, we understand that this might not possihle,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https://nig) ctc.ipss.ccppubccccc Pridde/CodeSubmn Sodie for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The experimental setup is clearly described in both the main experiment section and the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental selting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper adopts evaluation metries and assessment methods consistent with existing studics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of crrors is not verified.
For asymmetric distributions, the authors should he careful not to show in tables or figures
symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
22

---

8. Experiments compute resourees
Question: each experiment, does paper provide sufficient information on computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The computational resources used are descrihed in the experimental implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: The research conducted in this paper fully conforms to the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considcration duc laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: As discussed in the introduction, the study substantially reduces communication costs by avoiding multiple rounds of exchange, which is advantageous for deployment under stringent latency bandwidth limitations.
Guidelines:
The answer means there is no societal impact of work performed.
the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
legitimate to point out that an improvement in
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
23

---

11. Safeguards
Does the paper describe safeguards that been put in place responsible release data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper utilizes the fully open-source CLIP model and publicly available benchmark datasets that widely used in existing literature. Therefore, additional safeguards necessary for responsible relcase.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards allow for controlled use of model. for example by requiring users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have describe how they avoided releasing unsafe images.
scraped from the Internet could pose safety risks. The authors should
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: the paper cites all relevant works, and all utilized assets are open-source and used in accordance with their licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL..
The name of the license (c.g.. c. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should he provided. For popular datasets, |paperawi wearrswithhcdd.... thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should he provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code developed for this work is included as anonymized supplementary material
Guidelines:
The NA means that the paper does not release
Rescarchers should communicate the details of the dataset/codd/moddl as part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs it important, original. or non standaad component of the core methods in this research? Note that if the LLM used only for writing. editing. or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Answer: [NA]
Justification: This paper only uses LLMs for grammar checking and editing.
Guidelines:
The answer NA means that the method development in this rescarch does not involve LLMs any important, original, or non-standard components.
Please refer LLM policy (https: http:/ppp....eeooeeeeee/e/////// for what should or should described.
25


Reviews:
[{"review_id": "WlC9T4BZLB", "reviewer": "Reviewer_bm2r", "summary": "The paper introduces two modules to address the One-Shot Federated Learning task. The first module employs an attention masking mechanism to preserve textual knowledge, while the second module generates visual prototypes to ultimately enhance the textual prompt.", "questions": "1. The paper claims that previous methods neglect the preservation of generalization to unseen tasks. However, it is not clear why the proposed text-prompt masking mechanism and the visual representation-guided prompt are effective in addressing this issue. Could the authors clarify the connection between these two modules and explain how they jointly contribute to improving generalization?\n\n2. In Figure 1, the meanings of T_T, T_P, and T_EOS​ are unclear. Why can the attention maps of the three mechanisms be shown in a single figure? What exactly are the differences between the three masking strategies, and what results or benefits does each masking mechanism correspond to?\n\n3. I noticed that the baseline results reported in the experimental section are inconsistent with those in the original papers. For example, in Table 5, the reported performance of FedTPG differs from that in the original work across several datasets. Could the authors clarify the reasons behind these discrepancies?\n\n4. The novelty of the paper is somewhat unclear to me. I find the first module (text-prompt masking) conceptually unclear, and the second module is centered around prototype-based learning, which is a well-established technique (e.g., in metric learning). Could the authors clarify what the core contribution of the paper is beyond these existing ideas?", "limitations": "Please see the questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is evaluated on multiple datasets and demonstrates performance improvements over baselines.\n\nWeaknesses:\n1. The paper’s readability could be improved; for example, Figure 1 is somewhat confusing and lacks clarity in conveying the core mechanisms.\n2. The novelty of the work is questionable, as both proposed modules appear to be based on commonly used techniques for prompt enhancement.", "was_revised": true, "final_justification": "Thank the authors for answering my concerns very clearly, and I also read the positive comments from other reviewers. Therefore, I would like to increase my score.", "rebuttal": "We sincerely appreciate the professional comments provided by Reviewer bm2r\n\nW1: The paper’s readability could be improved; for example, Figure 1 is somewhat confusing and lacks clarity in conveying the core mechanisms.\n\nA1: The purpose of Figure 1 is to illustrate the three interaction types (full attention, hard masking, and λ-reweighting) and how these are integrated into the Transformer layers within the proposed GPR-NIAM framework. We agree that the current presentation may appear dense and less intuitive. In the revised version, we will redesign Figure 1 by (1) separating the attention types from the Transformer structure for better readability, (2) adding detailed annotations and a legend to highlight the roles of T_P, T_T, and T_EOS tokens, and (3) providing a clearer explanation in the caption and main text to guide readers through the core mechanisms. As the rebuttal policy does not allow uploading new figures, the updated version will be included in the revised manuscript.\n\nW2: The novelty of the work is questionable, as both proposed modules appear to be based on commonly used techniques for prompt enhancement. / The novelty of the paper is somewhat unclear to me. I find the first module (text-prompt masking) conceptually unclear, and the second module is centered around prototype-based learning, which is a well-established technique (e.g., in metric learning). Could the authors clarify what the core contribution of the paper is beyond these existing ideas?\n\nA2: Although attention masking and prototype optimization have been explored in other studies, our proposed GPR-NIAM framework is the first to integrate Non-Interfering Attention Masking (AttnIso) with Cross-Silo Collaborative Refinement (CSCR) in a unified manner, specifically tailored for the one-shot Federated Prompt Learning (FPL) setting. (1) Unlike conventional prompt-tuning methods where learnable prompts and text tokens interact freely, **the AttnIso module employs a non-interfering attention masking strategy that enforces unidirectional information flow through hard masking and λ-weighting.** This design not only preserves transferable knowledge but also enhances cross-task generalization, which is particularly crucial in the one-shot FPL setting. (2) Although prototype learning has been explored in prior studies (e.g., FedProto, FedProc), **these methods typically rely on aggregating local prototypes from clients to form a global prototype, which requires multiple rounds of interaction and thus cannot be applied in one-shot scenarios.** In contrast, **our approach fully leverages the potential of pre-trained models by freezing the visual encoder, which effectively reduces feature distribution discrepancies across clients.** This design enables our CSCR module to directly perform centralized optimization and alignment of the global prompt using multi-source visual prototypes on the server side.\n\nW3: The paper claims that previous methods neglect the preservation of generalization to unseen tasks. However, it is not clear why the proposed text-prompt masking mechanism and the visual representation-guided prompt are effective in addressing this issue. Could the authors clarify the connection between these two modules and explain how they jointly contribute to improving generalization?\n\nA3: Our design aims to balance task-specific adaptation with the retention of transferable knowledge to enhance generalization to unseen tasks. **The AttnIso module employs non-interfering attention masking to enforce unidirectional information flow, preventing learnable prompts from excessively interfering with the original text embeddings and thus preserving the general semantic structure of the pre-trained model.** Meanwhile, **the CSCR module leverages multi-source visual prototypes to perform centralized optimization and alignment of the global prompt on the server side, effectively mitigating the bias caused by client data heterogeneity.** The combination of these two modules enables the model to avoid overfitting to seen tasks at the local level while enhancing cross-domain representation capabilities globally, leading to stronger generalization on unseen tasks as verified by our experimental results.\n\nW4: In Figure 1, the meanings of T_T, T_P, and T_EOS are unclear. Why can the attention maps of the three mechanisms be shown in a single figure? What exactly are the differences between the three masking strategies, and what results or benefits does each masking mechanism correspond to?\n\nA4: $T_T$ denotes the original text tokens (e.g., ``a photo of a [CLASS]''), $T_P$ represents the learnable prompt tokens, and $T_{\\mathrm{EOS}}$ is the end-of-sequence token used to aggregate contextual information. Figure 1 shows the three mechanisms together because they are all controlled by a unified attention mask matrix $M$, corresponding to full attention ($0$), hard masking ($-\\infty$), and $\\lambda$-reweighting ($\\lambda$), respectively. Full attention allows unrestricted token interaction but may distort the original semantics; hard masking blocks the influence of $T_P$ on $T_T$, preserving the pre-trained semantic structure; and $\\lambda$-reweighting enables controlled information flow between $T_{\\mathrm{EOS}}$ and $T_T$, balancing task adaptation with knowledge retention. The combination of these strategies avoids overfitting while enhancing cross-task generalization, as verified in the ablation results (Table~3).\n\nW5: I noticed that the baseline results reported in the experimental section are inconsistent with those in the original papers. For example, in Table 5, the reported performance of FedTPG differs from that in the original work across several datasets. Could the authors clarify the reasons behind these discrepancies?\n\nA5: The discrepancies mainly arise because we re-implemented all baseline methods under a unified Federated Prompt Learning (FPL) framework to ensure fair comparisons. For example, the original FedTPG paper was not evaluated in a one-shot FPL setting, so we adapted it to a single-round communication protocol, which inevitably affects the results. In addition, we used a consistent setup with the pre-trained CLIP (ViT-B/16) backbone and a fixed prompt length (10 × 512), and simulated non-IID data across multiple clients using a Dirichlet distribution ($\\beta = 0.5 or 0.1$), which differs from the configurations in the original papers. Furthermore, some baseline implementations were unavailable or incompatible with our experimental environment, and we re-implemented them based on the original descriptions, which may lead to minor deviations."}, {"review_id": "a8JHgY2vtU", "reviewer": "Reviewer_7qN8", "summary": "This paper focuses on enhancing the generalization ability of Federated Prompt Learning within a one-shot communication setting. The authors propose GPR-NIAM. The core idea is to introduce a non-interfering attention masking mechanism that restricts the influence of learnable prompt tokens on the original text embeddings, thereby preserving transferable knowledge. In addition, a cross-silo collaborative refinement module is introduced to calibrate the global prompt using visual prototypes collected from distributed clients, improving coordination under data heterogeneity. Extensive experiments on 10 datasets across both task and domain generalization scenarios demonstrate that GPR-NIAM significantly outperforms existing state-of-the-art baselines.", "questions": "1. Is the model's performance sensitive to the choice of the textual prompt template, such as 'a photo of a [CLASS]'?\n\n2. Can the proposed method be extended to support multi-round communication?\n \n3. How does the method handle the case where a certain class contains only a single sample on a client?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed AttnIso module enforces unidirectional intervention from prompts to text tokens via masking, effectively preserving the transferable knowledge of the original tokens. It can also be seamlessly integrated into various existing methods.\n\n-  The paper presents a clear description of the methodological workflow with a well-organized structure.\n\n- Comprehensive experimental validation, including performance comparisons, ablation studies, hyperparameter analysis, and case studies of key modules, greatly aids reader understanding.\n\n**Weaknesses**\n\n- The paper does not appear to include a sensitivity analysis regarding the number of prompt tokens.\n\n- Can the authors clarify the actual dimensions of the attention mask matrix \nM? Is it square or used as a 1D masking vector?\n\n- The CSCR stage relies on a centralized prototype pool maintained at the server, and the paper should clearly articulate whether this design introduces potential privacy leakage risks.", "was_revised": true, "final_justification": "After reading the authors’ rebuttal and their responses to other reviewers, I find that my major concerns have been adequately addressed, particularly the analysis regarding the sensitivity of the method to prompt length. The clarifications and additional results improve my understanding of the proposed approach and its robustness. Therefore, I have decided to raise my score to 5", "rebuttal": "We sincerely appreciate the professional comments provided by Reviewer 7qN8\n\nW1: The paper does not appear to include a sensitivity analysis regarding the number of prompt tokens.\n\nA1: We have conducted additional sensitivity experiments regarding the number of prompt tokens. Specifically, we adjusted the prompt length to **5×512** and **20×512**. The results show that GPR-NIAM remains robust across different prompt lengths, while consistently maintaining an advantage over all baselines in every case.\n\n| 5×512   | Caltech101 Base | Caltech101 Novel | Caltech101 HM | DTD Base | DTD Novel | DTD HM | UCF101 Base | UCF101 Novel | UCF101 HM |\n|--------------|-----------------|------------------|---------------|----------|-----------|--------|-------------|--------------|-----------|\n| PromptFL     | 88.41           | **93.97**        | 91.11         | 59.25    | 52.77     | 55.83  | 76.78       | 69.91        | 73.18     |\n| FedTPG       | 86.65           | 93.37            | 89.88         | 60.18    | 56.15     | 58.10  | 74.30       | 70.63        | 72.41     |\n| PromptFolio  | 89.47           | 93.46            | 91.42         | **61.02**| **56.28** | 57.62  | 74.40       | **70.65**    | 72.48     |\n| GRP-NIAM_P   | **89.90**       | 93.37            | **91.60**     | 60.96    | 55.95     | **59.12** | **77.98**  | 70.39        | **73.99** |\n\n\n| 20×512     | Caltech101 Base | Caltech101 Novel | Caltech101 HM | DTD Base | DTD Novel | DTD HM | UCF101 Base | UCF101 Novel | UCF101 HM |\n|--------------|-----------------|------------------|---------------|----------|-----------|--------|-------------|--------------|-----------|\n| PromptFL     | 88.91           | **94.31**        | 91.53         | **62.50**| 52.53     | 57.08  | 76.83       | 69.12        | 72.77     |\n| FedTPG       | 88.84           | 92.86            | 90.81         | 58.10    | 54.83     | 56.41  | 76.30       | 73.41        | 74.83     |\n| PromptFolio  | 89.97           | 94.22            | 92.05         | 56.13    | 53.98     | 55.03  | 75.72       | 74.41        | 75.06     |\n| GRP-NIAM_P   | **90.02**       | 94.28            | **92.10**     | 61.43    | **56.23** | **58.71** | **77.23**  | **74.58**    | **75.88** |\n\nW2: Can the authors clarify the actual dimensions of the attention mask matrix M? Is it square or used as a 1D masking vector?\n\nA2: The attention mask matrix $M$ is a **square matrix** with dimensions $(N_p + N_t + 1) \\times (N_p + N_t + 1)$, where $N_p$ is the number of learnable prompt tokens, $N_t$ is the number of original text tokens, and the additional $1$ corresponds to the $[EOS]$ token. Each element $M_{i,j}$ specifies the masking rule for attention from token $i$ to token $j$, as described in Equation~(1). We use this full square matrix rather than a 1D masking vector because the masking strategy needs to control pairwise attention flows between all tokens (e.g., blocking $T_P$ from attending to $T_T$, reweighting $T_{EOS}$ to $T_T$, etc.).\n\nW3: The CSCR stage relies on a centralized prototype pool maintained at the server, and the paper should clearly articulate whether this design introduces potential privacy leakage risks.\n\nA3: The CSCR stage aggregates **class-wise visual prototypes**, where each prototype is computed as the **mean of multiple sample features** within the same class on a client, into a centralized prototype pool on the server. Importantly, **these prototypes are high-level feature representations rather than raw data**, extracted by the frozen pre-trained visual encoder (e.g., CLIP). As such, they do not directly reveal sensitive information or allow reconstruction of original client data, which largely mitigates privacy risks. Similar prototype-sharing approaches (e.g., FedProto, FedProc) have been widely adopted in federated learning literature with no reported privacy breaches under this setting. Moreover, we can be readily extended with existing privacy-preserving techniques such as differential privacy (adding calibrated noise to prototypes) or secure aggregation.  \n\nW4: Is the model's performance sensitive to the choice of the textual prompt template, such as 'a photo of a [CLASS]'?\n\nA4: The model's performance is not highly sensitive to the choice of textual prompt templates. Following prior works such as CLIP and PromptFL, we use **``a photo of a [CLASS]''** as the default template. Since our method mainly optimizes the learnable prompt tokens ($T_P$) while keeping the text tokens ($T_T$) fixed, the initial template primarily serves as a starting point for semantic alignment and has limited impact on the final performance. We also conducted a small-scale experiment comparing different templates (e.g., **``a picture of a [CLASS]'', an image showing [CLASS]''**), and the variation in Top-1 accuracy was less than 1\\%, indicating that the GPR-NIAM framework is robust to the choice of templates. We will include this observation in the revised manuscript.\n\n| Template                  | Caltech101 Base | Caltech101 Novel | Caltech101 HM | DTD Base | DTD Novel | DTD HM | UCF101 Base | UCF101 Novel | UCF101 HM |\n|---------------------------|-----------------|------------------|---------------|----------|-----------|--------|-------------|--------------|-----------|\n| A photo of a [CLASS]      | 91.59           | 94.22            | **92.89**         | 68.40    | 53.50     | 60.04  | 78.57       | **70.28**        | 74.19     |\n| a picture of a [CLASS]    | 91.43           | **94.34**        | 92.86         | 67.37    | 52.83     | 59.22  | 78.61       | 69.54        | 73.79     |\n| an image showing [CLASS]  | **91.52**       | 93.88            | 92.68     | **68.57**| **54.12** | **60.49** | **78.77**  | 70.14    | **74.20** |\n\nW5: Can the proposed method be extended to support multi-round communication?\n\nA5: Yes, the proposed GPR-NIAM framework can be naturally extended to support multi-round communication. In the current one-shot setting, the AttnIso module operates during local prompt tuning, and the CSCR module refines the global prompt using aggregated prototypes in a single round. For a multi-round extension, the server can update the global prompt after each round of prototype-guided refinement and broadcast it back to the clients. Clients can then continue local prompt tuning with the updated global prompt, repeating the process for multiple communication rounds.\n\nThis extension allows the model to progressively align local and global prompts, potentially improving performance under highly heterogeneous data. While our primary focus is on the one-shot scenario due to its communication efficiency, we have also conducted supplementary experiments under multi-round settings, and the results further confirm the effectiveness of our framework. These details will be added to the revised manuscript.\n\nW6: How does the method handle the case where a certain class contains only a single sample on a client?\n\nA6: In cases where a class on a client contains only a single sample, our method constructs the prototype by using the feature representation of that single sample extracted by the frozen pre-trained visual encoder (e.g., CLIP) and **adds a small Gaussian noise perturbation** to avoid overfitting and improve generalization. Although such a prototype is not the average of multiple features, the high-quality, semantically rich features from the pre-trained encoder combined with the noise perturbation ensure its robustness. We will include this clarification in the revised manuscript."}, {"review_id": "EF7BqGLMD1", "reviewer": "Reviewer_sLzh", "summary": "This paper introduces GPR-NIAM, a method designed to enhance one-shot federated learning (OSFL) for federated prompt learning (FPL) by addressing the limitations of existing approaches that rely on multi-round communications and lack cross-task generalization; GPR-NIAM achieves this through a non-interfering attention masking mechanism that restricts excessive interaction between learnable prompt embeddings and original text embeddings, preserving transferable knowledge across tasks, and further refines the global prompt using cross-silo collaborative refinement to mitigate data heterogeneity.", "questions": "1. Can the masking mechanism be dynamically adjusted? \n\n2. Can other attention mechanisms be used for NIAM?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. GPR-NIAM enables one-shot federated learning, significantly reducing communication overhead by eliminating the need for multiple rounds of client-server interaction.\n2. By designing a masking mechanism that restricts excessive interaction between learnable prompt embeddings and original text embeddings, GPR-NIAM preserves transferable knowledge across tasks, enhancing cross-task generalization capabilities.\n3. GPR-NIAM leverages visual representations extracted from multiple clients and utilizes multi-source visual supervision to refine the global prompt in a centralized manner, further mitigating the negative impact of data heterogeneity.\n\nWeaknesses:\n1. Although GPR-NIAM reduces communication overhead by enabling one-shot learning, the transmission of visual prototypes during the global prompt refinement stage may become a bottleneck in large-scale federated systems with limited bandwidth.\n2. GPR-NIAM relies on pre-trained models (e.g., CLIP) for its effectiveness. The choice of pre-trained model can influence the performance of GPR-NIAM, and the method may not perform as well if the pre-trained model is not well-suited to the target task or domain.\n3. GPR-NIAM involves a two-stage optimization strategy. This introduces complexity and increases the difficulty of implementation and deployment.", "was_revised": true, "final_justification": "The rebuttal answered my questions. I will keep the score unchanged.", "rebuttal": "We sincerely appreciate the professional comments provided by Reviewer sLzh\n\nW1: Although GPR-NIAM reduces communication overhead by enabling one-shot learning, the transmission of visual prototypes during the global prompt refinement stage may become a bottleneck in large-scale federated systems with limited bandwidth.\n\nA1: The communication overhead of GPR-NIAM mainly consists of prompt transmission and prototype transmission, where the cost of prompts is $L \\times 512$ and the cost of prototypes is $C \\times K \\times 512$, with $L$ being the prompt length, $C$ the number of classes, and $K$ the number of prototypes per class. \n\n| Prompt Transmission Cost       | Prototype Transmission Cost         |\n|-------------------------------|-------------------------------------|\n| $L \\times 512 $           | $ C \\times K \\times 512 $        |\n\nW2: GPR-NIAM relies on pre-trained models (e.g., CLIP) for its effectiveness. The choice of pre-trained model can influence the performance of GPR-NIAM, and the method may not perform as well if the pre-trained model is not well-suited to the target task or domain.\n\nA2: We agree that GPR-NIAM relies on the representational power of pre-trained models (e.g., CLIP) for its effectiveness, and the choice of pre-trained backbone can influence performance, especially if the model is not well-suited to the target task or domain. However, **this dependency is common to all prompt-based methods**. In practice, **we can flexibly select or replace the pre-trained backbone according to the target task or domain** (e.g., domain-specific CLIP variants or other vision-language models), allowing GPR-NIAM to adapt to different scenarios. We will add a discussion of this point to the revised manuscript.\n\nW3: GPR-NIAM involves a two-stage optimization strategy. This introduces complexity and increases the difficulty of implementation and deployment.\n\nA3: We acknowledge that GPR-NIAM adopts a two-stage optimization strategy (AttnIso-based local prompt tuning followed by CSCR-based global refinement), which introduces additional complexity compared to single-stage methods. However, this design is crucial for separating task-specific adaptation (local stage) from cross-client alignment (global stage), enabling both strong generalization and efficient knowledge aggregation.\n\nIn terms of implementation, each stage is built upon standard prompt tuning and prototype aggregation operations, both of which are lightweight and compatible with existing federated learning frameworks. To further reduce deployment difficulty, the two stages can be combined into a single pipeline by automating prototype extraction and global refinement on the server. We will add a note in the revised manuscript clarifying this trade-off between performance and complexity.\n\nW4: Can the masking mechanism be dynamically adjusted?\n\nA4: Yes, the masking mechanism in GPR-NIAM can be dynamically adjusted. While our current implementation uses fixed masking strategies (full attention, hard masking, and $\\lambda$-reweighting), the framework is flexible enough to support **adaptive or learnable masking schemes**. For instance, the value of $\\lambda$ can be made learnable or conditioned on the similarity between prompt tokens and text tokens, allowing the model to dynamically control the degree of interaction. Similarly, different attention heads or layers could adopt varying masking patterns based on task-specific signals. We will include a discussion of this potential extension in the revised manuscript.\n\nW5: Can other attention mechanisms be used for NIAM?\n\nA5: Yes, other attention mechanisms can be integrated into NIAM. Our current design is based on standard multi-head self-attention for simplicity and compatibility with CLIP, but NIAM is not limited to this choice. Alternative mechanisms such as sparse attention, could be adopted to reduce computational cost or emphasize specific token interactions. However, these attention variants are not specifically designed for the one-shot FPL scenario, and may require additional adaptation to handle the challenges of cross-client heterogeneity and limited communication. Our masking strategy is purposefully tailored to these challenges, which is why we chose standard attention combined with NIAM for this work. We will include a discussion of this flexibility and rationale in the revised manuscript."}, {"review_id": "qpb0XiXyne", "reviewer": "Reviewer_nDhj", "summary": "This paper focuses on One-Shot Federated Prompt Learning and proposes GPR-NIAM. The Attention Isolation module introduces a non-interfering attention mechanism to suppress the influence of learnable prompts on original text embeddings, ensuring a unidirectional flow of information that prevents overfitting and enhances generalization. The Cross-Silo Collaborative Refinement module constructs a unified global prompt using visual prototypes from clients, enabling cross-modal knowledge alignment and mitigating global bias caused by non-IID data. Evaluated on 10 benchmark datasets against 8 state-of-the-art methods, GPR-NIAM achieves either the best or second-best performance in both within-task and cross-domain generalization tasks.", "questions": "1. In the leave-one-domain-out task, is the test domain data completely unseen during the training phase?\n\n2. Is there any information flow dependency between the two modules?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The problem addressed in this paper is relatively well-defined, and the introduction offers a reasonably thorough analysis of the current challenges in Federated Prompt Learning and One-Shot Federated Learning.\n\n2. The proposed approach is compatible with existing methods and can be seamlessly integrated into frameworks such as PromptFL and TCP, demonstrating strong generalizability.\n\n3. The experiments cover ten diverse datasets and compare the proposed method against eight state-of-the-art baselines across multiple evaluation settings, demonstrating its effectiveness. In addition, the paper gives case studies that  provides insights into the underlying mechanisms driving its performance gains.\n\n\nMajor Weaknesses\n\n1. Although the method emphasizes a one-shot setting, the potential additional communication overhead introduced by the transmission of visual prototypes is not quantitatively analyzed.\n\n2. It is recommended to further elaborate on the advantages of the proposed prototype construction method, particularly in comparison to traditional approaches such as FedProto.\n\n\nMinor Weaknesses\n\n1. Avoid using the same symbol to represent multiple entities. For example, the symbol f is used to denote both the image encoder function and its output feature.\n\n2. The potential privacy risks associated with uploading visual prototypes are not discussed. \n\n3. The learnable prompt is fixed at a size of 10×512 throughout the experiments. It is recommended to include additional configurations such as 5×512 and 20×512 to assess the sensitivity and robustness of the method with respect to prompt length.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I have raised my evaluation accordingly.", "rebuttal": "We sincerely appreciate the professional comments provided by Reviewer nDhj\n\nW1: Although the method emphasizes a one-shot setting, the potential additional communication overhead introduced by the transmission of visual prototypes is not quantitatively analyzed.\n\nA1: Compared to learnable prompts, the visual prototypes (512-dimensional vectors per class per client) can indeed contribute more to the overall communication cost, particularly in large-scale federated systems with many classes and clients. We have provided a detailed analysis of this overhead in the revised manuscript, clarifying that the communication cost is primarily related to three factors: (1) the number of classes, (2) the number of participating clients, and (3) the dimensionality of the feature representations. This analysis can guide practical deployment by helping determine suitable trade-offs between performance and communication efficiency.\n\nW2: It is recommended to further elaborate on the advantages of the proposed prototype construction method, particularly in comparison to traditional approaches such as FedProto.\n\nA2: While FedProto and FedProc construct class prototypes by simply averaging all sample features within the class, our method employs a random-weighted aggregation strategy instead of a plain mean. This design introduces controlled diversity into the prototype representation, which prevents over-smoothing of features and better captures intra-class variations.\n\nSuch diversity-aware prototype construction is particularly beneficial in federated settings, where the data distribution across clients is heterogeneous and class features can be highly variable. By incorporating random weighting, our prototypes provide a richer representation space for global prompt refinement, which leads to improved generalization across clients and unseen tasks.\n\nW3: Avoid using the same symbol to represent multiple entities. For example, the symbol f is used to denote both the image encoder function and its output feature.\n\nA3: In the revised manuscript, we will refine the notations to ensure clarity. Specifically, we will use $f(\\cdot)$ exclusively to denote the image encoder function, and denote its output feature as \n$\\mathbf{z} = f(x)$, where $x$ is the input image. This adjustment will be applied consistently throughout the paper to avoid ambiguity between functions and feature representations.\n\nW4: The potential privacy risks associated with uploading visual prototypes are not discussed.\n\nA4: The CSCR stage aggregates **class-wise visual prototypes** into a centralized prototype pool on the server, where each prototype is computed as the **weighted mean feature vector of multiple samples** from the same class on a client. These prototypes are **high-level feature embeddings rather than raw data**, extracted by the frozen pre-trained visual encoder (e.g., CLIP), and thus they do not directly expose sensitive information or enable the reconstruction of original client data, which largely reduces privacy risks. Similar prototype-sharing strategies (e.g., FedProto, FedProc) have been widely used in federated learning literature without reported privacy breaches. Furthermore, our framework can be seamlessly combined with **privacy-preserving techniques** such as differential privacy (by adding calibrated noise to prototypes) or secure aggregation if stronger privacy guarantees are needed.\n\nW5: The learnable prompt is fixed at a size of 10×512 throughout the experiments. It is recommended to include additional configurations such as 5×512 and 20×512 to assess the sensitivity and robustness of the method with respect to prompt length.\n\nA5: we have added additional experiments with prompt lengths of 5×512 and 20×512. Preliminary results show that GPR-NIAM remains robust across different prompt sizes, with performance variations within 0.5–1.2% on average, indicating that the method is not overly sensitive to the specific prompt length. \n\nW6: In the leave-one-domain-out task, is the test domain data completely unseen during the training phase?\n\nA6: Yes, in the leave-one-domain-out (LODO) setting, the test domain data is completely unseen during the training phase. For each experiment, we use data from all but one domain for training (both local prompt tuning and global refinement) and reserve the left-out domain exclusively for evaluation. This ensures that the model’s performance on the test domain truly reflects its ability to generalize to unseen domains.\n\nW7: Is there any information flow dependency between the two modules?\n\nA7: There is no direct information flow dependency between the two modules; rather, they complement each other in a sequential manner. The AttnIso module operates during local prompt tuning on each client, where it regulates the interaction between learnable prompt tokens and text tokens to preserve transferable semantics. After local tuning, the CSCR module takes the locally updated prompts and their associated visual prototypes as input to perform centralized global prompt refinement. Thus, AttnIso and CSCR are functionally independent but jointly contribute to the final performance by addressing different challenges, AttnIso focuses on maintaining semantic generalization at the token level, while CSCR reduces cross-client heterogeneity at the global level."}]

Author rebuttal:

Meta-review: This paper proposes GPR-NIAM, a method for one-shot federated prompt learning that uses a novel attention masking mechanism and cross-silo collaborative refinement to improve communication efficiency and cross-task generalization without multi-round communication. The four reviewers' ratings are: Weak Accept, Weak Accept, Accept, Accept. The strong consensus on its empirical rigor and the significance of enabling one-shot federated learning for prompt-based methods aligns with NeurIPS's standards.
Decision: Accept (poster)