Title: Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning;Federated Prompt Learning;One-shot Federated Learning
Abstract: Federated Prompt Learning (FPL) enables communication-efficient adaptation by tuning lightweight prompts on top of frozen pre-trained models. Existing FPL methods typically rely on global information, which is only available after the second training round, to facilitate collaboration among client models. Therefore, they are inherently dependent on multi-round communication to fully exhibit their strengths. Moreover, existing one-shot federated learning methods typically focus on fitting seen tasks, but lack cross-task generalization. To bridge this gap, we propose the global prompt refinement with non-interfering attention masking (GPR-NIAM) method for one-shot FPL. The core idea is to design a masking mechanism that restricts excessive interaction between the original text embeddings and the learnable prompt embeddings. GPR-NIAM achieves this through the collaboration of two key modules. Firstly, the attention isolation module suppresses attention from the learnable prompt tokens to the original text tokens, and reweights the reverse attention which preserves generalization across tasks. Secondly, the cross-silo collaborative refinement module integrates decentralized visual knowledge into a unified base and calibrates the global prompt through multi-source cross-modal knowledge alignment, further mitigating the inconsistency caused by data heterogeneity. Extensive experiments conducted on ten benchmark datasets under two tasks show that GPR-NIAM outperforms eight state-of-the-art methods in both class-level and domain-level generalization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on One-Shot Federated Prompt Learning and proposes GPR-NIAM. The Attention Isolation module introduces a non-interfering attention mechanism to suppress the influence of learnable prompts on original text embeddings, ensuring a unidirectional flow of information that prevents overfitting and enhances generalization. The Cross-Silo Collaborative Refinement module constructs a unified global prompt using visual prototypes from clients, enabling cross-modal knowledge alignment and mitigating global bias caused by non-IID data. Evaluated on 10 benchmark datasets against 8 state-of-the-art methods, GPR-NIAM achieves either the best or second-best performance in both within-task and cross-domain generalization tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The problem addressed in this paper is relatively well-defined, and the introduction offers a reasonably thorough analysis of the current challenges in Federated Prompt Learning and One-Shot Federated Learning.\n\n2. The proposed approach is compatible with existing methods and can be seamlessly integrated into frameworks such as PromptFL and TCP, demonstrating strong generalizability.\n\n3. The experiments cover ten diverse datasets and compare the proposed method against eight state-of-the-art baselines across multiple evaluation settings, demonstrating its effectiveness. In addition, the paper gives case studies that  provides insights into the underlying mechanisms driving its performance gains.\n\n\nMajor Weaknesses\n\n1. Although the method emphasizes a one-shot setting, the potential additional communication overhead introduced by the transmission of visual prototypes is not quantitatively analyzed.\n\n2. It is recommended to further elaborate on the advantages of the proposed prototype construction method, particularly in comparison to traditional approaches such as FedProto.\n\n\nMinor Weaknesses\n\n1. Avoid using the same symbol to represent multiple entities. For example, the symbol f is used to denote both the image encoder function and its output feature.\n\n2. The potential privacy risks associated with uploading visual prototypes are not discussed. \n\n3. The learnable prompt is fixed at a size of 10×512 throughout the experiments. It is recommended to include additional configurations such as 5×512 and 20×512 to assess the sensitivity and robustness of the method with respect to prompt length.\n\nQUESTIONS:\n1. In the leave-one-domain-out task, is the test domain data completely unseen during the training phase?\n\n2. Is there any information flow dependency between the two modules?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GPR-NIAM, a method designed to enhance one-shot federated learning (OSFL) for federated prompt learning (FPL) by addressing the limitations of existing approaches that rely on multi-round communications and lack cross-task generalization; GPR-NIAM achieves this through a non-interfering attention masking mechanism that restricts excessive interaction between learnable prompt embeddings and original text embeddings, preserving transferable knowledge across tasks, and further refines the global prompt using cross-silo collaborative refinement to mitigate data heterogeneity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. GPR-NIAM enables one-shot federated learning, significantly reducing communication overhead by eliminating the need for multiple rounds of client-server interaction.\n2. By designing a masking mechanism that restricts excessive interaction between learnable prompt embeddings and original text embeddings, GPR-NIAM preserves transferable knowledge across tasks, enhancing cross-task generalization capabilities.\n3. GPR-NIAM leverages visual representations extracted from multiple clients and utilizes multi-source visual supervision to refine the global prompt in a centralized manner, further mitigating the negative impact of data heterogeneity.\n\nWeaknesses:\n1. Although GPR-NIAM reduces communication overhead by enabling one-shot learning, the transmission of visual prototypes during the global prompt refinement stage may become a bottleneck in large-scale federated systems with limited bandwidth.\n2. GPR-NIAM relies on pre-trained models (e.g., CLIP) for its effectiveness. The choice of pre-trained model can influence the performance of GPR-NIAM, and the method may not perform as well if the pre-trained model is not well-suited to the target task or domain.\n3. GPR-NIAM involves a two-stage optimization strategy. This introduces complexity and increases the difficulty of implementation and deployment.\n\nQUESTIONS:\n1. Can the masking mechanism be dynamically adjusted? \n\n2. Can other attention mechanisms be used for NIAM?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on enhancing the generalization ability of Federated Prompt Learning within a one-shot communication setting. The authors propose GPR-NIAM. The core idea is to introduce a non-interfering attention masking mechanism that restricts the influence of learnable prompt tokens on the original text embeddings, thereby preserving transferable knowledge. In addition, a cross-silo collaborative refinement module is introduced to calibrate the global prompt using visual prototypes collected from distributed clients, improving coordination under data heterogeneity. Extensive experiments on 10 datasets across both task and domain generalization scenarios demonstrate that GPR-NIAM significantly outperforms existing state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The proposed AttnIso module enforces unidirectional intervention from prompts to text tokens via masking, effectively preserving the transferable knowledge of the original tokens. It can also be seamlessly integrated into various existing methods.\n\n-  The paper presents a clear description of the methodological workflow with a well-organized structure.\n\n- Comprehensive experimental validation, including performance comparisons, ablation studies, hyperparameter analysis, and case studies of key modules, greatly aids reader understanding.\n\n**Weaknesses**\n\n- The paper does not appear to include a sensitivity analysis regarding the number of prompt tokens.\n\n- Can the authors clarify the actual dimensions of the attention mask matrix \nM? Is it square or used as a 1D masking vector?\n\n- The CSCR stage relies on a centralized prototype pool maintained at the server, and the paper should clearly articulate whether this design introduces potential privacy leakage risks.\n\nQUESTIONS:\n1. Is the model's performance sensitive to the choice of the textual prompt template, such as 'a photo of a [CLASS]'?\n\n2. Can the proposed method be extended to support multi-round communication?\n \n3. How does the method handle the case where a certain class contains only a single sample on a client?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces two modules to address the One-Shot Federated Learning task. The first module employs an attention masking mechanism to preserve textual knowledge, while the second module generates visual prototypes to ultimately enhance the textual prompt.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed method is evaluated on multiple datasets and demonstrates performance improvements over baselines.\n\nWeaknesses:\n1. The paper’s readability could be improved; for example, Figure 1 is somewhat confusing and lacks clarity in conveying the core mechanisms.\n2. The novelty of the work is questionable, as both proposed modules appear to be based on commonly used techniques for prompt enhancement.\n\nQUESTIONS:\n1. The paper claims that previous methods neglect the preservation of generalization to unseen tasks. However, it is not clear why the proposed text-prompt masking mechanism and the visual representation-guided prompt are effective in addressing this issue. Could the authors clarify the connection between these two modules and explain how they jointly contribute to improving generalization?\n\n2. In Figure 1, the meanings of T_T, T_P, and T_EOS​ are unclear. Why can the attention maps of the three mechanisms be shown in a single figure? What exactly are the differences between the three masking strategies, and what results or benefits does each masking mechanism correspond to?\n\n3. I noticed that the baseline results reported in the experimental section are inconsistent with those in the original papers. For example, in Table 5, the reported performance of FedTPG differs from that in the original work across several datasets. Could the authors clarify the reasons behind these discrepancies?\n\n4. The novelty of the paper is somewhat unclear to me. I find the first module (text-prompt masking) conceptually unclear, and the second module is centered around prototype-based learning, which is a well-established technique (e.g., in metric learning). Could the authors clarify what the core contribution of the paper is beyond these existing ideas?\n\nLIMITATIONS:\nPlease see the questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}