Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a lightweight federated learning framework where clients jointly learn to fuse the representations generated by multiple fixed pre-trained models rather than training a large-scale model.  The paper is generally well-written and in a good shape. Extensive experiments were conducted to show the effectiveness of the proposed FedPCL. 


strengths_and_weaknesses: Strengths: Using multiple pre-trained models to capture client-specific and class-relevant information is new in federated learning. FedPCL enjoys light computation and communication costs, compared to most other federated learning.  

Weaknesses:  
1.	Apart from the multiple pre-trained models, FedPCL is built on the idea of prototypical learning and contrastive learning, which are not new in federated learning. 
2.	The performance of FedPCL heavily relies on the selection of different pre-trained models, limiting its applications to more wide areas. As shown in Table 4, the model accuracy is quite sensitive to the pre-trained models. 

questions: 1. The generalization bound of FedPCL given in Theorem 4.1 is unclear without sufficient analysis. The authors claimed that ”the bound is associated with the complexity of training models and the size of prototype size”. The reviewer believe that this is a quite straightforward observation even without Theorem 4.1. Then, what is the purpose of Theorem 4.1. The reviewer are encouraged to analyze the convergence of FedPCL with the fixed pre-trained models. 

2. In addition, the derivations of Theorem 4.1 in the appendix is also not clear. For example, the derivation of (13) by McDiarmid’s inequality and the derivation of (14) by Rademacher complexity are not clear.  

limitations: This work adequately addressed the limitations. The authors developed a lightweight federated learning framework to reduce the computation and communication costs and integrated pre-trained models to extract prototypes for federated aggregation. The is a new try for federated learning.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies a specific federated learning problem, which considers to leverage the large-scale pre-trained models. It builds a federated prototype-wise contrastive learning framework to capture more client-speciﬁc and class-relevant information. A generalization analysis has been provided to discuss the effect of the built prototypes. The authors conduct a range of experiments to demonstrate the performance of the proposed method over state-of-the-art performance.
strengths_and_weaknesses: Strengths
(1) By constructing a FedPCL mechanism, the proposed framework trains the model in a lightweight manner and simultaneously shares the knowledge across the clients. Besides, the projection network provides the personalized feature selection for each client.

(2) A theoretical analysis points out the how the proposed prototype-based learning method affects the generalization bound, which helps how this type of methods differ from previous non-prototype-based works.

(3) A range of experiments have been conducted to show the promise of the proposed method over the state-of-the-art methods, and FedPCL can be further boosted by integrating different pretrained models.

Weakness
(1) The relationship between two loss functions in Eq.(8) and Eq.(9) have not sufficiently explained, since the latter provide the whole supervision information about the global C. In this perspective, why we need Eq.(8).

(2) Another question is about the pre-trained backbones. It seems that the proposed framework actually can be applied to the wider scenarios where there are not pre-trained models as the feature extractors. Limiting the model to this scenario raises the concerns that whether the prototype-based federated learning methods easily overfits the local data without the auxiliary feature extractors. The authors could discuss more about this point.

(3) It lacks of a baseline like [1] and the authors should discuss how your method is different from this work and other prototype-based methods like FedProto and why yours is better.

[1] FedProc: Prototypical Contrastive Federated Learning on Non-IID data
questions: Please explain the questions in the weakness part listed above.
limitations: NONE
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a lightweight framework that enables federated learning from multiple pre-trained models, which significantly reduces communication and computation costs and makes it possible to integrate large-scale models into federated learning framework. To efficiently learn and share learned knowledge across clients under this framework, a prototype-based contrastive learning algorithm is developed to fully leverage the representation ability of pre-trained models in the latent space. A comprehensive empirical study is conducted and shows promising results achieved by the proposed algorithm. 

strengths_and_weaknesses: Pros:
1. The problem studied in this paper is interesting and important. This work first proposes to integrate pre-trained models into federated learning to form a lightweight FL framework, which expands the scope of current FL frameworks and makes an important contribution to the FL community.

2. This paper has its originality and technically sound. The main claims regarding the proposed setting are well supported in the methodology and experimental parts. By transmitting prototype sets and optimizing the well-designed prototype-wise contrastive loss, class-specific knowledge can be fully extracted and shared during local training and server aggregation procedures, respectively.

3. Extensive and comprehensive experiments are conducted. Experimental results on three benchmark datasets under different non-IID settings show superior or comparable performance achieved by FedPCL compared to existing FL/PFL methods. Sufficient ablation studies are provided to demonstrate the advantages of the loss and prototypes in the approach. Detailed data and model settings are provided in the appendix for reproduction. 


Cons:
1. It is better to provide more analysis on the performance improvement achieved by FedPCL in Section 5.2 and 5.4 to let the readers well understand how contrastive loss and prototypes make their contribution to the improvement. 

2. Although there is a relatively thorough literature review in the related work part, I prefer to see a discussion on the relation of this work and the fine-tuning techniques in FL since both of them have the potential to integrate pre-trained foundation models into the framework.

questions: 1. Existing SOTA personalized FL methods that achieve high performance in traditional non-IID settings, e.g., pFedMe, FedRep, do not work well under this lightweight framework. What is their limitation when using pre-trained models? Can this be fixed by introducing some simple but efficient techniques?

2. Is it possible to allow clients to have different pre-trained models and keep sharing knowledge in this manner? This would be more common in real-world cases.

limitations: The authors have well addressed the limitations of this work in terms of specific target tasks and illustrated the potential directions of their future studies.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents Federated Prototype-wise Contrastive Learning (FedPCL) which aim to exploit pre-trained models in FL environment for reducing communication (COM) costs.


The authors claim the following contributions:
-  First to integrate pre-trained models into federated learning to reduce COM cost.
- Proposing FedPCL.
- Experiments showcasing FedPCL advantages over existing P/FL works.
strengths_and_weaknesses: ***Pros:***
- The paper is well written and easy to follow.
- The motivation of this paper is well explained - reducing COM liabilities is a major challenge in P/FL.
- The authors provided generalization bound for FedPCL


***Cons:***
 - The proposed method increases the computational demand from the clients (which in most cases are edge devices e.g. wearables, smartphones, etc.).
- From eq (6,7) I understand that the server will be able to "know" which classes each client has. This raises some privacy issues. For example in medical applications we don't want the server to know which client has/not illness. 
- Missing citations of important and strong PFL papers (e.g. [1,2]).
- Problematic results in the experimental part - in lots of cases Solo (local training) outperforms most PFL methods.
- It is not clear if and how the authors performed hyperparameter tuning for FedPCL / baseline approaches. In addition, some experimental details are missing such train/val/test split, etc.
- Experiments were conducted on limited numbers of clients (6 max). Rea-life P/FL systems obtain thousands / millions of clients. 


[1] Personalized federated learning using hypernetworks.

[2] Personalized Federated Learning with First Order Model Optimization.
questions: See above.
limitations: See above.
review_title: null

Meta-review: metareview: To train large-scale models for federated learning settings, this paper presents a Federated Prototype-wise Contrastive Learning (FedPCL) approach. FedPCL aims to solve federated learning problems utilizing the representations of multiple fixed pre-trained models. Specifically, it computes the class prototypes of each client, and their average. Then it sends them to each client, and builds contrastive loss function based on them. Experiments show effectiveness of FedPCL under the proposed lightweight framework.

However, this paper suffers from several limitations. Firstly, the algorithm violates the privacy principle of federated learning, since all the prototypes of each client are sent to all the clients. In addition, the t-test indicates that the difference between FedPCL and FedProto are not statistically significant for many cases in table 2.
confidence: Less certain
Decision: Accept