Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the self-supervised learning (SSL) framework for FL. Different SSL methods are investigated to study their feasibility under the FL setting. With the popular SimSiam framework, personalized federated SSL is proposed. The performance comparison indicates
that representation regularization-based personalization method is able to outperform other variants.
main_review: Strength:

+ The self-supervised learning topic for FL is emerging and has great practical value. 
+ Both non-personalized and personalized FL settings are investigated in the SSL regime. 
+ Experiments and ablation studies are conducted under different settings. 

Weakness:

- For the Per-SSFL framework, the local (personalized) model and global model are used. The memory consumption aspect should be discussed. For resource-constrained edge clients, high memory cost could be an issue. 
- Although it mentioned in the implementation setting that the client number selected per round is 10, it is not clear how many total clients are used in the FL setting. 
- In Table 1, what is the FL method under the supervised setting?
- In Figure 2, what does (0.5) for SSFL on IID and SSFL on non-IID mean? 
- As can be seen in Figure 2, the convergence rates for the IID and non-IID cases are quite similar. Can you provide an explanation for that?
- After reading Appendix D and Figure 10 in Appendix, the experimental setting on GLD-23K is still not quite clear. For example, how the local training set for each client is generated? What about the label distribution? Also, it seems that the number of clients used on the GLD-23K is different from that on CIFAR-10. 
- For ease of comparison and implementation, it would be good to evaluate the method on more commonly used datasets such as CIFAR-100 and Tiny-ImageNet and other datasets besides vision datasets (e.g., text) for FL. 
- It would be interesting to see the SSFL results under different numbers of selected participant clients.  
- Since $\lambda$ is an important parameter that balances consensus and personalization, its effect should be studied. 

Minor issues:

1. Section 2.1, “…  the local empirical risk over the heterogeneous dataset $D^k$.” -> $D_k$
2. Figure 10 appeared in Sec. 5.2 and 5.3, it should be Fig. 2 and Fig. 3.
3. In Figure 2(a), the colors for the curve (SSFL on non-IID) and its legend are different (pink vs. red). It should be made consistent. 
4. Sec. 3.2, “… contemporary self-supervised learning frameworks (e.g., SimSiam,
SwAV, BYOL)” -> should be “… (e.g., SimCLR, SwAV, BYOL)”

A careful proofread of the paper is highly recommended. 

summary_of_the_review: I appreciate the emerging topic of SSL in FL. The proposed personalized federated SSL is interesting. However, there are quite a few unclear details regarding the method and experimental settings.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors merge self-supervised learning in personalized federated learning to solve the limited label and data heterogeneity problems in the local clients. They test several current algorithms under their framework. Then they propose an algorithm named Per- SSFL considering the balance between the consensus and personalization. Finally, they provide experiment results to support their claims and comprehensive analysis of what they find. The main contribution is to design a self-supervised FL framework with supportive experiment results. They also provide suggestions to choose appropriate algorithms and hyper-parameters under different settings.
main_review: Strengths:
+ The label deficiency problem is a realistic gap before we put federated learning into practice. The research question itself is a good and convincing topic to explore
+ They cite appropriate related research work including SSL and Personalized FL.
+ The problem formulation and the optimization function define the research problem clearly. They provide experiment results to support the claims. The analysis and discussion after the experiment are sufficient and convincing.

Weakness:
- I am a little bit concerned about the technical novelty in this paper. To combine SSL and FL is fine but lacks the motivation and we may not encourage just merging existing techniques together in a framework.
- Also, by adding regulations, adding constraints, or other hyper-parameters to control the difference between the global models and locals, such approaches have been proposed already. The hidden ideas or we say the core idea is quite similar.
- More discussion is expected about how to choose hyper-parameters.  
summary_of_the_review: In FL, label deficiency and data heterogeneity are two of the main open research questions. Authors propose a natural idea to leverage SSL in FL with the support of testing, analysis, and experiments. However, I am not sure the novelty of this work is good for this conference. I might expect more novel technique ideas in this paper-increasing the FL field.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The submission is not anonymized, the author's name chaoyanghe appears frequently in the attached SSFL_Sumpplementary/SSFL-Source-Code. Therefore, it should be directly desk rejected.

--- Logging error ---
Traceback (most recent call last):
  File "/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py", line 1025, in emit
    msg = self.format(record)
  File "/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py", line 869, in format
    return fmt.format(record)
  File "/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py", line 608, in format
    record.message = record.getMessage()
  File "/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py", line 369, in getMessage
    msg = msg % self.args
main_review: See Summary Of The Paper for detail.
summary_of_the_review: See Summary Of The Paper for detail.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work is concerned with a very practical scenario of Federated Learning where the participating agents may not have access to labelled data. SimSiam architecture to learn useful feature representations with extensions that incorporate personalisation for local client models. The learnt representation are evaluated against a KNN classifier for analysing their usefulness.

main_review: 
Strengths
- Deals with an important and practical scenario of FL
- The analysis of personalised representations is interesting

Weakness
- Insufficient quantitative assessment of SimSiam approach against other self-supervised approaches. 
- It is unclear the feature representation can be used in practice, especially in 
the personalised setting. Perhaps explore the use as an auxiliary objective in some downstream applications?
- The effect of hyperparameters is under-explored. For instance, the size of representation which could overpower the benefits of self-supervision in the assessment with KNN.

summary_of_the_review: I think the work tackles an important question, however the experimentation is inadequate to assess the usefulness of the approach.
review_title: null

Meta-review: This manuscript proposes an extension of semi-supervised learning to the federated setting. The contributions include a thorough evaluation of performance and some method extensions. 

There are four reviewers. One reviewer points out a name leakage issue in the code that was missed and suggests deks-rejection. The area chair has chosen not to desk-reject the paper. Three other reviews agree that the manuscript addresses an interesting and timely issue -- indeed, label acquisition is a significant issue in federated learning. Three reviewers agree to reject the paper -- raising concerns about novelty compared to existing methods, some details of the evaluation, and some lack of clarity. The authors provide a good rebuttal addressing many of these issues. However, the reviewers are unconvinced that the method is sufficiently novel after reviews and discussion. Authors are encouraged to address the highlighted concerns for future submission of this work.
Decision: Reject