Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Summary: This work proposes an FL paradigm for SSL under client heterogeneity. The proposed method, Hetero-SSFL allows each client to train personalized and different self-supervised models. It then enables joint learning across clients by aligning the lower dimensional representation with those of a common dataset assumed to be available on the server. The work is an extension of Makhija et al., 2022 by changing the supervised learning task to SSL. Also, Herero-SSFL has the impractical assumption of having a common dataset on the server, which may limit its application in some real settings. The author compared Hetero-SSFL with other FL+SSL methods on benchmark datasets.
strength_and_weaknesses: Pros: Overall, the idea of allowing different network architecture across clients in FL for SSL is very interesting. The selected technique foundation (Makhija et al., 2022) is reasonable. 

Cons: 
1.Limited methodology novelty. Despite of the different learning paradigm, Hetero-SSL is very similar to Makhija et al., 2022 (for supervised learning tasks). The core techniques discussed in Sec 3, including general formulation, RAD, CKA, etc., seems directly borrow from Makhija et al., 2022. The authors may want to highlight the differences if there is any or give sufficient credits to the previous work. 

2.The assumption of requiring a common public dataset available in the server will raise many practical concerns. For example, what kind of data can be considered as the common dataset, how many data points need to be used, is it easy to find the privacy-free public data to serve this purpose, etc.

3.What do a and b mean in Assumption 4.1. I assume they are two iterations. Then don’t you want a,b > 0?

4.Lemma 4.5 is similar to Tan et al.  The authors should give proper reference here.

5.Are the results in Table 3 from IID or non-IID setting?

6.How to select the common (global) data is missing in Section 5.

7.Grammar and format issues should be corrected.

clarity,_quality,_novelty_and_reproducibility: Clarity: okay
Quality:  the paper written is not well-prepared and it requires proofread
Novelty: limited as core parts are similar to previous work
Reproducibility: okay
summary_of_the_review: The paper aims to address the interesting question, but the novelty and the written quality downgrade it from the other submissions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method for federated self-supervised learning with clients who may train on local tasks using heterogeneous model architectures. The authors adopt a Siamese structure for local self-supervised learning, implemented to BYOL or SimSiam, and validate it using both image and text classification tasks. To allow semantic aggregation of local clients' representation, they introduce a public dataset named Representation Alignment Dataset (RAD) that all local models utilize for implicitly transferring representations across clients by minimizing the CKA distance of RAD representation. Therefore, the proposed model doesn't need direct weight aggregation and allows communication with heterogeneous clients.

strength_and_weaknesses: **strength**
- propose a new method to transfer the knowledge across clients without direct weight aggregation, so that allows participation in heterogeneous self-supervised models 
- validate the model over text- and image-based classification problems
- provide the convergence analysis for the proposed model

**weakness**
- Missing communication cost analysis.
- Missing analyses of the RAD size and the construction of the RAD. To avoid the data privacy issue, the distribution of the RAD should be different from the distributions of local datasets (i.e., Non-IID). Therefore, the construction of the RAD can highly impact the model performance.
- Only two ResNet-variants are used for architecture heterogeneity. More diverse architectures are required for backbones, such as Transformers, simple CNNs, and MLPs.
- Performance gap with baselines decreases for larger participating clients at each round.

----
+ local clients from baselines do not train on the samples constructed to RAD?
clarity,_quality,_novelty_and_reproducibility: The construction of RAD is not clearly described. Otherwise, the paper is easy to read, and motivation is intuitive.
summary_of_the_review: The paper proposes a practical framework allowing model heterogeneity during federated self-supervised learning. The method is reasonable and provides convergence analysis, but there is a lack of analyses to understand the behavior of the proposed methods and the issue of RAD construction.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents Hetero-SSFL, a framework for SSL in FL that enables diverse clients (heterogeneous in terms of their compute capabilities and/or data distribution) to collaboratively learn a good SSL model. At the end of N rounds, the SSL model gets personalised (via linear evaluation protocol) to each client's data. Different from the majority of previous FL works, is the fact that clients in Hetero-SSFL do not communicate their locally-trained model with the server. Instead, what clients send are the embeddings outputted by the local model when using a common (i.e. same in all clients) dataset. These representations are then aggregated in the server and used in the following round to align (via a distance loss term) the representations each client learns during SSL. Hetero-SSFL is primarily evaluated on vision classification tasks but also text classification.
strength_and_weaknesses: ### Strengths

*    The paper is fairly well written, although it would benefit if Figure 1 was presented earlier in the text (maybe at the top of page 4?) Making it more self-contained would be good too.

*    The results in Tables 1,2,3 look super good compared to the other baselines but. Could the Authors explain why the figures in Table 2 are lower than in Table 1? It seems like all the baselines do better in the IID case (as you'd normally expect) for CIFAR-10 but Hetero-SSFL does worse... But then for CIFAR-100 in Table 2 (IID) all methods do worse than in Table 1 (non-IID). Could the Authors elaborate? Maybe the numbers got mixed up adding them to the table?

### Weaknesses
Some weaknesses and clarifications needed:

*    The Authors present this work as enabling collaboratively SSL in FL when clients are of different compute capabilities. However, only two types of networks ResNet-18/34 are considered and both train with batch size 200. That leaves the setting to just two types of clients. If the Authors considered a few more models to cover a wider spectrum of system capabilities (specially towards more lightweight models -- e.g mobilenet, efficientnet or similar?) and different batch sizes (which directly impacts on memory peak at training time), then I would be convinced that Hetero-SSFL is indeed addressing the client device heterogeneity problem.

*    The RAD is one of the key ingredients in Hetero-SSFL but the Authors do not provide details about it. Is it also a vision dataset? which one? How does the quality of the SSL change with different RADs?

*    The Authors state that the RAD "doesn't posses any special properties", could this be elaborated? I'm inclined to size and diversity in the RAD would be important, but to what extent? is MNIST a good RAD?

*    How the SSL training on the clients happens is clear. However, how is the linear evaluation protocol implemented in this work? Is there labelled "validation" set to train the classifier and then the evaluation is done on a, yet another, sub-set of data that the client has (i.e. a labelled test set)? Do these validation and test sets follow the same label distribution as the (unlabelled) training set each client has? 

In Table 3, the Authors present some results on how Hetere-SSFL extrapolates from the cross-silo (5 total clients and 5 clients per round -- as considered for Table1 and, I believe, Table 2 also) to a cross-device setting. I understand the cross-device setting is more challenging, specially when it comes to ensuring the alignment of representations helps the training process. 
*    Could the Authors comment upon why SSL suffers such a sharp drop in performance in cross-device settings?
*    It would be good to include in Table 3 the standard (non-SSL) FL baseline, which should be something in the high 50%s, given that FedAdam(https://arxiv.org/abs/2003.00295) achieves 52.5% in a setup with 500 clients (sampling 10 per round I believe). Adding that extra row is good for putting the results in context: SSL is really important and clearly the way to go for FL, but we aren't there yet.
clarity,_quality,_novelty_and_reproducibility: This paper is well written and addresses a known problem in an original way. 
summary_of_the_review: I am happy with this work but there are some open questions that need to be addressed ( see both the Strengths and Weaknesses lists) specially in regards to the RAD and results in Table1&2. I think the Authors can address these during the rebuttal period easily.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new approach called Hetero-SSFL, to conduct federated learning under self-supervised tasks with heterogeneous clients. Specifically, besides local training, all clients learn to align the lower dimensional representations on a common dataset. The paper provides a convergence guarantee theoretically and demonstrate the improved performance empirically.
strength_and_weaknesses: Strengths:
- The modification by adding an objective to make the representation on a common dataset aligned is intuitive and clear. 
- Authors gave a convergence guarantee for the proposed method, making the approach more thorough and convincing from the theoretical side.

Weaknesses:
- Experiments were not very comprehensive. Some important ablation studies were missing. For example, how will the coefficient $\mu$ will affect the performance? And what is the influence of the size of the common dataset RAD? It is not clear whether the performance gain was due to the additional data.
- It was not clear how the server utilized the representations from different local clients, given that the dimensions were not the same. In Eq. (1), the second term was the distance of two representations, while in Eq. (4), it was changed to two kernels. It seems that the paper did not mention how to aggregate representations.
- Comparison of communication efficiency in federated learning is necessary in terms of metrics such as the size of transmitted data in one round.
- Some implementation details were not complete. For example, how to obtain the RAD dataset? The authors also stated that weight vector $[w_1, w_2, \dots, w_N]$ can be learned, but how to achieve it? Will learning the weight vector require the communication between the server and clients just as FedAvg to average weights from different clients?
- There is a lack of discussion about privacy in federated learning and the authors should talk about potential privacy problems and whether the method can preserve differential privacy.


clarity,_quality,_novelty_and_reproducibility: Overall, the paper is easy to follow but as mentioned in weaknesses, some details were missing, which might cause confusion to the audience. The technical contributions of the paper were quite marginal despite its superior performance compared with baseline methods. As for reproducibility, it is only possible when the RAD dataset is available.
summary_of_the_review: The paper has some important details missing and the experimental part was not comprehensive enough. The current version is not ready for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper describes an FL paradigm for SSL under client heterogeneity. The proposed method, Hetero-SSFL allows each client to train personalized and different self-supervised models. It then enables joint learning across clients by aligning the lower dimensional representation with those of a common dataset assumed to be available on the server. The work is an extension of Makhija et al., 2022 by changing the supervised learning task to SSL. Also, Herero-SSFL has the impractical assumption of having a common dataset on the server, which may limit its application in some real settings. The author compared Hetero-SSFL with other FL+SSL methods on benchmark datasets. Overall, the idea of allowing different network architecture across clients in FL for SSL is very interesting. The selected technique foundation is reasonable.

However, there are several obvious weakness: 1) Limited methodology novelty. Despite of the different learning paradigm, Hetero-SSL is very similar to Makhija et al., 2022 (for supervised learning tasks). The core techniques discussed in Sec 3 seems directly borrow from Makhija et al., 2022. The authors may want to highlight the differences if there is any or give sufficient credits to the previous work. Although this paper discusses SSL in FL, the contribution (in my view) is incremental to Makhija et al., 2022. 2) The assumption of requiring a common public dataset available in the server will raise many practical concerns. For example, what kind of data can be considered as the common dataset, how many data points need to be used, is it easy to find the privacy-free public data to serve this purpose, etc. 3) A common concern shared by all the reviewers is about RAD, which plays a very important role in the proposed method. The assumption that RAD is public and comes from the same domain is strong. A key application of FL is privacy-sensitive data analysis. Thus, a representative public dataset can be difficult to acquire. Also, this RAD needs to communicate with clients. Thus, communication costs need to be highlighted. It will be helpful to highlight the number of RAD data points required. 4) The theoretical analysis does not show the SSL component. The assumptions are commonly used in the existing optimization literature and not specifically for SSL. The proof does not explicitly show how SSL played a role in the derivation. Overall, this paper may not be ready for publication at ICLR. The next version must be a strong paper if authors can take comments into consideration.
justification_for_why_not_higher_score: According to my expertise and reviewing process, this paper should belong to a Reject.
justification_for_why_not_lower_score: According to my expertise and reviewing process, this paper should belong to a Reject.
Decision: Reject