Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Collaborative self-supervised learning is considered based on a combination of known split federated learning and self-supervised contrastive leaning strategies. The former bases on model splitting between server and local clients while the latter depends on momentum-based key generation. The proposed combination, called MocoSFL, has some complexity/performance advantages over exiting FL-based solutions. A variation of this idea, based on pretraining (and later freezing) of feature extractors utilizing a small amount of clients' training data, is also proposed to improve communication overhead as well as resistance to MIA attacks.
strength_and_weaknesses: Strengths: The chosen topic of collaborative SSL is an important and timely topic. The paper is clearly written. Performance is shown to be better than SOTA in some cases.

Weaknesses: 

The novelty is limited as the work represents a combination of known methods. The design choices made during this combination all seem right but are also either already known or easily anticipated (like latent vector concatenation to create a large
batch size for the server-side model, sharing of features and more frequent synchronization). As for the target-aware variation, it is not clear whether the availability of local clients' training data (however small) is justified. Sure, as the authors mention, Bhagoji et al., 2019 considers the possibility of server being able to validate local models, but note that Bhagoji et al. mean to raise the bar for attackers (in an effort to prove that attacks are possible). In contrast, the present authors actually consider the possibility of using the local client data to guard against MIA attacks (and to reduce communication overhead following the feature extractor freezing). My impression is that this is a highly non-standard FL setting, and I would want to see more convincing references. 

Specific questions/issues include:

The obvious baselines FedEMA and FedU are missing from Table 2 and Table 3.
From Table 5, the results for Non-IID cases with Nc=5 or Nc =100 are better than IID cases which is counter-intuitive. Could the authors provide justification for this performance gain?
Can the authors explain why the latent vector size in Table 1 changed from 5000 to 39.1 MB? Is it just because the backward gradients are not included?
In Figure 7, how the GFLOPs for computation are measured? Specifically, does it include the FLOPs needed for backpropagation?
The authors claim this technique to be competitive in cross-silo setting. However, the paper requires having batch size in range of 100-200 at the server-end which seems impractical considering there exists millions of samples/clients. Could the authors comment on this?
Eq (2) finds that having a large batch-size can maintain hardness of negative samples, which seems to contradicts the empirical results in Table 6 where increasing the batch size further to 400 degrades the performance. Reasons?
Did the authors implemented/reproduced the main results of baselines works e.g. FedEMA? It seems like the results are exactly as in the original paper. 
How does this method compared against a FedMoco (Nanqing Dong, Irina Voiculescu: “Federated Contrastive Learning for Decentralized Unlabeled Medical Images”, 2021; arXiv:2109.07504) which also utilized MoCo in federated learning setup? 
Fig 3 and Fig 7 contains computation results per clients only. How much computation burden is added to the server side?

clarity,_quality,_novelty_and_reproducibility: The proposed algorithm is clear and the paper is well-written, but the novelty is questionable.
summary_of_the_review: The addressed topic - collaborative SSL - is important and timely. But the novelty here is limited as the work represents a combination of known methods. The design choices made during this combination all seem right but are also either already known or easily anticipated. As for the target-aware variation, the availability of local clients' training data at the server needs a stronger justification (is this a realistic/fair assumption? is it actually done in practice somewhere?). There are a number of questions/issues raised in the Weaknesses section above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to address the issues existing in current collaborative self-supervised learning (SSL) schemes and proposes MocoSFL, a collaborative SSL framework based on Split Federated Learning (SFL) and Momentum Contrast (MoCo). In MocoSFL, the large backbone model is split into a small client-side model and a large server-side model, and only the small client-side model is processed locally on the client’s local devices. 
Specifically, there are three key components in MocoSFL: (i) vector concatenation which enables the use of small batch size and reduces computation and memory requirements by orders of magnitude; (ii) feature sharing that helps achieve high accuracy regardless of the quality and volume of local data; (iii) frequent synchronization that helps achieve better non-IID performance because of smaller local model divergence. Numerous evaluations are conducted to evaluate the effectiveness of the proposed MocoSFL.

strength_and_weaknesses: Strength: 

1) The practical value offered by MocoSFL is interesting: I think MocoSFL is a very practical solution that can support a large number of clients (1,000 clients) to conduct collaborative self-supervised learning (SSL).
2) The proposed MocoSFL has experienced enormous evaluations: The proposed MocoSFL takes into account both IID and non-IID settings, as well as the cross-client and cross-silo cases; experiments are carried out in both simulated and real hardware devices.
3) The proposed scheme takes into account not only model performance, hardware requirements, and communication overhead, but also privacy concerns.
4) Because most IoT devices have limited memory and most data is unlabeled in reality, the investigated problem is both timely and important to the community.
5) The paper's writing is good, and the overall structure is clear.


Weaknesses:

1)  In addition to model inversion attacks, I am wondering whether MocoSFL with the TAResSFL module has the potential to defend against other privacy attacks, such as the Membership Inference Attack. 
2)  Although 1,000 clients are already a SOTA achievement, it would be great to also analyze how many clients can be supported by MocoSFL at most. Is there any turning point here? 
3) I am very interested in the possibility of landing MocoSFL in ubiquitous IoT devices, so I want to know what is the min hardware requirement to implement MocoSFL, It is better to provide more discussions on this part?

clarity,_quality,_novelty_and_reproducibility: 1) The overall structure of this paper and writing is simple and straightforward. The FL-SSL challenges are well-explained, and the proposed MocoSFL is well-motivated. The evaluation settings as well as the Hyper-parameters are clearly listed.

2) The paper is of high quality and free of technical errors. The proposed MocoSFL is a solid solution that is by far the most practical for assisting a large number of clients in conducting collaborative self-supervised learning (SSL). There are adequate evaluations: The proposed MocoSFL takes into account both IID and non-IID settings; the proposed MocoSFL takes into account not only the cross-client case but also the cross-silo case, and experiments are carried out in both simulated and real hardware devices.

3)  Although very few papers touched on the problem of FL-SSL, the identified two major challenges in deriving high accuracy in FL-SSL schemes for cross-client applications are novel. The proposed MocoSFL framework and the key components are also novel.

4) Appendix A.1 of this paper contains a detailed explanation of the MocoSFL training and evaluation hyperparameters, collaborative learning system hyperparameters, and TAResSFL module hyperparameters. These should be sufficient for reproducibility.

summary_of_the_review: The proposed MocoSFL takes into consideration not only model performance, hardware requirements, and communication overhead, but also privacy concerns. As a result, it would garner interest from a diverse community, including but not limited to FL, SSL, communication, privacy, and so on. MocoSFL also provides the industry with practical benefits by supporting a large number of clients and testing on real devices. This paper's overall structure is very clear, and the writing is also excellent. The paper as a whole is of high quality and of sufficient interest to the community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a federated self-supervised learning method named MocoSFL. based on Split Federated Learning (SFL) and MoCo. 

By leveraging a small client-side model, vector concatenation, and effective feature sharing, MocoSFL solves two challenges: high computing resources and large data requirements. Thus, MocoSFL makes it practical for a cross-client application which typically has a very large number (>100x) of clients, and each client holds a small amount of data and has limited computing capability. 

In addition, this paper introduces the TAResSFL module to effectively address the privacy threat and communication overhead of MocoSFL. MocoSFL with TAResSFL module can support 1,000 clients for the first time, and achieve over 1000x reduction in memory usage, 10000x reduction in computation, and 200x reduction in communication overhead. Moreover, compared to state-of-the-art methods, MocoSFL achieves better non-IID accuracy because of its small local model divergence.

strength_and_weaknesses: Strengths
(1)	This paper firstly identifies two practical limitations of FL-SSL methods that make them unable to support cross-client applications.

(2)	The roots of these two challenges are discussed well in Section 2.

(3)	This paper proposes a novel combination of SFL and MoCo and utilizes three components to get the best of the two. The resulting MocoSFL enables a small client-side model, a large equivalent batch size, and effective feature sharing hence mitigates two challenges.

(4)	The paper presents good reasoning on why the proposed MocoSFL is successful in mitigating large data requirements and improving non-IID performance.

(5)	Extensive experimental results demonstrate the advantages of the proposed method compared to state-of-the-art methods.

(6)	The paper honestly points out the remaining privacy and communication issues of MocoSFL, and correspondingly presents TAResSFL module to mitigate these issues.

(7)	At the end, the paper presents a hardware demo to showcase the practicality of the proposed method, showing that clients with a cheap device can participate in federated self-supervised learning without any problem.

Weaknesses

(1)	Experiment shows up to 1,000 clients. It would be interesting to see the accuracy performance when the number of clients is greater than 1,000.

(2)	MocoSFL results are limited - only for ResNet architectures. How it performs on other models?

(3)	In TAResSFL, the availability of a close-related source domain dataset and a small portion of target domain data is questionable and should be better justified.

(4)	Hyper-parameter choices of TAResSFL seems not so easy. Could you give more hints on how to wisely or automaticly choose hyper-parameters?

(5)	(minor) The privacy evaluation only uses mean square error (MSE) as measurement, did you ever consider other metrics such as structural similarity index measure (SSIM), Peak signal-to-noise ratio (PSNR)? 

clarity,_quality,_novelty_and_reproducibility: The paper clearly explains the problems and presents a novel combination of SFL and MoCo as solutions. The paper is well-written and organized. Moreover, the authors provide detailed experiment settings in appendices and promise the future release of code repository.
summary_of_the_review: This seems like a particularly interesting paper. This paper identifies practical problems in federated self-supervised learning and provides a novel MocoSFL framework to effectively solve them. The paper presents an in-depth discussion of both the problems and the solutions and shows extensive experimental results to support the arguments.  Finally, with the remaining issues addressed, the hardware demo leverages the importance and practicality of this work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes MocoSFL, a collaborative SSL framework based on SFL. The proposed framework addresses hardware resource requirement at client-side by enabling small batch size training and computation offloading. It also relieves the large data requirement of local contrastive learning by enabling effective feature sharing. To address the privacy issue and communication overhead of MocoSFL, this paper further introduces the TAResSFL module. Combined with TAResSFL, MocoSFL can support 1,000 clients for the first time, achieve better non-IID performance, and largely reducte memory usage, computation and communication cost. 
strength_and_weaknesses: Strengths

1. The whole paper is very well structured and well written.
2. The proposed MocoSFL can be scaled to a large number of clients (1,000 clients for the first time).
3. MocoSFL seems to be a very practical framework, which requires less memory, less communication, and consumes less power.
4. Authors also test the effectivenes of MocoSFL on a real Raspberry Pi 4B device, which validates its potential to land in industry.
5. The experimental design is solid and the results are convincing. All the key factors in a system are well considered and addressed, including model performance, memory usage, computation cost, communication cost, and privacy. 

Weaknesses

1. All the chosen datasets are CV centered, I want to know whether MocoSFL can be extended to the other types of data. More discussion is expected.
2. Seems FL-BYOL is the only baseline. Are there any potential baseline? Could you elaborate more on this?
3. Figure 5 can be better illustrated with x-axis and y-axis lines with arrows.

clarity,_quality,_novelty_and_reproducibility: Clarity: The limitations of current FL-SSL methods are clearly identified. The proposed MocoSFL is clearly explained.

Quality: The whole paper is well structured and well written. The experimental results are extensive and convincing.

Novelty: To my knowledge, the proposed MocoSFL and the combination with TAResSFL is novel. Also, seems MocoSFL is the first work that can deal with as large as 1,000 clients in federated self-supervised learning. Hence, I think this paper could be a milestone work in advancing the area of federated self-supervised learning.

Reproducibility: Detailed experiment settings in the paper are sufficient to reproduce the results. Authors also plan to release code upon acceptance in their REPRODUCIBILITY STATEMENT. 

summary_of_the_review: FL under self-supervised learning setting is still a very new but very practical problem. This paper clearly identifies the major challenges in deriving high accuracy in FL-SSL schemes for cross-client applications, and proposes a novel collaborative SSL framework called MocoSFL based on Split Federated Learning (SFL) and Momentum Contrast (MoCo). MocoSFL is the first work that can deal with as large as 1,000 clients in federated self-supervised learning, and tests its effectivenes on a real Raspberry Pi 4B device. Overall, this paper is very interesting and to my knowledge novel. This paper could be a milestone work in advancing federated self-supervised learning. Hence, I would like to vote for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a federated self-supervised learning method, combining federated Split Federated Learning (SFL) and MoCo.
Reviewers were consistently positive about the paper, and liked several of the contributions made. This includes the ability to deal with large number of clients, each potentially having only a small amount of data (while heterogeneous), and limited compute capacity. The novel methodology is thoroughly evaluates, so could have practical impact.

Overall, reviewers all agreed that the paper is bringing novelty, is well-written and deserves acceptance.

We hope the authors will incorporate the several minor points mentioned by the reviewers during the discussions. 

justification_for_why_not_lower_score: We have consensus this work is very solid and is the first to overcome several big limitations in previous systems
Decision: Accept: notable-top-5%