MocoSFL: enabling cross-client collaborative self-supervised learning
Authors: Jingtao Li
Lingjuan Lyu
Daisuke Iso
Chaitali Chakrabarti
Michael Spranger
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Self-supervised Learning
Collaborative Learning
Split Federated Learning
Momentum Contrast

Abstract: Existing collaborative self-supervised learning (SSL) schemes are not suitable for cross-client applications because of their expensive computation and large local data requirements. To address these issues, we propose MocoSFL, a collaborative SSL framework based on Split Federated Learning (SFL) and Momentum Contrast (MoCo). In MocoSFL, the large backbone model is split into a small client-side model and a large server-side model, and only the small client-side model is processed locally on the client's local devices. MocoSFL has three key components: (i) vector concatenation which enables the use of small batch size and reduces computation and memory requirements by orders of magnitude; (ii) feature sharing that helps achieve high accuracy regardless of the quality and volume of local data; (iii) frequent synchronization that helps achieve better non-IID performance because of smaller local model divergence. For a 1,000-client case with non-IID data (each client only has data from 2 random classes of CIFAR-10), MocoSFL can achieve over 84% accuracy with ResNet-18 model. Next we present TAResSFL module that significantly improves the resistance to privacy threats and communication overhead with small sacrifice in accuracy for a MocoSFL system. On a Raspberry Pi 4B device, the MocoSFL-based scheme requires less than 1MB of memory and less than 40MB of communication, and consumes less than 5W power. The code is available at https://github.com/SonyAI/MocoSFL.

Published as a conference paper at ICLR 2023
MOCOSFL: ENABLING CROSS-CLIENT COLLABORA- TIVE SELF-SUPERVISED LEARNING
Jingtao Li
Lingjuan Lyu
Daisuke Iso
Arizona State University jingtaol saaggaao@@auueeee
Sony Al
Sony Al da baisuke.. wsw.sony.ccoommmmoooooom.mm.moo.
1 ling 1 lvésony.com
Chaitali Chakrabarti
Michael Spranger
Arizona State University chaitali@asi edu
Sony AI michael. m www.aeeesssaangggg.y..ooom
ABSTRACT
Existing collaborative self-suue learning (SSL) schemes are suitable for cross-client applications because of their expensive computation and large local data requirements. address these issues, we propose MocoSFL, collabora- tive SSL framework based on Split Federated Learning (SFL) and Momentum Contrast (MoCo). MocoSFL, the large backbone model split into a small client-side model and large server-side model, and only the small client-side model is processed locally on the client's local devices. MocoSFL. has three key components: (i) vector concatenation which enables the use of small batch size and reduces computation and memory requirements by orders of magnitude; (ii) feature sharing that helps achieve high accuracy regardless of the quality and vol- ume of local data; (iii) frequent synchronization that helps achieve better non-lID performance because of smaller local model divergence. For a 1,000-client case with non-IID data (each client only has data from 2 random classes of CIFAR-10), MocoSFL. can achieve over 84% accuracy with ResNet- model. Next we present TAResSFL. module that significantly improves the resistance to privacy threats and communication overhead with small sacrifice accuracy for a MocoSFL system. On Raspberry Pi 4B device, the MocoSFL-based scheme requires less than 1MB memory and less than 40MB of communication, and consumes less than 5W power. The code is available at /tthtss:///ghuubbb... thub: Stt///oonAIA/MoS///
INTRODUCTION
Collaborative learning schemes have become increasingly popular, as clients can train their own local models without sharing their private local data. Current collaborative learning applications mostly focus on supervised learning applications where labels are available (Hard et al. [018 Roth al. 2020). However, availability of fully-labeled data may not be practical since labeling requires expertise and can be difficult to execute, especially for the average client.
Federated learning (FL) (McMahan et al. 2017) is the most popular collaborative learning framework. One representative algorithm is "FedAvg", where clients send their local copies of the model to the server and the server performs weighted average operation (weight depends on the amount of data)
to get new global model. FL has achieved great success in supervised learning, and has been used successfully wide range of applications, such next word prediction McMahan et (201)). visual object detection for safety [iu et al. (2020), recommendation Wu et al. (202a b), graph-based analysis Chen al. 2002)) Wu et al. (2022c), etc.
For collaborative learning on unlabeled data, prior works (Zhang et al. 2020 Zhuang et al. 2021 I
2022) combine FL scheme with classic self-supervised learning (SSL) methods such as BYO Grril al. 2020) and Moco (He et al. 2020). These methods can all achieve good performance when clients' data is Independent and Identically Distributed (IID) but suffer from poor performance
Work done during internship Sony AI. Corresponding Lingjuan Lyu.

---

Published as a conference paper at ICLR 2023
0 0:3G8
0,665
1.1GB
10K
1K
500
KNN Accuracy
80%
Memory Consumption
KNN Accuracy
80%
Number of Data
FL-BYOL FL-MoCoV2
Nard
Neasy
Online
Q
55%
FL-BYOL
55%
FL-MoCoVV2
Momentum
K
N
easy
30%
30%
N
hard
0.8
32
Batch Size
64
96
128
05
Number of Clients
50
100
Feature Space
(a)
(b)
(c)
Figure Challenges in FL-SSL schemes. (a) large batch size is necessary to achieve good performance (KNN validation accuracy (Wu et al., 2018) (b) Accuracy reduces with increasing number of clients since the amount local data is now smaller. (c) Hard negative keys are essential for the success of contrastive learning.
in non-IlD cases. Recently, Zhuang et al. (2022) mitigated the non-IlI performance drop with divergence-awe aggregation technique and provided statc-of-the-ard (SoTA) accuracy performance using a combination of FL and different SSL methods.
However, these SoTA FL-SSL schemes are not practical for cross-client applications. First, FL- SSL imposes a significant computation overhead and large memory requirement on clients. This is because SSL requires a large backbone architecture (Chen et al. 2020a) together with a large batch size to ensure good performance. As shown in Fig. Ta), accuracy drops dramatically when batch size is low for both FL.-SSL. methods. Second, FL-SSI. schemes fail to maintain a good accuracy when the number of clients is large (cross-client cases), as shown in Fig. [b). For a dataset with fixed size, when the number of clients increases, data per client decreases, resulting in accuracy degradation. The drop in accuracy is mainly because of the failure to meet data size requirement in performing contrastive learning. Zhang al. (2020); |Wu al. (202T) attempt to address this issue in FL-SSL with remote feature sharing. However, this introduces a high communication overhead due client-to-client feature memory synchronization; for 100-client system, costs around 2.46GB per synchronization per client.
To solve these challenges, we propose MocoSFL., scheme based on Split Federated Learn- ing (SFL) Thapa et al. (2020) that incorporates the feature memory bank and momentum model designs MoCo (He et al. 2020). We adopt the SFL scheme for three reasons: (i) SFL utilizes smaller client-side model and so reduces the computation overhead and has lower memory con- sumption and model parameters; (ii) SFL's latent vector concatenation enables a large equivalent batch size for the centralized server-side model, making micro-batch training possible for clients and thus reducing client's local memory; (iii) When combined with MoCo's key-storing mechanism, SFL's shared server-side model enables effective feature sharing, which removes the requirement of large amount of local data and makes the scheme possible for cross-client applications. As a result, MocoSFL achieves good accuracy with ultra-low memory requirements and computation overhead, and can support very large number of clients. MocoSFL. shows better non-IID performance since local model divergence is smaller. However, the of SFL brings extra communication overhead as well as data privacy concerns. Thus, we present target-aware ResSFL (TAResSFL) module as an effective solution to mitigate these issues with small accuracy drop.
Our main contributions are:
We identify two major challenges in deriving high aceuracy in FL-SSL schemes for cross- client applications. These are availability of a large amount of data that is required for contrastive learning and the ability to process them clients who may not have sufficient hardware resources.
We propose MocoSFL, SFL-based scheme to address the two challenges. MocoSFL uses a small client-side model, latent vector concatenation, and feature sharing. For cross-client case, MocoSFL is the only viable and practical solution. For cross-silo case, MocoSFL can achieve even better performance than SoTA FL-SSL schemes under non-IID setting because of smaller model divergence.
2

---

Published as a conference paper at ICLR 2023
To address communication overhead and privacy issues that inherent to SFL-based schemes, we propose target-domain-awe ResSFL (TAResSFL) that effectively reduces the communication cost and mitigates model inversion attacks.
2
BACKGROUND
2.1
SELF-SUPERVISED LEARNING
To learn from unlabeled data, SSL schemes based on contrastive learning such as SimCLR Chen et al. (2020a), B Brill et al. 2020), Simsiam Chen and He (2021) and MoCo He et al. (2020) have achieved great performance on popular benchmarks. Unlike other schemes (BYOL, SimCLR, etc.) that use other samples in the current data-batch as negative keys, MoCo uses previously computed positive keys as negative keys, and stores current positive keys in the feature memory for future iterations. The key-storing mechanism results in a relatively smaller batch size requirement that is beneficial for reducing device memory and also makes it easy for feature sharing implementations for our proposed scheme. For the loss function, MoCo relies on InfoNCE loss Oord al. (2018) as the contrastive mechanism to update its model parameters:
exp(Q K*/T))
LQ.KNN = -log eapp(Q K+/T) + ENEAs erp(Q N/T)
(I)
where query key Q and positive key K+ are the output vectors of server-side momentum model and the momentum model, respectively, obtained by processing two augmented views of the image. N denotes all negative keys in the feature memory of size M. Importantly. the success of MoCo scheme highly depends on the "hardness" of its negative keys (Kalantidis et al.) | Robinson et al.] (020). The "hardness" of negative key N in the feature memory bank, can be determined by the similarity (inner-product) between Qa (at step t) and N; the smaller the similarity, the better the "hardness". We notice the "hardness" of negative key N reduces quickly because model updates are in the direction of minimizing the InfoNCE loss in Eq. ( As result, MoCo adopts a slow-changing momentum model produce consistent negative keys to add the feature memory at the end of every training step and thereby maintains their hardness.
2.2 SPLIT FEDERATED LEARNING
Split Federated Learning (SFL) Thapa et al. (2020) is a recent collaborative learning scheme that focuses on high computation efficiency at the client side. It splits the original model architecture into two parts, the client-side model that contains all layers upto the "cut-layer" and the server-side model that contains the remaining layers. We distribute copies of client-side model C2 to client-i's local devices and instantiate the server-side model in a cloud server. To complete each training step, clients need to send the latent vectors (the output of client-side model) to the server, and the server processes latent vectors, computes the loss, performs backward propagation and returns the corresponding gradients to clients. Thapa al. (2020) present two possible ways for server to process latent vectors sent by clients. In this paper, we use SFL-VI where the server concatenates all clients' latent vectors and processes them altogether, which makes the equivalent batch size larger at the server and benefits contrastive learning. In contrast, in SFL-V2, client's latent vectors are processed sequentially in a first-come-firs--erre manner and thus does not benefit from the "large batch". We provide details of SFL-VI in Appendix A.2 A.2
3 MOTIVATION
As mentioned in Section | the two challenges in extending FL-SSL to cross-client applications are high computing resource requirement and large data requirement.
3.1 HIGH COMPUTING RESOURCE REQUIREMENT
The first challenge is the computing resource requirement of training SSL model locally. Using
a compact backbone model may be accurate for supervised learning, but is not suitable for SSL as has a much higher requirement the model capacity. (Shi et al. 2021 Fang al. 2021 show
3

---

Published as a conference paper at ICLR 2023
that compact architectures like Mobilenet-Vi V3 (Howard et al. 2019), EfficientNet Tan and Le (2019)
suffer from over 10% accuracy degradation compared to a larger ResNet-18 architecture, while an larger ResNet-50 model has over 15% better accuracy compared to ResNet- 18 on ImageNet dataset (Deng et al. 2 This means memory requirement for training an SSL. model with high accuracy very high. Using smaller batch size reduces accuracy dramatically, as shown in Fig. Fig. aa), and is thus not an option. A FL-SSL scheme (ResNet-18 with a batch size of 128) costs 590.6 MFLOPs per image and over 1100MB of memory per client, which is not practical.
3.2 LARGE DATA REQUIREMENT
The other major diffieulty for FL-SSL to generalize to the cross-client case is the large data require- ment. For cross-client applications, the amount of data available to each client can be very limited. For example, in a cross-silo medical application, a client can be a hospital with tons of data. In comparison, in a cross-client application, client be patient who has limited amount of data.
The root of the problem lies in the difficulty to find hard negative samples when clients do not have enough local data. When the amount of data is larger, the chance for hard negative samples to be present becomes much higher. As a result, existing FL-SSL can only be successful for cross-silo applications where clients have large amount of data and can perform effective contrastive learning locally. As demonstrated in Fig. [b), we observe high accuracy when clients have 10K samples of data, while the accuracy drops quickly to around 30% when clients have only 500 samples.
Unfortunately, remote feature sharing /Zhang et al. (0200) Wu et al. (022 2022d in FL-SSL schemes cannot solve the large data requirement. These schemes update the shared feature memory less frequently because of significant communication overhead with each update. Since clients must synchronize their local copies of the shared feature memory each time a minor change happens, in Zhang et al.] (2020); Wu et al. (2022d), new negative keys are added to the feature memory only once per epoch. Even so, the total communication overhead of remote feature sharing scales quadratically with the number of clients makes it not practical for cross cross-clientee case.
4 METHOD
4.1 PROPOSED MOCOSFL
Our proposed MocoSFL is an innovative combination of SFL-V1 and MoCo-V2 V2 (Chen et al., 2020b) as shown in Fig. There three key components. First, in each training step, the latent vectors sent by all clients are concatenated before being processed by the server-side model. This helps achieve large equivalent batch size in order to support mini-batch training. Second, use shared feature memory which is updated by positive keys contributed by all clients in every training step. Third, we improve the non-IID performance by using higher synchronization frequency.
Next, we will elaborate on how these three components in the proposed MocoSFL address the two challenges in Section 4.2 and Section 4.3 Section 4.4 describes how MocoSFL achieves better non- IID performance and Section 4.5 addresses the privacy and communication issues of the proposed scheme.
4.2 REDUCE HARDWARE RESOURCE REQUIREMENT
Choice of SFL helps reduce computational overhead and memory consumption at the client-end because of the much smaller client-side model. For example, on a CIFAR-10 ResNet-18 model with a batch size of 128, client-side model with 3 layers only costs 13.7% of the FLOPs compared to the entire model, and its memory cost 227MB, merely one fourth of the entire model. Furthermore, we reduce the batch size to 1 (also known as "micro-batch"",, to further reduce the memory consumption. The use of micro-batch in local model training is only possible thanks to the latent vector concatenation mechanism which basically aggregates latent vectors sent by all clients into big batch before sending it to the server. In addition, in a micro-batch setting, we replace the batch normalization layer by group normalization Wu and He (2018) and weight standardization (Qiao al. 2019) to gain better accuracy performance. In Fig. 3 b), we compare the computation and memory consumption of the proposed MocoSFL. with the FL-SSL scheme. MocoSFL. with cut-layer of 3 achieves ~ 288x
4

---

Published as a conference paper at ICLR 2023
Client
|00 Ax*
DeQueue
Server
Xkta
X
Augmentation
Client i momentum
N
000
Aqi
Shared Feature Memory
EnQueue
Concatenate
Client i online
A k+
K
InfoNCE
Wireless communication
Loss
Client] j
000 Avtj
Server momentum
x
Concatenate
x
Augmentation
Client j momentum
A
X
000
Ayj
avg
Server online
Client online
Frequent Sync
Figure 2:
MocoSFL scheme. Three highlighted components are (1) latent vector concatenation, (2) shared feature memory, and (3) frequent synchronization.
reduction in memory consumption than FL-SSL and has 25%100% higher accuracy. Details of accuracy evaluation included in Section 5.
4.3 MITIGATE LARGE DATA REQUIREMENT
As indicated by (Kalantidis et al. 2020; Robinson et al. 2020), the "hardness" of a negative key N heavily depends on its similarity with the current query key Q, given that N and Q have different ground-truth labels. To evaluate the hardness of negative key No residing in the feature memory, we use the similarity measure (inner-product) between No and Qt. a freshly calculated query key at time t. In FL-SSL with feature sharing, the negative key is only updated after a long period of time to reduce communication overhead. As result, the hardness diminishes quickly. In contrast, MocoSFL frequently updates its feature memory maintain good hardness. At every training step, freshly calculated positive key K is added to the tail of the queue, and the oldest one is popped out.
However, frequent updates of feature memory is not enough to ensure high level of hardness. We also find it necessary to use a large batch size and a large feature memory. This finding agrees with the study in Bulat al. (2021) and also explains the accuracy drop for small batch size in FL-MocoV2 in Fig. aa). ) To illustrate this, we consider the total similarity measure at time ł and make the following assumptions: For the newest batch of negative keys N, K₁-1 of size B in the feature memory at time f, we assume the similarity measure between N₂ and Qe is a constant y for all We also assume similarity of every batch of negative keys feature memory gets reduced by constant factor y (y < 1) after each model update to represent the degradation caused by model updates. Thus, for a freshly computed query Qt, its total similarity measure with negative keys the feature memory can be represented as:
hardness Buyy + Bup² +... Bin LM/BI LM/BBB
1
= B x
1-7
where B is the batch size and M is the feature memory size. We see that using a large batch size helps bring more fresh negative keys and maintain better hardness, Also, using
B is beneficial a larger feature memory (increasing M) can keep enough negative keys and contribute to better total hardness, In the cross-client case, FL-SSL schemes can hardly meet these two requirements because of the small clients' memory. Nonetheless, MocoSFL. can easily fulfill them because (1) latent vector concatenation enables large equivalent batch size, and (2) feature memory hosted by the cloud server can be much larger.
4.4 IMPROVING NON-IID PERFORMANCE
We found that use of SFL results in fewer model parameters at the client side and hence smaller model divergence. Furthermore, introducing frequent synchronization in MocoSFL. provides additional
5

---

Published as a conference paper at ICLR 2023
100
FL-SSL
MocoSFL-1
MocoSFL-3
SyncFreq
Divergence
SyncFreq=5
1600
1627.8
SyncFreq=10
Metrics
1200
4-
9214
800
2-
400
223.8
0,8
MF MFLOOS//mage
Memory (MB)
FL-SSL
M-coSFL-S MccoSFL-3 MocoSFL-1
(h)
Figure 3: (a) Proposed MocoSFL reduces model divergence. (b) Computation overhead (FLOP counts of inference per image of the online model at client-end) comparison between FL-SSL. scheme and MocoSFL schemes. MocoSFL-L: client-side model has L layers.
reduction in model divergence and greatly improves the non-Ill) performance. According to Zhang al. (2020); Zhuang al. (202T) 2022). the model divergence between two models is calculated as the L2 norm of the weight difference. Following the same idea, the total divergence measure of a cross-client system can be measured as the average weight divergence of local models w.r.t. the global model during training:
divergence ENe ΣΣΣ
W oa Walll
where L denotes the number of layers in the client-side model, E denotes the total number of synchronizations, .ec denotes the number of clients, and 1, e, are the respective indices for L, Nc- W* is the average of all client models W'. MocoSFL reduces the model divergence with two orthogonal mechanisms. The first mechanism the reduction of client-side model size, which directly results in lower model divergence. As shown in Fig. 3 a), compared to FL-SSL scheme, MocoSFL has a much lower model divergence when the client-side model has less than layers. The other mechanism is frequent model synchronization which helps reduce the model divergence. This is only possible in SFL because of communication overhead of sending weights in a smaller client-side model, smaller. Fig. 3 also illustrates how model divergence further reduces we increase the synchronization frequency.
4.5 IMPROVING PRIVACY AND COMMUNICATION OVERHEAD OF MOCOSFL
The proposed MocoSFL scheme is based SFL and suffers from issues high overall commu- nication overhead due to transmitting and receiving latent vectors and vulnerability to Model Inver- sion Attack (MIA) Fredrikson al. (2015), in which the server can reconstruct clients' raw inputs from latent vectors. making clients data privacy questionable. (We leave the details of its threat model and working mechanism in Appendix A.3) To address the privacy and communication issues (TAResSFL.). TAResSFL. extends ResSFL al.) (2022) for self-supervised learning through: (1) target-data-awarc self-supervised pre-training, and (2) freezing feature extractor during SFL training. TAResSFL also utilizes the bottleneck layer design from ResSFL to reduce the communication overhead.
in MocoSFI. we propose Srgget-Aware-Reese
In ResSFL Li et al. (2022). the server performs pretraining to build up the resistance to MIA using data from a different domain since it does not have access to clients' data. Next, the pretrained resistant client-side model is transferred the clients and gets fine-tuned using SFL. TA improves the pretraining step by assuming that the server can get access to a small subset (<1%) of training data, together with large amount of data from another domain, and perform pre-training using self-supervised learning. Such a pretrained client-side model has better transferability. and can thus stay frozen during SFL process, thereby avoiding the expensive fine-tuning, As shown in Fig 4 we blend the source dataset X, with tiny subset of target dataset X, during the feature extractor training. The attacker aware training has the min-max form:
min max L(S(Ws; C(We: cs [zy. E&+ 1)) + AR(G C(Wc; W. Wo
(5)
Contrastive Loss
Inversion Score

---

Published as a conference paper at ICLR 2023
where R in the inner maximization denotes similarity measure, for which we use the structural similarity index (SSIM) score Zhao et al. (2016). The inner maximization is used to train the simulated attack model G, whose function reconstructing the activation back to raw input state that is similar with ground-truth X-- The outer minimization step goes in the direction of lowering contrastive loss, where the regularization term makes the model accurate well as resistant attack. These two steps are done alternatively to make the feature extractor resistant, and also be able to achieve good accuracy on the target dataset.
Fig. 4 presents the Target-aware ResSFL scheme. We use the resistant feature extractor to initialize client-side models during transfer step, as shown by the pink arrow in Fig. Unlike ResSFL, here we freeze its parameter completely to maintain the resistance since any parameter change can cause resistance drop. However, freezing brings noticeable accuracy drop of larger than 3%, even with the use of CIFAR-10 100 as source dataset if we perform pre-training by only using the source dataset (see Table 4 But if we blend the source data with small portion of target data during pre-training. the accuracy can be greatly improved. Since the model accuracy validation is done by the server for monitoring purpose (Bhagoji et al. 2019), we believe that the server can separate out a small proportion of validation data meet the target data availability assumption.
The freezing also greatly benefits hardware resource requirement since: (1) clients only need to transmit the latent vectors to the server and do not need to perform backward propagation using gradients from the server; (2) client-side model synchronization is not needed, As a result, MocoSFL. 128x (=5001.4/33
with FFL component achieves
overall communication reduction Here MocoSFL methods undergo 200 model synchronizations for 200 epochs while FL-SSL methods need 100 synchronizations (using the
compared to the original MocoSFL, as shown in Table [
same setting in Zhuang et al. (2022)).
Simulated Attacker
Initialize
Communication
Weights (MB)
Latent Vectors (MB)
Total (Relative (MB) Ratio)
Xig
Aro
Online Freeze models
Overbead
Resistant Feature Extractor
Zhunng al. 2022
FL.SSL methods
$269.2
0.0
8269.2 (1.000s)
MOOA
A...
Mnnnn Q, K?
Moco8PL-1
5001.4 (0.005x)
Xx Xex=
0000
Arx
Qu,Ki
InfoNCE loss
MocoSPL-3
57.9
5000
5057.9 9 (0.6124)
Mix-
MreoSFL-.LTTARSSPP
0.0
39.1
39.1 (0.055))
Feature Extractor
Tail model Contrastive head
39.1
(0,005))
Figure 4: Target-aware ResSFL scheme: (1) Target Table I: Communication overhead per client. FL- domain data used in pretraining; (2) Client-side SSL: 100 times of synchronization; MocoSFL. model is frozen during training.
200 times of synchronization.
EXPERIMENTAL RESULT
Experimental Setting, We simulate the multi-client MocoSFL scheme on a Linux machine, where we use different CPU threads to simulate different clients and a single RTX-3090 GPU to simulate the cloud server. We use ResNet-18 (He et al. (016)) for the majority of the experiments to better compare with existing SoTA (Zhuang et al. 2022). We use CIFAR-10 10 as the main dataset and also present results on CIFAR- 100 and ImageNet 12-class subset as in Li et al. (202I). For the IID case, we assume the entire dataset is divided randomly and equally among all clients. For non- IID experiments, we mainly consider the pathological (aka. class-wise) non-lllD distribution as in McMahan et al. (2017); Zhuang et al. (022) where assign 2 classes of CIFAR-10//mageNer
data 20 classes of CIFAR- 100 data randomly to each client. We perform MocoSFL training for a total of 200 epochs, using SGD as the optimizer with an initial learning rate of 0.06. For accuracy performance evaluation, we adopt similar linear probe methods as in Grill et al. (2020); Zhuang et al. (2022). where we train a new linear classifier on the outputs generated by the MocoSFL backbone model. We include details of
choices and evaluations in Appendix A.I
5.1 ACCURACY PERFORMANCE
Improved non-III performance. Fig. shows how the increased synchronization frequency can significantly improve the non-III accuracy. We present results for the cut-l choices of 1 and 3 convolutional layers in the client-side model, represented by "MocoSFL-I" and "MocoSFL-3", respectively. We attribute the improved accuracy to the reduction in model divergence.
7

---

Published as a conference paper at ICLR 2023
Comparison with FL-SSL. When synchronization frequency of the MocoSFL is set 10 (per epoch) for the 5-client cases, MocoSFL. achieves significantly better non-lID accuracy performance than Zhuang et al. (2022) CIFAR-10 dataset due to lower model divergence (see Table 2 On CIFAR-100 dataset, with N = 5, we observe our method has lower accuracy than FL-SSL. methods. We hypothesize that the performance of MocoSFL. is more sensitive the model complexity, and thus has limited performance for a more complex task like CIFAR-100, When model complexity high enough, for instance, on a larger ResNet-50 model, accuracy our method ~4% higher than Zhuang et al. (2022) as shown in Appendix B.7 Furthermore, our methods outperform FL-SSL methods by a large margin 20-client cases thanks to the feature sharing aspect.
Linear Probe
CIFAR-10
CIFAR-100
87%
Method
85%
MocoSFL-1
Ng
No 20
No
No
MocoSFL-3
FL-BYOL Zhuang et al. 2022
83.34
75.77
61.78
52.78
10
20
Sync Frequency
MocoSFL.- (nurs)
87.81
85.84
58.78
57.80
MocoSFL-3 (ours)
87.29
85.32
57.70
57.52
Figure 5: Effeet of increasing synchronization
Table 2: Non-IID performance comparison (linear probe accuracy)
Cross-client Performance. Our proposed MocoSFL generalize from cross-silo application (with upto 20 clients) to cross-client application with 100, 200, and 1000 clients. Note that none of the previous FL-SSL methods can scale such large number of clients. For the
choices, we follow two principles introduced in Appendix A.I] we let each client use a batch size of 1 and use the synchronization frequency of fs (1000/ 10000/NNo Vepoch, and we the client sampling ratio to 100/Nc. to keep the same equivalent batch size at the server end. The results are shown the 1000-client case. MocoSFL's accuracy for IID case is high when Nc increases from 100 to 1,000, though its accuracy drop by % for non-IID case. This small drop because model divergence scales with number clients as described in Section
in Table 3 Note that each client has only 50 data samples
Table 3: MocoSFL cross-client accuracy performance (linear probe accuracy) of ResNet- 18 model on CIFAR-10, CIFAR- 100 and Imagenet- 12 datasets with different number of clients No.
IID
non-IID
Method
Dataset
Ng 100
Nc 200
Ne
1000
NG 100
No 200
Ne 1000
CIFAR-10
87.29
87.38
87.51
87.71
87.39
86.46
MocoSFL-1
CIFAR-100
58.91
59.15
58.85
59.22
58.90
56.75
Net-e2
92.02
91.73
91.76
92.24
91.44
91.28
CIFAR-10
87.29
87.15
87.25
87.10
85.22
84.75
MocoSFL-3
CIFAR -100
58.41
58.30
58.80
58.69
58.59
56.88
ImageNet-
92.08
92.24
92.02
92.60
91.83
91.28
5.2
PRIVACY EVALUATION
Target Data
Ground-
Method
Metric
Truth
0.0%
0.5%
1.0%
MocoSVL..1
Accuracy (%)
01.14=10.47
80.78+t1.34
79. 79.96:222.96
MocoSFL
Attack MSE
0.0391:0.005
0.033=0.014
0.039-10.002
MocoSVL-3
Accuracy (%)
11.19-2.32
80.51++1.499
83.13+-2.40
MocoSFL +TAResSFL
Attack MSE
0.045:10.003
0.035=0.003
0.039-10.002
Table 4: Accuracy and MIA resistance performance (Attack Figure 6: Visualization of MIA recon- MSE) of MAA (Averaging 3 random seeds) structed images using TAResSFI.
We use G to perform MIA attack; the architecture information of given in Appendix A.3 We assume the real-time attacker also uses attack model with the same architecture as G. For the pre-training step, we use a fixed hyper-parameter choice of À = 2.0 and a target SSIM level of 0.6 to limit the strength of regularization. We assume 0.0%, 0.5% and 1.0% of the target dataset CIFAR-10 accessible, and we use CIFAR- 100 dataset as source dataset to assist the pre-training We set the cut-layer choices and 3 and set the #clients 100 for the training process.
8

---

Published as a conference paper at ICLR 2023
Successful mitigation of MIA. As shown in Table 4
applying TAResSFL can achieve good accuracy performance (>81%) as well as high enough MIA resistance (>0.020) for most cases. Fig. shows the visualization of MocoSFL-3 when 1.0% target data is available. We observe the reconstructed images are much more noisy and blurry such that the subject can be successfully hidden.
Larger cut-layer allows better resistance-accuraat tradeoff. Using smaller cut-layer of seems a better choice terms of accuracy and hardware requirement. However, as shown Table 4 the accuracy and resistance tradeoff seems much better by using cut-layer of 3. We believe that the extra client-side model complexity helps in the optimization of both accuracy and resistance objectives. So, applying TAResSFL with larger cut-layer more favorable option.
5.3 HARDWARE DEMONSTRATION
Finally, we compare the total hardware resource cost of the proposed MocoSFL and "Mo- coSFL+TAN TARReSFLL with synchronization frequency of 1/epoch for 200 epochs, and FL-SSL (Zhuang al. 2022) with 500 local epochs per client and synchronization frequency set per 5 local epochs (original setting). For MocoSFI, we use 1 1,000 clients with batch size of 1, and cut-layer of 3. For FL.-SSI. to achieve similar accuracy, we use 5 clients with batch size 128. And we assume the data follows the default 2-class non-IlD setting. To evaluate overhead, we use commonly-used libraries such as 'fvcore' for FLOPs and forch.cuda.memory allocated' for memory (please see Appendix[A.3) for detail.) For power measurement,
Raspberry Pi 4B equipped with 1GB memory one real client and simulate all other clients on the PC.
0.1
100
100000
794873
Computation (GFLOPg)
734
2187
Battery
Server
Communication [MS
8209
35
5058
TCP
Memory (MB)
921
Client
10.8
MacoSFL-TAARsSSK
Figure 7: Hardware resource comparison between FL-SSL and MocoSFL schemes.
Figure 8:
Raspberry setup. The Raspberry connects to the output via measurement tool, and communicates wirelessly with the server using websocket (TCP).
As shown in Fig. FL-SSL requires 7,949 TFLOPs for the entire training session as each client needs to perform 500 local epochs on 10K data and memory consumption of 921 MB. The communication overhead due synchronization has high cost of around 8,269 MB. Thus, FL-SSL is only suitable TARRSSSSF.AA hardware requirements are reasonable since
for cross-silo situation. For computation is only 734 GFLOPs and communication is around 39 MB in total. The memory consumption is tiny 0.8 MB mostly due to the weight parameters since TAResSFL does not require local training. As shown in Fig. X our measurement using USB multimeter shows the proposed MocoSFL running on the Raspberry Pi only draws power of 2.26W, in average, and consumes around 9,100 mAh on 5V battery.
CONCLUSION
We propose MocoSFL, a collaborative SSL framework based on SFL. The proposed framework addresses hardware resource requirement at client-side by enabling small batch size training and computation offloading. It also relieves the large data requirement of local contrastive learning by enabling effective feature sharing. The proposed scheme is the only one that can support a large number of clients. In combination with ResSFL-based module. it addresses privacy concerns of MocoSFL. Finally, achieves even better IID/non-IIl performance with much lower hardware requirement than the SoTA Fl-baased SSL methods.

---

Published as a conference paper at ICLR 2023
7 ETHICS STATEMENT
In this work, we address two practical issues of hardware (compute and memory) resources and large data requirement in collaborative SSL by using SFL-based scheme. Compared the conventional approaches, the resulting MocoSFL scheme has significant advantages in affordability and is much easier to deploy. Apart from being environmental -riendly. MocoSFL makes possible reach more clients, especially for those in poverty or under-representee minorities, and thus eliminate potential human-bias and unfairness issues of the ML service.
We address the privacy issues in the proposed MocoSFL. by proposing TAReSSFL. module to use in sensitive applications. Fig. shows that the subject of the raw image can successfully hidden.
More discussion on the privacy of "MocoSFL=TAResSH"" as instance encoding (Huang et al. 2020; Yala et al. 202T). which try to protect users' data by transforming the original dataset to a distorted version such that they cannot be recognized by humans while an arbitrary deep learning model can still learn useful information from it (i.e. achieve high accuracy classification task). MocoSFL, especially with the TAResSFL, where the client-side model frozen, have some similarities since the frozen client-side model can be seen as a transfor- mation, and the collection of latent vectors be regarded as the transformed dataset. However, (IoccOSFL+TAResSSS
We notice line of works known
scheme has two favorable properties that make it distinct from instance encoding methods. Domain dependency. First, SFL only finds a transformation method for a given task. Since the transformation method itself (the client-side model) heavily depends on the target domain information (as we need access to the target domain data train the client-side model), cannot work on data from another domain. As the output of the "client-side model transformation", latent vectors are only useful for current problem without any transferability guarantee. However, instance encoding methods intend to derive a general transformation method that can work across domains, targeting harder problem. Risk Control. Instance encoding methods publish the trans- formed dataset the wide public which is risky. As pointed out by Carlini al. (2021): all the raw data will be leaked if successful decryption method invented in the future even if it does not exist While in proposed scheme, the latent vectors from clients will only be accessible to the server party, thereby reducing the risk. Since we already provide TAResSFL. to mitigate the MIA attack that can be possibly launched from the honest-but-curious server, the risk is minimized. To eliminate possibility of future advanced attack, we can introduce a protocol that requests the server party regularly delete these latent vectors immediately after the training is done.
REPRODUCIBILITY STATEMENT
To make it easier for readers reproduce the results in this work, we open-source code https://githur com/ Sooo//SonyAI//ocoS// We also provide detailed explanation on the Mo- coSFL training and evaluation hyperparameters, collaborative learning system hyperparameters in Appendix A.I and TAResSFL module hyperparameters in Appendix A.4
9 ACKNOWLEDGMENT
This work is funded by Sony AI.
This common practice according to http::///og.cc.ggo//rrvvayyy//ce
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Arjun Nitin Bhagoji. Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pages 634-643. PMLR, 2019.
Adrian Bulat, Enrique Sánchez-Lozzno, and Georgios Tzimiropoulos. Improving memory banks for unsuper- vised learning with large mini-batch, consistency and hard negative mining. In ICASSP 2021-202/ IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1695-1699. IEEE, 2021.
Nicholas Carlini, Samuel Deng, Sanjam Garg, Somesh Jha, Saced Mahloujifar, Mohammad Mahmoody, Abhradcep Thakurta, and Florian Tramèr. Is private learning possible with instance encoding? In 202/ IEEE Symposium on Security and Privacy (SP), pages 410-427. IEEE, 2021.
Chaochao Chen, Jun Zhou, Longfei Zheng, Huiwen Wu, Lingjuan Lyu, Jia Wu, Bingzhe Wu, Ziqi Liu, Li Wang. and Xiaolin Zheng. Vertically federated graph neural network for srivacy-preservee node classification, In IJCAI, 2022.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003,00 042997, 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. 2009 IEFE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. arXiv preprint arXiv:2101.00 10170431,1,,, 2021.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence informa- tion and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 1322-1333, 2015. 2015.
Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33: 21271-21284, 2020.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner. Chloe Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 1eI 103604, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
Andrew Howard, Mark Sandler. Grace Chu, Liang- Chinh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
Yangsibo Huang, Zhao Song, Kai Li, and Sanjeev Arora. Instahide: Instance-hiding schemes for private distributed learning. In International conference on machine learning, pages 4507 4518. PMLR, 2020.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances Neural Information Processing Systems, 33:21798-21809, 2020.

---

Published as a conference paper at ICLR 2023
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby, Big transfer (bit): General visual representation learning. European conference on computer vision, pages 491-507. Springer, 2020.
Jingtao Li, Adnan Siraj Rakin, Xing Chen, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti. Ressfl: A resistance transfer framework defending model inversion attack in split federated learning, 2022.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. Advances Neural Information Processing Systems, 34: 14900-191191,, 2021.
Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tianjian Chen, Han Yu, and Qiang Yang. Fedvision: An online visual object detection platform powered by federated learning. In IAAI, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas, Communication efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: / 807.03748, 2018
Siyuan Qiao. Huiyu Wang. Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batcl training with batch-channel normalization and weight standardization. arXiv preprint arXiv:/903, 2019.
Joshua David Robinson, Ching-Yao Yao Chuang. Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2020.
Holger R Roth, Ken Chang, Praveer Singh, Nir Neumark, Wenqi Li, Vikash Gupta, Sharut Gupta, Liangqiong Qu, Alvin Ihsani, Bernardo C Bizzo, et al. Federated learning for breast density classification: real- world implementation. In Domain adaptation and representation transfer, and distributed and collaborative learning, pages 181-191 Springer, 2020.
Haizhou Shi, Youcai Zhang, Siliang Tang, Wenjie Zhu, Yaqian Li, Yandong Guo, and Yucting Zhuang. On the effi- cacy of small self-supervised contrastive models without distillation signals. arXiv preprint 117 2021.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, PMLR, 2019.
Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. arXiv preprint arXiv:2004. 200::200441288,, 2020.
Praneeth Vepakomma, Abhishek Singh, Otkrist Gupta, and Ramesh Raskar. Nopeek: Information leakage
reduction to share activations in distributed deep learning. In 2020 International Conference on Data Mining Workshops (ICDMW), pages 933-942. IEEE, 2020.
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xic. Communication-cffcionn federated learning via knowledge distillation. Nature communications, 1-1))--8, 2022a.
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu. Yongfeng Huang, and Xing Xie. Fedetr: Federated native ad ctr prediction with cross platform user bchavior data. ACM Transactions Intelligent Systems and Technology (TIST), 2022b.
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Tao Qi Yongfeng Huang, and Xing Xie. A federated graph neural network framework for preseevin personalization. Nature Communications, 13(1):1-10, 2022c.
Yawen Wu, Dewen Zeng. Zhepeng Wang, Yi Sheng, Lei Yang. Alaina J James, Yiyu Shi, and Jingtong Hu. Federated contrastive learning for dermatological discase diagnosis via de-device learning. In 202! IEEE/ACM International Conference Computer Aided Design (ICCAD), pages 1-7, IEEE, 2021.
Yawen Wu, Zhepeng Wang, Dewen Zeng, Meng Li, Yiyu Shi, and Jingtong Hu. Decentralized unsupervised learning of visual representations. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI, pages 2326-2333, 2022d.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-1 2018
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non parametric instance discrimination. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 3733-3742, 2018.
12

---

Published as a conference paper at ICLR 2023
Adam Yala, Homa Esfahanizadeh, Rafael GL D' Oliveira, Ken R Duffy, Manya Ghobadi, Tommi S Jaakkola, Vinod Vaikuntanathan, Regina Barzilay, and Muriel Medard. Neuracrypt: Hiding private health data random neural networks for public training. arXiv preprint arXiv:2106 024444 2021.
Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Federated unsupervised representation learning. arXiv preprint arXiv:2010.08982 2020.
Hang Zhao, Orazio Gallo, luri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks.
IEEE Transactions on computational imaging, 3(1):47-55 57, 2016.
Weiming Zhuang. Xin Gan, Yonggang Wen. Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation Icarning from decentralized data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4912-4921, 2021.
Weiming Zhuang. Yonggang Wen, and Shuai Zhang. Divergence- aware federated self-supervised learning. arXiv preprint 2203:22044043885, 2022.
13


Reviews:
[{"review_id": "m5e81Gu2KE", "reviewer": "Reviewer_zHbH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes MocoSFL, a collaborative SSL framework based on SFL. The proposed framework addresses hardware resource requirement at client-side by enabling small batch size training and computation offloading. It also relieves the large data requirement of local contrastive learning by enabling effective feature sharing. To address the privacy issue and communication overhead of MocoSFL, this paper further introduces the TAResSFL module. Combined with TAResSFL, MocoSFL can support 1,000 clients for the first time, achieve better non-IID performance, and largely reducte memory usage, computation and communication cost. ", "summary_of_the_review": "FL under self-supervised learning setting is still a very new but very practical problem. This paper clearly identifies the major challenges in deriving high accuracy in FL-SSL schemes for cross-client applications, and proposes a novel collaborative SSL framework called MocoSFL based on Split Federated Learning (SFL) and Momentum Contrast (MoCo). MocoSFL is the first work that can deal with as large as 1,000 clients in federated self-supervised learning, and tests its effectivenes on a real Raspberry Pi 4B device. Overall, this paper is very interesting and to my knowledge novel. This paper could be a milestone work in advancing federated self-supervised learning. Hence, I would like to vote for acceptance.", "strength_and_weaknesses": "Strengths\n\n1. The whole paper is very well structured and well written.\n2. The proposed MocoSFL can be scaled to a large number of clients (1,000 clients for the first time).\n3. MocoSFL seems to be a very practical framework, which requires less memory, less communication, and consumes less power.\n4. Authors also test the effectivenes of MocoSFL on a real Raspberry Pi 4B device, which validates its potential to land in industry.\n5. The experimental design is solid and the results are convincing. All the key factors in a system are well considered and addressed, including model performance, memory usage, computation cost, communication cost, and privacy. \n\nWeaknesses\n\n1. All the chosen datasets are CV centered, I want to know whether MocoSFL can be extended to the other types of data. More discussion is expected.\n2. Seems FL-BYOL is the only baseline. Are there any potential baseline? Could you elaborate more on this?\n3. Figure 5 can be better illustrated with x-axis and y-axis lines with arrows.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "jfCmrE5yiT", "reviewer": "Reviewer_Sd6w", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address the issues existing in current collaborative self-supervised learning (SSL) schemes and proposes MocoSFL, a collaborative SSL framework based on Split Federated Learning (SFL) and Momentum Contrast (MoCo). In MocoSFL, the large backbone model is split into a small client-side model and a large server-side model, and only the small client-side model is processed locally on the client’s local devices. \nSpecifically, there are three key components in MocoSFL: (i) vector concatenation which enables the use of small batch size and reduces computation and memory requirements by orders of magnitude; (ii) feature sharing that helps achieve high accuracy regardless of the quality and volume of local data; (iii) frequent synchronization that helps achieve better non-IID performance because of smaller local model divergence. Numerous evaluations are conducted to evaluate the effectiveness of the proposed MocoSFL.\n", "summary_of_the_review": "The proposed MocoSFL takes into consideration not only model performance, hardware requirements, and communication overhead, but also privacy concerns. As a result, it would garner interest from a diverse community, including but not limited to FL, SSL, communication, privacy, and so on. MocoSFL also provides the industry with practical benefits by supporting a large number of clients and testing on real devices. This paper's overall structure is very clear, and the writing is also excellent. The paper as a whole is of high quality and of sufficient interest to the community.", "strength_and_weaknesses": "Strength: \n\n1) The practical value offered by MocoSFL is interesting: I think MocoSFL is a very practical solution that can support a large number of clients (1,000 clients) to conduct collaborative self-supervised learning (SSL).\n2) The proposed MocoSFL has experienced enormous evaluations: The proposed MocoSFL takes into account both IID and non-IID settings, as well as the cross-client and cross-silo cases; experiments are carried out in both simulated and real hardware devices.\n3) The proposed scheme takes into account not only model performance, hardware requirements, and communication overhead, but also privacy concerns.\n4) Because most IoT devices have limited memory and most data is unlabeled in reality, the investigated problem is both timely and important to the community.\n5) The paper's writing is good, and the overall structure is clear.\n\n\nWeaknesses:\n\n1)  In addition to model inversion attacks, I am wondering whether MocoSFL with the TAResSFL module has the potential to defend against other privacy attacks, such as the Membership Inference Attack. \n2)  Although 1,000 clients are already a SOTA achievement, it would be great to also analyze how many clients can be supported by MocoSFL at most. Is there any turning point here? \n3) I am very interested in the possibility of landing MocoSFL in ubiquitous IoT devices, so I want to know what is the min hardware requirement to implement MocoSFL, It is better to provide more discussions on this part?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aSuBSSIWDlt", "reviewer": "Reviewer_bsEM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Collaborative self-supervised learning is considered based on a combination of known split federated learning and self-supervised contrastive leaning strategies. The former bases on model splitting between server and local clients while the latter depends on momentum-based key generation. The proposed combination, called MocoSFL, has some complexity/performance advantages over exiting FL-based solutions. A variation of this idea, based on pretraining (and later freezing) of feature extractors utilizing a small amount of clients' training data, is also proposed to improve communication overhead as well as resistance to MIA attacks.", "summary_of_the_review": "The addressed topic - collaborative SSL - is important and timely. But the novelty here is limited as the work represents a combination of known methods. The design choices made during this combination all seem right but are also either already known or easily anticipated. As for the target-aware variation, the availability of local clients' training data at the server needs a stronger justification (is this a realistic/fair assumption? is it actually done in practice somewhere?). There are a number of questions/issues raised in the Weaknesses section above.", "strength_and_weaknesses": "Strengths: The chosen topic of collaborative SSL is an important and timely topic. The paper is clearly written. Performance is shown to be better than SOTA in some cases.\n\nWeaknesses: \n\nThe novelty is limited as the work represents a combination of known methods. The design choices made during this combination all seem right but are also either already known or easily anticipated (like latent vector concatenation to create a large\nbatch size for the server-side model, sharing of features and more frequent synchronization). As for the target-aware variation, it is not clear whether the availability of local clients' training data (however small) is justified. Sure, as the authors mention, Bhagoji et al., 2019 considers the possibility of server being able to validate local models, but note that Bhagoji et al. mean to raise the bar for attackers (in an effort to prove that attacks are possible). In contrast, the present authors actually consider the possibility of using the local client data to guard against MIA attacks (and to reduce communication overhead following the feature extractor freezing). My impression is that this is a highly non-standard FL setting, and I would want to see more convincing references. \n\nSpecific questions/issues include:\n\nThe obvious baselines FedEMA and FedU are missing from Table 2 and Table 3.\nFrom Table 5, the results for Non-IID cases with Nc=5 or Nc =100 are better than IID cases which is counter-intuitive. Could the authors provide justification for this performance gain?\nCan the authors explain why the latent vector size in Table 1 changed from 5000 to 39.1 MB? Is it just because the backward gradients are not included?\nIn Figure 7, how the GFLOPs for computation are measured? Specifically, does it include the FLOPs needed for backpropagation?\nThe authors claim this technique to be competitive in cross-silo setting. However, the paper requires having batch size in range of 100-200 at the server-end which seems impractical considering there exists millions of samples/clients. Could the authors comment on this?\nEq (2) finds that having a large batch-size can maintain hardness of negative samples, which seems to contradicts the empirical results in Table 6 where increasing the batch size further to 400 degrades the performance. Reasons?\nDid the authors implemented/reproduced the main results of baselines works e.g. FedEMA? It seems like the results are exactly as in the original paper. \nHow does this method compared against a FedMoco (Nanqing Dong, Irina Voiculescu: “Federated Contrastive Learning for Decentralized Unlabeled Medical Images”, 2021; arXiv:2109.07504) which also utilized MoCo in federated learning setup? \nFig 3 and Fig 7 contains computation results per clients only. How much computation burden is added to the server side?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OyYlUWCtB4", "reviewer": "Reviewer_Ddhx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a federated self-supervised learning method named MocoSFL. based on Split Federated Learning (SFL) and MoCo. \n\nBy leveraging a small client-side model, vector concatenation, and effective feature sharing, MocoSFL solves two challenges: high computing resources and large data requirements. Thus, MocoSFL makes it practical for a cross-client application which typically has a very large number (>100x) of clients, and each client holds a small amount of data and has limited computing capability. \n\nIn addition, this paper introduces the TAResSFL module to effectively address the privacy threat and communication overhead of MocoSFL. MocoSFL with TAResSFL module can support 1,000 clients for the first time, and achieve over 1000x reduction in memory usage, 10000x reduction in computation, and 200x reduction in communication overhead. Moreover, compared to state-of-the-art methods, MocoSFL achieves better non-IID accuracy because of its small local model divergence.\n", "summary_of_the_review": "This seems like a particularly interesting paper. This paper identifies practical problems in federated self-supervised learning and provides a novel MocoSFL framework to effectively solve them. The paper presents an in-depth discussion of both the problems and the solutions and shows extensive experimental results to support the arguments.  Finally, with the remaining issues addressed, the hardware demo leverages the importance and practicality of this work.", "strength_and_weaknesses": "Strengths\n(1)\tThis paper firstly identifies two practical limitations of FL-SSL methods that make them unable to support cross-client applications.\n\n(2)\tThe roots of these two challenges are discussed well in Section 2.\n\n(3)\tThis paper proposes a novel combination of SFL and MoCo and utilizes three components to get the best of the two. The resulting MocoSFL enables a small client-side model, a large equivalent batch size, and effective feature sharing hence mitigates two challenges.\n\n(4)\tThe paper presents good reasoning on why the proposed MocoSFL is successful in mitigating large data requirements and improving non-IID performance.\n\n(5)\tExtensive experimental results demonstrate the advantages of the proposed method compared to state-of-the-art methods.\n\n(6)\tThe paper honestly points out the remaining privacy and communication issues of MocoSFL, and correspondingly presents TAResSFL module to mitigate these issues.\n\n(7)\tAt the end, the paper presents a hardware demo to showcase the practicality of the proposed method, showing that clients with a cheap device can participate in federated self-supervised learning without any problem.\n\nWeaknesses\n\n(1)\tExperiment shows up to 1,000 clients. It would be interesting to see the accuracy performance when the number of clients is greater than 1,000.\n\n(2)\tMocoSFL results are limited - only for ResNet architectures. How it performs on other models?\n\n(3)\tIn TAResSFL, the availability of a close-related source domain dataset and a small portion of target domain data is questionable and should be better justified.\n\n(4)\tHyper-parameter choices of TAResSFL seems not so easy. Could you give more hints on how to wisely or automaticly choose hyper-parameters?\n\n(5)\t(minor) The privacy evaluation only uses mean square error (MSE) as measurement, did you ever consider other metrics such as structural similarity index measure (SSIM), Peak signal-to-noise ratio (PSNR)? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%