SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
Authors: Chaoyang He
Zhengyu Yang
Erum Mushtaq
Sunwoo Lee
Mahdi Soltanolkotabi
Salman Avestimehr
Venue: ICLR 2022 Submitted (2022)
Keywords: federated learning

Abstract: Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data privacy, reduce data migration costs, and break regulatory restrictions. An essential, but rarely studied, challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL, compared to centralized training, due to the fact that FL users are often reluctant to label their private data and edge devices do not provide an ideal interface to assist with annotation. Addressing label deficiency is also further complicated in FL, due to the heterogeneous nature of the data at edge devices and the need for developing personalized models for each user. We propose a self-supervised and personalized federated learning framework, named SSFL, and a series of algorithms under this framework which work towards addressing these challenges. First, under the SSFL framework, we analyze the compatibility of various centralized self-supervised learning methods in FL setting and demonstrate that SimSiam networks performs the best with the standard FedAvg algorithm. Moreover, to address the data heterogeneity at the edge devices in this framework, we have innovated a series of algorithms that broaden existing supervised personalization algorithms into the setting of self-supervised learning including perFedAvg, Ditto, and local fine-tuning, among others. We further propose a novel personalized federated self-supervised learning algorithm, Per-SSFL, which balances personalization and consensus by carefully regulating the distance between the local and global representations of data. To provide a comprehensive comparative analysis of all proposed algorithms, we also develop a distributed training system and related evaluation protocol for SSFL. Using this training system, we conduct experiments on a synthetic non-I.I.D. dataset based on CIFAR-10, and an intrinsically non-I.I.D. dataset GLD-23K. Our findings show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable. The performance comparison indicates that representation regularization-based personalization method is able to outperform other variants. Ablation studies on SSFL are also conducted to understand the role of batch size, non-I.I.D.ness, and the evaluation protocol.

Under review as a conference paper at ICLR 2022
SSFL: TACKLING LABEL DEFICIENCY IN FEDERATED LEARNING VIA PERSONALIZED SELF-SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) is transforming the MI. training ecosystem from a cen- tralized over-the-cloud setting distributed training over edge devices in order strengthen data privacy, reduce data migration costs, and break regulatory restric- tions. An essential, but rarely studied, challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL. compared to centralized train- ing, due to the fact that FL users are often reluctant to label their private data and edge devices do not provide an ideal interface to assist with annotation. Addressing label deficiency is also further complicated in FL, due to the heterogeneous nature of the data at edge devices and the need for developing personalized models for each user. We propose a sulf-supervised and personalized federated learning frame- work, named (SSFL), and series of algorithms under this framework which work towards addressing these challenges. First, under the SSFL L framework, we analyze the compatibility of various centralized self-supervised learning methods in FL setting and demonstrate that SimS i am networks performs the best with the stan- dard FedAvq algorithm. Moreover, to address the data heterogeneity at the edge devices in this framework, we have innovated a series of algorithms that broaden existing supervised personalization algorithms into the setting of self-supervised learning including perFedAvg, Ditto, and local fine-tuning, among others. We further propose novel personalized federated self-supervised learning algorithm, Ver-SSFL, which balances personalization and consensus by carefully regulating the distance between the local and global representations of data. To provide a comprehensive comparative analysis of all proposed algorithms, also develop distributed training system and related evaluation protocol for SSFL. Using this training system, we conduct experiments on synthetic non-L.L.D. dataset based CIFAR-10, and an intrinsically non-I.I.D. dataset GL.D-23K. Our findings show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable. The performance comparison indicates that representation regularization-basee form other variants. Ablation studies on SSFL are also conducted to understand the role of batch size, non-I.I.D I...D.Dess., and the evaluation protocol.
personalization method able to outper-
INTRODUCTION
Federated Learning (FL) is contemporary distributed machine learning paradigm that aims strengthening data privacy, reducing data migration costs, and breaking regulatory restrictions (Kairouz et al., 2021; Wang al., 2021). has been widely applied to computer vision. natural language processing, and data mining. However, there are two main challenges impeding its wider adoption in machine learning. One is data heterogeneity, which natural property of FL. which diverse clients may generate datasets with different distributions due to behavior preferences (e.g., the most common cause of heterogeneity is skewed label distribution which might result from instances where some smartphone users take more landscape pictures, while others take more photos of daily life). The second challenge is label deficiency at the edge, which relatively less studied. This issue is more severe at the edge than in centralized setting because users reluctant annotate their private and sensitive data, and/or smartphones and IoT devices do not have a inse-friendly interface to assist with annotation.

---

Under review as a conference paper at ICLR 2022
Reguuattindin Ponsonlization Vil Consensud)
Server
alop-gred
tiobel
Personal Simlertty
Global
S milurity
prodictor
Mupugnad
prockctor h
stop grad
Model
Glooa Model
encoder
encodorr
encoder (Slebal
encoder1 (Slobal
Client
Client
Porsonaland
Globol
Parsorattord Model
Giotal
Image
Per SSFL Learning Framework
Federated Learning with Per-SSFL,
Figure 1: Depiction of the Self-supervised and Personalized Federated Learning (SSFL) framework.
To mitigate the data heterogeneity issue among clients, researchers have proposed algorithms for training a global model FedAvg (McMahan et al., 2017), FedProx (Li et al., 2018), FedNova (Wang al., 2020), FedoPT (Reddi et al., 2020), as well personalized FL frameworks (e.g., PF edeMM Ditto, Per Per-FedAvgy FedAvg). These algorithms all depend on the strong assumption that the data at the edge has sufficient labels. To address the label deficiency issue in FL, recent works (Liu et al., 2020; Long et al., 2020; Itahara et al., 2020; Jeong et al., 2020; Liang et al., 2021; 1; Zhao al., 2020; Zhang et al., 2020a;b) assume that the server or client has a fraction of labeled data and use semi-supervised methods such as consistency loss (Miyato et al., 2018) or pseudo labeling (Lee, 2013) to train global model. more realistic but challenging setting is fully unsupervised training. Although recent work in FL (Saced et al., 2020) attempts to address this challenge through Siamese networks proposed around thirty years ago (Bromley et al., 1993), its design does not tackle data heterogeneity for learning personalized models, and it only trains small-scale sensor data in IoT devices. Moreover, these existing works in FL. have not examined recent progress in the Self-Supervised Learning (SSL) community where methods such SimCLR (Chen et al., 2020), SwAV(Caron et al., 2021), BYOL (Grill et al., 2020), and SimSi am (Chen & He, 2020) have shown tremendous improvement in reducing the amount of labeled data required to achieve state-of-the-an performance. As such, it remains still unclear how these SSL methods can be incorporated into FL and how well they would perform, especially when intertwined with the data heterogeneity challenge that does not exist in centralized training.
In this paper, we propose Supervised Federated Learning (SSFL), unified supervisdd and personalized federated learning framework, and series of algorithms under this framework to address these challenges. As shown Figure this framework brings state-of-the-ar supervisdd learning algorithms to the realm of FL in order to enable training without using any supervision, while also integrating model personalization to deal with data heterogeneity (Section 3. More specifically, under the SSFL framework, we analyze the compatibility of various centralized self- supervised learning methods in the FL setting and demonstrate that S immiiam networks performs the best with the standard FedAvg algorithm (Section 3.2). Moreover, address the data heterogeneity edge devices, we have innovated a series of algorithms that broaden the reach of existing supervised personalization algorithms into the setting of self-supervised learning, including pe =Fvvv (Fallah et al., 2020), Ditto (Li et al., 2021), and local fine tuning, among others. We further propose a novel personalized federated self-supervised learning algorithm, per- SSFL (Section 3.3), which balances personalization and consensus by carefully regulating the distance between the local and global representations of data (shown as the yellow block in Figure/ 1).
To provide comprehensive and comparative analysis of the proposed algorithms, we also develop a distributed training system and evaluation protocol for SSFL Using this training system, we conduct experiments on synthetic non-I.I.D. dataset based on CIFAR-10 and natural non-L.L.D. dataset GLD-23K. Our experimental results demonstrate that all algorithms in our framework work reliably. In FL, the gap of evaluation accuracy between supervised learning and unsupervised learning is small. Personalized SSFL performs better than FedAvg- based SSFL. We also conduct ablation studies fully understand the SSFL framework, namely the role of batch size, different degrees of non-I.I.D D.ness and performance in more datasets. Finally, our unified API design can serve as a suitable platform and baseline, enabling further developments of more advanced SSFL algorithms.
2

---

Under review as a conference paper at ICLR 2022
2
PRELIMINARIES
SSFL builds upon two fundamental areas in machine learning: federated optimization and self- supervised learning. Thus, we first introduce some basics and formulations in these areas.
2.1 FEDERATED OPTIMIZATION
Federated optimization refers the distributed optimization paradigm that network of devices collaboratively solve a machine learning task. In general, it can be formulated as distributed optimization problem with the form (McMahan et al., 2017): ming Ek=1 TDI
Del [(0, D₁).. Here, each device k has a local dataset D2 drawn from local distribution Xg. The combined dataset D = L Dj is the union of all local datasets Dr. 0 represents the model weight of client model. the client's local loss function that measures the local empirical risk over the heterogeneous dataset D Under this formulation, to mitigate the non-L..D. issue, researchers have proposed algorithms such as FedP rox (Li et al., 2018), FedNova (Wang et al., 2020), and F FedoPT (Reddi et al., 2020) for training global model, as well as personalized FL frameworks such as Ditto (Li et al., 2021), and Per gevvgg (Fallah et al.). All these algorithms have a strong assumption that data at the edge have sufficient labels, meaning that their analysis and experimental settings based on a supervised loss function, such as the cross-entropy loss for image classification.
2.2 SELF-SUPERVISED LEARNING
Self-supervised learning (SSL) aims to learn meaningful representations of samples without human supervision. Formally, it aims learn an encoder function fo : X' + Rd where 9 is the parameter of the function, X' is the unlabeled sample space (e.g. image, text), and the output is a d dimensional image classification and seg-
vector containing enough information for downstream tasks such mentation. The key to SSL's recent success is the inductive bias that ensures a good representation encoder remains consistent under different perturbations of the input (i.e. consistency regularization).
One prominent example among recent advances in modern SSL frameworks is the Siamese architec- ture (Bromley et al., 1993) and its improved variants Si SiMCLR (Chen et al., 2020). SwAV (Caron et al., 2021), BYOL (Grill et al., 2020), and $ $imSii am (Chen & He, 2020). Here we review the most elegant architecture, Bimsi iam, and defer the description and comparison of the other three to Appendix A. SimSiiiiaai am proposes two-head architecture in which two different views (augmentations) of the same image are encoded by the same network fe- Subsequently, predictor Multi Layer Perceptron (MLP) he and stop-gradiens operation denoted by 2 are applied to both heads. In the SSL context, "stop gradient" means that the optimizer stops at a specific neural network layer during the back propagation and views the parameters in preceding layers constants. Here, 0 is the concatenation of the parameters the encoder network and the predictor MLP. The algorithm aims to minimize the negative cosine similarity -(.--) between two heads. More concretely, the loss is defined as
1 DI
Lss (0, D)
Σ D(fe(T(a)), ZED
(I)
where T represents stochastic data augmentation and D is the data set.
SSFL: SELF-SUPERVISED FEDERATED LEARNING
In this section, we propose SSFL, a unified framework for self-supervised federated learning. Specif- ically, we introduce the method by which SSFL works for collaborative training of global model and personalized models, respectively.
3.1 GENERAL FORMULATION
We formulate self-supervised federated learning as the following distributed optimization problem:
min e
G G (01, C(Kx θ Xx))
(2)
(0A (bakk)
3

---

Under review as a conference paper at ICLR 2022
where 0g is the parameter for the local model (feg. h θ is the parameter for the global model (fe. he); [(f). Xx)) is a loss measuring the quality of representations encoded by for and fe on the local distribution Xg; and G(.) denotes the aggregation function (e.g. sum of client losses weighted by ). To capture the two key challenges in federated learning (data heterogeneity and label deficiency), we hold two core assumptions the proposed framework: (1) X of all clients are heterogeneous (non-I.LD.), and (2) there no label.
To tackle the above problem, we propose a unified training framework for federated self-supervised learning, as described in Algorithm 1. This framework can handle both non-personalized and personalized federated training. In particular, if one enforces the constraint 0% θ for all clients [ the problem reduces to learning a global model. When this constraint is not enforced, 9 can be different for each client, allowing for model personalization.
is the local optimizer the client side which solves the local sub-problem in a self-supervised manner. Se Serve r TOPT takes the update from the client side and generates a new global model for all clients.
Algorithm 1: SSFL: A Unilied Framework for Self-supervised Federated Learning
input T, ( -10 " (кe(K] CLIENTSSLOPT SERVEROPT
for = =0, O...., T do
Server randomly selects subset of devices s(O
Server sends the current global model () S(()
for device & E 5(0) paralle! do
Solve local sub-problem equation
O2. e + CLIENTSSLOPT (0(", (C0,,,
Send (t))
@
o back
((++1) SERVEROPT (e(*) (1 hAStt)
return: :fe (Be)eejK) 9
Next, we will introduce specific forms of C1 ilintt.S5..PT and ServerOPT personalized training.
for global training and
3.2 -SOBBA--SSFL: COLLABORATION TOWARDS GLOBBA MODEL. WITHOUT SUPERVISION
To train global model using SSFL, we design a specific form of ClientsSLOPT using SimSiam amm. We choose SimS i am over other contemporary self-supervised learning frameworks (c.g.. SimSi iam, SWAV, BYOL) based on the following analysis as well as experimental results (see Section 5.1).
The simplicity in neural architecture and training method. S SimSiam's 's architecture and training method are relatively simple. For instance, compared with S SiMCLR, SimSiam am has simpler loss function; compared with SwAV, SimSia inssian does not require an additional neural component (prototype vectors) and Sinkhorn- algorithm; compared with BYOL, SimSiam am does not need to maintain an additional moving averaging network for an online network. Moreover, the required batch size of S SimSia am is the smallest, making it relatively friendly for resource-constrained federated learning. A more comprehensive comparison can be found in Appendix A.
Interpretability of SimSiam leads to simpler local optimization. More importantly, SimS inniiaa am is more interpretable from an optimization standpoint which simplifieds the local optimization. In particular, can be viewed as an implementation of Soppectation-Maximzzzinn (EM) like algorithm, meaning that optimizing Lss in Equation is implicitly optimizing the following objective
[IIo(T(())) min E T BNX O.77
Here, fo is the encoder neural network parameterized by O. 17 is an extra set of parameters, whose size is proportional to the number of images, and T/e refers to using the image index of x to access sub-vector of 17. This formulation is w.r.t. both 0 and 7) and can be optimized via an alternating algorithm. At time step t, the n' update takes the form 7/2 + E [ (T(x))), indicating that n is assigned the average representation of over the distribution of augmentation. However, it is impossible to compute this step by going over the entire dataset during training. Thus, SimSiam uses one-step optimization approximate the EM-like two-step iteration by introducing the predictor he
4

---

Under review as a conference paper at ICLR 2022
to approximate and learn the expectation (i.e. he (2) (F])((((2))) for any image z. After this approximation, the expectation Er[:] is ignored because the sampling of T is implicitly distributed across multiple epochs. Finally,
obtain the self-supervised loss function in Equation 1, in which negative cosine similarity D is used in practice (the equivalent L2 distance is used in Equation 3 for the sake of analysis). Applying equation as c ClientSSIOPT simplifies the local optimization for each client in self-supervised manner.
3.3 PER-SSFL: LEARNING PERSONALIZED MODELS WITHOUT SUPERVISION
In this section, we explain how SSFL addresses the data heterogeneity challenge by learning person- alized models. Inspired by the interpretation in Section 3.2, we define the following sub-problen for each client & E [
A I1 - P6.:13
min
E
[lfe (T(x)) Th. 112 +
XN
2
(4)
D&
s.t. Or, H
arg min 0,H
E [Ilfe(T(x)) -.. 2~X,
Compared global training, we additionally include O, the global model parameter, and the global version of 17, and the expected representations which correspond to the personalized parameters O₂ and Tk In particular, through the term |/1 - Hill l| g. we aim for the expected local representation of any image x to reside within neighborhood around the expected global representation æ. Therefore, by controlling the radius of the neighborhood, hyperparameter À helps to balance consensus and personalization.
Algorithm Per-SSFL.
input :K, A, @ [0,0) ]ke[K]. number of local iteration, (3: learning rate for 0.....
Server randomly selects subset devices 5(0
Server sends the current global model (*)) g(t)
for device g(() paraliel
Sample mini-batch B& from local dataset DA. and do local iterations
/A optimi2e ze the global parameter @ locally
A
Z1, Za (T (oro
P. hato (Z.) (Z2)
@ ( -8Ver (N1.22)) where sstands for stop-gradient
/x optimize the local parameter 0g
for (T(.)) fu, (T(B))
p1,pa + he, (a1), he,
9t Vo. Pipl.5)
Piet. P Pe (420E..
CLIENTSSLOPT
Send (1 = 0 e back to server
1
(+++)) ( Evego
IP
D
A
return (8.)se (
We see that Equation 4 in the above objective is optimization problem w.r.t. both and 17. However, as the above target is intractable in practice, following an analysis similar to Section 3.2, we use the target below surrogate:
A
min 0x
Lss (0., D+)
ΣD
D (ho, (fez (T(x)) her (fe- (T(x))))
(5)
D&
zED
s.t. O* arg min Lss D)
(6)
In practice, can be optimized independently of 6² through the FedAvg (McMahan et al., 2017) algorithm. To make the computation more efficient, also apply the symmetrization trick proposed in (Chen & He, 2020). We refer to this algorithm as Per-SSFL and provide a detailed description in Algorithm 2 (also illustrated in Fig. 1).
Regarding the theoretical analysis. To our knowledge, all sulf-vuper learning frameworks do not have any theoretical analysis yet, particularly the SimSiam dual neural network architecture.
5

---

Under review as a conference paper at ICLR 2022
Our formulation and optimization framework are interpretable, they are built based an EM-like like algorithm for SimSiam and minimizing the distance between the private model and the global model's data representation.
Innovating baselines to verify SSFL. Note that we have not found any related works that explore a Siamese-like SSL architecture in an FL setting. As such, to investigate the performance of further propose several other algorithms that can leverage the SSFL framework. 1. LSFS.FL. We apply FedAvg (McMahan et al., 2017) on the S SimSiam am. loss Lss for each client to obtain a global model. We perform one step of SGD the clients' local data for local adaption: 2. MAML-SSFL. This algorithm is inspired by perFedAvg (Fallah et al.) and views the personalization on each devices as the inner loop of MAML (Finn al.). It aims learn an encoder that can be easily adapted to the clients' local distribution. During inference, we perform one step of SGD on the global model for personalization; BiLevel-SSFL. Inspired by Ditto (Li et al., 2021), we learn personalized encoders on each client by restricting the parameters of all personalized encoders be close global encoder independently learned by weighted aggregation. More details of these algorithms, formulation, and pseudo code are introduced in Appendix B. In Section 5.3, we will show the comparison results for these proposed SSFL algorithmic variants.
our proposed algorithm,
4 TRAINING SYSTEM AND EVALUATION PIPELINE FOR SSFL
A Distributed Training System to accelerate the algorithmic exploration in SSFL framework. We also contributed to reproducible research via our distributed training system. This is necessary for two reasons: (1) Running a stand-alone simulation (training client by client sequentially) like most existing FL works requires a prohibitively long training time when training a large number of clients. In SSFL, the model size (e.g.. ResNet-18 v.s. shallow CNNs used in the original FedAvg paper) and the round number for convergence (e.g.. 800 epochs in the centralized SimSiam framework) is relatively larger than in FL literature. By running all clients in parallel on multiple CPUs/GPUs, we can largely accelerate the process. (2) Given that SSFL. is unified and generic learning framework, researchers may develop more advanced ways to improve our work. As such, we believe it is necessary to design unified APIs and system frameworks in line with the algorithmic aspect of SSFL. See Appendix for more details on our distributed training system.
Evaluation Pipeline. In the training phase, we use a KNN classifier (Wu et al., 2018) as an online indicator to monitor the quality of the representations generated by the SimS1 i am encoder. For Global- L we report the KNN test accuracy using the global model and the global test data, while in SSFL, we evaluate all clients' local encoders separately with their local test data and report their averaged accuracy. After self-supervised training, to evaluate the performance of the trained encoder, we freeze the encoder and attach linear classifier to the output of the encoder. For Global-SSFL, we can easily verify the performance of SimSiam am encoder by training the attached linear classifier with FedAvg. However, for Per-SSFI, each client learns a personalized sims encoder. As the representations encoded by personalized encoders might reside in different spaces, using a single linear classifier trained by FedAvg to evaluate these representations unreasonable (see experiments in Section 5.4.3). As such, we suggest an additional evaluation step to provide a more representative evaluation of Per-SSFL's performance: for each personalization encoder, we use the entire training data to train the linear classifier but evaluate on each client's 's local test data.
EXPERIMENTS
In this section, we introduce experimental results for SSFL with and without personalization and present performance analysis on a wide range of aspects including the role of batch size, different degrees non-IIDness, and understanding the evaluation protocol.
Implementation. We develop the SSFL training system to simplify and unify the algorithmic exploration. Details of the training system design can be found in Appendix C. We deploy the system in distributed computing enviromment which has 8 NVIDIA A100-SXM4 GPUs with sufficient memory (40 GB/GPU) to explore different batch sizes (Section 5.4.1). Our training framework can multiple parallel training workers in single GPU, so it supports federated training with large number of clients. The client number selected per round used in all experiments is 10, which is a reasonable setting suggested by recent literature (Reddi et al., 2020).

---

Under review as a conference paper at ICLR 2022
Learning Task. Following SimCLR (Chen et al., 2020), SimSi am (Chen & He, 2020), BYOL (Grill et al., 2020), and SwAV (Caron et al., 2020) in the centralized setting, we evaluate SSL for the image classification task and use representative datasets for federated learning.
Dataset. We run experiments on synthetic non-I.... dataset CIFAR-10 10 and intrinsically non-I.I.D.
dataset Google Landmark-233 (GLD-23K), which are suggested by multiple canonical works in the FL community (Reddi et al., 2020; He et al., 2020; Kairouz et al., 2019). For the non-L.LD, setting, we distribute the dataset using a Dirichlet distribution (Hsu et al., 2019), which samples P. Dir(a) (we assume a uniform prior distribution) and allocates a Po.k proportion of the training samples of class e to local client k. We provide a visualization of the data distribution in Appendix E.I.
Model Architecture. For the model architecture, ResNet- 18 is used as the backbone of the SimSiar framework, and the predictor is the same as that in the original paper.
Next, we focus on results from the curated CIFAR-10 dataset and defer GLD-23K to Appendix D.
5.1 COMPARISONS ON SIMSIAM, SIMCLR, SWAV, AND BYLL
Our first experiment determines which SSL method is ideal for FL settings. We experiments using FedAvg for these four methods and obtain two findings: (1) SimSi am outperforms SimCIR in terms of accuracy; (2) BYOL and SwAV do not work in FL; we tested wide range of hyper-parameters, but they still are unable to converge to reasonable accuracy. These experimental results confirm our analysis in Section 3.2.
5.2
EVALUATION ON GLOBAL-
0.8 Accuracy
Loss
0.6
SSRL on LLL (0.5)
SSFL on ILD (0.5)
on non-I (0.5)
on non-LID (0.5)
0.4
00.6
0.2
0.8
round
round
0
200
400
600
200
400
600
(a) Training Time Accuracy
(b) Training Loss
Figure 2: Training and Evaluation using SSFL
The goal of this experiment is to understand the accuracy gap between supervised and supevsuee federated learning in both I.I.D. and non-I.I.D, settings where we aim to train global model from private data from clients.
Setup and Hyper-parameters. We evaluate Global-SSFL. using non-L.I.D. data from CIFAR-10; we set a = 0.1 for the Dirichlet distribution. For supervised learning, the test accuracy is evaluated on global test dataset. For self-supervised training, we follow the evaluation protocol introduced in Section 4. We use SGD with Momentum as the client-side optimizer and a learning rate scheduler across communication rounds. We searched the learning rate on the grid of (0.1, 0.3. 0.01, 0.03) and report the best result. The experiment is run three times using the same learning rate with fixed random seeds to ensure reproducibility. The training lasts for 800 rounds, which is sufficient to achieve convergence for all methods. More hyperparameters are in Appendix E.2.
We display the training curves in Figure 10 which demonstrates that SSFL converge reliably in both I.I.D. and non-I.I.D. settings. For the I.I.. data, we find that SSFL can achieve the same accuracy as the centralized accuracy report in the SimSi iammiammmm am paper (Chen & He, 2020). For the non-L.L.D. data, SSFL achieves a reasonable accuracy compared to the centralized accuracy. The accuracy comparisons in different dimensions (supervised v.s. self-supervised; L.L.D v.s. non-I.LD.) are summarized in Table 1.
Accuracy
Method
KNN Indicator
Evaluation
Supervised
Sell-Supervised
Acc. Gap
LA-SSF2
0.9217
0.8013
L.ID B.n.L.I.D
0.932
0.914
0.018
MAML-SSFL
0.9355
0.8235
0.8812
0.847
0.0342
B1 Teeeel 188PP.
0.9304
0.8137
Acc. Gup 0.0508
0.06
N/A
Per-SSFL
0.9388
0.8310
Table 1: Evaluation accuracy compari- son between supervised FL and SSFL.
Table 2: Evaluation Accuracy for Various Per-SSFL Methods.

---

Under review as a conference paper at ICLR 2022
5.3 EVALUATION ON PER-SSFL
Based on the results of SSFL with FedAvg, we further add the personalization components for SSFL introduced in Section 3.3 (Per-SSFL).
Setup and Hyper-parameters. For fair comparison, we evaluate Per-SSFI. on non-I.I.D, data from CIFAR-IO and a 0.1 for the Dirichlet distribution. For Per-SSFL training, we follow the evaluation protocol introduced in Section 4. Similar to SSFL, we use SGD with Momen- tum as the client-side optimizer and the learning rate scheduler across communication rounds. We search for the learning rate on a grid of f0.1, 0.3, 0.01, 0.03) and report the best result. For Per-SSFL and Bi BiLevll- LeeeellssFF we also tune the À of the regularization term with a search space {1, 10, 0.1, 0.01, 0.001 J. The experiments are run three times with the same learning rate and with fixed random seeds to ensure reproducibility. The training also lasts for 800 communication rounds, which is the same Global SSFL. Other hyperparameters can be found in Appendix E.2.
Loss
Accuracy
0.2
00.4
MAML 6SSR
Blevel-SFL
40.6
LA-S8FL
MAML 807L
0.8
round
LA-SSTL
round
0
200
400
600
800
200 400
600
800
(a) Training Loss
(b) Averaged Personalized Accuracy
Figure 3: Training and Evaluation using SSFL
We illustrate our results in Figure 10 and Table 2. To confirm the convergence, we draw loss curves for all methods in Figure 10(b) (note that they have different scaled values due to the difference of their loss functions). Figure 10(b) indicates that Per SSFL performs best among all methods. MAML-SSFF is also suggested method since it obtains comparable accuracy. LA- SSFL is practi- cal method, but it does not perform well in the self-supervised setting. In Figure 10(b), the averaged personalized accuracy of LSFF diverges in the latter phase. Based BiLevel-SSFU result, we can conclude such a method is not a strong candidate for personalization, though it shares similar objective functions
This indicates that regularization through representations encoded by S sims1 i am outperforms regularization through weights.
5.4
PERFORMANCE ANALYSIS
5.4.1 ROLE OF BATCH SIZE
FL typically requires a small batch size to enable practical train- ing on coourrc-constraiieed edge devices. Therefore, under- standing the role of batch size in SSFL is essential to practical deployment and applicability. To investigate this, we use dif- ferent batch sizes and tune the learning rate to find the optimal accuracy for each one. The results Figure show that SSFL. requires a large batch size (256); otherwise, it reduces the accu- racy or diverges during training. Since system efficiency is not gradient accumulation, which
0.8- Accuracy
0.6
Batch Size: 256
Batch Size:
0.4
Size:
Betch Size: 16
Batch Size:
the focus of this paper, a simple yet effective method. We fix the batch size at 32 and use accumulation step 8 for all experiments. For an even larger batch size (e.g., 512), the memory cost is significant, though there is notable gain in accuracy. Therefore, discontinue the search for batch sizes larger than larger than 256. A more advanced method includes batch-size-one training and knowledge distillation. We defer the discussion to Appendix F.
02
round
Figure 4: Results for batch sizes
5.4.2 ON DIFFERENT DEGREES OF NON-I.I.. NESS
We investigate the impact of the degree of data heterogeneity on the SSFL performance. We compare the performance between 0.1 and 0.5. These two settings provide a non-negligible gap
8

---

Under review as a conference paper at ICLR 2022
in the label distribution in each client (see visualization in Appendix E.1). Figure 5(a) and 5(b) shows the learning curve comparisons. It is clearly observed that the higher degree data heterogeneity makes it converge more slowly, adversely affecting the accuracy.
0.94 Accuracy
0.8
Accuracy
Loss
0.02
0.7
0.92
0.04
0.06
0.6
0.9
alpha (0.5)
alpha (0.5)
0.5
LA-SSFL
alpha (0.1)
-0.1
alpha (0.1)
Per-SSFL
0.88
round
0.12
round
0.4
round
200
400
600
800
200
400
600
800
60
80
100
(a) Averaged Personalized Accuracy
(b) Training Loss
Figure 6: Understanding the
Figure 5: Evaluation on Different Degress of Non-I.L.D.ness
Evaluation Protocol
5.4.3 UNDERSTANDING THE LINEAR EVALUATION OF PERSONALIZED ENCODERS
As discussed in 4, in SSFL, federated linear evaluation; however, in SSF-, each client learns a personalized SimSi AimSia am encoder. Such heterogeneity in diverse encoders makes fair evaluation difficult. To demonstrate this, we run experiments with naive federated linear evaluation personalized encoders and surprisingly find that such an evaluation protocol downgrades the performance. As shown in Figure 6, the federated linear evaluation for Per-SSFL, performs worse than even LA-SSFI. This may be attributed to the fact that the naive aggregation drags close to the parameter space of heterogeneous encoders, making the encoder degenerate in terms of personalization.
easily verify the quality of the S SimSi encoder using
RELATED WORKS
Federated Learning (FL) with Personalization. PFedMe (Dinh et al.), perFedAvg (Fallah al., 2020), and Ditto (Li et al., 2021) are some representative works in this direction. However, these methods all have strong assumption that can provide reliable annotations for their private and sensitive data, which we argue to be very unrealistic and impractical.
Label deficiency in FL. There are few related works tackle label deficiency in FL (Liu al., 2020; Long et al., 2020; Itahara et al., 2020; Jeong et al., 2020; Liang et al., 2021; Zhao et al., 2020; Zhang et al., 2020b). Compared to these works, our proposed SSFL does not use any labels during training. FedMatch (Jeong et al., 2020) and FedCA (Zhang et al., 2020a) requires additional communication costs synchronize helper models or public labeled dataset. (Saeed et al., 2020) addresses the fully unsupervised challenge on small-scale sensor data in IoT devices. However, compared to our work, it uses the Siamese networks proposed around thirty years ago (Bromley et al., 1993), lacking consideration the advance in the past two years (i.e., SimCLR (Chen et al., 2020), SwAV(Caron et al., 2021), BYOL (Grill al., 2020), and SimSi imm (Chen & Hc, 2020)). Moreover, these works does not have any design for learning personalized models.
CONCLUSION
We propose Self-supervised Federated I Learning (SSFL. framework and series of algorithms under this framework towards addressing two challenges: data heterogeneity and label deficiency. SSFL can work for both global model training and personalized model training. We conduct experiments on synthetic non-I.I.D. dataset based CIFAR- 10 and the intrinsically non-I.LD. GLD-23K dataset. Our experimental results demonstrate that SSFL can work reliably and achieves reasonable evaluation accuracy that is suitable for use in various applications.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatch: Semi-supervised learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations, a.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A. Raffel. MixMatch: holistic approach to semi-supervised learning. In Neural Information Processing Systems, pp. 5049-5059, b.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication Citmuunicatioo-Effiiennn Efficient Learning of Deep Networks from Decentralized Data. arXiv e-prints, art. arXiv:1602.05629, February 2016.
Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04):669-688, 1993.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. 2020.
Marco Cuturi. Sinkhom distances: Lightspeed computation of optimal transportation distances, 2013.
Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized Federated Learning: A Meta-Learning Approach.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: meta- learning approach. arXiv preprint arXiv:2002.07945 2020.
Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. URL
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
Chaoyang He, Songze Li, Jinhyun So, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr. Fedml: A research library and benchmark for federated machine learning. arXiv preprint wwXiv:2007.13518 2020.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: arXivv/704.000 /704.0484/ 2017.
MtugMing Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint /809:0/0900033 2019.
10

---

Under review as a conference paper at ICLR 2022
Zijian Hu, Zhengyu Yang, Xuefeng Hu, and Ram Nevatia. SimPLE: Similar pseudo label exploitation for supi-super classification. URL http: /ttp:::///xiv..../// arwiv.
Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation- based superviie federated learning for communication non-iid private data, arXiv preprint ar arXiv:2008.061 00:0:0000800180, 2020.
collaborative training with
Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency. arXiv preprint arXiv:2006. 12097, 2020.
Kairouz, B. McMahan, B. Avent, Aurélien Bellet, M. Bennis, Bhagoji, Keith Bonawitz, Zachary B. Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira. S. Rouay- heb, David Evans, Josh Gardner, Zachary Garrett, A. Gascón, Badih Ghazi, Phillip B. Gibbons, M. Gruteser, Z. Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, T. Javidi, Gauri Joshi, M. Khodak, Jakub Konecný, Aleksandra Korolova, F. Koushan- far, O. Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, Mariana Raykova, Hang Qi. D. Ramage, R. Raskar, D. Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, A. T. Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang. F. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14:1-210, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv: 2019.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning.
Dong-Hyun Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. 2013.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06122 2018.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith, Ditto: Fair and robust federated learning through personalization. 2021. URL httL:://///a/.i..t./
abs 221/2222.... 04221.
Xinle Liang, Yang Liu, Jiahuan Luo, Yuanqin He, Tianjian Chen, and Qiang Yang. Self-supervised
cross-silo federated neural architecture search. arXiv preprint 1xXiv:2101.11896 2021.
Yi Liu, Xingliang Yuan, Ruihui Zhao, Yifeng Zheng, and Yefeng Zheng. Re-ssfl: Towards ro- bust and commumication-efficcee semi-supervised federated learning system. arXiv preprint arXiv:2012.04432 2020.
Zewei Long, Liwei Che, Yaqing Wang, Muchao Ye, Junyu Luo, Jinze Wu, Houping Xiao, and Fenglong Ma. Fedsemi: An adaptive federated semi-supervised learning framework. arXiv preprint arXiv:2012.03292,, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas Communication-efiee efffccientt learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282. PMLR, 2017.
Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and supervied learning. 41(8):1979-1993 doi: 10.1 10910T9MMM..000 2.8588....
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning, IEEE transactions pattern analysis and machine intelligence, 4
2018.
Sashank Reddi, Zachary Charles, Manzil Zaheer. Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00 00095. 2020.
11

---

Under review as a conference paper at ICLR 2022
Aaqib Saeed, Flora D Salim, Tanir Ozcelebi, and Johan Lukkien. Federated self-supervised learning of multisensor representations for embedded intelligence. IEEE Internet of Things Journal, 8(2): 1030-1040, 2020.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Neural Information Processing Systems, pp. 1171-1179.
Tim Salimans and Diederik Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. CoRR, abs/1602.07868,, 2016. URL http:// arxiv. orgg abs 602 07868.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14. PMLR, 2019.
Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.0744 2020.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. McMahan, B. A. Areas, Maruan Al- Shedivat, Galen Andrew, S. Avestimehr, Katharine Daly, Deepesh Data, S. Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, A. Ingerman, Martin Jaggi, T. Javidi, P. Kairouz, Satyen Kale, Sai Praneeth Reddy Karimireddy, Jakub Konecný, Sanmi Koyejo, Tian Li, Luyang Liu, M. Mohri, Hang Qi, Sashank Reddi, Peter Richtárik, K. Singhal, Virginia Smith, M. Soltanolkotabi, Weikang Song, A. T. Suresh, Sebastian U. Stich, Ameet S. Talwalkar, Hongyi Wang. Blake E. Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, M. Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A field guide to federated optimization. ArXiv, abs/2107.06917 2021.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination, 2018.
Fengda Zhang, Kun Kuang. Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yuet- ing Zhuang, and Xiaolin Li. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08982. 2020a.
Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E Gonzalez, and Michael W Mahoney. Improving semi-supervised federated learning by reducing the gradient diversity of models. arXiv preprint arXiv:2008. 2020b.
Yuchen Zhao, Hanyang Liu, Honglin Li, Payam Barnaghi, and Hamed Haddadi. Semi-supervised federated learning for activity recognition. arXiv preprint arXiv:201 2.00000...00,00000000
12


Reviews:
[{"review_id": "Zj87BXq__QR", "reviewer": "Reviewer_7Cc7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The label deficiency problem is a realistic gap before we put federated learning into practice. The research question itself is a good and convincing topic to explore\n+ They cite appropriate related research work including SSL and Personalized FL.\n+ The problem formulation and the optimization function define the research problem clearly. They provide experiment results to support the claims. The analysis and discussion after the experiment are sufficient and convincing.\n\nWeakness:\n- I am a little bit concerned about the technical novelty in this paper. To combine SSL and FL is fine but lacks the motivation and we may not encourage just merging existing techniques together in a framework.\n- Also, by adding regulations, adding constraints, or other hyper-parameters to control the difference between the global models and locals, such approaches have been proposed already. The hidden ideas or we say the core idea is quite similar.\n- More discussion is expected about how to choose hyper-parameters.  ", "summary_of_the_paper": "The authors merge self-supervised learning in personalized federated learning to solve the limited label and data heterogeneity problems in the local clients. They test several current algorithms under their framework. Then they propose an algorithm named Per- SSFL considering the balance between the consensus and personalization. Finally, they provide experiment results to support their claims and comprehensive analysis of what they find. The main contribution is to design a self-supervised FL framework with supportive experiment results. They also provide suggestions to choose appropriate algorithms and hyper-parameters under different settings.", "summary_of_the_review": "In FL, label deficiency and data heterogeneity are two of the main open research questions. Authors propose a natural idea to leverage SSL in FL with the support of testing, analysis, and experiments. However, I am not sure the novelty of this work is good for this conference. I might expect more novel technique ideas in this paper-increasing the FL field.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y2VoUpLkhV1", "reviewer": "Reviewer_TsoV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n+ The self-supervised learning topic for FL is emerging and has great practical value. \n+ Both non-personalized and personalized FL settings are investigated in the SSL regime. \n+ Experiments and ablation studies are conducted under different settings. \n\nWeakness:\n\n- For the Per-SSFL framework, the local (personalized) model and global model are used. The memory consumption aspect should be discussed. For resource-constrained edge clients, high memory cost could be an issue. \n- Although it mentioned in the implementation setting that the client number selected per round is 10, it is not clear how many total clients are used in the FL setting. \n- In Table 1, what is the FL method under the supervised setting?\n- In Figure 2, what does (0.5) for SSFL on IID and SSFL on non-IID mean? \n- As can be seen in Figure 2, the convergence rates for the IID and non-IID cases are quite similar. Can you provide an explanation for that?\n- After reading Appendix D and Figure 10 in Appendix, the experimental setting on GLD-23K is still not quite clear. For example, how the local training set for each client is generated? What about the label distribution? Also, it seems that the number of clients used on the GLD-23K is different from that on CIFAR-10. \n- For ease of comparison and implementation, it would be good to evaluate the method on more commonly used datasets such as CIFAR-100 and Tiny-ImageNet and other datasets besides vision datasets (e.g., text) for FL. \n- It would be interesting to see the SSFL results under different numbers of selected participant clients.  \n- Since $\\lambda$ is an important parameter that balances consensus and personalization, its effect should be studied. \n\nMinor issues:\n\n1. Section 2.1, “…  the local empirical risk over the heterogeneous dataset $D^k$.” -> $D_k$\n2. Figure 10 appeared in Sec. 5.2 and 5.3, it should be Fig. 2 and Fig. 3.\n3. In Figure 2(a), the colors for the curve (SSFL on non-IID) and its legend are different (pink vs. red). It should be made consistent. \n4. Sec. 3.2, “… contemporary self-supervised learning frameworks (e.g., SimSiam,\nSwAV, BYOL)” -> should be “… (e.g., SimCLR, SwAV, BYOL)”\n\nA careful proofread of the paper is highly recommended. \n", "summary_of_the_paper": "This paper introduces the self-supervised learning (SSL) framework for FL. Different SSL methods are investigated to study their feasibility under the FL setting. With the popular SimSiam framework, personalized federated SSL is proposed. The performance comparison indicates\nthat representation regularization-based personalization method is able to outperform other variants.", "summary_of_the_review": "I appreciate the emerging topic of SSL in FL. The proposed personalized federated SSL is interesting. However, there are quite a few unclear details regarding the method and experimental settings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Km-JKYd-l5-", "reviewer": "Reviewer_zbEP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n- Deals with an important and practical scenario of FL\n- The analysis of personalised representations is interesting\n\nWeakness\n- Insufficient quantitative assessment of SimSiam approach against other self-supervised approaches. \n- It is unclear the feature representation can be used in practice, especially in \nthe personalised setting. Perhaps explore the use as an auxiliary objective in some downstream applications?\n- The effect of hyperparameters is under-explored. For instance, the size of representation which could overpower the benefits of self-supervision in the assessment with KNN.\n", "summary_of_the_paper": "This work is concerned with a very practical scenario of Federated Learning where the participating agents may not have access to labelled data. SimSiam architecture to learn useful feature representations with extensions that incorporate personalisation for local client models. The learnt representation are evaluated against a KNN classifier for analysing their usefulness.\n", "summary_of_the_review": "I think the work tackles an important question, however the experimentation is inadequate to assess the usefulness of the approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EWBjWta06kW", "reviewer": "Reviewer_bsjD", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "See Summary Of The Paper for detail.", "summary_of_the_paper": "The submission is not anonymized, the author's name chaoyanghe appears frequently in the attached SSFL_Sumpplementary/SSFL-Source-Code. Therefore, it should be directly desk rejected.\n\n--- Logging error ---\nTraceback (most recent call last):\n  File \"/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py\", line 1025, in emit\n    msg = self.format(record)\n  File \"/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py\", line 869, in format\n    return fmt.format(record)\n  File \"/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py\", line 608, in format\n    record.message = record.getMessage()\n  File \"/Users/chaoyanghe/opt/anaconda3/envs/fedml/lib/python3.7/logging/__init__.py\", line 369, in getMessage\n    msg = msg % self.args", "summary_of_the_review": "See Summary Of The Paper for detail.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes an extension of semi-supervised learning to the federated setting. The contributions include a thorough evaluation of performance and some method extensions. 

There are four reviewers. One reviewer points out a name leakage issue in the code that was missed and suggests deks-rejection. The area chair has chosen not to desk-reject the paper. Three other reviews agree that the manuscript addresses an interesting and timely issue -- indeed, label acquisition is a significant issue in federated learning. Three reviewers agree to reject the paper -- raising concerns about novelty compared to existing methods, some details of the evaluation, and some lack of clarity. The authors provide a good rebuttal addressing many of these issues. However, the reviewers are unconvinced that the method is sufficiently novel after reviews and discussion. Authors are encouraged to address the highlighted concerns for future submission of this work.
Decision: Reject