Divergence-aware Federated Self-Supervised Learning
Authors: Weiming Zhuang
Yonggang Wen
Shuai Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: Federated Learning
Self-supervised Learning
Unsupervised representation learning

Abstract: Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data. Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (e.g., from cameras and phones), often resulted from privacy constraints. Extensive attention has been paid to SSL approaches based on Siamese networks. However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture. We aim to fill in this gap via in-depth empirical study and propose a new method to tackle the non-independently and identically distributed (non-IID) data problem of decentralized data. Firstly, we introduce a generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods. In this framework, a server coordinates multiple clients to conduct SSL training and periodically updates local models of clients with the aggregated global model. Using the framework, our study uncovers unique insights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients in FedSSL is particularly beneficial for non-IID data. Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Exponential Moving Average update (FedEMA). FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynamically measured by model divergence. Extensive experiments demonstrate that FedEMA outperforms existing methods by 3-4% on linear evaluation. We hope that this work will provide useful insights for future research.

Published as a conference paper at ICLR 2022
DIVERGENCE-AWARY FEDERATED SELF-SUPERVISER LEARNING
Weiming Zhuang!., Yonggang Wen', Shuai Zhang
SS-Lab, NTU, Singapore 2NTU, Singapore Seenee Time Research
weiming001@e ntu. edu. sg, ygwenentu. edu. sg, zhangshuaiiseensttiiee
ABSTRACT
Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data. Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (c.g.. from cameras and phones), often resulted from privacy constraints. Extensive attention has been paid to SSL approaches based on Siamese networks. However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture. We aim to fill in this gap via dn-thpth empirical study and propose new method to tackle the non- independently and identically distributed (non-lID) data problem of decentralized data. Firstly, we introduce a generalized FedSSL framework that embraces ex- isting SSL. methods based on Siamese networks and presents flexibility catering to future methods. In this framework, server coordinates multiple clients conduct SSL training and periodically updates local models of clients with the aggregated global model. Using the framework, our study uncovers unique in- sights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients FedSSL is particularly beneficial for non-lID data. Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Ex- ponential Moving Average update (FedEMA). FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynam- ically measured by model divergence. Extensive experiments demonstrate that FedEMA outperforms existing methods by 3-4% on linear evaluation. We hope that this work will provide useful insights for future research.
INTRODUCTION
Self-supervised learning (SSL) has attracted extensive research interest for learning representations without relying on expensive data labels. In computer vision, the common practice is to design proxy tasks facilitate visual representation learning from unlabeled images (Doersch al. 2015
Noroozi & Favaro 2016; Zhang et al., 2016; Gidaris et al. 2018). Among them, the state-of-the-an SSL methods employ contrastive learning that uses Siamese networks minimize the similarity of
two augmented views of images (Wu et al., 2018 Chen et al., 2020a He et al., 2020; Grill et al.
2020 Chen & He 2021). D. All these methods heavily rely on the assumption that images are centrally available in cloud servers, such as public data on the Internet.
However, the rapidly growing amount of decentralized images may not be centralized due to increas- ingly stringent privacy protection regulations (Custers et al. 2019). The increasing number of edge devices, such as street cameras and phones, are generating large number of unlabeled images, but these images may not be centralized they could contain sensitive personal information like human faces. Besides, learning representations from these images could more beneficial for downstream tasks deployed in the same scenarios (Yan al. 2020). A straightforward method is to adopt SSL methods for each edge, but it results in poor performance (Zhuang et al. 2021a) as decentralized data mostly non-independently and identically distributed IID) (Li al. 2020a).
Federated learning (FL) has emerged as a popular preaeeevinn method to train models from decentralized data /McMahan et al. 2017), where clients send training updates to the server instead

---

Published as a conference paper at ICLR 2022
of raw data. The majority of FL methods, however, are not applicable for unsupervised representa- tion learning because they require fully labeled data (Caldas et al. 2018), or partially labeled data in either the server clients (Jin et al. 2020a Jeong et al. 20.II.. Recent studies implement FL with SSL. methods that are based on Siamese networks, but they only focus on a single SSL method. For example, FedCA (Zhang et al.
2020a) is based on SimCLR (Chen et al., (020a) and FedU (Zhuang) ict al., 2021a) is based on B YOL (Grill et al. (020). These efforts have not yet revealed deep insights into the fundamental building blocks of Siamese networks for federated self-supervised learning.
In this paper, we investigate the effects of fundamental components of federated self-supervised learning (FedSSL) via in-depth empirical study. To facilitate fair comparison, we first introduce a generalized FedSSL framework to embrace existing SSL methods that differ in building blocks of Siamese networks. The framework comprises of a server and multiple clients: clients conduct SSL, training using Siamese networks an online network and target network; the server aggregates the trained online networks to obtain a new global network and uses this global network to update the online networks of clients in the next round of training. FedSSL primarily focuses on the cross-silo FL where clients are stateful with high availability (Kairouz et al. 2019).
We conduct empirical studies based on the FedSSL framework and discover important insights of FedSSL. Among four popular SSL methods (SimCLR (Chen et al. 2020a). MoCo (He et al. 2020), BYOL (rril et al. 2020), and SimSiam (Chen & He 1021). FoL achieves the best perfor- mance, whereas FedSimSiam yields the worst performance. More detailed analysis uncover the following unique insights: 1) Stop-gradient operation, essential for SimSiam and BYOL, is not al- ways essential in FedSSL; 2) Target networks of clients are essential to gain knowledge from online networks: 3) Keeping local knowledge of clients is beneficial for performance on non-IID data.
Inspired by the insights, we propose new approach, Federated Divergence-aware Exponential Mov- ing Average update (FedEMA) D. to address the non-IID data problem. Specifically, instead of updating online networks of clients simply by the global network, FedEMA updates them via ex- ponential moving average (EMA) of the global network, where the decay rate of EMA is measured by the divergence of global and online networks dynamically. Extensive experiments demonstrate that FedEMA outperforms existing methods in a wide range of settings. We believe that important insights from this study will shed light future research. Our main contributions are threefold:
We introduce a new generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods.
We conduct in-depth empirical studies of FedSSL based on the framework and discover deep insights of the fundamental building blocks of Siamese networks for FedSSL.
Inspired by the insights, we further propose a new model update approach, FedEMA, that adaptively updates online networks of clients with EMA of the global network. Extensive experiments show that FedEMA outperforms existing methods in a wide range of settings.
2 RELATED WORK
Self-supervieed Learning In computer vision, self-supervised learning (SSL) aims to learn visual representations without any labels. Discriminative SSL methods facilitate learning with proxy tasks (Pathak et al., 22 contrastive learning (Oord et al.] 2018 Bachman al. 2019) has become promising principle. It uses Siamese networks to minimize the similarity of two augmented views (positive pairs) and max- imize the similarity of two different images (negative pairs). These methods are either contrastive or non-contrastive ones: contrastive SSL methods require negative pairs (Chen et al., 2020a; He et al.) to prevent training collapse; non-contrastive SSL methods (Grill al. [020]] Chen He, are generally more efficient as they maintain remarkable performances using only positive pairs. However, these methods do not perform well on decentralized non- -ID data (Zhuang et al.)
& Favaro 2016 Zhang et al.] 2016 Gidaris et al.) 2018). Among them,
2020
2021
2021a).
We analyze their similarities and variances and propose a generalized FedSSi. framework.
Federated Learning Federated learning (FL) is a distributed training technique for learning from decentralized parties without transmitting raw data to central server (McMahan et al. 2017).
Intuitivelyy FedSSL is analogous to SuperClass in object-oriented programming (OOP), then FedEMA is SubClass that inherits FedSSL. and overrides the model update method.
2

---

Published as a conference paper at ICLR 2022
Server
Online W
Online W
Online W
Online W
Client
Client 2
Client
Online W
Online W
Online W
image
image
image
Loss L
Loss L
Loss L
Target W
Target W2
Target W
2
22
22
Online Network
Target Network
Model Upload
Model Aggregation
Model Update
Overview of federated self superviid learning (FedSSL) framework. comprises an end- to-end training pipeline with four steps: 1) Each client k conducts local training on unlabeled data DR with Siamese networks client k uploads Wa to the server; 3) The server aggregates them to obtain new global network W 4) The server updates We of client with W.
Figure I:
an online network We and a target network W 2) After training,
Among many studies that address the non-llD data challenge (Zhao et al. 2018 Li al. 2020b
Wang et al.] 2020; Zhuang et al. 2021c). Personalized FL (PFL) aims to learn personalized mod- els for chents Tan et al., 2021). Although some PFL methods interpolate global and local models (aazelly et al. 2020 Mansour et al. 2020 yuyang deng et al., 2021). our proposed FedEMA differ in the motivation, application scenario, and measurement of the decay rate. Besides, the majority of existing works only consider supervised learning where clients have fully labeled data. Although re-
cent works propose federated semi-supervised learning (Jin al. 2020b Zhang al. 2020b Jeong et al. 2021) or federated domain adaptation (Peng et al. [220 Zhuang et al.) (2021b), they still need labels in either the server or clients. This paper focuses on purely unlabeled decentralized data.
Federated Unsupervised Learning Learning representations from unlabeled decentralized data while preserving data privacy is still nascent field. Federated unsupervised representation learning is first proposed by van Berlo et al. (2020) based on autoencoder, but it neglects the non-III data challenge. (Zhang et al. al.[ (2020a) address the non-lID issue with potential privacy risk for sharing features. Although Zhuang et al. (2020) address the issue based on BYOL our FedEMA, they do not shed light on why BYOL works best. Since SSL methods are evolving rapidly and new methods emerging, introduce generalized FedSSL framework and deeply investigate the fundamental components to build up practical guidelines for the generic FedSSL framework.
AN EMPIRICAL STUDY OF FEDERATED SELF-SUPERVISED LEARNING
This section first defines the problem and introduces the generalized FedSSI. framework. Using the framework, we then conduct empirical studies to reveal deep insights of FedSSL..
3.1 PROBLEM DEFINITION
FedSSL aims to learn generalized representation from multiple decentralized parties for down- stream tasks in the same scenarios. Each party & contains unlabeled data Dj {X₂, } that cannot be transferred to the server or other parties due to privacy constraints. Data is normally non-III among decentralized parties (Li et al. 2020a); each party could contain only limited data categories (e.g., two out of ten CIFAR- 10 classes) (Luo et al. 2019). As a result, each party alone is unable to obtain a good representation (Zhuang al.] 2021a). The global objective function to learn from multiple parties is min (w) = Σ n "* (w), where K is the number of clients, and n Σ ni n
is the total data amount. For client k, fa(w)= = ExgoPy [ff((w; æx)] is the expected loss over data distribution Pk, where 2ạ is the unlabeled data and fa (w; .x)) is the loss function.
3.2 GENERALIZED FRAMEWORK
We introduce a generalized FedSSL framework that empowers existing SSL methods based on Siamese networks to learn from decentralized data under privacy constraints. Figure depicts the
3

---

Published as a conference paper at ICLR 2022
end-to-end training pipeline of the framework. It comprises of three key operations: 1) Local Train- ing in clients; 2) Model Aggregation in the server; 3) Model Communication (upload and update) between the server and clients. We implement and analyze four popular SSL methods SimCLR (Chen et al.; 2020a), MoCo (V1 (He et al., 2020) and V2 (Chen et al. 2020b)), SimSiam (Chen & He 2021), and BYOL (Grill al., 2020). Variances in Siamese networks of these methods lead to differences in executions in these three operations 7
Local Training Firstly, each client &s conducts self-supervised training on unlabeled data Dj. based on the same global model We downloaded from the server. Regardless of SSL methods, clients train with Siamese networks - an online network W and a target network W for E local epochs using cooresponding loss functions L. We classify these SSL methods with two major differences (Figure [ in Appendix AA: 1) Only SimSiam and BYOL contain predictor in the online network, so we denote their online network W = (Wx, k. WF), where Wa is the online encoder and W is the predictor; As for SimCLR and MoCo, W = WR. 2) SimCLR and SimSiam share identical weights between the online encoder and the target encoder, so W and BYOL update the target encoder with EMA of the online encoder in every mini-batch: W mW, + (1 - W, where m is the momentum value normally set to 0.99.
We. In contrast, MoCo
g.
Model Communication After local training, client & uploads the trained online network WT the server and updates it with the global model W after aggregation. Considering the differences of SSL methods, we upload and update encoders and predictors separately: 1) we upload and update the predictor when it presents in local training; 2) we follow the communication protocol Zhuang al. a upload and update only the online encoder W₁ when encoders are different.
Model Aggregation When the server receives online networks from clients, it aggregates them to obtain a new global model W = ik " We. W = (W,, WP) if predictor presents, otherwise W gg where W, is the global encoder. Then, the server sent W to clients to update their online networks. The training iterates these three operations until it meets the stopping conditions. At the end of the training, we use the parameters of W as the generic representation W for evaluation.
3.3 EXPERIMENTAL SETUP
We provide basic experimental setups in this section and describe more details in Appendix B
Datasets We conduct experiments using CIFAR-I0 and CIFAR- 100 datasets (Krizhevsky et al., 2009). To simulate federated settings, we equally split dataset into K clients. We simulate non- IID data with label heterogeneity, where each client contains limited classes - = {2, 4, 6, 8, 10) number of classes for CIFAR-10 and 1 = {20, 40, 60, 80, 100} for CIFAR-100. The setting is IID when each client contains 10 (100) classes for CIFAR- 10 (C0)
Implementation Details We implement FedSSL in Python using popular deep learning framework PyTorch (Paszke et al. 2017). To simulate federated learning, we train each client on one NVIDIA
V100 GPU. These clients communicate with the server through NCCL backend. We use ResNet- 18 HHe et al. 2016) as default network for the encoders and present results of ResNet-50 in Appendix The predictor a two-layer multi-layer perceptron (MLP). By default, we train for R 100 rounds with K = 5 clients, E- = 5 local epoches, batch size B = 128, learning rate 0.032 with (1 20) for CIFAR-10 (CIFAR-100).
cosine decay, and non-III data
Linear Evaluation We evaluate the quality of representations following linear evaluation Kolesnikov et al., 2019 Grill et al.) 2000) protocol. We first learn representations from the FedSSL. framework. Then, we train new linear classifier on the frozen representations.
3.4 ALGORITHM COMPARISONS
We benchmark and compare the SSL methods using the FedSSL framework. To denote the imple- mentation of an SSL method, We add prefix Fed to the name of the SSL method. For example, FedBYOL denotes using BYOL in the FedSSL framework.
Intuuitively, FedSSL. is analogous to SuperClass in OOP, then implementation of each method is Sub- Class that inherits FedSSL. and overrides methods of local training, model communication, and aggregation.
4

---

Published as a conference paper at ICLR 2022
Table 1: Top-1 accuracy comparison of SSL methods using the FedSSL framework on non-IID
CIFAR datasets. FedBYOL, performs perforrs the best, whereas FedSimSiam performs the worst.
Type
Method
CIFAR-10 (%)
CIFAR-100 (%)
FedSimCLR
78.09 + 0.14
55.58 + 0.13
Contrastive
FedMoCoV1
78.21 + 0.04
56.98 + 0.29
FedMoCoV2
79.14 0.13
57.47 0.65
FedSimSiam
76.27 + 0.18
48,94 + 0.22
Non-contrastive
FOLBYOL
79.44 + 0.99
57.51 +: 0.09
KNN Acz. (%)
75
60
60
KNN Acc.
FedB YOL
acc (%)
30
40
00
70
10
40
60
70
Rounds
Rounds
w/o stop-grad
75.73
FedRiVO
FedRVOI
FedRiOl
FecRiVl
FeeSinSiam ped
FadSenSarr
FudSinSaa w/s stop-grad
FecSirSuum
w/ stop-grad
79.44
Figure 2: Comparison of non-contrastive FedSSL methods with and without (w/o) predictor (pred) or stop-gradient (stop-grad) on non-IID CIFAR-10 dataset. Without predictor, both FedSimSiam and FedB drops performance on kNN testing accuracy (left plot). Without stop- soopggrddeent FedB YOL retains competitive results on kNN testing accuracy (middle plot) and linear evaluation (right table).
Table compares linear evaluation results of these methods under the non-IlI setting of CIFAR datasets. On one hand, contrastive FedSSL methods obtain similar performances. As SimCLR is previously reported to need large batch size (e.g., B 4096) (Chen et al. 2020a), it is surprising that FedSimCLR obtains competitive results using the same batch size = 128 as the others. On the other hand, the results of non contrastte FedSSL methods have large variances: FedB YOL achieves the best performance, whereas FodSimSiam yields the worst performance. Since SimSiam is capable to learn powerful representations BYOL (Chen & He 2021), well considering that non- contrastive methods are conceptually simpler and more efficient (Tian et al.] (021). we focus on non-contrastive methods and further investigate the effects of their fundamental components.
3.5
IMPACT OF FACTORS OF NON-CONTRASTIVE METHODS
This section analyzes the impact of fundamental components of non-contrastive FedSSL methods. From empirical studies, we obtain the following insights: 1) predictor is essential; 2) EMA and stop- gradient improves performances; 3) Local encoders should retain local knowledge of the non-IlD data; 4) Target encoder should gain knowledge from the online encoder. Details are as followed.
Predictor is essential. Figure (left plot) presents the kNN testing accuracy monitoring process for FedB YOL and FedSimSiam with and without predictors. Without predictors, both methods can barely learn due to collapse local training. affirms the vital role of predictor (Chen & He 2021 202I) even when learning from decentralized data.
Tian et al.
Stop-gradient operation is previously indicated as an essential component for SimSiam and BYO (Tian et al., 2021), but it is not essential for FedB YOL. Stop-gradient prevents stochastic gradi- ent optimization on the target network. Figure 2 shows that FedSimSiam without sto--gradien collapses, whereas FedB without stop-gradient still achieves competitive performance. It is because online and target encoders are significantly different in FedB as the online encoder updated by the global encoder every communication round. In contrast, SimSiam or FedSiamSiam share weights between online and target encoders, so removing stop- stop-gradient leads to collapse.
Exponential Moving Average (EMA) is not essential, but it helps improve performance. Table (first and second rows) shows that FedB YOL outperforms FedSimSiam at different levels of non- IID data, which is represented by {2. 4, 6, 8. 10) classes per client of CIFAR- 10. EMA is the main difference between SimSiam and BYO, indicating that EMA is helpful to improve performance. Based these results, further analyze the underlying impact of EMA the encoders below.
5

---

Published as a conference paper at ICLR 2022
Table 2: Top-1 accuracy comparison on various non-IID levels the number of classes per client on the CIFAR-10 10 dataset. Update-both means updating both Wa and W with W
# of classes per client (%)
Method
2
4
10 (iid)
FodBYOL 79.44
82.82
83.02
84.57
84.20
FedSimSiam
76.27
79.34
80.17
80.92
80.50
FedB YOL, update-both
74.50
78.77
83.02
84.56
83.80
KNN Acc. (%)
75
FodB
acc (%)
45-
w/o EMA
50.20
10
20
30
40
50
60
70
80
90
100
w/o EMA and stop-grad
11.97
Rounds
w/o EMA and stop-grad, update-both
68.75
FodBYOL w/o EMA FedBYOL w/o and sg. update-both
s6
original
79.44
Figure 3: Comparison of FOL without exponential moving average (EMA) and nop-gradient (sg) on the non-llD CIFAR-I0 10 dataset. FedB YOL w/o EMA and sg can hardly learn, but updating both W2 and WE with W, (update-both) enables it to achieve comparable results.
Encoders that retain local knowledge of non-lll) data helps improve performance. EMA in Fed- BYOL allows the parameters of the online encoder be different from the target encoder. As result, the global encoder only updates the online encoder, not the target encoder. We hypothesize that retaining such local knowledge of data in the target encoder is beneficial especially when the data distribution is highly skewed. For comparison, we remove such local knowledge by updating both online and target encoders with the global model. Table 2 shows that FoL with both en- coders updated leads to lower performance than YOL: It achieves results close to FedSimSiam when the data distribution is more skewed (2 or classes per client). These results demonstrate the importance of keeping local knowledge in the encoders. Besides, the results of {6, 8, 10} classes per client also indicate the benefit of EMA.
Target encoder is essential to gain knowledge from the online encoder. Figure 3 shows that Fed- BYOL without EMA can merely learn, and FedB YOL without EMA and stop-graad moop-gradient (sg) degrades in performance. In both cases, the target encoder is either never updated (w/o EMA) or is updated only through backpropagation (w/o EMA and sg) - not updated by the online encoder. On the other hand, also identify that FedSSL. methods in Table D which achieve competitive results, all update target encoders with knowledge of online encoders (the global encoder is the aggregation of the online encoder). We argue that target encoder crucial gain knowledge from the online encoder to provide contrastive targets. We further validate it by using the global encoder to update both online and target encoders when removing both EMA and stop-gradien gradient. Figure shows that such method improves performance and achieves comparable results.
DIVERGENCE-AWARE DYNAMIC MOVING AVERAGE UPDATE
Built on the FedSSL framework, we propose Federated Divergence-awarr EMA update (FedEMA) to further mitigate non-IID data challenges. Since FedB contains all components that help improve performance, adopt it as the bascline and optimize the model update operation.
Non-IID data causes the global model to diverge from centralized training (Zhuang et al., 202 Inspired by the insight that retaining local knowledge of non-IID data helps improve performance. we propose to update the online network via EMA of the global network. Compared with FedB YOL that replaces the online network with the global network, FedEMA fuses local and global knowledge effectively through EMA update, where the decay rate of EMA is dynamically measured by model divergences. Figure 4 depicts our proposed FedEMA method. The formulation is as followed:

---

Published as a conference paper at ICLR 2022
W = wW + +11 wo
(1)
W = p +11 w
(2)
min [A 1
(3)
where WI and Wh are the online encoder and predictor of client k at training round r; W and WP. are the global encoder and predictor; μ is the decay rate, measured by the divergence between global and online encoders; À is a scaler to adjust the level of model divergence, which is measured by calculating the norm of the global and online encoders. We summarize FedEMA in Algorithm FedEMA can be regarded as a generalization of FedB YOUB they are the same when À = 0.
Scaler plays a vital role adapt FedEMA for different levels of divergence caused by the data. The divergence between global and online encoders varies when the settings of federated learn- ing change. For example, different degrees of non-IID settings would result in different diver- propose
gences. Since characteristics of data are unknown before training they are unlabeled, a practical autoscaler to calculate a personalized 0 for each client automatically. The formula is where [0, 1] is the expected value of μ at round r. We calculate X only once at the earliest round that client k is sampled for training. When the same set of clients are calculated at round 1.
A ||W; WW'l"
sampled for training, A [[] WPT
The intuition of FedEMA is to retain more local knowledge when divergence is large and incorporate more global knowledge when divergence is small. When model divergence large, keeping more local knowledge is more beneficial for the non-IID data. Since the global network is the aggregation of online networks, representing global knowledge from clients. When divergence is small, adapting more global knowledge help improve model generalization. Since model divergence is larger the start of training (Figure 6). it is practical to choose larger E [0.5, 1); T = 1 is not considered because only local knowledge is used when 7 We use 0.7 by default in experiments.
Algorithm Our proposed FedEMA
Global Encoder W
Global Predictor W
1: ServerExecution:
2: Init We and Wp.0 init A to be null
3: for each round r 0, 1,.,. R do
Divergence-sware
Server
EMA Update
4: St + (Selection of K clients)
5:
for client St parallel do
Client k.
6:
Wh, WP.r +Client( WWr. (.))))
Online Encoder
Predictor W
W
7:
W + Ekes,
De WE
8:
Wh.+1 + Ekes, WPr
image
Augmentation
EMA
[bbl]]
(3L 322
[1xa|l
9:
for client St do
Target Encoder
10:
AR
I| will
if Da is null
222
Wt
stop-gradient
11: Return WR
Client (W- W r. As):
Figure 4: Illustration of our proposed Feder- ated Divergence-aware Exponential Moving Av- erage update (FedEMA). Compared with Fed- BYOL that simply updates the online network W of client & with the global network Wo. we pro- pose to update them via EMA of the global net- work following [qn where the decay rate I is dynamically measured the divergences be- tween the online encoder W₁ and the global en- coder W, (Eqn (3). The online network, W (We, W?). is the concatenation of the online coder We and the predictor W.P
13: if d is null or not selected in then
WR. Wt WT + WT. WT. War
15: else
14 + mA | - 1I
16:
17:
We + wW k + (1 W.
WT - ---W - + -1 (1-1)
19:
for local epoch e 0, 1.., 1 do for b E B data batches with size B do W < We 7/ C w: (WF: b) W mWk + (1 m)
20:
21:
22:
23:
Return WT, WP.
7

---

Published as a conference paper at ICLR 2022
Table 3: Top-1 accuracy comparison under linear probing CIFAR datasets. Our proposed
FedEMA outperforms all other methods. Full results are in Table
CIFAR-10 (%)
CIFAR-100 (%)
Method
IID
Non-IID
IID
Non-IID
Standalone training
82.42 + 0.32
74.95 + 0.66
53.88 + 2.24
52.37 + 0.93
FedBYOL
84.29 0.18
79.44 + 0.99
54.24 0.24
57.51 0.09
FedU (Zhuang et al. 2021a
83.96 + 0.18
80.52 + 0.21
54.82 0.67
57.21 + 0.31
FedEMA (A
0.8)
85.59 = 0.25
82.77 + 0.08
57.86 + 0.15
61.21 + 0.54
FedEMA (autoscaler, 0.7)
86.26 + 0.26
83.34 + 0.39
58.55 + 0.34
61.78 + 0.14
BYOL (Centralized)
90.46 + 0.34
65.54 0.47
Table 4: Top-1 accuracy comparison on 1% and 10% labeled data for semi-supervised learning non-Ill) CIFAR datasets. FedEMA outperforms other methods. Full results are in Table 7
CIFAR- 10 (%)
CIFAR 100 (%)
Method
1%
10%
1%
10%
Standalone training
61.37 0.13
69.06 + 0.24
21.37
0.73
39.99 0.87
FedB YOL
70.48 + 0.30
76.95 + 0.46
30.21 + 0.40
47.07 + 0.14
FedU (Zhuang et al.
2021a
69.52 + 0.73
77.06 + 0.55
29.00 + 0.27
46.67 + 0.06
FedEMA (A
72.78 + 0.66
79.01 + 0.30
32.49 0.22
49.82 0.36
FedEMA (autoscaler, r 0.7)
73.44
0.22
79.49 + 0.34
33.04 0.23
50.48 0.11
BYOL (Centralized)
87.67 = 0.15
87.89 + 0.05
40.96 + 0.58
56.60 + 0.33
5
EVALUATION
This section follows the experimental setup in Section 3.3 to evaluate FedEMA in the linear evalua- tion and supe-supe learning. We also provide ablation studies of important hyperparameters.
5.1 ALGORRTHM COMPARISONS
To demonstrate the effectiveness of FedEMA, we compare it with the following methods: 1) stan- dalone training, where a client learns independently using BYOL; 2) FedCA, which is proposed in Zhang al. (2020a);
FodB the baseline; FedU, which is proposed in (Zhuang al., 2021a). lap. Besides, we also present results of possible upper bounds that learn representations with centralized data using BYOL.
Linear Evaluation Table shows that FedEMA outperforms other methods on different settings of CIFAR datasets. Specifically, the performance is more 3% higher than existing methods in most settings. Besides, our proposed autoscaler achieves similar results as X = 0.8. More experiments on larger number of clients and random selection clients are provided in Table Appendix C
Semi-supervised Learning We also assess the quality of representations following the semi- supervised learning protocol (Zhai al., 2019: Chen et al. 2020a) - we add new two-layer MIP top of the encoder and fine-tune the whole model with limited (1 1 and 10%) labeled data 4 indicates that FedEMA consistently outperforms other methods on non-IID settings of CIFAR datasets and our autoscaler outperforms manual-selected À
for 100 epochs. Table
5.2 ABLATION STUDIES
Ablation FedEMA We analyze whether need to update both online encoder (Eqn 1 and predictor (Eqn 2 in FedEMA. [ggure]sshww that updating only the encoder or predictor leads to better performance; only updating predictor also leads to faster convergence. Their combination
8

---

Published as a conference paper at ICLR 2022
Method
acc (%)
80
Divergence
1.0
0.6
FedBYOL
79.44
0.4
WWW
FedEMA predictor only
81.13
23 30
40
50 61 70 80
100
0.0
0
10
20 30
40 50 60 70 so 90 100 Rounds
FedEMA encoder only
82.39
Rounds
FedBYOL
FedEMA ancoder only
Client 0
Client Client
Client
FedEMA (ours) 82.77
pred
Client
Figure 5: Ablation studies of FedEMA: applying EMA either
Figure 6: Changes of diver- gence throughout training.
predictor or encoder leads to better performance on CIFAR-10.
Top Acc. (%)
Top-1 Ace. (%)
96
.00
Top-1 Acc.
Ours
FodBYOL
60-
10
30
0.0.02 0.4 0.600. 0.5 1.2 1.1 1.6 1.8.2.2.
0.1
0.3
0.5
U.7
0.9
Staler
Decay Rate
4
8
10 (id)
CIFAR-10
CIFAR-100
Encoder
Predictor
Both
Classes Per Client
(a) Scaler À
(b) Constant μ
(c) Non-IID level
Figure 7: Ablation study on scaler X, decay rate μs, and non-IlD levels of the CIFAR-10 dataset: (a) analyzes the impact of scaler À on performance; (b) compares using constant μ on encoder, predictor, or both; (c) studies the impact of different non-IID levels.
sults in the best performance. These results demonstrate the effectiveness of updating both predictor and encoder in FedEMA. More results on other settings are provided in Table Appendix C
Changes of Divergence
that the divergence between global encoder and online encoder (Eqn 3) decreases gradually as training proceeds. validates our intuition that more local knowledge is used at the start of training when divergence is larger. Besides, clients can update at their own pace depending on the divergence caused by their local dataset.
Scaler À We study the impact of À with values in [0, 2] with interval of 0.2 in Figure 7
X
leads to a significant performance drop because results in a = 1 at the start of training on CIFAR datasets, implying that aggregated global network is used. When À (0, 1), the performances are consistently better than FedB YOL (A 0) as both local and global knowledge are effectively aggregated. These analyses mainly suitable for our experiment setting. The range values of depend on the characteristics of data and the hyper-parameters (e.g., local epoch) of FL settings. A practical way tune manually is understand the divergence by running the algorithm for several rounds and choose the À that scales to (0.5, 1). Nevertheless, we recommend using autoscaler and provide ablation study of autoscaler Figure T0a Appendix C
Constant Values of We further demonstrate the necessity of dynamic EMA by comparing with using constant values of μ in Eqn and 2 Figure shows that a good choice of constant fr can outperform FedB YOL. but FedEMA outperforms using constant y for the online encoder, predictor, or applying both. We also provide results that encoder and predictor use different μr Appendix C
Non-IID Level Figure compares the performance of different non-lII levels, ranging from 2 to 10 classes per client on the CIFAR-10 dataset. We use autoscaler for these experiments. FedEMA consistently outperforms FedB YOL in these settings.
CONCLUSION
We uncover important insights of federated self-supervised learning (FedSSL) from in-depth em- pirical studies, using newly introduced generalized FedSSL framework. Inspired by the in- sights, propose new method, Federated Divergence-aware Exponential Moving Average update (FedEMA), to further address the non-IID data challenge. Our experiments and ablations demon- strate that FedEMA outperforms existing methods in wide range of settings. In the future, plan to implement FedSSL and FedEMA on larger-scale datasets. We hope that this study will provide useful insights for future research.

---

Published as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
To facilitate reproducibility of experiment results, we first provide basic experimental setups in Sec- tion 3.3 including datasets, implementation details, and evaluation protocols. Then, we describe more experimental details in Appendix [B] including datasets, data transformation, network archi- tecture, training details, and default settings. Also, we indicate the settings and hyper-paramelers of experiments when their settings are different from the default. Moreover, we plan to open-source the codes the future.
ACKNOWLEDGMENTS
We would like to thank reviewers of ICLR 2022 for their constructive and helpful feedback. This study is in part supported by the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s): the National Research Foundation, Singapore under its Energy Programme (EP Award IRF2017EWT--PP00322222 administrated by the Energy Market Authority of Singapore, and its En- engy Research Test-Bed and Industry Partnership Funding Initiative, part of the Energy Grid (EG) 2.0 programme, and its Central Gap Fund ("Central Gap" Award No. NRF2020NRF-CC0011-00 Singapore MOE under its Tier grant call, Reference number RG96/20.
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter, Learning representations by maximiz- ing mutual information across views. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https http:::s////occeedinnge meapeeedddinnnu.nigiieeeee www.ins.unccoom oa/ (tteer/222////ledddeees ort.:334....aacacaaaaoeoeee000a0
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750- 15758, June 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020b.
Bart Custers, Alan M. Sears, Francien Dechesne. Ilina Georgieva, Tommaso Tani, and Simone van der Hof. EU Personal Data Protection in Policy and Practice. Springer, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 18X3v:1803.007 2018
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot. koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. ht /ttppps//ppocccceggg.g.. 1188
21271-21284 Curran Associates, Inc., 2020.
cc/papper/222//11
10

---

Published as a conference paper at ICLR 2022
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. In Advances Neural Information Processing Systems. Curran Associates, Inc., 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Wonyong Jeong, Jachong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learn- ing with inter-client consistency & disjoint learning. In International Conference on Learning Representations, 2021. URL
Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. Towards utilizing unlabeled data in federated learning: A survey and prospective. arXiv e-prints, pp. arXiv-2002, 2020a.
Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. A survey towards federated semi-supervised learning. arXiv preprint arXiv:2002. 1545, 2020b.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint 1rXiv:1912.04990 2019.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1920-1929, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tian Li. Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37:50-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429-450, 2020b.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL http::::///peeevieee. net forum?
id Skg89Scxx
Jiahuan Luo, Xueyang Wu, Yun Luo. Anbu Huang, Yunfeng Huang, Yang Liu, and Qiang Yang. Real-world image datasets for federated learning. arXiv preprint arXiv: 1089, 2019.
Yishay Mansour, Mehryar Mohri, Jac Ro, and Ananda Theertha Suresh. Three approaches for
personalization with applications to federated learning. arXiv preprint arXiv:2002. 10619, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication--ffiiienn learning of deep networks from decentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, vol- ume 54 of Proceedings of Machine Learning Research, pp. 1273-1282, PMLR, 2017. URL http::/prroceedinngg mww. press v54 mcmahan 7 https///////mmmmaammaaaa html
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European conference on computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1 003748887888 2018.

---

Published as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitioo Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Xingchao Peng. Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adap- tation. In International Conference Learning Representations, 2020. URL https://
openreview wwwprrvview.ee//oorureee
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. arXiv preprint arXiv:2103.007/ 2021.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th In- ternational Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10268-10278 PMLR, 18-24 Jul 2021. URL httphhtts//ppeee /ttp:psss/prrrceeddgs....m
Bram van Berlo, Aaqib Saeed, and Tanir Ozcelebi. Towards federated unsupervised representation learning. In Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking, pp. 31-36, 2020.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Represen- tations, 2020. URL /ttphttts///ppeereettt
net feee//ooormmiddBiugllll
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin, Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Xi Yan, David Acuna, and Sanja Fidler. Neural data server: A large-scale search engine for transfer learning data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3893-3902, 2020.
yuyang deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. 2021. URL
meet/oouu?iddggaee jp07
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1476-1485, 2019.
Fengda Zhang, Kun Kuang, Zhaoyang You. Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yuet- ing Zhuang, and Xiaolin Li. Federated unsupervised representation learning. arXiv preprint 2rXiv:2010.08982 2020a.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Zhengming Zhang, Zhewei Yao, Yaoqing Yang, Yujun Yan, Joseph E Gonzalez, and Michael W Mahoney. Benchmarking semi-supervised federated learning. arXiv preprint arXiv:2008. 11364, 17, 2020b.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. CoRR, hts/1800.00550 2018. URL 18C6600583
Weiming Zhuang, Yonggang Wen, Xuesen Zhang, Xin Gan, Daiying Yin, Dongzhan Zhou, Shuai Zhang, and Shuai Yi. Performance optimization of federated person re-identification via bench- mark analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 955-963, 2020.
12

---

Published as a conference paper at ICLR 2022
Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsuper- vised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021a.
Weiming Zhuang, Xin Gan, Yonggang Wen, Xuesen Zhang, Shuai Zhang, and Shuai Yi. Towards unsupervised domain adaptation for deep face recognition under privacy constraints via federated learning. arXiv preprint arXiv:2105.07606 2021b.
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Joint optimization in edge-cloud continuum for federated unsupervised person re-identification. In Proceedings of the 29th ACM International Conference on Mullimedia, pp. 433-441, 2021c
Weiming Zhuang, Xin Gan, Yonggang Wen, and Shuai Zhang. Easyfl: A low-code federated learn- ing platform for dummies. IEEE Internet of Things Journal, 2022.
13

---

Published as a conference paper at ICLR 2022
A DIFFERENCES OF SELF-SUPERVISED LEARNING METHODS
We study four SSL methods using the FedSSL framework in Section These four SSL methods have two major differences that impact the executions of local training, model communication, and model aggregation. Figure 8 depicts these differences: 1) BYOL and SimSiam have predictors, whereas MoCo and SimCLR do not have them; 2) SimSiam and SimCLR share weights between two encoders, whereas BYL and MoCo have different parameters for the online and target encoders.
With Predictor
similarity
similarity
stop- grad
stop- grad
Predictor W
Predictor W
EMA
Online W
Target W 22
Online W.
Online Wa
2
322
image æ
image
Non-identical Encoders
BYOL
SimSlam
Identical Encoders
similarity
stop- grad
similarity &
dissimilarity
Memory Bank
EMA
Online We
Online Wi
Online W2
Target W 422
31
image
image 2
MoCo V1
Without Predictor
SimCLR
Figure 8: Illustration of differences among four Sulf-supe Learning (SSL) methods.
B EXPERIMENTAL DETAILS
In this section, we provide more details about the dataset, network architecture, and training and evaluation setups.
B. DATA
Datasets CIFAR-10 and CIFAR-100 are two popular image datasets (Krizheskky et al., 2009). Both datasets consist of 50,000 training images and 10,000 testing images. CIFAR-10 contains 10 classes, where each class has 5,000 training images and 1,000 testing images. While CIFAR-100 contains 100 classes, where each class has 500 training images and 100 testing images. To simulate federated learning, we equally split the training set into clients. We simulate non-llI) data using label heterogeneity
data among clients is more skewed when each client contains less number of classes. Hence, we simulate different levels of non-IID data with / number of classes per client, where {2, 4, 6, 8, 10) for CIFAR-IO and / {20, 40, 60, 80, 100) for CIFAR-I00. For example, when simulating 5 clients with = 4 classes per client in CIFAR- 10, we need 5 x 4 = 20 total sets of data over 10 classes. Thus, we split the training images of each class equally into two sets (2,500 images in each set) and assign random four sets without overlapping classes to client. The setting III when each client contains all classes of a dataset. By default, we run experiments with K clients with non-IlD IID setting I = 2 classes per client for CIFAR-10 dataset and [ 20 classes per client for CIFAR-100 dataset.
Transformation In local training of the FedSSL framework, we take two augmentations of each image as the inputs for online and target networks, respectively. We obtain the augmentations by transforming the images with a set of transformations: For SimCLR, BYOL, SimSiam, and Mo- CoV2, we adopt the transformations from Chen et al. (2020a); For MoCoVI, we use the transfor- mation described in its paper (He et al. 2020).
14

---

Published as a conference paper at ICLR 2022
Table 5: Top-1 accuracy comparison under linear evaluation protocol on CIFAR datasets.
Our
proposed FedEMA outperforms all other methods on non-III settings.
CIFAR-10 (%)
CIFAR-100 (%)
Method
Architecture
Param.
IID
Non-IID
IID
Non-IID
Standalone training
ResNet-18
11M
82.42
74.95
53.88
52.37
FedSimCL.R
ResNet-18
11M
82.15
78.09
56.39
55.58
FedMoCoVI
ResNet-18
11M
83.63
78.21
59.58
56.98
FedMoCoV2
ResNet-18
11M
84.25
79.14
58.71
57.47
FedSimSiam
ResNet-18
11M
81.46
76.27
49.92
48.94
FedB YOL.
ResNet-18
11M
84.29
79.44
54.24
57.51
FedU (Zhuang et al. 2021a)
ResNet-18
11M
83.96
80.52
54.82
57.21
FedEMA predictor only (ours)
ResNet-18
11M
84.97
81.13
55.52
57.53
FedEMA encoder only (ours)
ResNet-18
11M
82.88
82.39
56.06
59.74
FedEMA (A 0.8)
ResNet-18
11M
85.59
82.77
57.86
61.21
FedEMA (autoscaler, T 0.7)
ResNet-18
11M
86.26
83.34
58.55
61.78
Standalone training
ResNet-50
23M
83.16
77.84
57.21
55.16
FedSimCLR
ResNet-50
23M
82.24
80.37
57.46
56.88
FedMoCoVI
ResNet-50
23M
87.19
82.18
64.74
59.73
FedMoCoV2
ResNet-50
23M
87.19
79.62
63.75
59.52
FedSimSiam
ResNet-50
23M
79.64
76.7
46.28
48.8
FedB YOL
ResNet-50
23M
83.90
81.33
57.75
59.53
FedCA Zhang et 2020al
ResNet-50
23M
71.25
68.01
43.30
42.34
FedU (Zuang et al. 2021a)
ResNet-50
23M
86.48
83.25
59.51
61.94
FedEMA predictor only (ours)
ResNet-50
23M
83.66
81.78
57.79
60.11
FedEMA encoder only (ours)
ResNet-50
23M
84.66
84.91
58.52
62.51
FedEMA (A = 0.8)
ResNet-50
23M
86.12
85.29
60.96
62.53
FedEMA (autoscaler, 7 = 0.7)
ResNet-50
23M
85.08
84.31
59.48
62.77
BYOL (Centralized)
ResNet-18
11M
90.46
65.54
BYOL (Centralized)
ResNet-50
23M
91.85
66.51
Table 6: Top-1 accuracy comparison on larger numbers of clients with client subsampling: 1) ran- domly selecting 5 out of 20 clients per round (5/20); randomly selecting out of 80 clients per round (8/80). FedEMA, trained with autoscaler, consistently outperforms FedB YOL in both set- tings.
5/20 clients (%)
8/80 clients (%)
Method
CIFAR-10
CIFAR-
CIFAR-10
CIFAR- 100
IID
Non-IID
IID
Non-IID
IID
Non-IID
IID
Non-IID
TedBYOL.
83.25
74.92
49.49
47.09
73.58
63.28
41.19
41.58
FedEMA (ours)
84.98
75.77
55.41
52.78
73.96
64.19
41.97
43.05
B.2 NETWORK ARCHITECTURE
The network architecture of the predictor is a two-layer multilayer perceptron (MLP). The two-layer MLP starts from fully connected layer with 4096 neurons. Followed by one- dimension batch normalization and a ReLU activation function, ends with another fully connected layer with 2048 neurons.
Predictor
Encoder We use ResNet-18 He et al. (006)) as the default network architecture of the encoder in the majority of experiments. Besides, also provide results of ResNet-50 in Table 5 and
15

---

Published as a conference paper at ICLR 2022
Table 7: Top-1 accuracy comparison on using 1% and 10% of labeled data for semi-supervised learning on the non-IlD settings of CIFAR datasets. FedEMA outperforms all other methods.
CIFAR-I0 (%)
CIFAR-100 (%)
Method
Architecture
Param.
10%
1%
10%
Standalone training
ResNet-18
11M
61.37
69.06
21.37
39.99
FedSimCLR
ResNet-18
11M
63.79
73.49
21.55
41.90
FedMoCoV1
ResNet-18
11M
60.57
73.95
21.83
43.49
FedMoCoV2
ResNet-18
11M
62.89
73.65
26.93
45.27
FedSimSiam
ResNet-18
11M
67.57
74.96
25.13
41.96
FedBYOL
ResNet-18
11M
70.48
76.95
30.21
47.07
FedU (Zhuang et al.
2021a
ResNet-18
11M
69.52
77.06
29.00
46.67
FedEMA (A =A=1
ResNet-18
11M
72.78
79.01
32.49
49.82
FedEMA (autoscaler, 0.7)
ResNet-18
11M
73.44
79.49
33.04
50.48
Standalone training
ResNet-50
23M
63.65
74.30
23.18
41,43
FedSimCLR
ResNet-50
23M
63,00
73.56
19.30
41.13
FedMoCoV1
ResNet-50
23M
61.85
75.53
22.12
46.43
FedMoCoV2
ResNet-50
23M
64.25
73.96
25.79
42.52
FedSimSiam
ResNet-50
23M
61.46
15.25
16.03
29.76
FedB YOL
ResNet-50
23M
69.99
76.69
26.57
45.46
FedCA Zhang et al.
2020a
ResNet-50
23M
28.50
36.28
16.48
22.46
FedU (Zhuang al.
2021a
ResNet-50
23M
69.76
80.25
28.42
48.42
FedEMA (A 1)
ResNet-50
23M
74.64
81.48
31.42
49.92
FedEMA (autoscaler, r 0.7)
ResNet-50
23M
72.52
80.68
29.68
50.75
BYOL (Centralized)
ResNet-18
11M
87.67
87.89
40.96
56.60
BYOL (Centralized)
ResNet-50
23M
89.07
89.66
41.49
60.23
Table 8: Top-1 accuracy comparison on various non-IID levels - the number of classes per client on the CIFAR-100 dataset. Update-both means updating both We and W with Wg.
of classes per client (%)
Method
2
10 (iid)
FoL
57.51
56.96
55.14
54.96
54.24
FedSimSiam
48.94
51.08
49.05
48.09
49.92
FedB YOL, update-both
49.53
54.17
51.50
52.70
53.41
Our ResNet architecture differs from the implementation in PyTorch (Paszke et al. 2017) in three aspects: 1) We use kernel size 3 x 3 for the first convolution layer instead of 7 x 7: 2) We use an average pooling layer with kernel size
4 before the last linear layer instead of adaptive average pooling layer; 3) We replace the last linear layer with two-layer MLP. The network architecture of MLP the same as the predictor.
B.3 TRAINING AND EVALUATION DETAILS
We implement FedSSL in Python using EasyFL (Zhuang et al. 2022). an casy-to-use federated learning platform based on PyTorch (Paszke et al. 2017). The tollowing the details of training and evaluation.
Training We Stochastic Gradient Descent (SGD) as the optimizer in training. We 0.032 as the initial learning rate and decay the learning with a cosine annealing (Loshchilov & Hutter 2017), which is also used in SimSiam. By default, train R 100 rounds with local
16

---

Published as a conference paper at ICLR 2022
Table 9: Comparison FedB without exponential moving average (EMA) and sop-gradien (sg) on the CIFAR datasets. FedBYOL w/o EMA and sg can hardly learn, but updating both Wa and W with W (update-botll enables it to achieve comparable results.
CIFAR- 10 (%)
CIFAR- 100 (%)
Method
IID
Non-IID
IID
Non-IID
FOL w/o EMA
54.11
50.20
23.82
25.83
FedBYOL w/o EMA and stop-grad
21.21
11.97
3.74
2.79
YOL w/o EMA and stop-grad, update-both
82.29
68.75
48.74
41.91
FedB YOL
84.29
79.44
54.24
57.51
KNN Acc. (%)
Rounds (%)
70-
Method
100
200
300
400
60
FedBYOL
FedEMA
50)
FedBYOL
79.08
82.23
83.77
86.09
50
100
150 200
250 300 Rounds
350 400
FedEMM (ours)
80.78
82.41
84.08
86.51
Figure 9: Comparison of FedB YOL and FedEMA various total training rounds R on the non-III setting of the CIFAR-10 dataset. FedEMA consistently outperforms FedB YOI..
epochs E 5 and batch size B 128 using K clients. We simulate training of clients on K NVIDIA V100 GPUs and employ the PyTorch Paszke et al. (2017) communication backend (NCCL) for communications between clients and the server. not specified, we use À by default or autoscaler with - 0.7 for FedEMA. As for experiments of FedU, we follow the hyper- parameters described in paper (Zhuang et al. 2021a).
Cross-silo FL Cross-device FL This paper primarily focuses cross-silo FL where clients are stateful with high availability. Clients can cache local models and carry these local states from round round. Extensive experiments demonstrate that FedEMA achieves the best performance under this setting. On the other hand, cross-device FL assumes there are millions of stateless clients that might participate in training just once. Due the constraints of experimental settings. the majority of studies conduct experiments with at most hundreds of clients (Wang et al. [2000] Jeong al. 202T). FedEMA
work under such experimental settings by caching the states of clients in the server. When the number of clients scales to millions, FedEMA degrades to FedB YOL that updates both encoders without keeping any local states.
Evalaution We assess the quality of learned representations using linear evaluation (Kolesnikov al. 2019 Grill et 2020) and semi-supervised learning (Zhai al,, 2019, Chen al. (2020a) protocols. We first obtain a trained encoder (or learned representations) using full training set for lin- ear evaluation and 99% or 90%
training set for munii-supevvieed learning (excluding the 1% or 10% for fine-tuning). Then, we conduct evaluations based on the trained encoder. For linear evalua- tion, we train a new fully connected layer on top of the frozen trained encoder (fixed parameters) for 200 epochs, using batch size 512 and Adam optimizer with learning rate 3e-3. For semi-supervised learning, we add new two-layer MLP on top of the trained encoder and fine-tune the whole model using 1% or 10% of data for 100 epochs, using batch size 128 and Adam optimizer with learning rate 1e-3. In both evaluation protocols, we remove the two-layer MLP of the encoder by replacing with an identity function.
C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
In this section, we provide more experimental results of algorithm comparisons and further analyze FedEMM in different data amounts, training rounds R, and batch sizes B.
17

---

Published as a conference paper at ICLR 2022
Top-1 Acc. (%)
80
Top-1 Accuracy (%)
83
22
70
81
Ours
FedBYOL
60
80
50
0.2 0.5 0.7 0.9
1.0
for Autoscaler
Mo
M==0 1 0.111.1 0.30.3 pu pu
p.=0.5 8.70.7
CIFAR-10
CIFAR-100
15m0.1
1..0.55 pum0.3 pl.0.5 p, ny ky
Encoder and Predictor
(a) for autoscaler
(b) Combinations of μo and jpp
Figure 10: Ablation study on for autoscaler and combinations of constant μu: (a) analyzes the impact of on performances: 2) presents top-I accuracy of using different combinations of constant Mo on the online encoder and constant Mp on the predictor.
Table 10: Top-1 accuracy comparison various batch sizes B on the non-IlI setting of CIFAR-10 dataset. The batch size should not be either too small or too large. Besides, FedEMA outperforms FedBYOL.
Batch Sizes B (%)
Method
16
32
64
128
256
512
FedB YOL
68.74
72.90
78.58
79.44
79.80
77.74
FedEMA (ours)
74.03
79.06
82.18
83.34
82.19
80.51
C.1 MORE EXPERIMENTAL RESULTS
Table presents top-1 accuracy comparison under linear evaluation of a wide range of methods CIFAR datasets using both ResNer-18 and ResNet-50. supplements the algorithm comparisons Section 3.4 and 5.1 Interestingly, FedMoCoV1 achieves good performances on IID settings of CIFAR- 100 dataset. Since decentralized data are mostly non-lID, we focus more on the non-IID setting. FedEMA outperforms all the other methods non-lID IID settings of CIFAR datasets. We use A = 0.8 when using ResNet-18 8 and À = when using ResNet-50.
Table 6 shows results of scaling to larger numbers of clients K with subsampling clients in each training round. We run two sets of experiments: 1) randomly selecting 5 out of 20 clients in each round with local epoch E = 5 and total rounds clients in each round with local epoch E 2 and total rounds R 800. We run FedEMA with autoscaler. FedEMA consistently outperforms FedBYOL with both encoders updated. We conduct these experiments using R8sNet-18.
400; 2) randomly selecting out 80
Table supplements the semi-supervised learning results on Table providing additional results using ResNet-50 as the network architecture for the encoder. FedEMA consistently outperforms all the other methods.
Besides. Table and 9 compare FedSimSiam, FedBYOL, and variances of FedBYOL to further demonstrate the insights from empirical studies. They supplement results in Table and Figure
C.2 FURTHER ANALYSIS
for Autoscaler We analyze the impact of on performances in Figure 10a Generally, using autoscaler with 7 (0, 1) better than FedBYOL (T = 0). The performance of yields worse results because only local knowledge are used in model update (the global knowledge is neglected) as discussed in Section 4 Besides, performances of 7 E [0.5, 1) are generally better other values, which verifies intuition discussed in Section These results also show that can achieve higher performance on the CIFAR-100 dataset on Table 3 by tunning T. We run experiments on non-III settings using R8.Net-18.
18

---

Published as a conference paper at ICLR 2022
Table 11:
Comparison of needed communication rounds to reach target accuracy using different local epochs E on the non-lll setting of the CIFAR-10 dataset. E = is unable to reach 80% in 100 rounds. A larger can reduce communication costs by increasing the computation cost.
Communiation (rounds)
Computation (epochs)
Target accuracy
E 1
==5
E
10
E=
E=
=
E 10
E 20
70%
90
40
10
8
90
200
100
160
80%
80
50
40
400
500
800
Top-1 accuracy comparison of various data amounts in clients and different numbers of clients. Increasing the number of clients does not improve performance, whereas increasing the data amount of clients results in better performance.
Table 12:
# of clients
K =
K = 10
K = 20
Data amount
10%
25%
50%
100%
50%
25%
FedB YOL
43.27
65.14
76.11
78.25
75.10
63.95
FedEMA (ours)
44.33
67.46
79.49
82.54
79.20
66.61
Constant μ To further illustrate the effectiveness of our dynamic EMA, we provide results of using different combinations of constant on the online encoder and constant μp on the predictor in Figure TOb] The results of f 0.9 and Mp 0.9 is only 54.52%, which is far lower than the others. Among these combinations, fo = 0.5 and μP = 0.3 achieve the best performance. It suggests that better performances may be achieved if we can construct different dynamic μ for the encoder and the predictor, while we leave this interesting insight for future exploration. Although good choices of constant No and H4p achieve better performance than FedB YOL., FedEMA consistently outperforms all these methods. These results complement Figure 7b in the main manuscript.
Impact of Training Rounds Figure FedB and FedEMA with increasing number of training (communication) rounds R. Performances of both FedB YOL and FedEMA increases as training proceeds and FedEMA consistently outperforms YOI. We run these experiments with À 0.5 for FedEMA the non-llD setting of CIFAR-10 dataset.
Impact of Batch Size B We investigate the impact of batch size in Table I0 The performances of batch size B = 128 and B 256 are similar, outperforming the other batch sizes. It indicates that the batch size should not either too small or too large. Besides, FedEMA outperforms FedB YOL. in all batch sizes. We run the experiments with autoscaler CIFAR- 10 dataset.
0.7) the non-IID setting of the
Communication Computation Cost Table 11 shows the needed communication rounds and computation epochs to reach a target accuracy using different local epochs E with FedEMA. In- creasing reduces communication cost as the needed rounds decrease, but it generally requires higher computation cost. For example, compared with E = 5 that needs 80 rounds to reach 80% with 400 epochs of computation, E 20 only uses 40 rounds but needs 800 epochs computation cost. These results indicate the trade-off between communication cost and computation cost.
Table 12 shows that increasing the data amount improves the performance signifi- cantly. By default, we split the CIFAR- 10 dataset into 5 clients, each client contains 10,000 training images, denoting as 100% data amount. As a result, p% data amount means that each client contains 10, 000 5% images. For example, 25% data amount means that each client contains 2,500 images. With lesser data points each client, we can construct more clients to conduct training as the to- tal data amount is fixed. Table 12 shows that when the data amount is same in clients, increasing the number of clients in each training round do not improve performance. However, increasing the data amount in each client increases the performance significantly. These results indicate that it is important for clients to have sufficient data to participate in training in FedSSL.
Data Amount
19


Reviews:
[{"review_id": "UH5sYmcGDk4", "reviewer": "Reviewer_wF5G", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "NOVELTY & SIGNIFICANCE\n\nThe key innovation of this paper is the specific formulae -- Eq. (3) -- that computes the decay rate in the EMA recipe which is driven by a scaler and the normalized L2-distance between the local & global online net. While this appears minimal in terms of algorithmic novelty, the empirical validation is quite extensive and shows strong improvement over existing baselines. \n\nThus, despite the simple technical innovation, I have a positive opinion of this paper. In fact, I think this is a pretty well-structured investigative process that arrives at a fine-tuned version of federated SSL with improved performance. It also re-affirms or discovers relatively useful empirical practices that boosts the performance of federated SSL, particularly in highly non-IID settings. \n\nOn another note, however, it seems the performance would depend on whether we have a correct choice of lambda. For Tables 3-4, how would it impact performance in the IID setting when lambda is set to 1 instead? I think this is likely a substantial restriction of the proposed work in case we do not know the data heterogeneity of the clients. Could the authors comment more on this? Any practical guidelines on choosing lambda here? Fig. 1 suggest a range of (0,1) for lambda but does it apply generically or only to CIFAR data?\n\nAnother look at the supplement results in Table 6 shows that the performance is also sensitive somewhat to the choice of the architecture. How would this impact the default range of good values for lambda?\n\nEXPERIMENT\n\nThe experiment is quite extensive & interesting. But I do have a few follow-up questions:\n\nIn Tables 3 and 4, what are FedEMA's performance on IID setting when lambda = 1. \n\nHow many runs were used to produce the average accuracy and standard deviation?\n\nWhat is the exact non-IID setting used in Tables 3 and 4?\n\nThere seems to be a decrease in terms of accuracy in CIFAR-100 when one moves from non-IID to IID settings. This is the opposite of what we saw in CIFAR-10 settings. Could the authors comment on this?\n\nIt seems some baselines such as FedMoCoV1 and FedMoCoV2 were missing from Table 7. Could the authors provide the missing info?", "summary_of_the_paper": "This paper investigates a generic federated SSL recipe that applies FedAvg to a range of existing SSL works including SimCLR, MoCo, BYOL and SimSiam. Each of these SSL blocks comprises two encoding networks, including an online net and a target net. The two nets were trained via optimizing a similarity loss such that the distance between encodings of different augmented versions (e.g. rotation) of the same input is small and vice versa. Depending on the specific SSL block, the parameterization of the two nets might be identical (SimCLR, SimSiam) or not (MoCo, BYOL). At each iteration, the online net of each client is uploaded to a server to be averaged. Then, the (global) averaged version is sent back to each client. Each client then reset its online net as a weighted average of the global and local version of the online net (to account for potential data heterogeneity), and continue updating the SSL block via gradient updates and so on.    \n\nHere, the main contribution of the paper is a series of ablation studies (Table 1, Figure 2, Table 2, Figure 3) that measure the isolated impact of several fundamental components of the aforementioned SSL blocks (e.g., SimCLR, SimSiam, MoCo and BYOL). This includes the predictor, stop-gradient, exponential moving average (EMA), non-identical online and target encoding net. The results indicate that BYOLD contains all the essential elements (predictor, EMA and non-identical online & target net & no stop-gradient) so the federated SSL recipe is rotated towards having BYOL as the model choice. Then, a customized, divergence-aware EMA is proposed for this recipe that is shown to improve substantially over existing FL-SSL baselines (Tables 3-4; Tables 6-7). There is also a bit of ablation study that measures the isolated impact of the new EMA which is also substantial as shown in Figure 5.", "summary_of_the_review": "This is an empirical paper. The key contribution is an investigative, ablation studies that build up intuitions and practical guidelines to fine-tune the generic federated SSL framework. The fine-tuned version is shown to have significant performance improvement. But on another note, this success might be dependent on a correct choice of lambda which is only possible if one knows the data heterogeneity of the clients. Perhaps this is a key restriction of the method. It would be great if the authors can expand on this during the rebuttal.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I1w7vVNWOaH", "reviewer": "Reviewer_Nu54", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper follows a clear red line and is easy to follow. The empirical insights are interesting and mostly well motivated and explained. I have a few open question related to the paper and some suggestions which would make this a very good paper. \n\nThe most successful methods, as well as FedEMA rely on the availablitiy of a local target network. I.e. the clients are state-full between rounds. In the general (sightly more mature) supervised FL literature, this setting is called the 'cross-silo' setting as opposed to the cross-device setting, in which we assume the federation to consist of millions of devices, each of which might take part in the federation just once. This distinction is missing in this paper and I encourage the authors to make this more explicit. \nRelated to this comment is my biggest issue with the experimental claims in this paper. For the federated setting $K=5$ clients is not enough. Even for a paper targeting cross-silo FL, I would suggest to experiment with more clients to make the insights more valuable. Table 11 considers up to $K=20$ clients, however the authors also reduce the amount of data per-client (in addition to spreading the available data over a larger amount of clients, as far as I understand). \nAlong the same lines, FL usually does not assume full client availability, i.e. not all $K$ clients are available for updates at every round. Subsampling clients is especially interesting in the context of state-full clients, since their local models might become stale. Also as the amount of data locally becomes smaller, methods that rely on a local target encoder might overfit, something that FedEMA might remedy. I would encourage the authors to explore that angle or clearly position their work in terms of which niche of the heterogeneous  FL space they target.\nHow representative of SSL methods are small-scale data-sets such as cifar10/cifar100? Maybe the authors could compare their scale to the literature. I encourage them to evaluate some experimental insights on larger data-sets such as Imagenet. \n\nServer-side optimization benefits from more advanced update rules than simply averaging client updates. E.g. https://arxiv.org/abs/2003.00295 explore this design space. Is there a reason the authors choose to do averaging only? Otherwise I'd encourage the authors to explore the influence of these optimisers. \n\nYour discussion of the scaler $\\lambda$ mentions that setting $\\lambda>1$ leads to $\\mu=1$. I figure that would depend a lot on the federation characteristics that determine the divergence between local and global model over time. A such, it is a very general statement. Maybe you could track the value of $\\mu$ over time for different settings of $\\lambda$ for your experiments.\n\nExperimental setting:\n- What hyper-parameter ranges have the authors explored for their experiments?\n- Is centralised BYOL done with same hyperparameters? E.g. can you do centralised performance with small batch-sizes?\n- Is linear evaluation done on the locally fine-tuned models locally or on the most recent global model with data at the server?\n\nMisc:\n- What is the difference between the pink line/model in Figure 3 and the last row in Table 2?\n- What time is indexed by $t$ in 3.2? Is that local time-steps per client or the global update steps? I.e. is the target encoder updated every mini-batch or every global round only? How is this different from equation 1 and 2? Are you describing 2-level updates? Algorithm 1 suggests the answer, but I encourage to make this more clear in the text. I also suggest adding superscript $0$ to lines 13/14 in Alg 1. \n\nDisclaimer: I am not an expert in SSL methods, which is why I appreciated Appendix A and corresponding discussions in the main text.\n", "summary_of_the_paper": "The authors introduce an abstraction of self-supervised learning (SSL) algorithms in the federated setting, which they term FedSSL. They empirically evaluate several existing SSL methods in the federated setting and derive a new algorithm, FedEMA based on their insights. ", "summary_of_the_review": "This paper provides an interesting insight into self-supervised training in FL. It motivates its proposed method by empirical insights on cifar10(0). My major issue with the paper is that its empirical scale is quite small in terms of what makes the FL scenario challenging. On the other hand, they have very broad empirical analyses. \nIf the authors extend their experimental evaluation along these scale-related dimensions and clarify their positioning within the FL literature, I will consider raise my score.\n\n=== Post-Rebuttal ===\nThe authors have addressed my concerns sufficiently and I have updated my score accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AvRgCptzuXn", "reviewer": "Reviewer_AcAS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think overall, the idea to solve the non-IID problem in Federated learning is good. However, it is not immediately clear to me why the solution works from a theortical perspective. The authors have demonstrated that the approach is superior by a quite significant margin emperically on CIFAR-10 and CIFAR-100.\n\n(1) The superscript and subscript of W have been defined. But W itself has not been defined in the paper. From the context, I assume these are the parameters of your model.\n\n(2) How do you select \\lambda?", "summary_of_the_paper": "The authors propose a new approach called Federated Divergence-aware Exponential moving Average update (FedEMA) to avoid the IID assumption. FedEMA is built onto of FedSSL which is a framework for self-supervised learning in a Federated Learning context. The authors proposes a new approach to fuse the local and global knowledge effectively through a EMA update, where the decay rate of EMA is dynamically measured by a divergence.\n", "summary_of_the_review": "Overall an interesting idea. However, I thought the novelty proposed is a bit incremental and should have been more thoroughly investigated. For example more experiments could have been run and it would be good to theoretically show why this works. Although from Section 3.5, I can understand the intuition behind why this works. Overall, I felt that the contribution that this paper proposes does not meet the requirement to be published at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-XEfYn43JBR", "reviewer": "Reviewer_48jX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strengths\n1. Unsupervised FL is a well-motivated and timely topic.\n\n\n* Weaknesses\n1. This paper is a simple combination of BYOL and FedAvg. I cannot see significant novelty in both BYOL and FedAvg-like algorithm. Especially, its relationship to ICCV 2021 paper FedBYOL [1] is not fully discussed. \n\n2. Why is SimSaim [2] not used as the SSL framework? It has better interpretability in optimization; it requires smaller batch size; it doesn't require moving averaging. I believe the performance of SimSiam would be better than BYOL in FL context.\n\n3. The definition of unlabeled non-I.I.D. dataset is not clear. According to Section 3.3, the authors use CIFAR-10 and CIFAR 100 to partition the dataset in an unbalanced manner with respect to the number of classes in each client. This is contradicted to the unsupervised assumption because we don't know users' labels, then it seems we cannot claim their data is non-I.I.D. in terms of the label. A careful definition of \"unsupervised non-I.I.D. data\" is required in this work.\n\n4. Note that FL clients should not be stateful. When numerous clients exist, we need to do client sampling for a new round of training. The newly sampled client do not have any cached states to conduct adaptive optimizer or moving averaging-like stateful algorithms [4]. Please explain how Algorithm 1 can work in cross-device FL. \n\n5. The batch size is a challenge in FL context since FL clients normally are resource-constrained. Batch size = 128 is too big for smartphones.\n\n6. The source code is not provided for reproducibility. I doubt some experiment results and would like to reproduce some results to confirm the efficacy. \n\n7. It would be better to discuss the privacy/security concern of the proposed FedSSL framework.\n\n---\n[1] Collaborative unsupervised visual representation learning from decentralized data. ICCV 2021.\n\n[2] Exploring Simple Siamese Representation Learning. CVPR 2021.\n\n[3] Federated Reconstruction: Partially Local Federated Learning. NeurIPS 2021.\n", "summary_of_the_paper": "This paper combines self-supervised learning framework BOYL and FedAvg to improve the performance of unlabeled non-IID datasets.", "summary_of_the_review": "I suggest NOT accepting this paper at the current version, given that a few issues mentioned above are critical.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on self-supervised learning (SSL) in the federated learning setting (FedSSL). Research in this area is timely and of significance. The authors phrase their work as primarily being an empirical study providing insights into the building blocks of FedSSL. The evaluation in the paper is quite thorough and the authors have been active in a detailed exchange regarding questions raised in the reviews. I would encourage the authors to fully implement the changes they promised into the revised manuscript and work towards timely release of open-source code. (I appreciate internal policies of various institutions, but I do agree with the reviewers that it is more important that the code and experimental details be made public for papers such as this one, compared to some other papers.) I have chosen to disagree with some of the concerns raised in one of the reviews, in particular, I do agree with the authors that insights into the building blocks through empirical studies is a significant contribution, and also that FedEMA is a novel contribution. The discussion on this forum will remain for interested readers to come to their own conclusions about the relative performance of various methods.
Decision: Accept (Poster)