Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?
Authors: Lirui Wang
Kaiqing Zhang
Yunzhu Li
Yonglong Tian
Russ Tedrake
Venue: ICLR 2023 poster (2023)
Keywords: Decentralized Learning
Heterogeneous and Unlabeled Data
Federated Learning
Self-Supervised Learning
Representation Learning

Abstract: The success of machine learning relies heavily on massive amounts of data, which are usually generated and stored across a range of diverse and distributed data sources. Decentralized learning has thus been advocated and widely deployed to make efficient use of distributed datasets, with an extensive focus on supervised learning (SL) problems. Unfortunately, the majority of real-world data are unlabeled and can be highly heterogeneous across sources. In this work, we carefully study decentralized learning with unlabeled data through the lens of self-supervised learning (SSL), specifically contrastive visual representation learning. We study the effectiveness of a range of contrastive learning algorithms under a decentralized learning setting, on relatively large-scale datasets including ImageNet-100, MS-COCO, and a new real-world robotic warehouse dataset. Our experiments show that the decentralized SSL (Dec-SSL) approach is robust to the heterogeneity of decentralized datasets, and learns useful representation for object classification, detection, and segmentation tasks, even when combined with the simple and standard decentralized learning algorithm of Federated Averaging (FedAvg). This robustness makes it possible to significantly reduce communication and to reduce the participation ratio of data sources with only minimal drops in performance. Interestingly, using the same amount of data, the representation learned by Dec-SSL can not only perform on par with that learned by centralized SSL which requires communication and excessive data storage costs, but also sometimes outperform representations extracted from decentralized SL which requires extra knowledge about the data labels. Finally, we provide theoretical insights into understanding why data heterogeneity is less of a concern for Dec-SSL objectives, and introduce feature alignment and clustering techniques to develop a new Dec-SSL algorithm that further improves the performance, in the face of highly non-IID data. Our study presents positive evidence to embrace unlabeled data in decentralized learning, and we hope to provide new insights into whether and why decentralized SSL is effective and/or even advantageous.

Published as a conference paper at ICLR 2023
DOES LEARNING FROM DECENTRALIZED NON-IID UNLABELED DATA BENEFIT FROM SELF SUPERVISION?
Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, Russ Tedrake MIT CSAIL
ABSTRACT
The success machine learning relies heavily on massive amounts of data, which are usually generated and stored across range of diverse and distributed data sources. Decentralized learning has thus been advocated and widely deployed make efficient of distributed datasets, with extensive focus on super- vised learning (SL) problems. Unfortunately, the majority of real-world data are unlabeled and can be highly heterogeneous across sources. In this work. we carefully study decentralized learning with unlabeled data through the lens of self-supervised learning (SSL), specifically contrastive visual representation learning. We study the effectiveness of range of contrastive learning algorithms under a decentralized learning setting, on relatively large-scale datasets including ImageNet- 100, MS-COCO, and new real-world robotic warehouse dataset. Our experiments show that the decentralized SSL (Dec-SSL) approach is robust to the heterogeneity of decentralized datasets, and learns useful representation for ob- ject classification, detection, and segmentation tasks, even when combined with the simple and standard decentralized learning algorithm of Federated Averaging (F (eeAAy). This robustness makes it possible to significantly reduce communi- cation and reduce the participation ratio of data sources with only minimal drops in performance. Interestingly, using the same amount of data, the repre- sentation learned by Dec-SSL can not only perform on par with that learned by centralized SSL which requires communication and excessive data storage costs, but also sometimes outperform representations extracted from decentralized SI. which requires extra knowledge about the data labels. Finally, we provide theo- retical insights into understanding why data heterogeneity is less concern for Dec-SSL objectives, and introduce feature alignment and clustering techniques develop a new Dec-SSL algorithm that further improves the performance, in the face of highly non-IID data. Our study presents positive evidence to embrace labeled data in decentralized learning, and we hope to provide new insights into whether and why decentralized SSL effective and/or even advantageous.
INTRODUCTION
The success of machine learning hinges heavily the access large-scale and diverse datasets. In practice, most data are generated from different locations, devices, and embodied agents, and stored in distributed fashion. Examples include a fleet of self-driving cars collecting a massive amount of streaming images under various road and weather conditions during everyday driving, or individuals using mobile devices to take photos of objects and scenery all over the world. Besides being large- scale, these datasets have two salient features: they are heterogeneous across data sources, and mostly unlabeled. For instance, images of road conditions, which are expensive to label, vary across cars driving on highways vs. rural areas, and under sunny vs. snowy weather conditions (Figure 19).
Methods that can make the best use of these large-scal scale distributed datasets can significantly advance the performance of current machine learning algorithms and systems. This has thus motivated surge of research in decentralized learning/laarning from decentralized data? (Konečný al., 2016; Hsich et al., 2017; McMahan et al., 2017; Kairouz al., 10 Nedic, 2020), where usually global model is trained on the distributed datasets using communication between the local data sources and
Code is available at https: / thub. com/1 / 1 krui wwwbec--S.. Hereafter, often decentralized learning shorthand for learning from decentralized data.

---

Published as a conference paper at ICLR 2023
centralized server, or sometimes even only among the local data sources. The goal is typically to reduce or eliminate the exchanges of local raw data to save communication costs and protect data privacy. How to mitigate the effect of data heterogeneity remains one of the most important research questions in this area (Zhao et al., 2018; Hsieh et al., 2020; Karimireddy et al., 2020; Ghosh et al., 2020; Li et al., 2021a), as it can heavily downgrade the performance of decentralized learning. Moreover, most existing decentralized learning studies focused on supervised learning (SL) problems that require data labels (McMahan al., 2017; Jeong al., 2020; Hsieh al., 2020). Hence, it remains unclear whether and how decentralized learning can benefit from large-scale, heterogeneous, and especially unlabeled datasets typically encountered in the real world.
On the other hand, people have developed effective methods of learning purely from unlabeled data and demonstrated impressive results. Self-supervised learning (SSL.), technique that learns representations by generating supervision signals from the data itself, has unleashed the power of unlabeled data and achieved tremendous successes for a wide range of downstream tasks in com- puter vision (He et al., 2020; Chen et al., 2020; He et al., 2021b), natural language processing (Devlin et al., 2018; Sarzynska- Wawer et al., 2021), and embodied intelligence (Sermanet et al., 2018: Florence et al., 2018). These SSL algorithms, however, are usually trained in centralized fashion by pooling all the unlabeled data together, without accounting for the heterogeneous nature of the decentralized data sources. Very recently, there have been a few contenporaaceous/oonuureen attempts (He et al., 2021a; Zhuang et al., 2021; 2022; Lu et al., 2022; Makhija et al., 2022) that bridged unsupervised/sell-spppees supperised learning and decentralized learning, with focuses on design- ing better algorithms that mitigate the data heterogeneity issue. In contrast, we revisit this new paradigm and ask the question:
Does learning from decentralized non-IID unlabeled data really benefit from SSL?
We focus on understanding the use of SSL in decentralized learning when handling unlabeled data. We aim to answer whether and when decentralized SSL (Dec-SSL) is effective (even combined with simple and off-the-shelf decentralized learning algorithms, e.g., F FedAve (McMahan et al., 2017)); what are the unique inherent properties of Dec-SSL compared to its SL counterpart; how do the properties play a role in decentralized learning. especially with highly heterogeneous data? We also aim to validate our observations on large-scald and practical datasets. We defer more detailed comparison with these most related works to SA.
In this paper, we show that unlike in decentralized (supervised) learning. data heterogeneity can be less concerning in decentralized SSL, with both empirical and theoretical evidence. This leads to more communication-efficen even outperform their supervised counterpart that assumes the availability of label information. Among the first studies bridge decentralized learning and SSL. our study provides positive ev- idence to embrace unlabeled data in decentralized learning, and provides new insights into this setting. We detail our contributions as follows.
and robust decentralized learning schemes, which can sometimes
Contributions. (i) We show that decentralized SSL, specifically contrastive visual representation learning, a viable learning paradigm to handle relatively large-scale unlabeled datasets, even when combined with the simple F FedAvg algorithm. Moreover, we also provide both experimental evi- dence and theoretical insights that decentralized SSL can be inherently robus! to the data heterogene- ity across different data sources. This allows more local updates, and can significantly improve the communication efficiency in decentralized learning. (ii) We provide further empirical and theoretical evidences that even when labels available and decentralized supervised learning (and associated representation learning) is allowed, Dec-SSL still stands out in face of highly non-llI) data. (iii) To further improve the performance of Dec-SSL, design new Dec-SSL algorithm, FeatARG by using an iterative feature alignment and clustering procedure. Finally, we validate our hypothesis and algorithm in practical and large-scale data and task domains, including a new real-world robotic warehouse dataset.
PRELIMINARIES AND OVERVIEW
Consider docentralized learning setting with K different data sources, which might correspond to different devices, machines, embodied agents, or datasets/users that can generate and store data locally. The goal is collaboratively solve a learning problem, by exploiting the decentralized data from all data sources. More specifically, consider each data source & E [K] has local dataset
[Pul and Fht X C Rdd are identically and independently distributed (IID) samples Da = ( [1
2

---

Published as a conference paper at ICLR 2023
from probability distribution Dj. i.e., k,,s Dx. Note that the distributions Dr: is in general different across data sources k, yielding an overall heterogeneous (i.e., non-lID) data distribution for the data from all the sources. Let D UkEjK] DR denote the set of all data samples. Moreover, are interested in situations where no label is provided alongside the data To effectively utilize the large-scale unlabeled data, we resort to self-supervised learning approaches.
Specifically, SSL approaches extract representations from these unlabeled data, by finding an em- bedding function Jw X - RT, where to is the parameter of the embedding function. = = (w)( the representation vector that can be useful for downstream tasks, e.g., classification or segmenta- tion. We summarize several popular SSL approaches here that will be used later in the paper.
Self-supervised representation learning. Now consider given data source [K]. There are two popular methods in the SSL community. In contrastive learning (Chen et al., 2020; He et al., 2020) specifically, sample x is used to provide supervision signals along with two generated positive samples x* and z (overloaded for notational simplicity) and (possibly multiple) negative samples r sampled from the training batch. The goal of SSL is to find an embedding (w that makes x and at close, while keeping and x apart, if negative samples are used.
One commonly used loss for SSL is the InfoNCE loss (Oord et al., 2018), which has been used in popular SSL approaches as SimCLR (Chen al., 2020) and MoCo (He al., 2020):
1 DR
PP
cxp(-
Cn(w)=
log
(Dp(-D((((z..) for E (xpp:)//))) exp(- (u (2.1)
where 7 > 0 is a temperature hyperparameter, j is the index for negative samples, (.: .) is a dis- tance function such as the cosine distance, i.e., (D 22) approaches, such as BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), remove the terms related to negative samples in (2.1). These methods also add an additional function g, the feature predictor, which only applies to x to create an asymmetry and to avoid the collapsed solutions. This usually leads to the following objective: [x(w) Do. EPP ((((f(())) (xt.))) In our experiments, make use of both losses and the SSL. approaches associated with them.
21 |a 12 Some other effective SSL
Decentralized SSL. To exploit the heterogeneous data distributed at different locations/devices, decentralized SSL optimizes the following global objective:
Del
min
Cu(),,
(2.2)
KE[K]
which can be solved using many existing decentralized learning algorithms. For instance, FedAvg (McMahan et al., 2017) is one of the most representative, easy-to-implement, and communication- efficient decentralized learning algorithms which optimizes this objective without data-sharing among data sources. At each iteration 4, the server first samples a set of data sources M, with size [Mel pK and run local update steps on each of the local dataset. Then, each data source M₁ sends back the updated local model weight w to the central server, and the server aver- ages them to be the global model w/+1 = [.. Σkem, wk 1.6 for the next round I. + 1. The server then broadcasts the global model to each data source to reset w 1+1.0 as w The number of local updates (6) determines the communication efficiency (larger 8 means less communication); in the experiments, we use E to denote the number of epochs of local updates (as a surrogate for 8). Both E and the participation rate P are important factors that determine the efficiency of decentralized learning. The learned representation (w) can then be used in downstream supervised learning tasks. There are many real-world applications of decentralized SSL, including self-driving cars, warehouse robots, and mobile devices. A further discussion can be found in Appendix $D.
2.1 OVERVIEW OF OUR STUDY
Terminology & setup. We separate our experiment pipeline into representation learning (pre- training phase) and downstream evaluation (evaluation phase). Our main focus is on the afore- mentioned Dec-SSL approach. We use FedAvg (McMahan al., 2017) with SimCLR (Chen al., 2020) as the default method. Moreover, we will also compare with settings where the label in- formation is available, i.e., the classical decentralized (supervised) learning, which should be more favorable for learning. See Figure I for summary of different settings. The first setting is Dec-
3

---

Published as a conference paper at ICLR 2023
SL: we simply run FedAvg the decentralized labeled data, for end-to-end classification.Do DeriicatttionDe-SSLS. does not learn representations explicitly, and serves as a natural baseline when labels are available. The second setting is representation learning from Dec-SL, where train supervised learning with FedAvg, and then use the feature extractor network as the backbone for downstream tasks. This way, we can also learn the representation from decentralized labeled data, and make the comparison with Dec-SSL more fair, since both are learning features for various downstream tasks. We term this setting as Rec-SLRep.
Dec-SL
SeepSLRep
Dec-SSL
The evaluation phase tests the representations from Dec-SSL or Dec-SLRep. We consider two protocols in the evaluation phase: linear probing for image classification (Zhang et al., 2016) and finetuning for object delectioon/eemmntatioon (Doersch al., 2015). For classification, we train a linear classifier on top of the frozen pretrained network and evaluate the top-1 classification accuracy, For object detectioo//semmenatiinn we finetune the network by using the pretrained weights as initialization and training in an end-to-end fashion, and then we evaluate the mean Average Precision (mAP) metric. Downstream tasks are performed on centralized train and test dataset. Please refer Appendix SC.1 for implementation details and Table 3 for experiment setups.
Labeled Data
Labeled Duta
Unlabeled Dota
D1
D2
D3
DR
Exla.
Pretraining
D
Evaluation
Tisk
Tesk
Figure I: Comparisons among Dec-SL, Dec-SLRep, and Dec-SSL.
Questions of interest. Through extensive experiments on large-scale datasets, and theoretical anal- ysis in simplified settings, we seek to answer the following questions: (i) How well can decentralized SSL, even instantiated with the simple FedAvg algorithm, rival the performance of its centralized counterpart, and handle the non-IIDness of decentralized unlabeled data? (ii) Is there any unique and inherent property of Dec-SSL, compared to its supervised learning counterpart; how and why may the property benefit decentralized learning, even when the label information is available? (iii) Is there way to further improve the performance of Dec-SSL in face of highly non-llD data? Our hypothesis is that SSL, whose objective is not particularly dependent on the r to y mappings, learns relatively uniform representation across decentralized and heterogeneous unlabeled datasets, thus leading to more efficient and robust decentralized learning schemes. We aim to validate this hypoth- esis and answer these questions in the following sections.
3 DEC-SSL IS EFFICIENT AND ROBUST TO DATA HETEROGENEITY
We first seek to address question (i) in $2.1 how well decentralized SSL performs, in face of non- IID and decentralized unlabeled data. To this end, we first introduce the notion of data heterogeneity in decentralized learning. which usually categorized as input heterogeneity, label distribution her- erogeneity, and the heterogeneity in the relationships between the features and labels, respectively (Hsieh et al., 2020). We create label heterogeneity by distributing each data source with different proportion of classes; construct the heterogeneity via either sampling from Dirichlet process with hyperparameter a or via skewness partitioning (Hsich et al., 2020) with hyperparameter 3. We also create input heterogeneity by leveraging the feature space of pretrained network on the data. See $C.2 for more details on how we create data heterogeneity across data sources.
3.1 EXPERIMENTAL OBSERVATIONS
CIFAR classification under differrntt types of non-IIDness. In this experiment, we construct input and label mon-Is using 5 data sources the CIFAR-10 (Krizhevsky al., 2009) dataset based on the Dirichlet Process. The sources of non-Illnee are the feature clusters and labels, respectively. We control parameter a to create datasets from very IID (each data source has roughly uniform distribution over 10 classes / 5 feature clusters) to very non-lII (each data source has data from 2 classes feature clusters). Recall that denotes the number of epochs for local updates and p denotes the participation ratio of data sources at each round. We use E 50 epochs of local updates in this experiment, which is equivalent to around = 1000 iterations, i.e., each local data source updates 50 epochs independently before averaging. The results are shown in Figure 2. Surprisingly,
4

---

Published as a conference paper at ICLR 2023
Feature Non- Mon-sIDeess (x) = 1,K = 5 50]
Label Non-IIDness (Y) 1,K =, 50]
1.0
centra ized SL
centiwied SSL
nonilld. Dec
0.5
0.5
nomi.id. Dec-SL
5=5
0.01 0=5
0.01
Figure 2: SSL objective is robust to different types of X and Y heterogeneity on the CIFAR-10 dataset. the pie chart below, each pie denotes one data source, and color denotes the sample number of one source of non-IIDness (left to right, more non-IID). We observe that Dec-SSL is surprisingly robust to the non- -IDesssesss both input (X) and label (Y) and also behaves closer to its centralized counterpart. Y-axis denotes accuracy.
the performance of downstream classification, with representations trained using decentralized SSL, is very insensitive to the non- -In-sss across the datasets and only bears a slight performance drop. This robustness over data non-IIDness is encouraging, and stands in sharp contrast with most exist- ing decentralized supervised learning algorithms, which are known to suffer from the data hetero- geneity in general (Hsieh et al., 2020). As baseline, we consider the classical decentralized SL. approach of F FedAvg, trained over the same non-llI data, but with label information. Indeed, the performance of decentralized SL can drop significantly as the non-IIDness increases. Finally, we note that the simple use of FedAvg in SSL can achieve performance comparable to the centralized SSL, showing that Dec-SSL is an effective decentralized learning scheme to handle unlabeled data.
Finetuning ImageNet representation for COCO detection. In this experiment, representations learned from ImageNet to COCO detection benchmark (Lin et al., 2014) with the Detectron pipeline (Girshick al., 2018). Specifically, we use ImageNet-100 100 with ResNet-18 and 1x training schedule for Mask R-CNN (He et al., 2017) with a ResNet18 FPN being the back- bone. Compared the contemporary works (Zhuang et al., 2022: Lu et al., 2022) on federated self-supervised learning, our setup is more relevant to real-world applications, as it works on larger- scale and more practical datasets and tasks.
finetune the
We run Dec-SSL on ImageNet-100 dataset with 5 data sources, and with E = 1 epoch of local updates, which corresponds to around 8 500 local updates, to learn the global representation using F FedAvg. On Table left, we observe that the representation from Dec-SSL almost reaches the performance of the representation from centralized SSL and improves upon baselines that train the model from scratch, i.e., the no pretrain row. This conveys that SSL can learn useful representations in decentralized settings, avoiding the heavy communication cost of centralized learning.
Decentralized SSL for real-world package segmentation. The issue of data heterogeneity and communication efficiency is significant for real-world applications such as those in Amazon ware- houses, whose fleets of working robots can generate millions of images per day (sce Figure 21 for an illustration). We provide details about the Amazon dataset in $D.1. We use data from one sample warehouse site Amazon, and split the data based the session ID (which is usually a sequence of days). Each decentralized learner is only allowed to access the local data at one session, which is equivalent the non-llI case where skewness 3 O. We then deploy decentralized self-supervised learning on subset of the enormous warehouse data, which has around 80000 images with contour labels output by the Amazon work-cells. We use SimCLR with FedAvg and communication effi- ciency E- = 1 number of local update epochs, as the pretraining method.
On the right subtable of Table 1, we compare different ways initialize weights for finetuning, and show that the representations learned from decentralized SSL outperforms training from scratch and even matches centralized SSL on the Amazon dataset. We also experiment with finetuning segmentation task using Mask R-CNN on different fractions of the data, and show that Dec-SSL can further improve the performance of training from scratch, when there is no as much labeled data.
3.2 THEORETICAL INSIGHTS
We now provide some theoretical insights into why the objective of Dec-SSI. leads to more robust performance in face of data heterogeneity. In particular, we analyze the property of the solutions to the local and global objectives of Dec-SSL in a simplified setting, and show that the global objective is not affected significantly by the heterogeneity of local datasets. Our setup is inspired by the very recent work (Liu et al., 2021), where the effect of imbalanced data in centralized SSL was studied in simplified setting, In particular, to a decentralized and 2K- one, carefully design the generation of data distribution across data sources, and establish analyses for both local and global objectives in decentralized SSL. We also
generalize the centralized and 3- way classification setting
5

---

Published as a conference paper at ICLR 2023
D1 aa..d
D2 Fa..d
Dk
62
ex+1
SL
SSL
SSL
6,
e1.x-1
Figure 3: The learned feature space of SSL is more insensitive heterogeneity under the linear settings. $3.2, we consider a decentralized learning setting where each local dataset has a skewed distribution with most data points (each color is a class) concentrated on one axis. Each basis vector inside the sphere denotes how well it is represented in the learned subspace. For contrastive objectives, the learned feature space (green sphere) of the local model more uniform and close to the global model. On the other hand, the SL objective (red sphere) tends to overfit to local dataset, and the learned feature spaces become hetcrogeneous.
ImageNet-100
MS MOCC
Amazon
Amazon (AP
Pretrain
Ap
Apmk
Pretrain
100%
10%
1%
pretrain
20.5
19.4
no pretrain
60.8
59.2
47.0
Central-SI-Rep
21.2 (4-0.)
20.1 (4-07)
Central-SSI.
61.6 (-0x)
60.4 (+122
49.5 (+225)
Central-SSI.
23.2 (4-2.)
22.1 (422.)
Dec-SSI.
61.2 (-0.4
60.1 (40.9)
48.8 (-118)
Dec- -LRep
19.8 (-0.7)
19.7
(+0.3)
Dec-SSI.
22.1 (4-1.6)
20.7
(41.3)
The model is pretrained on ImageNet-100 100 (Tian et al., 2020a) dataset and then finetune MS-COCO with metries bounding- bbx mAP (AP"") and mask mAP (AP*)). Right: Finetuning results on the Amazon package segmentation dataset with representations pretrained on the Amazon dataset. We observe that Dec-SSL reaches similar performance (AP*)) as centralized SSL and also outperforms training from scratch. Note that 100%, 10%, 1% denote the portion of the data used for finetuning.
Table 1: Left: Object detection and semantic segmentation finetuned on COCO:
improve some analysis therein, and design new metrics to characterize the performance adapted to the decentralized setting. Due to space limitation, we include abridged introduction here, and defer more details to Appendix $E.
Setup. Consider a Dec-SSL problem with K data sources. Similar to the SimSiam approach, we first augment x, an anchor sample from the dataset, by sampling E ~ NN I) IID from the Gaussian distribution. Consider the linear embedding function (w) wx, where ma and m 2 2K. The SSL objective for data source is given by
(w) == [[w((RR + T + (k, + 2 wwl.
(3.1)
where Ê is taken expectation over the empirical dataset Thi ~ DR. and the randomness of Éx.; and S. Morcover, recall the global objective is given in (2.2). Note that (3.1) instantiates SimSiam loss with the negative inner-produce (a, b) as the distance function D(a, b) and no feature predictor, and with a regularization term for mathematical tractability, as in Liu et al. (2021).
Data heterogeneity. The K data sources collaboratively solve (2.2) to learn representation for 2 2-way classification task. The K local datasets are generated in a way that for each fixed k E [K]. the labels are skewed in that data from classes 2k 1 and 2k constitute the majority of the data, while other classes are rare, or even unseen. More details on the specifications of data heterogeneity can be found in I. We visualize the heterogeneity of the data distributions in Figure 3.
To compare the representations learned across data sources and that learned from jointly solving (2.2), we introduce the following definition on the representability of the representation space.
Definition 3.1 (Representability vector). Let S C Rdd be the subspace spanned by the rows of the learned feature matrix w E Rmxd, where the embedding function fw. = wr. The representability of S is defined as a vector Is(e;) E Rd is the projection of standard basis e, onto S, and thus r (ei, (ei, uz)² where (S) and (uj,...,,0 }
[ri,... ,ra]" Fd such that r; [e.s(e))] 11 for E [d]. where
v.} is set of orthonormal bases for S.
The intuition of this definition is that a good feature space should have the property that many standard unit bases among eI, Ea. which can be used to represent any vectors in IRd, can be represented well by the feature space, i.e., have large projections onto it. Note that as a vector, r provides quantitative way to compare the representability of two feature spaces across different directions (i.e,, different unit basis). In the following theorem, we compare the representability learned by local objectives and the global one, for Dec-SSI..

---

Published as a conference paper at ICLR 2023
Comm. Efficiency |p = K,K = 5 0.1]
Partial Participation [K = 20, = =, =0.]
0.7
0.7
0.6
0.6
0.5
SURep
0.4
SURep
0.4
20
80
100
0.2
Figure 4: Dec-SSL performance on ImageNet-100 dataset. Compared to supervised learning, we observe that under non-IID settings, decentralized SSL can perform better under communication constraints (left) and partial participation constraints (right).
Theorem 3.2 (Representability of local v.s. global objectives for Dec-SSL). For decentralized SSL in the setting described above, with high probability. the representability vector learned from any local objective of source k, denoted by r. = [rt, ... . satisfies that 1 - (d S r S for all E [K]] {{... Moreover, the representability vector learned from the global objective, denoted by 1.. -Fa]" satisfies that 1 Od (dd-4/5) S f, for all E [K].
Theorem 3.2 states that the feature spaces learned from local SSL objectives are relatively uniform, in the sense that for the K basis directions e EK that generate the data, any two data sources have similar representability in all of them but two directions, especially when the dimension d of the data is large. Furthermore, when solving the global objective (2.2), the learned representation also uniform, and its representability differs at most one direction from that of each local data source. Note that the results hold with highly heterogeneous data across data sources. In other words, Dec-SSL. not affected significantly by the non-IIDness of the data, justifying the empirical observations in $3.1. Illustration of the results can also be found in Figure 3.
Intuition & implication. The main intuition behind Theorem 3.2 is that, the objective of SSL is not biased by the heterogeneous distribution of labels at each local dataset, and tends to learn uniform representations. Related arguments have also been made in the recent works on the theoretical un- derstanding contrastive learning/SSL (Wang & Isola, 2020; Liu et al., 2021). In the decentralized setting, this insensitivity to data heterogeneity becomes even more relevant, as it potentially allows each local data source perform much more local updates, without drifting the iterates signifi- cantly. This enables more communication--ffiiiee decentralized learning schemes, in contrast to most existing ones that are vulnerable to data non-IIDness. We validate these points next.
4 DEC-SSL CAN BE FAVORABLE EVEN WHEN LABELS ARE AVAILABLE
We here seek to address question (ii) in $2.1 how does the unique property of Dec-SSL, such as the robustness to data heterogeneity, benefit decentralized learning? While lack of labels seems limitation, we show that this might not be the case in decentralized learning with heterogeneous data. First. it is known that decentralized SL in general performs poorly when the data is highly het- erogeneous (Zhao et al., 2018; Hsieh et al., 2020). Further, even in the decentralized representation learning setting when labels are available, Dec-SSL still stands out in face of highly non-Ill) data.
To make a fair comparison, we mainly compare Dec-SSL with Dec-SLRep (recall the definition in $2.1), which are both decentralized representation learning approaches. We defer the comparison with Dec-SL to Appendix $B. We conduct experiments on both ImageNet and CIFAR-10 datasets, and evaluate the performance of the learned representations in terms of the variations of com- monly used metrics in decentralized learning the number of local updates epochs E, and the participation ratio of data sources p. We observe consistently that Dec-SSL indeed outperforms Dec-SL in learning representations in terms of communication efficiency and participation ra- tio, especially with highly non-IlI) data. We remark that such observations are also consistent with those on object detection and semantic segmentation given in Table 1.
4.1 EXPERIMENTAL OBSERVATIONS
In this experiment, we train and evaluate the feature backbone on ImageNet-1(00 in decentralized setting. We create non-IIDness across the local datasets based on label skewness and use = 0.1 (each data source has only 10% of its data coming from the uniform class distributions).
efficiency under high non-IIDness. In Figure 4, we show that under the non- IID scenario, averaging weights with an infrequent communication schedule causes less trouble to Dec-SSL than to Dec-SLRep. might sound sub-optimal, but we notice that decentralized SSL is very robust with respect this parameter. Intuitively, the robustness of Dec-SSL allows each local model to drift longer, leading to lower communication frequency for decentralized learning.
Communication
FedAvg, the idea of averaging weights after multiple epochs
7

---

Published as a conference paper at ICLR 2023
Algorithm Comm. f0- 1,K = 5,a = 0.02]
Metbod/ Seeting
IIB
DOn-IID
1.00
FURL (Zhung 2020a
71.25
68.01
0.78
EMA (Zhuang el al., 2022)
86.26
83,34
Per-SSFI. (He al., 202
N/A
83.10
0.50
FealARC
FEDU (Zhuang et al, 2021)
83.96
80.52
0.25
FedAvg
ZeatARC (Ours)
86.74
84.63
0.00
50
100
500
CIFAR-IOD
CIFAR-10
Algorithm Ablation Study 1p
1,K
E
5]
Pretrain
100%
10%
1%
0.85
pretrain
0.31
0.27
0.25
0.80
Custar Quly
Algen Only
Fedrig
Dec-SIRRep ITD
Dee-SSL IID
0.71
0.65
0.60
0.67
0.57
0.47
FeerARC
Dec-SI Rep Non-ITD
0.43
0.35
0.32
0.75
Des-SSL Noo-IID
0.70
0.66
0.57
WMT Figure 5: Ablation study on the FeatARC algo- rithm. We observe that under non- -IDness and com- munication constraints, FeatARC outperforms the baseline variants of the algorithm and FedAvg.
0.70
Table 2: Top). Algorithm performance com- parison. Bottom). CIFAR-IO Linear probing on the representation of CIFAR-100. Our al- gorithm surpasses previous works on federated SSL. both in the IID and non-IID settings.
Participation ratio under high non-IIDness. In this experiment, we split ImageNet-100 into 20 data sources and use local update E = epochs. We measure the performance of decentralized learning algorithms with respect to the participation ratio of data sources at each round. For instance, when p = 1, at each round, all data sources update their local weights and upload to the server, while P = 0.05 means that each round a single random data source is selected for update. On the right of Figure 4, we show that with non-IID data, the convergence of Dec-SSL is more stable to less participants compared to Dec-SLRep This allows more efficient decentralized learning, especially when deployed with extremely large number of data sources and unstable communication channels.
4.2 THEORETICAL INSIGHTS
To shed light on the above observations, we provide analysis for the feature spaces learned by the local objective of Dec-SLRep, under the same setup as in $3.2. For Dec-SLRep and each data source k we consider learning a two-layer linear network Iu (x) UA 24 classifier, where ue E Rmxd and U/k E Re and use u.. the learned representation for downstream tasks. The network is learned by minimizing (1xx) "ull? Tux)) + [1()))) 'v& 12. subject to the margin constraint that (9 (x)], 19 (3)] + for all data (2, y) in the local dataset & with all y' * y. We now have the following proposition on the representations learned by Dec-SLRep across data sources.
Proposition 4.1 (Representations learned by Rec-SLRep across heterogeneous data sources). With high probability, the features U4 = [ug.1.-.. uk,m] I E Rmxd learned from the local dataset Dj satisfies that Σm= (Mt,i, e2)² < Odd + ). j E [K=); while Σx= (MA,i, cx)2 2 1-0 1 In other words, the correlation between the learned features in wr and ej is small for all j E [K]
{k}, while the correlation between the features and ex is large.
The proposition suggests that the feature spaces learned by Dec-SLRep differ significantly across local data sources, given the highly heterogeneous data. More specifically, we show that most of the unit bases in {ei, ,e} }o  have small correlations with the features learned at each local data source, while these feature spaces themselves vary significantly across data sources. The unit bases that are not learned might be significant for various other downstream tasks, making the learned representations less favorable. This heterogeneity among local solutions is not in favor of local updates, as too many local updates would drift the iterates towards its local solution, and the iterates would become too far away from each other, hurting the convergence of decentralized learning. Hence, compared with the Dec-SSL case and Theorem 3.2, Dec-SLRep can be less robust to data heterogeneity and less come from using more data, since we use exactly the same data for training Rec-SLRep and Dec- SSL. The intuition is also illustrated in Figure 3. Finally, we remark that the uniformity of features, which is believed to be the key to better transfer performance in SSL (Wang & Isola, 2020; Caron et al., 2020), is not always preferred given specific learning tasks (Burgess et al., 2018).
We note that the advantage of Dec-SSL does not
OUR ALGORITHM - FeatARC (FEATURE ALIGNMENT AND CLUSTERING)
Although Dec-SSL tends to learn relatively uniform features that are robust across datasets, the uni- formity itself might not imply the alignment of features across datasets: the representation network from different local data sources can still map the same data point to different regions in the feature space. This misalignment becomes more significant when the data is highly non-IID and can have an adverse effect on the model aggregation process in decentralized learning (Zhang et al., 2020a).
8

---

Published as a conference paper at ICLR 2023
To mitigate this issue and address question (iii) in $2.1, we propose to use the same feature distance loss as an auxiliary local objective to align the local models with the global model. The alignment between two features is defined as the negative cosine distance metric [2)
To further improve the Dec-SSL algorithm, we propose learn multiple models using clustering- based approach. In particular, instead of learning single global model as in (2.2), we learn C models and separate the K data sources into C' clusters. The update of C' models and the assignment of data sources to C' clusters are conducted alternatively. When C' K, the algorithm reduces to learning local models; when C 1, reduces to learning a single global one. The clustering approach intuitively learns multiple models to interpolate the performance between learning single global model and K local models, thus achieving a good bias-variance tradeoff when testing on each local dataset (Mansour et al., 2020; Ghosh al., 2020). However, unlike the supervised learning case, we do not use the loss of the decentralized learning (i.e., (2.1)) as the metric for clustering. This is because for contrastive learning, it has been observed that the SSL loss might not be indicative enough for the performance of the representation on downstream tasks (Robinson et al., 2021). Hence, we here again use the feature alignment distance (D: :) the metric for clustering.
We adopt the alignment regularization and clustering techniques, and developed a new Dec-SSL. algorithm FeatARC, summarized in Algorithm 1 and Algorithm 2 in Appendix. We show the per- formance of F eeatARC in Figure 5, in comparison with different baselines including FedAvg, under different levels of data heterogeneity and communication frequency. performs the baselines consistently, including the variants that only uses alignment ("Align Only") or clustering ("Cluster Only"). Moreover, on the top of Table we show that FeatARC also out- performs other recent decentralized self-supervised learning algorithms on CIFAR-10 dataset.
shown that FeatARG out-
EXTENSIONS
In this section, diseuss few extended experiments of our framework. Please see Appendix $B for a thorough set of experiments and ablation studies with visualizations.
FULLY DECENTRALIZED CASE AND DIFFERENT NETWORK TOPOLOGY
We conduct experiments on the fully decentralized learning in Appendix $B.5, where the local data sources are only allowed communicate with their neighbors over peer-to-peer network, without a centralized server. In short, most observations we had regarding Dec-SSI. in the setting with a centralized server still hold, even under several different network topologies. This aligns with our theoretical insight provided in Section 3, which came from the benign properties of the solution to the Dec-SSL. objective, instead of the properties of specific algorithms (averaging the iterates via star or other network topologies) that achieves the solution.
6.2 EXTREMELY HETEROGENEOUS CASE FOR DECENTRALIZED LEARNING
In Figure 13, we show that even in the extremely heterogeneous case where each local source only owns one class, the Dec-SSI. framework still robust to the non-IIDness of the data. This also holds true when we scale to more clients, as shown in Figure 15. The Dec-SSL objective would not be biased by the highly heterogeneous class labels at each local dataset, while the Dec-SL objective could be biased by it. This is also consistent with our theoretical insights in Section $3.2 and the key reason for the success of Dec-SSL that, despite only having one single class, the information of features obtained from local datasets may still be useful for the jointly classifying of all the classes.
6.3 COMPARISON OF FEATARC WITH OTHER ALGRRITTS
We also compare our algorithm with the Dec-SSL algorithms that are combined with other feder- ated learning algorithms, including Li et al. (2020a) (FedProx) and Li et al. (2020b) (FedBN). In Figure 16 (Left), show that our proposed FeatARC outperform these two baselines.
7 CONCLUSION
We propose the framework of decentralized SSL that learns representations from non-IID unlabeled data and conduct an empirical study on the robustness of Dec-SSL to different types of heterogene- ity, communication constraints, and participation rates of data sources. We also provide findings and theoretical analyses of Dec-SSL compared to its supervised learning counterpart, well as developing a new algorithm to further address the high heterogeneity in decentralized datasets.

---

Published as a conference paper at ICLR 2023
Acknowledgement. This work is supported in part by Amazon.com Services LLC, PO2D- 06310236 and Defense Science & Technology Agency, DST000EC120300823. L.W. was sup- ported by the MIT EECS Xianhong Wu Graduate Fellowship. K.Z. also acknowledges support from Bimene-Berkeley Research Fellowship. We thank MIT Supercloud for providing compute re- sources. The authors would like to thank many helpful discussions from Phillip Isola MIT and Andrew Marchese at Amazon.
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint TrXX:::102.00229, 2019.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.000
2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Christopher Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in backslash beta-vae. arXiv preprint arXiv:1804.03399 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2959-2968. 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-99244, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15758, 2021.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 04805, 2018.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, 1(3):211-21) 1936.
Peter R Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint 100. 2018.
10

---

Published as a conference paper at ICLR 2023
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-traiie trannntt- CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019. URL https::p///.........
/ttps://oppeee /tppnreeeeeeeeoeeeeee net wwt.tnetoooooeee forum? (d-Byghjj099)
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586- 19597, 2020.
Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron. https://gix http://gtttt.../g.//e hub. wtw.ttttooooooo....oooooooootttt httpm///cccboooeeeerceeeeeeeeeee
2018.
Bean-Bastiens Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-21288 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supprvised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34, 2021.
Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. Ssfl: Tackling label deficiency federated learning via personalized self- supervision. arXiv preprini arXiv:21 10.02470, 2021a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021b.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learn- ing can improve model robustness and uncertainty. Advances in Neural Information Processing Systems, 32, 2019.
Kevin Hsieh, Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, Gregory Ganger, Phillip Gibbons, and Onur Mutlu. Gaia: (Geo-Distributel) machine learning approaching [LAN} speeds. In USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pp. 629-647, 2017.
Kevin Hsich, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pp. 4387- 4398. PMLR, 2020.
Wonyong Jeong, Jachong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. In International Conference on Learn- ing Representations, 2020.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations, 2018.
Chao Jia Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2023
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning, Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021.
Sai Praneeth Karimireddy, Satyen Kale. Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143, PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and
Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:/610.05492 2016.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar-100 datasets.
URI:
https://www: cs. toronto. edu/kriz/cifar html, 6(1):1, 2009.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In European conference on computer vision, pp. 577-593, Springer, 2016.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. Predicting what you already know helps: Provable sulf-supervised learning. Advances in Neural Information Processing Systems, 34, 2021.
Qinbin Li. Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos:
An experimental study. arXiv preprint 10XV:2202.000799 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429-454 2020a.
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. FedBN: Federated learn- ing non-iid features via local batch normalization. In International Conference on Learning Representations, 2020b.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2
2021b.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In NeurlPS 202/ Workshop on Distribution Shifts: Connecting Methods and Applications, 2021. URL https: wpwpeevveeeeeeeeeeeeeeeeeee n forum? id=v JP RLpGx.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
Nan Lu, Zhao Wang, Xiaoxiao Li, Gang Niu, Qi Dou, and Masashi Sugiyama. Federated learning from only unlabeled data with mass-conditional-shaiig clients. In International Conference on Learning Representations, 2022.
Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. arXiv preprint arXiv:2205. 12493, 2022.
12

---

Published as a conference paper at ICLR 2023
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXhv:1802.03426, 2018.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effcieett learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Angelia Nedic. Distributed gradient methods for convex machine learning problems in networks: Distributed optimization. JEEE Signal Processing Magazine, 37(3):92-104 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807. 1807.0374, 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint 10:::22033000000 2021.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can con- trastive learning avoid shorteut solutions? Advances in Neural Information Processing Systems, 34, 2021.
Justyna Sarzynska-
Barzynska-Wawer, Wawer, Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukasz Okruszek. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304: 14135, 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Timt-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020a.
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In European Conference Computer Vision, pp. 266-282. Springer, 2020b.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1 1179-1206. PMLR, 2021.
Roman Vershynin. High- probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere, In International Conference on Machine Learning, pp. 9929-9939. PMLR, 2020.
Zhirong WU, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Interna- fional Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2023
Fengda Zhang, Kun Kuang, Zhaoyang You. Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yuet- ing Zhuang, and Xiaolin Li. Federated unsupervised representation learning. arXiv preprint 0rX1::2010.08982, 2020a.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized feder- ated learning with first order model optimization. arXiv preprint arXiv:2012.08565, 2020b.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint arXiv:/806.0000 00582, 2018.
Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsuper- vised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4912-4921, 2021.
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. In International Conference Learning Representations, 2022.
14


Reviews:
[{"review_id": "XhEW1lxoEsX", "reviewer": "Reviewer_fhez", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the use of self-supervised learning (SSL) techniques for decentralized learning, with the emphasis on the understanding of how and why it is effective. It focuses particularly on the task of decentralized SSL that enables representation learning from unlabeled data that are owned separately by multiple clients, and compares it with other standard setups for decentralized supervised learning. The paper demonstrated that decentralized SSL works pretty well even when client data are highly non-iid (i.e., heterogeneous) both empirically and theoretically. This finding holds also for labeled data settings. Furthermore, the paper presents a new decentralized SSL method named FeatARC, which groups client data into several clusters to train multiple global models, and confirmed that it worked further better than FedAvg.", "summary_of_the_review": "I think that this seems to be a solid piece of work with sufficient technical contributions and excellent presentation. Nevertheless, I also believe that this work could become further stronger if other federated learning algorithms (e.g., FedProx, FedBN) are also evaluated. Otherwise it remains unclear if the provided results hold just for a specific algorithm of FedAvg or can work in general.", "strength_and_weaknesses": "## Strong points\n- Overall, the paper is excellently written and easy to follow. The paper is very well organized, and the connections to related work, proof of theoretical claims, and details of experiments are well documented in the appendix, making the paper complete.\n- The proposed work is well motivated. Although enabling SSL in decentralized (or federated) learning setups is a known topic with some existing work, to my knowledge there has been no study at this level of why SSL should work through extensive empirical evaluation and theoretical analysis. This would seem to be a solid technical contribution.\n- The fact that SSL works well for representation learning even from highly non-iid and unlabeled data, which was supported both empirically and theoretically, is a nice finding, because collecting labeled data is often identified as difficult in federated learning. \n- Experimental evaluation is thorough. Decentralized SSL is compared against decentralized SL approaches under a variety of conditions, such as multiple datasets, multiple levels of non-iidness, multiple SSL methods (SimCLR, SimSiam, and BYOL), and multiple hyperparameters for federated learning algorithms (FedAvg). \n\n## Weak points\n- Although the experimental result is already extensive, this work could have been further stronger if it confirmed the effectiveness of DecSSL with other federated learning algorithms. Currently, all the results are obtained using FedAvg, a very standard algorithm of federated learning. On the other hand, the performance of FedAvg is known to degrade when data is highly non-iid [a] or when the model includes batch normalization [b]. It remains not perfectly obvious if the current results hold only for FedAvg or various other algorithms such as FedProx and FedBN. This is a critical question to me to judge the significance of this work.\n- Another that was not very clear in the paper is the effect of heterogeneity in terms of data size |D_k|. As shown in Eq 2.2, this heterogeneity is considered in the global objective. How does it affect the performance of DecSSL empirically and theoretically? \n\n[a] Li et al., \"Federated Optimization in Heterogeneous Networks\", MLSys 2020\n\n[b] Li et al., \"FedBN: Federated Learning on Non-IID Features via Local Batch Normalization\", ICLR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ue__nIV-jDB", "reviewer": "Reviewer_bG2C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A decentralized self-supervised learning is proposed that learns representations from non-IID unlabeled data. The paper presents empirical studies on the robustness to different types of heterogeneity, communication constraints, and partial participation of data sources, as well as theoretical findings. ", "summary_of_the_review": "Because of the weaknesses mentioned above, primarily on experiments not being extensive in supporting the claims, I do not suggest acceptance of this paper. ", "strength_and_weaknesses": "Strengths\n\n- Theoretical insights are provided to support the design choices\n- Well written paper\n- Extensive ablation studies and analyses\n- Sec. 4 is very important for the future of ML.\n\nWeaknesses\n\n- The experimental section is quite weak. Only computer vision tasks are focused, with synthetic distortions. There are many real-world datasets that suffer from being non-i.i.d. with decentralized learning being important, such as from healthcare and finance. \n- Self-supervised learning is adopted in a very standard way, the methodological novelty is low. Not all self-supervised learning methods are considered extensively extensively.\n- There are bold claims about robustness to data heterogeneity but demonstrations are quite weak. There is no dataset used in experiments with features with very different characteristics.\n- It is unclear how hyperparameters are chosen.\n- Clustering idea in Sec. 5 is quite naive and ad-hoc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N0HIxGME4c4", "reviewer": "Reviewer_AXwb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper shows that decentralized self-supervised learning (Dec-SSL) can be used on unlabeled data and is robust to the heterogeneity of data aggregated over several data sources. It combines FedAvg, a decentralized learning algorithm, and SimCLR, a self-supervised representation learning algorithm, to create a Dec-SSL algorithm and evaluates this on multiple tasks for varying degrees of data heterogeneity against multiple baselines. It provides some theoretical considerations as to why Dec-SSL is robust to data heterogeneity and proposes an extension to the aforementioned Dec-SSL algorithm called FeatARC and shows that FeatARC outperforms baselines, which include ablations to FeatARC.", "summary_of_the_review": "The paper motivates addressing an important problem in a growing learning paradigm, explores this problem well in a specific domain, and proposes theoretical insights and an extension to an algorithm that's the combination of two existing algorithms. The paper is clearly written and understandable, even for those with limited experience in decentralized learning. However, the paper is limited by its use of only the visual domain and lack of results on representations learned, especially since it is primarily evaluatory of existing algorithms and minor extensions of these algorithms.", "strength_and_weaknesses": "The paper motivates the problem it addresses well--namely, the problem that much real-world data is unlabeled and can differ greatly across data sources, both of which cause issues in existing decentralized learning algorithms when compared to situations in which data is labeled or there is homogeneity across data sources. Decentralized learning is becoming more prevalent over time, so addressing this problem is becoming more important.\n\nThe primary objective of the paper is to evaluate a Dec-SSL algorithm that is a combination of two pre-existing algorithms on multiple vision tasks on large-scale visual datasets under varying degrees of heterogeneity. The paper does a good job at this. It conducts several experiments that assess the performance of Dec-SSL with respect to baselines that intuitively should and do act as \"upper bounds\" to Dec-SSL and show that Dec-SSL is robust to both feature and label non-IIDness (i.e., data hetereogeneity). The paper shows that Dec-SSL outperforms a decentralized supervised learning algorithm (Dec-SLRep) on object detection and semantic segmentation tasks on MS-COCO and segmentation tasks on an Amazon package segmentation dataset on multiple well-accepted metrics. Overall, the benchmarks the paper evaluates Dec-SSL on are large and diverse, and the baselines it compares Dec-SSL to are reasonable.\n\nThe paper provides theoretical insights as to why Dec-SSL outperforms Dec-SLRep in a clear, concise manner. However, I'm unsure whether the paper shows that these insights are clearly supported by empirical results. Figure 3 seems to be an artificial, illustrative example, but I don't see empirical results mirroring this example.\n\nThe paper further shows that Dec-SSL can be beneficial compared to Dec-SLRep even when labels are available by evaluating both algorithms under communication and participation constraints on ImageNet-100.\n\nThe paper finally motivates and introduces a new Dec-SSL algorithm, FeatARC, an extension to Dec-SSL that uses well-known techniques to improve multiple-model performance. FeatARC outperforms Dec-SSL on CIFAR tasks, and the paper shows ablations of the techniques used to extend FeatARC from Dec-SSL to show that these techniques consistently improve FeatARC over varying degrees of heterogeneity.\n\nThe paper would benefit from\n1. Extending the work to problem settings and domains other than tasks on visual data\n2. Further visualizing and comparing representations learned through Dec-SLRep, Dec-SSL, and FeatARC. Rigorously quantifying these comparisons (e.g., comparing Bhattacharyya distances between local models and global model in Dec-SSL vs FeatARC) under varying levels of heterogeneity and constraints would be interesting\n3. Explaining how the Amazon data was curated and whether it is publicly available (apologies if I missed this--I didn't see it in the main body or Appendix)\n4. Evaluating the performance of FeatARC to other algorithms on more than just CIFAR\n5. Highlighting limitations of more uniform representations (i.e., those learned by Dec-SSL vs Dec-SLRep). Are there settings in which having such \"spread out\" representations leads to worse performance? What about interpretability? Would less uniform representations generally be more interpretable, distinct, or disentangled (for example, comparing VAE to Beta-VAE with beta > 1)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-BI0kYH1z0", "reviewer": "Reviewer_84PM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel decentralized learning algorithm based on the recently developed self-supervised learning algorithms to address the heterogeneous data issues. Specifically, the authors applied the SSL to the Federated Learning settings and presented conclusions based on the empirical findings from a couple popular benchmarking datasets. They claimed that the Dec-SSL approach is robust against the data heterogeneity. Additionally, they provided theoretical insights into understanding why data heterogeneity is not a significant concern for Dec-SSL. They also designed a new algorithm called FeatARC to make learn multiple models based on clusters. The empirical evidences validated their claims in the paper.\n", "summary_of_the_review": "This paper presents a novel decentralized learning algorithm based on the recently developed self-supervised learning algorithms to address the heterogeneous data issues. The authors concluded that in their proposed Dec-SSL framework, the data heterogeneity is less of a concern. The author designed a new algorithm to run multiple models based on clusters and provided some theoretical insights. Overall, this paper provides some new insights on FL with unlabeled data and good empirical evaluation. However, there is confusion in learning settings and the theoretical contribution is marginal, which requires more work from the authors to make it technically solid and sound.", "strength_and_weaknesses": "Strength:\n\nThis paper presented a new direction that involves the self-supervised learning techniques to address the unlabeled and heterogeneous data issues in decentralized learning settings. Indeed, unlabeled data in decentralized learning has not sufficiently been explored, while the past years have witnessed the extensive studies on the decentralized supervised learning. The insight from this work seems new for the community as it shows that the proposed Dec-SSL framework is robust against data heterogeneity. Also, the authors provided sufficient and promising empirical evidences. Overall the paper is easy to follow and organized well.\n\nWeaknesses:\n\n1. The notation is confusing. Throughout the whole paper, the authors just focused on Federated Learning settings, which are NOT decentralized. The significant difference between these two settings is that for FL, communication only happens between server and workers, while for decentralized settings, there is no server and communication happens among workers. Even though they might share some similarities, in theory and practice, results can be completely different. So I wonder why the authors emphasized decentralization in this work. Probably FL is more appropriate in this work. Sine the title can be somewhat confusing. Also, if the setting changes, would the conclusions drawn from the results still hold true? The simpliest correction would be to focus on FL.\n\n2. The number of data sources is limited. It looks like throughout the paper, the authors always used 5 agents in the empirical results. Given the current status in FL and decentralized learning, 5 is too small and the relevant results are not that convincing as there is no clue to know if the same conclusions would apply to a large number of agents. For example, if CIFAR 100 and 50 or 100 agents were used, would Dec-SSL still perform robustly? Given that this paper is more applied and that conclusions are more based on experimental results, it is better for the authors to run the same experiments with a large number of agents/workers. Another issue that naturally emerges is that in decentralized learning, different topologies are quite critical. The authors should consider this topic in the paper and see how the topology would affect the performance of Dec-SSL framework. \n\n3. The theoretical contribution is marginal. Though the authors have provided the theoretical insights for Dec-SSL. However, the analysis of convergence of FeatARC is missing. Even if in the paper we have seen the slight model performance improvement over the baselines, that could be attributed to noise or variance. Then theoretical bound is important here since we can know if the proposed algorithm has the similar rate or not.\n\n4. Can Dec-SSL be applicable to an extreme case, where each agent only owns a unique class in the real decentralized learning, instead of the presented FL in this work? We have seen that many existing algorithms have failed under this scenario. Additionally, was Dec-SSL designed specifically for unlabeled data? Or it might perform better if the data was unlabeled or partially unlabeled?\n\n*******************************Post-rebuttal********************************\nI appreciate the detailed responses, careful revisions and additional results from the authors. After carefully reviewing the responses and clarification from the authors, I think most of my concerns have been addressed. Though there is some room for the authors to improve the theoretical analysis in this work, I feel it has provided some good insights for the community on the decentralized learning by leveraging SSL. While I would like to raise my score, the authors should still consider other comments from other reviewers to make the draft more technically solid and sound. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster