Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper shows that decentralized self-supervised learning (Dec-SSL) can be used on unlabeled data and is robust to the heterogeneity of data aggregated over several data sources. It combines FedAvg, a decentralized learning algorithm, and SimCLR, a self-supervised representation learning algorithm, to create a Dec-SSL algorithm and evaluates this on multiple tasks for varying degrees of data heterogeneity against multiple baselines. It provides some theoretical considerations as to why Dec-SSL is robust to data heterogeneity and proposes an extension to the aforementioned Dec-SSL algorithm called FeatARC and shows that FeatARC outperforms baselines, which include ablations to FeatARC.
strength_and_weaknesses: The paper motivates the problem it addresses well--namely, the problem that much real-world data is unlabeled and can differ greatly across data sources, both of which cause issues in existing decentralized learning algorithms when compared to situations in which data is labeled or there is homogeneity across data sources. Decentralized learning is becoming more prevalent over time, so addressing this problem is becoming more important.

The primary objective of the paper is to evaluate a Dec-SSL algorithm that is a combination of two pre-existing algorithms on multiple vision tasks on large-scale visual datasets under varying degrees of heterogeneity. The paper does a good job at this. It conducts several experiments that assess the performance of Dec-SSL with respect to baselines that intuitively should and do act as "upper bounds" to Dec-SSL and show that Dec-SSL is robust to both feature and label non-IIDness (i.e., data hetereogeneity). The paper shows that Dec-SSL outperforms a decentralized supervised learning algorithm (Dec-SLRep) on object detection and semantic segmentation tasks on MS-COCO and segmentation tasks on an Amazon package segmentation dataset on multiple well-accepted metrics. Overall, the benchmarks the paper evaluates Dec-SSL on are large and diverse, and the baselines it compares Dec-SSL to are reasonable.

The paper provides theoretical insights as to why Dec-SSL outperforms Dec-SLRep in a clear, concise manner. However, I'm unsure whether the paper shows that these insights are clearly supported by empirical results. Figure 3 seems to be an artificial, illustrative example, but I don't see empirical results mirroring this example.

The paper further shows that Dec-SSL can be beneficial compared to Dec-SLRep even when labels are available by evaluating both algorithms under communication and participation constraints on ImageNet-100.

The paper finally motivates and introduces a new Dec-SSL algorithm, FeatARC, an extension to Dec-SSL that uses well-known techniques to improve multiple-model performance. FeatARC outperforms Dec-SSL on CIFAR tasks, and the paper shows ablations of the techniques used to extend FeatARC from Dec-SSL to show that these techniques consistently improve FeatARC over varying degrees of heterogeneity.

The paper would benefit from
1. Extending the work to problem settings and domains other than tasks on visual data
2. Further visualizing and comparing representations learned through Dec-SLRep, Dec-SSL, and FeatARC. Rigorously quantifying these comparisons (e.g., comparing Bhattacharyya distances between local models and global model in Dec-SSL vs FeatARC) under varying levels of heterogeneity and constraints would be interesting
3. Explaining how the Amazon data was curated and whether it is publicly available (apologies if I missed this--I didn't see it in the main body or Appendix)
4. Evaluating the performance of FeatARC to other algorithms on more than just CIFAR
5. Highlighting limitations of more uniform representations (i.e., those learned by Dec-SSL vs Dec-SLRep). Are there settings in which having such "spread out" representations leads to worse performance? What about interpretability? Would less uniform representations generally be more interpretable, distinct, or disentangled (for example, comparing VAE to Beta-VAE with beta > 1)


clarity,_quality,_novelty_and_reproducibility: The paper is primarily intended to evaluate the viability of decentralized learning with unlabeled data and data heterogeneity. The clarity of the paper is generally excellent. The quality is very good but is limited to the visual domain and common vision tasks. The novelty is limited, as most of it is in the form of experimental design and theorems of limited importance instead of algorithmic advances (as explained above for Dec-SSL and FeatARC) or deep insights that shifts the research community towards new lines of research. The reproducibility of the work is excellent--the experimental design, algorithms, optimizers, architectures, hyperparameters, etc. are clearly presented in the main body and Appendix.
summary_of_the_review: The paper motivates addressing an important problem in a growing learning paradigm, explores this problem well in a specific domain, and proposes theoretical insights and an extension to an algorithm that's the combination of two existing algorithms. The paper is clearly written and understandable, even for those with limited experience in decentralized learning. However, the paper is limited by its use of only the visual domain and lack of results on representations learned, especially since it is primarily evaluatory of existing algorithms and minor extensions of these algorithms.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel decentralized learning algorithm based on the recently developed self-supervised learning algorithms to address the heterogeneous data issues. Specifically, the authors applied the SSL to the Federated Learning settings and presented conclusions based on the empirical findings from a couple popular benchmarking datasets. They claimed that the Dec-SSL approach is robust against the data heterogeneity. Additionally, they provided theoretical insights into understanding why data heterogeneity is not a significant concern for Dec-SSL. They also designed a new algorithm called FeatARC to make learn multiple models based on clusters. The empirical evidences validated their claims in the paper.

strength_and_weaknesses: Strength:

This paper presented a new direction that involves the self-supervised learning techniques to address the unlabeled and heterogeneous data issues in decentralized learning settings. Indeed, unlabeled data in decentralized learning has not sufficiently been explored, while the past years have witnessed the extensive studies on the decentralized supervised learning. The insight from this work seems new for the community as it shows that the proposed Dec-SSL framework is robust against data heterogeneity. Also, the authors provided sufficient and promising empirical evidences. Overall the paper is easy to follow and organized well.

Weaknesses:

1. The notation is confusing. Throughout the whole paper, the authors just focused on Federated Learning settings, which are NOT decentralized. The significant difference between these two settings is that for FL, communication only happens between server and workers, while for decentralized settings, there is no server and communication happens among workers. Even though they might share some similarities, in theory and practice, results can be completely different. So I wonder why the authors emphasized decentralization in this work. Probably FL is more appropriate in this work. Sine the title can be somewhat confusing. Also, if the setting changes, would the conclusions drawn from the results still hold true? The simpliest correction would be to focus on FL.

2. The number of data sources is limited. It looks like throughout the paper, the authors always used 5 agents in the empirical results. Given the current status in FL and decentralized learning, 5 is too small and the relevant results are not that convincing as there is no clue to know if the same conclusions would apply to a large number of agents. For example, if CIFAR 100 and 50 or 100 agents were used, would Dec-SSL still perform robustly? Given that this paper is more applied and that conclusions are more based on experimental results, it is better for the authors to run the same experiments with a large number of agents/workers. Another issue that naturally emerges is that in decentralized learning, different topologies are quite critical. The authors should consider this topic in the paper and see how the topology would affect the performance of Dec-SSL framework. 

3. The theoretical contribution is marginal. Though the authors have provided the theoretical insights for Dec-SSL. However, the analysis of convergence of FeatARC is missing. Even if in the paper we have seen the slight model performance improvement over the baselines, that could be attributed to noise or variance. Then theoretical bound is important here since we can know if the proposed algorithm has the similar rate or not.

4. Can Dec-SSL be applicable to an extreme case, where each agent only owns a unique class in the real decentralized learning, instead of the presented FL in this work? We have seen that many existing algorithms have failed under this scenario. Additionally, was Dec-SSL designed specifically for unlabeled data? Or it might perform better if the data was unlabeled or partially unlabeled?

*******************************Post-rebuttal********************************
I appreciate the detailed responses, careful revisions and additional results from the authors. After carefully reviewing the responses and clarification from the authors, I think most of my concerns have been addressed. Though there is some room for the authors to improve the theoretical analysis in this work, I feel it has provided some good insights for the community on the decentralized learning by leveraging SSL. While I would like to raise my score, the authors should still consider other comments from other reviewers to make the draft more technically solid and sound. 
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of this paper are decent. Particularly, the empirical novelty looks great.
summary_of_the_review: This paper presents a novel decentralized learning algorithm based on the recently developed self-supervised learning algorithms to address the heterogeneous data issues. The authors concluded that in their proposed Dec-SSL framework, the data heterogeneity is less of a concern. The author designed a new algorithm to run multiple models based on clusters and provided some theoretical insights. Overall, this paper provides some new insights on FL with unlabeled data and good empirical evaluation. However, there is confusion in learning settings and the theoretical contribution is marginal, which requires more work from the authors to make it technically solid and sound.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A decentralized self-supervised learning is proposed that learns representations from non-IID unlabeled data. The paper presents empirical studies on the robustness to different types of heterogeneity, communication constraints, and partial participation of data sources, as well as theoretical findings. 
strength_and_weaknesses: Strengths

- Theoretical insights are provided to support the design choices
- Well written paper
- Extensive ablation studies and analyses
- Sec. 4 is very important for the future of ML.

Weaknesses

- The experimental section is quite weak. Only computer vision tasks are focused, with synthetic distortions. There are many real-world datasets that suffer from being non-i.i.d. with decentralized learning being important, such as from healthcare and finance. 
- Self-supervised learning is adopted in a very standard way, the methodological novelty is low. Not all self-supervised learning methods are considered extensively extensively.
- There are bold claims about robustness to data heterogeneity but demonstrations are quite weak. There is no dataset used in experiments with features with very different characteristics.
- It is unclear how hyperparameters are chosen.
- Clustering idea in Sec. 5 is quite naive and ad-hoc.
clarity,_quality,_novelty_and_reproducibility: Clarity - Overall pretty good but the paper organization can be improved.

Quality - Pretty decent, except experimental sections.

Novelty - Not high, straightforward extension.

Reproducibility - Unclear. Not all experimental details are presented.
summary_of_the_review: Because of the weaknesses mentioned above, primarily on experiments not being extensive in supporting the claims, I do not suggest acceptance of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the use of self-supervised learning (SSL) techniques for decentralized learning, with the emphasis on the understanding of how and why it is effective. It focuses particularly on the task of decentralized SSL that enables representation learning from unlabeled data that are owned separately by multiple clients, and compares it with other standard setups for decentralized supervised learning. The paper demonstrated that decentralized SSL works pretty well even when client data are highly non-iid (i.e., heterogeneous) both empirically and theoretically. This finding holds also for labeled data settings. Furthermore, the paper presents a new decentralized SSL method named FeatARC, which groups client data into several clusters to train multiple global models, and confirmed that it worked further better than FedAvg.
strength_and_weaknesses: ## Strong points
- Overall, the paper is excellently written and easy to follow. The paper is very well organized, and the connections to related work, proof of theoretical claims, and details of experiments are well documented in the appendix, making the paper complete.
- The proposed work is well motivated. Although enabling SSL in decentralized (or federated) learning setups is a known topic with some existing work, to my knowledge there has been no study at this level of why SSL should work through extensive empirical evaluation and theoretical analysis. This would seem to be a solid technical contribution.
- The fact that SSL works well for representation learning even from highly non-iid and unlabeled data, which was supported both empirically and theoretically, is a nice finding, because collecting labeled data is often identified as difficult in federated learning. 
- Experimental evaluation is thorough. Decentralized SSL is compared against decentralized SL approaches under a variety of conditions, such as multiple datasets, multiple levels of non-iidness, multiple SSL methods (SimCLR, SimSiam, and BYOL), and multiple hyperparameters for federated learning algorithms (FedAvg). 

## Weak points
- Although the experimental result is already extensive, this work could have been further stronger if it confirmed the effectiveness of DecSSL with other federated learning algorithms. Currently, all the results are obtained using FedAvg, a very standard algorithm of federated learning. On the other hand, the performance of FedAvg is known to degrade when data is highly non-iid [a] or when the model includes batch normalization [b]. It remains not perfectly obvious if the current results hold only for FedAvg or various other algorithms such as FedProx and FedBN. This is a critical question to me to judge the significance of this work.
- Another that was not very clear in the paper is the effect of heterogeneity in terms of data size |D_k|. As shown in Eq 2.2, this heterogeneity is considered in the global objective. How does it affect the performance of DecSSL empirically and theoretically? 

[a] Li et al., "Federated Optimization in Heterogeneous Networks", MLSys 2020

[b] Li et al., "FedBN: Federated Learning on Non-IID Features via Local Batch Normalization", ICLR 2021
clarity,_quality,_novelty_and_reproducibility: - **Clarity**: The paper is very clearly written. The motivation, approaches, and results of the work are all clear.
- **Quality**: Overall, I think the work is of very high quality with excellent presentation and sufficient technical contributions.
- **Novelty**: As far as I know, this level of detailed experimental evaluation and theoretical analysis for how SSL works well under decentralized learning setups is novel.
- **Reproducibility**: Details of experimental setups are presented in the appendix. Code is also submitted. 
summary_of_the_review: I think that this seems to be a solid piece of work with sufficient technical contributions and excellent presentation. Nevertheless, I also believe that this work could become further stronger if other federated learning algorithms (e.g., FedProx, FedBN) are also evaluated. Otherwise it remains unclear if the provided results hold just for a specific algorithm of FedAvg or can work in general.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the use of self-supervised learning (SSL) techniques for decentralized learning, with the emphasis on the understanding of how and why it is effective. It focuses particularly on the task of decentralized SSL that enables representation learning from unlabeled data that are owned separately by multiple clients, and compares it with other standard setups for decentralized supervised learning. The paper demonstrated that decentralized SSL works pretty well even when client data are highly non-iid (i.e., heterogeneous) both empirically and theoretically. This finding holds also for labeled data settings. The model is validated on diverse tasks in computer vision. 


justification_for_why_not_higher_score: The main concern is they only include computer vision tasks in their experimental evaluation. If they could include one or two other domain applications, that will make the paper stronger.





justification_for_why_not_lower_score: Overall, the paper is excellently written and easy to follow. The level of the investigation into SSL in Federated learning is more thorough and deeper compared with other existing works. Therefore, this work has made a great contribution to the federated learning literature.
Decision: Accept: poster