Question: I cannot see significant novelty in both BYOL and FedAvg-like algorithm. Will you fully discuss the relationship to ICCV 2021 paper FedBYOL?
Evidence: Self-supervised Learning In computer vision, self-supervised learning (SSL) aims to learn visual representations without any labels. Discriminative SSL methods facilitate learning with proxy tasks. Among them, contrastive learning has become a promising principle. It uses Siamese networks to minimize the similarity of two augmented views (positive pairs) and maximize the similarity of two different images (negative pairs). These methods are either contrastive or non-contrastive ones: contrastive SSL methods require negative pairs to prevent training collapse; non-contrastive SSL methods are generally more efficient as they maintain remarkable performances using only positive pairs. However, these methods do not perform well on decentralized non-IID data. We analyze their similarities and variances and propose a generalized FedSSL framework.
Federated Learning Federated learning (FL) is a distributed training technique for learning from decentralized parties without transmitting raw data to a central server. Among many studies that address the non-IID data challenge, Personalized FL (PFL) aims to learn personalized models for clients. Although some PFL methods interpolate global and local models, our proposed FedEMA differ in the motivation, application scenario, and measurement of the decay rate. Besides, the majority of existing works only consider supervised learning where clients have fully labeled data. Although recent works propose federated semi-supervised learning or federated domain adaptation, they still need labels in either the server or clients. This paper focuses on purely unlabeled decentralized data.
Federated Unsupervised Learning Learning representations from unlabeled decentralized data while preserving data privacy is still a nascent field. Federated unsupervised representation learning is first proposed by van Berlo et al. (2020) based on autoencoder, but it neglects the non-IID data challenge. Zhang et al. (2020a) address the non-IID issue with potential privacy risk for sharing features. Although Zhuang et al. (2020) address the issue based on BYOL as our FedEMA, they do not shed light on why BYOL works best. Since SSL methods are evolving rapidly and new methods are emerging, we introduce a generalized FedSSL framework and deeply investigate the fundamental components to build up practical guidelines for the generic FedSSL framework.
Answer: Thank you for raising the concern. We would like to clarify that our methods are not a simple combination of BYOL and FedAvg. Although ICCV 2021 paper [1] proposes a new method (FedU) based on BYOL, they do not shed light on why BYOL works best. Since SSL methods are evolving rapidly and new methods (like SimSiam [2], MoCoV2, SimCLR) are emerging, it is crucial to understand the fundamental components of FedSSL. In this paper, we first introduce a generalized FedSSL framework that embraces existing SSL methods and has flexibility catering to future methods. Then, we took an investigative approach to deeply investigate the fundamental components of FedSSL to build up intuitions and practical guidelines for the generic FedSSL framework. Based on these insights, we propose a new model update approach, FedEMA, which improves the performance substantially. We have revised the Related Work section with these discussions. Besides, we would like to point out that FedEMA outperforms [1], as shown in Table 3, 4, 5, and 7 in the manuscript.
[1] Collaborative unsupervised visual representation learning from decentralized data. ICCV 2021.