Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper combines self-supervised learning framework BOYL and FedAvg to improve the performance of unlabeled non-IID datasets.
main_review: * Strengths
1. Unsupervised FL is a well-motivated and timely topic.


* Weaknesses
1. This paper is a simple combination of BYOL and FedAvg. I cannot see significant novelty in both BYOL and FedAvg-like algorithm. Especially, its relationship to ICCV 2021 paper FedBYOL [1] is not fully discussed. 

2. Why is SimSaim [2] not used as the SSL framework? It has better interpretability in optimization; it requires smaller batch size; it doesn't require moving averaging. I believe the performance of SimSiam would be better than BYOL in FL context.

3. The definition of unlabeled non-I.I.D. dataset is not clear. According to Section 3.3, the authors use CIFAR-10 and CIFAR 100 to partition the dataset in an unbalanced manner with respect to the number of classes in each client. This is contradicted to the unsupervised assumption because we don't know users' labels, then it seems we cannot claim their data is non-I.I.D. in terms of the label. A careful definition of "unsupervised non-I.I.D. data" is required in this work.

4. Note that FL clients should not be stateful. When numerous clients exist, we need to do client sampling for a new round of training. The newly sampled client do not have any cached states to conduct adaptive optimizer or moving averaging-like stateful algorithms [4]. Please explain how Algorithm 1 can work in cross-device FL. 

5. The batch size is a challenge in FL context since FL clients normally are resource-constrained. Batch size = 128 is too big for smartphones.

6. The source code is not provided for reproducibility. I doubt some experiment results and would like to reproduce some results to confirm the efficacy. 

7. It would be better to discuss the privacy/security concern of the proposed FedSSL framework.

---
[1] Collaborative unsupervised visual representation learning from decentralized data. ICCV 2021.

[2] Exploring Simple Siamese Representation Learning. CVPR 2021.

[3] Federated Reconstruction: Partially Local Federated Learning. NeurIPS 2021.

summary_of_the_review: I suggest NOT accepting this paper at the current version, given that a few issues mentioned above are critical.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates a generic federated SSL recipe that applies FedAvg to a range of existing SSL works including SimCLR, MoCo, BYOL and SimSiam. Each of these SSL blocks comprises two encoding networks, including an online net and a target net. The two nets were trained via optimizing a similarity loss such that the distance between encodings of different augmented versions (e.g. rotation) of the same input is small and vice versa. Depending on the specific SSL block, the parameterization of the two nets might be identical (SimCLR, SimSiam) or not (MoCo, BYOL). At each iteration, the online net of each client is uploaded to a server to be averaged. Then, the (global) averaged version is sent back to each client. Each client then reset its online net as a weighted average of the global and local version of the online net (to account for potential data heterogeneity), and continue updating the SSL block via gradient updates and so on.    

Here, the main contribution of the paper is a series of ablation studies (Table 1, Figure 2, Table 2, Figure 3) that measure the isolated impact of several fundamental components of the aforementioned SSL blocks (e.g., SimCLR, SimSiam, MoCo and BYOL). This includes the predictor, stop-gradient, exponential moving average (EMA), non-identical online and target encoding net. The results indicate that BYOLD contains all the essential elements (predictor, EMA and non-identical online & target net & no stop-gradient) so the federated SSL recipe is rotated towards having BYOL as the model choice. Then, a customized, divergence-aware EMA is proposed for this recipe that is shown to improve substantially over existing FL-SSL baselines (Tables 3-4; Tables 6-7). There is also a bit of ablation study that measures the isolated impact of the new EMA which is also substantial as shown in Figure 5.
main_review: NOVELTY & SIGNIFICANCE

The key innovation of this paper is the specific formulae -- Eq. (3) -- that computes the decay rate in the EMA recipe which is driven by a scaler and the normalized L2-distance between the local & global online net. While this appears minimal in terms of algorithmic novelty, the empirical validation is quite extensive and shows strong improvement over existing baselines. 

Thus, despite the simple technical innovation, I have a positive opinion of this paper. In fact, I think this is a pretty well-structured investigative process that arrives at a fine-tuned version of federated SSL with improved performance. It also re-affirms or discovers relatively useful empirical practices that boosts the performance of federated SSL, particularly in highly non-IID settings. 

On another note, however, it seems the performance would depend on whether we have a correct choice of lambda. For Tables 3-4, how would it impact performance in the IID setting when lambda is set to 1 instead? I think this is likely a substantial restriction of the proposed work in case we do not know the data heterogeneity of the clients. Could the authors comment more on this? Any practical guidelines on choosing lambda here? Fig. 1 suggest a range of (0,1) for lambda but does it apply generically or only to CIFAR data?

Another look at the supplement results in Table 6 shows that the performance is also sensitive somewhat to the choice of the architecture. How would this impact the default range of good values for lambda?

EXPERIMENT

The experiment is quite extensive & interesting. But I do have a few follow-up questions:

In Tables 3 and 4, what are FedEMA's performance on IID setting when lambda = 1. 

How many runs were used to produce the average accuracy and standard deviation?

What is the exact non-IID setting used in Tables 3 and 4?

There seems to be a decrease in terms of accuracy in CIFAR-100 when one moves from non-IID to IID settings. This is the opposite of what we saw in CIFAR-10 settings. Could the authors comment on this?

It seems some baselines such as FedMoCoV1 and FedMoCoV2 were missing from Table 7. Could the authors provide the missing info?
summary_of_the_review: This is an empirical paper. The key contribution is an investigative, ablation studies that build up intuitions and practical guidelines to fine-tune the generic federated SSL framework. The fine-tuned version is shown to have significant performance improvement. But on another note, this success might be dependent on a correct choice of lambda which is only possible if one knows the data heterogeneity of the clients. Perhaps this is a key restriction of the method. It would be great if the authors can expand on this during the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduce an abstraction of self-supervised learning (SSL) algorithms in the federated setting, which they term FedSSL. They empirically evaluate several existing SSL methods in the federated setting and derive a new algorithm, FedEMA based on their insights. 
main_review: The paper follows a clear red line and is easy to follow. The empirical insights are interesting and mostly well motivated and explained. I have a few open question related to the paper and some suggestions which would make this a very good paper. 

The most successful methods, as well as FedEMA rely on the availablitiy of a local target network. I.e. the clients are state-full between rounds. In the general (sightly more mature) supervised FL literature, this setting is called the 'cross-silo' setting as opposed to the cross-device setting, in which we assume the federation to consist of millions of devices, each of which might take part in the federation just once. This distinction is missing in this paper and I encourage the authors to make this more explicit. 
Related to this comment is my biggest issue with the experimental claims in this paper. For the federated setting $K=5$ clients is not enough. Even for a paper targeting cross-silo FL, I would suggest to experiment with more clients to make the insights more valuable. Table 11 considers up to $K=20$ clients, however the authors also reduce the amount of data per-client (in addition to spreading the available data over a larger amount of clients, as far as I understand). 
Along the same lines, FL usually does not assume full client availability, i.e. not all $K$ clients are available for updates at every round. Subsampling clients is especially interesting in the context of state-full clients, since their local models might become stale. Also as the amount of data locally becomes smaller, methods that rely on a local target encoder might overfit, something that FedEMA might remedy. I would encourage the authors to explore that angle or clearly position their work in terms of which niche of the heterogeneous  FL space they target.
How representative of SSL methods are small-scale data-sets such as cifar10/cifar100? Maybe the authors could compare their scale to the literature. I encourage them to evaluate some experimental insights on larger data-sets such as Imagenet. 

Server-side optimization benefits from more advanced update rules than simply averaging client updates. E.g. https://arxiv.org/abs/2003.00295 explore this design space. Is there a reason the authors choose to do averaging only? Otherwise I'd encourage the authors to explore the influence of these optimisers. 

Your discussion of the scaler $\lambda$ mentions that setting $\lambda>1$ leads to $\mu=1$. I figure that would depend a lot on the federation characteristics that determine the divergence between local and global model over time. A such, it is a very general statement. Maybe you could track the value of $\mu$ over time for different settings of $\lambda$ for your experiments.

Experimental setting:
- What hyper-parameter ranges have the authors explored for their experiments?
- Is centralised BYOL done with same hyperparameters? E.g. can you do centralised performance with small batch-sizes?
- Is linear evaluation done on the locally fine-tuned models locally or on the most recent global model with data at the server?

Misc:
- What is the difference between the pink line/model in Figure 3 and the last row in Table 2?
- What time is indexed by $t$ in 3.2? Is that local time-steps per client or the global update steps? I.e. is the target encoder updated every mini-batch or every global round only? How is this different from equation 1 and 2? Are you describing 2-level updates? Algorithm 1 suggests the answer, but I encourage to make this more clear in the text. I also suggest adding superscript $0$ to lines 13/14 in Alg 1. 

Disclaimer: I am not an expert in SSL methods, which is why I appreciated Appendix A and corresponding discussions in the main text.

summary_of_the_review: This paper provides an interesting insight into self-supervised training in FL. It motivates its proposed method by empirical insights on cifar10(0). My major issue with the paper is that its empirical scale is quite small in terms of what makes the FL scenario challenging. On the other hand, they have very broad empirical analyses. 
If the authors extend their experimental evaluation along these scale-related dimensions and clarify their positioning within the FL literature, I will consider raise my score.

=== Post-Rebuttal ===
The authors have addressed my concerns sufficiently and I have updated my score accordingly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new approach called Federated Divergence-aware Exponential moving Average update (FedEMA) to avoid the IID assumption. FedEMA is built onto of FedSSL which is a framework for self-supervised learning in a Federated Learning context. The authors proposes a new approach to fuse the local and global knowledge effectively through a EMA update, where the decay rate of EMA is dynamically measured by a divergence.

main_review: I think overall, the idea to solve the non-IID problem in Federated learning is good. However, it is not immediately clear to me why the solution works from a theortical perspective. The authors have demonstrated that the approach is superior by a quite significant margin emperically on CIFAR-10 and CIFAR-100.

(1) The superscript and subscript of W have been defined. But W itself has not been defined in the paper. From the context, I assume these are the parameters of your model.

(2) How do you select \lambda?
summary_of_the_review: Overall an interesting idea. However, I thought the novelty proposed is a bit incremental and should have been more thoroughly investigated. For example more experiments could have been run and it would be good to theoretically show why this works. Although from Section 3.5, I can understand the intuition behind why this works. Overall, I felt that the contribution that this paper proposes does not meet the requirement to be published at ICLR.
review_title: null

Meta-review: The paper focuses on self-supervised learning (SSL) in the federated learning setting (FedSSL). Research in this area is timely and of significance. The authors phrase their work as primarily being an empirical study providing insights into the building blocks of FedSSL. The evaluation in the paper is quite thorough and the authors have been active in a detailed exchange regarding questions raised in the reviews. I would encourage the authors to fully implement the changes they promised into the revised manuscript and work towards timely release of open-source code. (I appreciate internal policies of various institutions, but I do agree with the reviewers that it is more important that the code and experimental details be made public for papers such as this one, compared to some other papers.) I have chosen to disagree with some of the concerns raised in one of the reviews, in particular, I do agree with the authors that insights into the building blocks through empirical studies is a significant contribution, and also that FedEMA is a novel contribution. The discussion on this forum will remain for interested readers to come to their own conclusions about the relative performance of various methods.
Decision: Accept (Poster)