Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: -- Summary --
The paper proposes a new approach to model aggregation at each round performed on the server using a combination of Bayesian ensembling, model distillation, and stochastic weight averaging. The proposed method is essentially a drop-in replacement for weighted averaging based model aggregation used by FedAvg. The new method, FedBE, is extensively evaluated experimentally on federated image classification datasets where the authors show improved performance over multiple baselines.


-- Overall evaluation --
I find the paper well written and the idea of using ensembles and distillation for model aggregation in FL settings very interesting. Although the overall idea is identical to FedDF by Lin et al. (2020) who proposed to use ensembling and distillation and the authors cite and compare with that work, I think forming the ensemble distribution, sampling from it, as well as using SWA for distillation are all novel components; the contribution of each introduced component is empirically analyzed in ablation studies. Overall, while I don't find the approach extremely novel or insightful, it seems like a nice contribution to the literature. Below are some comments and questions that I hope will help further clarify and improve the paper.


-- Comments and questions --

- Section 3.3: While I see how different distributions over w are formed (Gaussian or Dirichlet), I'm not sure I understand why these distributions approximate the actual posterior? In other words, is it true that the distributions described by the authors minimize KL (or some other) divergence to the true posterior? Can you justify that?

- Regarding the Dirichlet (eq. 7), how are the gamma parameters estimated? Or are these simply hyperparameters?

- It's nice to have a toy example in section 3 (Figure 1) that illustrates the approach. However, some details are missing from the description. For example, how alphas were selected? How many clients per round were used? How did the test data distribution differ from the data on the clients?

- Computational complexity: I wonder how much is the computational overhead one has to pay to run FedBE on the server instead of FedAvg? In some practical settings, FedBE could be a great replacement for FedAvg, but in other settings, it might be too compute-intensive. It would be nice if the authors can discuss tradeoffs and analyze computational complexity.

- Related to the previous question: I wonder if distillation at each round is actually necessary? Given an ensemble (which is relatively cheap to obtain), why not sample multiple w's from it and initialize local models on clients using these samples? Do the authors expect it to work worse?


-- Experiments --

- Table 1 clearly demonstrates that FedBE dominates other methods on CIFAR-10. However, I'm puzzled by the reverse trends for centralized SGD and FL with the increase in the complexity of the model: more complex models trained with FL have worse performance than simpler models, and the opposite is true for centralized training. I wonder if ResNets were actually undertrained to some extent? Can the authors explain this phenomenon?

- Section 5.1, paragraph 2: how come the number of local epochs is computed as E = K / batch_size? Also, which metric was used exactly to tune the number of local epochs for FedAvg? After how many rounds the performance of FedAvg was evaluated and used for selecting E?


-- Minor --

- Is there any empirical evidence that supports the claim in footnote 2?

- Eq. 2 assumes that all clients have local datasets of the same size. Appendix A (eq. 10) provides the correct expression, but why not give that expression directly in the main text? The difference is literally 2 symbols. Same for eqs. 11-13.
review_title: A bit limited novelty, but an interesting contribution

final_score: rating: 7: Good paper, accept
review_content: The paper presents FEDBE, a novel method for federated learning. Models are trained locally in the clients' side then communicated to the server. The global model is constructed by Bayesian model ensemble. Knowledge distillation is used to construct the model that is passed to the clients for the next round. FEDBE is simple and shows strong empirical performance.

Strong points:

- The method demonstrates strong performance versus common baselines
- Extensive experiments were conducted.
- The paper is well written.


Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.
- I recommend to accept this paper.
- This type of work is definitely needed to enable more powerful models that preserves the privacy of the clients.


Questions:

- What is the effect of number of epochs between model collections (25 steps vs. 50 or more)? Is this what is shown in Figure 10? If so please clarify more.
- What is the effect of the step size decaying on SWA? IS the method sensitive to this choice?
- What are the hyperparameters that the method is sensitive for?
review_title: Nice combination of different techniques to construct a good algorithm with extensive evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper focuses on improving model aggregation on the server of multi-round federated learning. For this purpose, this paper proposes FedBE, which combines the idea of Bayesian model ensembling and distillation based on stochastic weight averaging. A set of unlabelled data on the server is required for distillation. Experiments are conducted on CIFAR-10/100 and tiny-ImageNet with both IID and non-IID cases.

The paper is well written and easy to follow. Though Bayesian model ensemble and distillation based on SWA are not new topics, using the idea of them in the field of federated learning is novel to the best of my knowledge. 

My major concern is about the experiment setting for using unlabeled data. According to my understanding, from the main body of the paper (Sec 5.2), 10K training images are used as unlabeled data saved on the server and the rest is distributed to each client. For baseline method like FedAvg, these 10K images are useless since it does not need unlabeled data for model average on the server. The different number of total training instances between FedBE (40K labeled + 10K unlabeled) and FedAvg (40K labeled) makes the comparison unfair. 

Besides, the analysis about either computational complexity or empirical wall clock time is needed. FedBE replaces the original model average with ensemble prediction & training, which is very slow. And these computations on the server seem cannot be parallelized with the computation on the client.  As a result, the computation on the server maybe the bottleneck of FedBE. So providing detailed analysis is important for better evaluating FedBE.
review_title: Questions about using unlabeled data

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper proposes a new model aggregation approach named FedBE in federated learning. In each round, FedBE first fits the local models into a Gaussian/Dirichlet distribution as a posterior distribution of the global model. Then, FedBE uses the sampled models from the distribution, the local models, and the averaged model as an ensemble to label a public dataset. The labeled dataset is used to train a global model. The experiments show that FedBE can outperform FedAVG on CIFAR-10 using ConvNet and ResNet.

Pros:

(1) The paper is well written and easy to follow.

(2) The studied problem is important and the proposed idea is interesting.

(3) The experiments are comprehensive and cover many aspects.

I have the following concerns, mainly about the experiments.

(1) In Algorithm 1, only the sampled models are used to predict the unlabeled dataset. However, in the experiments, the sampled models, the local models, and the averaged model are all used as an ensemble. Please keep consistency. Also, if the local models may perform badly due to client’s model drift, why the local models are also included in the ensemble?

(2) In Section D.6, SCAFFOLD has a bad test accuracy. I don’t understand why FedBE can improve SCAFFOLD a lot. If the averaged model is bad in SCAFFOLD, the sampled models are also very likely to be bad. However, the authors show that the accuracy of FedBE+SCAFFOLD is still close to FedBE+FedAvg. Can the authors explain it?

(3) From Table 7, FedBE with Dirichlet distribution is worse than FedBE with Gaussian distribution although it introduces an additional parameter $\alpha$. Is there any explanation?

(4) The size of unlabeled public data is large in the experiments. The paper uses CIFAR100+tiny-imagenet as the public dataset, which has a much larger size than the training dataset (i.e., CIFAR10). However, in reality, the data are usually private and the server may not be able to collect a large related public dataset. Can the authors show the performance of FedBE with a smaller public dataset?

(5) The authors can add a figure to show the test accuracy of different approaches versus the number of communication rounds. It is more clear to see the performance of different approaches with such a figure.

(6) The paper presents a simple averaging version in the main text and a weighted averaging version in Appendix. Which version is used in the experiments?

Minor issue: Figure 1 uses Dirichlet distribution as an example. Since the paper focuses on the Gaussian distribution, the authors can present an example based on Gaussian distribution instead.


==========Post rebuttal============

Thanks for the authors' response. The authors have addressed some of my concerns. I have raised my score accordingly.
review_title: Interesting idea but some experimental results need further clarification

Meta-review: This is a well written paper with good experimentation.  The paper builds on the work of FedDF and does ablation studies to demonstrate its improvements.  The key original idea is the use of a common pool of unlabeled data which is used in transmitting partial results between local and global servers.  The results seem pretty good.

From a practical viewpoint, the unlabelled common data will, in most cases, need to be generated/artificial data since it will need to be public (to the other servers at least).  This option should be tested to demonstrate feasability.

AnonReviewer2 was concerned about whether it was fair to provide additional unlabelled data.  The authors tested this out and showed it was OK.  Regardless, the different servers could easily generate artificial data for this purpose.  AnonReviewer1 had a number of issues which the authors largely addressed. The other two reviewers appreciated the paper.  All reviewers gave constructive suggestions.

Decision: Accept (Poster)