Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: -- Summary --
The paper proposes a method for federated learning of a mixture of experts model (FedMix). The approach allows training an ensemble of models each of which specializes to a subset of clients with similar data characteristics. The authors argue that this way of training an ensemble reduces the gradient divergence/interference, improves the overall performance, and sometimes reduces the communication overhead. The new method is evaluated a few federated image classification datasets.


-- Overall evaluation --
I find the idea of training a model ensemble instead of a single global model extremely appealing in the context of federated learning. The fact that the proposed approach allows models in the mixture (i.e., experts) to automatically specialize to different subsets of clients throughout training without having to run any clustering on the server makes the approach particularly appealing. Having said that, I also believe that the paper in the current form has many weaknesses, including:
- a significant lack of clarity throughout section 2 (see comments and questions below),
- in terms of methods, fairly ad-hoc changes introduced into the evidence lower-bound objective given in eq. 5 (removal of the entropy term and addition of a different regularize, which is justified in a very handwavy way),
- weak experimental results, which indicate that FedMix is, in fact, worse than the baselines both in terms of performance and the communication cost (unless I'm misreading Table 1 and Figure 4).

I believe that the paper has interesting and novel ideas, but falls short on both presenting them as well as getting them to work empirically. I would not recommend accepting the paper in the current form.


-- Comments and questions --

- Using MoE in federated learning makes a lot of sense intuitively. The authors further specialize MoE to FL by introducing a local gating network and then tying everything together through a global q distribution that approximates the posterior. These design choices are justified by stating that running federated optimization on the objective given in eq. 3 does not work well. I wonder, however, why not use a shared gating network instead of having to approximate the objective?

- I find the way lower bound in eqs. 5-6 is introduced and then the entropy term is dropped and substituted with some regularize (see 2nd paragraph on page 4) very ad-hoc. Typically, one would use a graphical model such as the one given in Figure 2 to derive a loss function in a principled way, without additional "hacks." Why use the graphical model formalism in the first place, if the final loss is being designed using some additional intuitive heuristics? Relatedly, perhaps there's a way to change the graphical model somehow and show that the final loss corresponds to the evidence lower bound in a slightly different probabilistic model? (e.g., with a different prior)

- Since $q_\phi(z \mid y)$ is a distribution over the experts in the mixture conditional on the class of the data point, what happens if K > # classes? It seems like q will select a separate model for each output class, so each expert will specialize to a single class only. How would that affect performance? Is K < # classes a requirement?


-- Experiments --

- If I'm not misreading the results in Table 1, FedMix adds a drastic communication overhead (due to multiple experts in the mixture) and does not perform as well as biased FedAvg on the CIFAR 10/100 datasets. Results for two baselines for EMNIST are missing.

- Similarly, results presented on Figure 4 for CIFAR datasets demonstrate that the baselines are better.

- For FedAvg, was the accuracy computed for the global model or for the global model after fine-tuning locally? Since FedMix targets to improve personalized FL, it should be compared with FedAvg with a simple fine-tuning-based personalization.


-- Minor points --

- Figure 1: are the bold curves some kind of running averages? Also, if the figure is given in the main text, it's strange that the definition of gradient divergence or alignment is provided in the appendix. I would recommend giving it somewhere in the main text.
- Figure 3: which dataset and task are these results for?
- Section 4, paragraph 1: I believe the EMNIST dataset has 3400 users if the version from tensorflow federated is used.
- Very minor: In Eq. 3, the notation for the sum over S is strange; maybe change to $\sum_{s=1}^S$?
review_title: Interesting approach, but subpar results and needs more clarification

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose using an ensemble of experts, but where the mixing proportions depend on both the data and the client (providing the personalization in the method). The proposed method is very interesting, but there are several issues which need to be addressed.

The method requires significantly higher communication cost to train, since multiple versions of the model parameters need to be communicated, as well as parameters for other models used for the mixing. The authors propose suggestions for reducing the number of models that need to be communicated, but they rely on very special behaviour of the different models (namely that they separate well so that clients do not actually need all of them). 

The clients need to transmit mixing parameters $q(z|s)$ as well as the $\phi$ which generate them. Given access to $N_s$ and the discrete nature of $y$, this would allow the server to reconstruct the label distribution of each user's data, which is a huge reduction in the privacy achieved by federated learning. 

The algorithm is quite complicated, and appears fairly fragile, as is demonstrated by the different functions for $h_k(x)$ required for just 3 data sets. Selection of tuning parameters in federated learning is a challenging and task, and having one which is a function makes the method quite challenging to use in practice.

Experimental results are not very compelling, with a simple personalized bias term outperforming the proposed model in terms of both number of communication rounds and total volume of data transmitted, and does not produce the privacy violation the proposed method produces. Although the simple personalized bias does not generalize beyond label distribution, there are many many similar personalization proposals for federated learning which do, and it is not clear whether the proposed method is competitive with those. Of course comparison with all existing methods is not feasible, but given the method must be entirely judged by empirical results, comparison with at least some should be done.

Overall it is not clear that the extra costs induced by the method are justified by the improvement in convergence speed, especially compared to more simple personalization methods which do not incur any of these extra costs (either in communication or privacy). 
review_title: A (personalized) mixture of experts, but with heavy communication and privacy cost

final_score: rating: 4: Ok but not good enough - rejection
review_content: Strength:
* The idea of training an ensemble of specialized models in FL is interesting, although not novel.
*  The evaluation is solid. I would be more impressed if the authors could also evaluate some SOTA language models.

Weakness:
*  The technical contribution is limited and the novelty is not sufficient. I am not against "A+B" research if it does bring drastic practical improvement. However, the proposed method is worse than some prior works on CIFAR 10 even using ten-time bandwidth (correct me if I interpret the results wrongly).
*  There are many other works with similar technical consideration although their intention might be different. Please check the line of works about personalized FL and compare with them if possible.
*  The writing is not clear given that the idea is actually simple.
review_title: The idea is interesting but improvements are expected

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a novel algorithm, which is a federated form on mixture of experts, called Federated Mixture of Experts (FedMix). In FedMix, an ensemble of specialized models is trained instead of a single global model. This strikes a compromise between training a single global model and one model per client. A gating mechanism is employed, to choose an expert model that is responsible for a given data point, thus aligning the gradient updates across experts and alleviating the consequences of non-i.i.d. data.

Pros:
- the paper is well written and easy to follow
- it tackles an important problem, namely how to cope with non-iid data in FL
- experiments are performed on different datasets

Cons:
- A major drawback of the paper is the lack of  in-depth experimental evaluation and theoretical insights. On Cifar-10 and Cifar-100 the proposed method performs worse than related state-of-the-art methods. On the rotated MNIST experiments in 4.2 it performs better, however, this section is very short and the insights from the presented results are limited.

- The STC approach presented in "Robust and Communication-Efficient Federated Learning from Non-IID Data" performs even better than biased FedAve on Cifar-10. It may well be that it also performs well in the rotated MNIST experiments. Anyways I am not convinced that the proposed method is superior to state-of-the-art techniques in non-iid settings (on Cifar-10 and -100 it is clearly not).

- The paper does not provide any new theoretical insights, it only reports empirical results. For instance, can you give guarantees that the proposed approach will result in K specialised models if the client data was generated from K different distributions (see  proof in Sattler et al. 2020).

- The authors mention the conceptual similarity of the proposed approach to Clustered Federated Learning (Sattler et al. 2020, Briggs et al., 2020), that allow to jointly train specialised models. Why do you not compare the proposed approach to these methods? I assume that Clustered FL will solve the rotated MNIST problem (although it may be worse in terms of communication-efficiency). More insightful experiments would be great.

Overall, I am not convinced by the proposed method and be the reported results.
review_title: Interesting approach, more in-depth experimental evaluation and theoretical insights needed

Meta-review: This paper presents an approach, FedMix, for federated learning using mixture of experts (MoE). The basic idea is to learn an ensemble of models and user-specific combination weights (mixing proportions).

The reviewers appreciated the MoE formulation for federated learning. However, there were multiple concerns from the reviewers, which include lack of clarity (regarding the variational lower bound that is being used), significant communication cost and privacy concerns (the server can infer the users' label distributions), weak experimental results, and lack of any theoretical support (which isn't that big an issue if the paper were stronger on other aspects). The author feedback was considered and the reviewers engaged in some discussions (also with the authors). In the end, however, the reviewer were still not convinced that the paper is ready to be published in its current state. Based on my own reading of the paper, the reviews, and the author response, I agree with this assessment.

The authors are advised to take into account the feedback from the reviewers and resubmit to another venue.
Decision: Reject