Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a federated multi-task learning approach to learn personalized models.
main_review: Pros.
This paper has proposed a theretical sound algorithm for federated MTL. The idea is valid with convergence results supprt.

However, the work can be improved by addressing the following comments.

1. Section 2.1 is not well-written and self-contained. The impossibility result is not formally established and only some intuitions are given. In addition, the SSL problem is not well defined and no citation is given as well. This makes me hard to justify the motivation of the proposed work.

2. There are more recent published papers in personalized FL. For example,

a. FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

b. Personalized Cross-Silo Federated Learning on Non-IID Data

These two apporaches, APFL and SCAFFOLD are able to handle clients sampling. Thus it is better to compare with them than pFedMe and clustered FL.

3. For the synethic data, the prospoed work should be compared with MOCHA which is the most closest work to the proposed work for linear model.
limitations_and_societal_impact: Not applied
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes an approach to learn local models in a federated learning setup where local distributions are assumed to be a mixture of a given set of ground distributions. The idea is to learn a ground model for each ground distribution and the mixture coefficients for the local distributions, and then obtain a local model as the weighted average of ground models (weights are given by the mixture coefficients). 
main_review: The paper is well-written, presents the contributions clearly, and discusses relevant related work. The proposed method is sound and novel. The idea of learning a set of distributions and their mixture coefficients is an interesting take on the non-iid problem in federated learning, especially given that the paper shows this to be a generalization of other frameworks. The empirical evaluation is solid with a decent number of datasets, a large number of clients (100), and a variety of model types. 

The paper puts a lot into the appendix, which is fair, but I felt that some of the results (especially the fully decentralized algorithm and the federated surrogate optimization) feel a bit displaced in the main text and could be moved to the appendix to rather provide more details on the main contributions in the main text. 

Overall, the paper provides a solid contribution that is backed by theory and empirical results. I vote for acceptance.

Detailed comments:

- the negative result is presented too informally and in its formulation only holds for equal feature distributions. At the same time, the result seems quite trivial: if one client holds no relevant information for another client, then no form of collaboration can improve local learning. 
- way too many arxiv citations, including older ones (2016, 2018). For some, this is just due to lazy bibliography writing. E.g., [28] is actually published: Kang, Jiawen, et al. "Incentive design for efficient federated learning in mobile networks: A contract theory approach." 2019 IEEE VTS Asia Pacific Wireless Communications Symposium (APWCS). IEEE, 2019. and [32] was published at the Workshop on Private Multi-Party Machine Learning.
- In order to overcome the limitation of assumption 2, the authors might consider using the results in [1], which deal with different feature distributions p_m(x), but do not address different conditionals. 

[1] Li, Xiaoxiao, et al. "FedBN: Federated Learning on Non-IID Features via Local Batch Normalization." International Conference on Learning Representations. 2020.

----------------------------------------------------------------------------------

I have read my fellow reviewer's reviews and the authors rebuttal. Many valid points of critique have been adequately addressed by the authors. While some points still remain open, e.g., the proper discussion of related work in PFL, I am still convinced that this is a good paper with an interesting take on the problem of non-iid local data. So, while this is surely not a perfect paper, I think it is a valuable contribution to the conference and I keep my vote on acceptance.
limitations_and_societal_impact: The paper addresses the limitations of the approach. The paper discusses societal impact in sufficient detail.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors consider model personalisation as an important aspect of federated learning. Based on an impossibility result, the authors are motivated to learn a mixture model $p(y|x)=\sum_m^M\pi_m p(y|x, \theta_m)$ as opposed to a single mapping $p(y|x,\theta)$. As opposed to standard FedAvg approaches, the mixture weights $\pi_m$ are local, i.e. specific to each client $t$. 
The authors propose an EM-type algorithm for learning the parameters $\theta_m$ and $\pi_{m,t}$ in a federated environment, coining their algorithm FedEM. Furthermore they propose a decentralised variant of FedEM. 
The authors show convergence proofs under appropriate assumptions and show that FedEM and D-FedEM are special cases of  more general federated surrogate optimization.

== after rebuttal ==

I have read the rebuttal and raised my score accordingly
main_review: Thx for presenting this interesting paper in a relevant topic in FL research. I enjoyed reading it, it is presented concisely.

General paper outline:
The paper consists of several elements, the connection of which I would like to see improved. The main technical motivation for FedEM seems to be 2.1, i.e. the impossibility result. The authors state that "The main consequence for FL is that, without further assumptions, a client cannot provably benefit from larger amounts of data available at other clients.". How does this motivate a mixture modelling approach? Further, without having read [3], how is the following not true: Imagine client 1 learning a feature extractor through some unsupervised fashion on a small vs. a huge amount of data. Client 2 will benefit from the shared feature extractor trained on a large data-set more than from the model trained on the small data-set, presumably through its better learned representation. 
Going further, the authors introduce their mixture model, before showing how it generalizes existing approaches. Comparing against [47], though, FedEM requires sending all M=C models to all clients, whereas [47] only ever communicates 1 model. 
The authors proceed to describe learning in the mixture model, both in the centralised and in the federated setting. Then the authors introduce a decentralised variant as well as a generalised 'federated surrogate optimization' few, without showing experiments in the later section. It seems to me that the paper would be improved by moving those two extensions to the appendix / a separate paper, as it doesn't add to the main contribution of the paper, which is the mixture model optimised with EM in the FL setting. 

The experimental section is characterised by breath of data-sets considered, as well as competing methods. 
There are, however, a few trivial baselines that I would like to see included:
The authors consider personalised model accuracy, i.e. not the performance of a global model at the server (as in FedAvg). As such, the authors need to include the performance of FedAvg when the last global model is allowed to be fine-tuned on the local data-sets for an appropriate amount of steps. Assuming $D_{train,t} \sim p(D_t)$ and  $D_{test,t} \sim p(D_t)$, fine-tuning should improve the performance of the global model. Moreover, due to the limited size of local training-sets, overfitting needs to be taken into account - similarly to how the authors speculate about $\pi_t$ to overfit due to its localised nature. 

Another trivial baseline to compare FedEM(m) with is a model with m-times the parameter count. It needs to be established that the additional model performance is not due to additional model parameters and instead indeed due to the mixture formulation. Furthermore it would be interesting to see the entropy of $\pi$ to understand if specialisation is happening. Is there a guarantee or at least an empirical observation that EM can recover a 'true' clustering, i.e. assuming all clients within a cluster share the same data-distribution? 

The authors are talking about a synthetic dataset, but do not include what that is about (also not in the appendix). 

I find it peculiar that except for Figure 1, all experiments do not include client sub-sampling. Full participation is a strange assumption to make in FL and I would consider the experimental section much stronger if the authors consider the more established settings. E.g. FEMNIST usually has >3.4k clients, sampling around 100 clients per round. Otherwise the authors should re-focus their claims for the cross-silo FL setting. 
Furthermore the authors make the in my opinion very strong assumption that there will be a large-enough labelled training set available at a new client. What I would like to see is the average performance of new clients with zero labelled examples (and therefore uniform $\pi$, adding progressively more labelled examples, up until the full local labelled training set is assumed to be available. 

I see that the code to reproduce experiments is attached in the supplementary, however I would appreciate a table of hyper parameters in the actual paper (appendix). 

I have not read the proofs of various propositions/theorems in the paper. 

If the authors improve their empirical evaluation along the lines that I have suggested, I will consider raising my score.

limitations_and_societal_impact: The authors discuss the 'Fairness' of their method, both in the experimental section as well as in Appendix L. While it is clear to me what the exact definition of fairness is that the authors relate to, I suggest acknowledging the overloaded nature and general unclear definition of that term, also in connection with FL. More specifically it might be the case that "Even clients with the worst personalized models are still better off when FedEM is used for training", however it might still not be 'fair' as some clients benefit more (absolutely or relatively) than others.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a new approach for personalized federated learning (PFL) by formulating PFL as multi-task learning (MTL). by assuming that each local data distribution is a mixture of unknown underlying distributions the model aims to learn a linear combination of clients' models based on their data distribution.

The authors claim the following contributions:
-  Showing that federated learning (FL) is impossible without assumptions on local data distributions
- Proposing FedEM and D-FedEM an expectation-maximization (EM) based approaches for PFL
- The proposed methods achieve better accuracy and generalize better than SOTA PFL methods.




main_review: **Pros/Cons**:

[+] The paper is well written and structured.

[+] Good Theoretical analysis of FedEM's convergence.

[+] The proposed methods achieve higher performance compared to previous methods over different datasets and federation sizes.

[-] The authors did not address more recent approaches in PFL [1,2] and did not compare to them. FedFomo[1] aims to learn a linear combination of clients' weights W/O assuming anything about underlying data distributions and pFedHN[2] performs better than pFedMe under identical data heterogeneity learning settings.

[-] The authors should provide STD analysis on their experimental results (i.e. running on multiple seeds)

[-] "the learning rate and the other hyper-parameters were tuned via grid search" - on "Datasets and models" subparagraph the authors describe the data splitting as follow " For all tasks, we randomly split each local dataset into training (80%) and test (20%) sets. **This implies that the HPs were chosen based on the test split!**

[-] Generalization to unseen clients - “it is not clear if and how the other personalized FL algorithms can tackle the same goal.” I totally disagree with this statement. PFL methods can handle situations in which new clients join the federation. Let us take pFedMe for example, after the training process is finished the Hub/Server obtains a global model which is optimized based on the federation. This model can be sent to the new clients to be fine-tuned locally. 

[-] All the clients in the federation take part in each communication round - this is a big limitation for large-scale federations, in real life FL systems federations can contain millions of clients such that training this federation will cost lots of computational power and money. Other PFL methods sample a subset of clients at each COM round (most of them sample $5$ clients per round).
This limitation raises another question, have the authors run all methods on the same number of communications? each COM round in FedEM performs $T$ communications and $T$ denotes the number of clients while for pFedMe $5$ communications occur per COM round. In FL we care about the number of communications it takes the model to converge.

**References**:

[1] Zhang, Michael, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. "Personalized Federated Learning with First Order Model Optimization." arXiv preprint arXiv:2012.08565 (2020).

[2] Shamsian, Aviv, Aviv Navon, Ethan Fetaya, and Gal Chechik. "Personalized Federated Learning using Hypernetworks." arXiv preprint arXiv:2103.04628 (2021).
limitations_and_societal_impact: The authors should discuss the possible societal impact of their work.
review_title: null

Meta-review: The reviewers are generally in favor of accepting the paper for its algorithmic and theoretical contributions on federated multi-task learning. Based on that, I recommend acceptance. However, please make sure to incorporate the reviewers' comments and the rebuttal into the final version. 
Decision: Accept (Poster)