Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes the use of Federated Averaging for achieving personalised user embedding. Federated Learning is used whether they propose a particular split of model parameters with user embedding (private) and the overall BLSTM model (shared). Federated Averaging is used for the global update.

The key contribution of this paper is not clear. It seems to be the introduction of the notion of split-personalisation-constraint, and it shows that the modeling each user with a “private” embedding that feeds to a global MLP with a global BLSTM as another input (named as FURL) can achieve the constraint so that FL can be used. The originality is limited.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose using federated learning (FL) to train personalized models, which improves the scalability and privacy preservation of the existing personalization techniques. The empirical results show good performance.

However, in general, I think the contribution is limited. The reasons are as follows:

1. The proposed algorithm, FURL, is a direct and simple combination of personalized model and FL. Although the authors claim that there is significant improvement in the performance, such improvement comes from the personalization. And, the personalization itself is not a novel thing (I think the personalized model used in this paper is similar to [1] or some other references. Please correct me if the personalized model used in this paper is new, since I'm not an expert in personalization.) Thus, in general, this paper simply use FL to replace fully synchronous SGD in the training of the personalized models. All the benefits claimed in the introduction, including scalability, privacy preservation, and improvement of performance, come from either vanilla personalization or vanilla FL. I fail to find any new contribution in this combination.

2. The authors emphasize a lot on the "independent aggregation constraint". Although it sounds like such constraint is designed especially for FL + personalization, it is actually a feature only for personalization, which has nothing to do with FL. Note that when doing inference/prediction, each user uses his/her own private part of the model. Different users' private part of models will never affect each other. It is equivalent to training a global model, which concatenates the private parts of models into a big model, and each user update the global model in a sparse manner. Thus, we can also train such personalized model with fully synchronous SGD with sparse gradients, which also does not synchronize the private parts. The private part is never shared by different users, no matter trained by fully synchronous SGD or FL.


------------
References

[1] Jaech, Aaron, and Mari Ostendorf. "Personalized language model for query auto-completion." arXiv preprint arXiv:1804.09661 (2018).
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: Authors proposed a formal training scheme (FURL) for personalized federated models. The claimed benefits of such models are 1) preservation of user privacy by keeping the personalized parameters locally on each user's device, and 2) reduced data exchange to make the training complexity grow linearly with the number of users. Authors approached the problem with defining the constraint of split personalization, and argued that common FL setting such as Federated Averaging could satisfy this constraint. 

Authors designed a personalized classification deep network for two data sets, namely Stickers and SubReddit. Both tasks could benefit personal preference in addition to textual features: Stickers CTR depends on user's adoption of the feature; SubReddit categorization depends on user's past activities in each sub-Reddit. A clearly conducted experiment showed that personalization has a significant contribution in non-federated setting, and using FURL in federated setting achieved similar performance while non-personalized FL may suffer bigger loss (in the case of SubReddit). Authors also compared the conversage curve and visualized final embeddings to show that federated learning produces acceptable convergence and equally reasonable embeddings.

The paper is well written and all claimed contributions are well articulated. Reviewer didn't find any significant problems.

Reviewer has limited knowledge of previous work in the personalized FL field, thus is only able to confirm the novelty from Authors' related work section.

One comment about formatting: in Figure 5, the color dots in the legend could be larger for easier identification. Please also consider some color-independent label/description to help readers with difficulties in color perception. For example, you can name the color in the legend (i.e. "Red") and provide some text labels in the embedding chart to tell which part is mostly red.

review_title: Official Blind Review #3

Meta-review: This manuscript personalization techniques to improve the scalability and privacy preservation of federated learning. Empirical results are provided which suggests improved performance.

The reviewers and AC agree that the problem studied is timely and interesting, as the tradeoffs between personalization and performance are a known concern in federated learning.  However, this manuscript also received quite divergent reviews, resulting from differences in opinion about the novelty and clarity of the conceptual and empirical results. Reviewers were also unconvinced by the provided empirical evaluation results. 
Decision: Reject