Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies the application of techniques from meta-learning (a method
to train a single model which can then be easily adjusted to perform well on
multiple tasks) to federated learning (the task of distributed training of
models on distributed datasets).  The paper notes that standard meta-learning
algorithms are similar to standard federated learning algorithms, and uses
this perspective to produce a merged method and evaluate it empirically.

Pros.
+ The motivation of the paper is clear and indeed these methods seems similar,
 and meta-learning can help with federated learning.

Cons.
- The resulting method appears somewhat underdeveloped; it is simply to run
 some amount of federated learning and then some amount of meta-learning,
 whereas the first parts of the paper led me to believe that a single
 simultaneous merge of the methods is the way to go.  The paper does not
 report any fine-grained evaluation of various such choices, thus I don't know
 why they did that they did, and thus do not find their choices compelling.
- The Reptile method is already presented in the original paper with
 a distributed counterpart, so why not just run that?  I am not convinced that
 some more minor modification of Reptile could not already do well on this
 paper.
- The empirical evaluation is not very extensive, so I am also not convinced
 there, and in particular I need convincing of this type to believe that
 regular reptile is beaten by FedAvg+reptile.

Minor comments.
Page 1, second paragraph, the word "outperform".  I'm not sure what the
performance measure is; in federated learning, we care about many things, for
instance privacy, keeping the work on the distributed clients low, etc.
Page 2, the "three objectives".  I feel meta-learning is doing all three too.
Page 3, Algorithm 1.  I realize space is a concern, but this was hard to read.
Page 4, Algorithm 2.  "relatively larger" is vague.
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: Update: I thank authors for the rebuttal. I agree that direction of exploring personalization in FL is interesting. With a stronger methodological contribution, this could become a good paper.

----------------------------------------------------------------------------------------------------------------
The main contribution of this paper is to notice the connection between Federated Averaging (FedAvg) and Model Agnostic Meta Learning algorithms (MAML). Authors also consider an algorithm that first trains with FedAvg and then continues training using Reptile.

Pros:

Interpretation of FedAvg as a meta-learning algorithm is interesting.

Cons:

Very limited methodological contribution. Proposed algorithm is essentially two existing algorithms applied one after another.

Experiments are not conducted rigorously enough. There are many arbitrary hyperparameter choices which may bias the conclusions made in the paper. Statement "We tried SGD with a range of other learning rates, and in all cases we observed this value to work the best." is alarming suggesting that authors tried a variation of settings observing test data performance and reported a few selected runs. Although "each experiment was repeated 9 times with random initialization", the train/test split of the clients was fixed. Randomizing over client train/test split could help to improve the reliability of the results.

EMNIST-62 is the only dataset analyzed in some detail. This dataset has drastically varying P(y|x) across clients, i.e. some people write zeros as some others write 'o's. This suggests that it is very hard to train a good global model and personalization is necessary. However this doesn't mean that Shakespeare dataset "does not provide any interesting insights". Perhaps, it is indeed more interesting and challenging, demanding more advanced methodology.

In Figure 1, number of communication rounds may be impractical for FL (considering also addition 200 Reptile rounds). On Shakespeare, FedAvg paper reports 54% accuracy achieved in under 50 communication rounds in one of the settings. There are also recent works on improving communication efficiency that were not discussed or studied for personalization quality, e.g. FedProx from "Federated Optimization in Heterogeneous Networks" and PFNM from "Bayesian Nonparametric Federated Learning of Neural Networks".

Questions about Figure 2 experiments:
1. Fine-tuning requires 200 extra epochs over the initially trained model. What's the initial model accuracy when FedAvg is further trained with Adam optimizer for 200 extra communication rounds?
2. The personalized test accuracy with FedAvg and Reptile fine-tuning reaches the same value in 10 update epochs, even when Reptile fine-tuning gets 200 extra initial training epochs. Does Reptile fine-tuning provide additional benefits to the initial model as compared to running FedAvg for more number of epochs?
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper considers personalization federated learning problem in which the goal is to personalize the global model on a given client/device based on available data on that device/client.  The paper claims not only their proposed method can lead to fast convergence time but also provide a solid initial model per device/client and results in a better-personalized model. To evaluate the performance of their method, EMNIST-62 and Shakespeare data are used.

Even though personalization in federated learning is very interesting and challenging, I am not sure about the contribution of this paper and what is exactly proposed in this paper: 

1)  Section 2: this paper shows the relationship between FedAvg and MAML. In my view, the connection is very straight forward and can be shown in a couple of sentences. I might be missing something here, but it is not obvious to me what this paper adds to the connection between MAML and FedAvg.  

2)  Personalized FedAvg Section: The same is about section 3. In my view, Algorithm 2 doesn't say anything new rather than to use Adam in local machine and SGD on global models and to optimize for "E" steps. But what if we use other datasets rather than EMNIST-62 and Shakespeare? will these recommendations still hold, i.e. using SGD on server and Adam on the devices? Per section 3 of this paper, Algorithm 2 indeed is the result of the experimental adaptation of the FedAvg algorithm so generalization to other datasets won't be obvious and it is a big question to me.
 
3) Also, the paper mentioned that this method can work even if there is no local data available on some of the devices/clients. I wasn't able to understand how personalization possible if there is no data to personalize. Wouldn't a device/client just use the global model?

In summary, I find the contribution and novelty of this paper limited and the empirical findings of this paper can't be always applicable to other datasets and scenarios. Plus, I am not convinced this paper shows anything different than FedAvg rather than some recommendations about local and global optimizer selections.
review_title: Official Blind Review #1

Meta-review: The reviewers have reached consensus that while the paper is interesting, it could use more time.  We urge the authors to continue their investigations.
Decision: Reject