Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 1. Strength:

Targeting an important problem of FL: reducing the communication cost.


2. Weakness:

This work simply applies the meta-learning method into the federated learning setting. I can’t see any technical contribution, either in the meta-learning perspective or the federated perspective. The experimental results are not convincing because the data partition is not for federated learning. Reusing data partition in a meta-learning context is unrealistic for a federated learning setting. 

The title is misleading or over-claimed. Only the adaptation phase costs a few rounds, but the communication cost of the meta-training phase is still high.

The non-IID partition is unrealistic. The authors simply reuse the dataset partitions used in the meta-learning context, which is not a real federated setting. Or in other words, the proposed method can only work in the distribution which is similar to the meta-learning setting.

Some meta earning-related benefits are intertwined with reducing communication costs. For example, the author claimed the proposed method has better generalization ability, however, this is from the contribution of the meta-learning. More importantly, this property can only be obvious when the data distribution cross-clients meet the assumption in the context of meta-learning.

The comparison is unfair to FedAvg. At least, we should let FedAvg use the same clients and dataset resources as those used in Meta-Training and Few-Rounds adaptation.

“Episodic training” is a term from meta-learning. I suggest the authors introduce meta-learning and its advantage first in the Introduction.

Few-shot FL-related works are not fully covered. Several recent published knowledge distillation-based few-shot FL should be discussed. 



3. Overall Rating

I tend to clearly reject this paper because: 1) the proposed framework is a simple combination of meta-learning and federated learning. I cannot see any technical contribution. 2) Claiming the few round adaptations can reduce communication costs for federated learning is misleading, since the meta-training phase is also expensive. 3) the data partition is directly borrowed from meta-learning, which is unrealistic in federated learning.

---------after rebuttal--------

The rebuttal does not convince me with evidence, thus I keep my overall rating. I hope the author can obviously compare the total cost of meta-learning phase plus FL fine-tuning phase with other baselines. 


review_title: This work attempts to reduce the communication cost in federated learning by applying meta-learning. The proposed method requires a meta-training phase and a few-round training phase. The authors demonstrate its effectiveness using a dataset in the context of meta-learning to simulate the dataset in federated learning.

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper studied the combination of federated learning tasks in a meta-learning setting. In particular, with the assistance of the pre-trained meta-model, the new FL model's training can be completed within limited communication rounds. It was inspired by the meta-learning method used in few-shot learning scenario. This paper proposed a few-round learning (FRL) algorithm and designed global prototype-assisted learning (GPAL) scheme to assist training. It is an interesting topic to combine meta-learning with federated learning. 

The weaknesses of this paper are summarized below.

1. The proposed method updates meta-model in each client. However, the meta-learning task consumes lots of computation resources and highly relies on the large number of classes. These make it hard to train a meta-model in a local client in a federated system. Although the setting sounds useful, it is hard to realize in real-world applications.

2. This paper is relevant to two widely-known few-shot learning methods, MAML, and prototypical network. So, it is better to consider MAML+FL and/or ProtoNet+FL as baselines to make the proposed methods more convincing and prove the efficacy of the proposed loss functions.

3. Given the complexity of the proposed algorithm and associated hyperparameters, the authors could anonymously release the source code in the reviewing stage. More details about the experimental platform used in this paper should be given. 

4. As illustrated in the Experimental setup on page 6, the meta/pre-training phase needs a large number of communication rounds. Is it appropriate for the bandwidth-limited or time-sensitive applications? Will this be a distracter in few-round learning scenarios?

5. For the 5-way setup in Table 1, there are 5 classes are randomly sampled from the dataset in each episode, which means that all the clients contain all the training classes, 64 classes for miniImageNet and 351 classes for tieredImageNet, locally. This is impractical because most local clients only have limited information to share.

6. The representation of trainable parameters in Algorithm 1 is a little bit confusing. For example, \theta and \phi are actually the same parameters. The only difference between them is that \theta is updated during local update using the support set, while \phi is updated during local meta-update using query set. Since the algorithm is an important part of this paper, the definition and use of these parameters should be much clearer. If possible, the authors can add a detailed interpretation of these two parameters.


review_title: The paper needs more clarification to explain the proposed complex setting and procedure.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary

This paper proposes a new paradigm to train federated learning models. In particular, following the spirit of meta-learning for few-shot learning, the authors propose to meta-train an initial model so that starting from this point, only $R$ (eg, 3) rounds of FL are needed to produce a satisfying test accuracy.

## Pros
1. The authors made significant efforts in designing the meta-learning strategy for few-round FL.
2. The proposed algorithm has the potential to redefine FL training paradigm. But there should be more validations. My questions and concerns are stated in the next section.

## Cons
The major concern I have is about the way they construct the dataset and evaluate the algorithm. The training task the authors selected is more like a meta-learning standard setting and is not common in federated learning. So I doubt its performance in realistic FL settings. It would be great if the authors can evaluate their algorithm in a standard FL dataset, otherwise it is not convincing.
1. When constructing the meta-learning datasets for each episode, the authors sample several classes from the whole dataset  and then simulate 10 clients based on the selected samples. However, in FL setting, this is infeasible, as the server cannot access the whole dataset. The authors should describe how to construct the meta-learning procedure given hundreds or even thousands of clients without accessing their local data. For example, Shakespeare dataset has 715 train clients and 715 test clients. How to construct the meta-learning procedure from this decentralized data and how the algorithm performs are unclear.
2. The scale of FL is relatively small. At each episode, there is only 10 clients. However, in practical on-device FL, there can be thousands of clients for training and testing. For example, in [1], StackOverflow has 342,477 training clients and 204,088 test clients. Even EMNIST dataset has 3400 test clients. The performance of the proposed algorithm is unclear in these realistic large-scale FL problems.
3. The meta-train algorithm require the computation of full-batch loss at each round, which consumes more computational resources than vanilla FedAvg. The authors are supposed to discuss this additional overhead.

## Post-rebuttal comments
Thanks the authors for the response! I've read it and other reviewers' comments. I feel the authors didn't directly answer my questions and just reiterate what they have in the paper. Unfortunately, it is still unclear to me how to perform meta-training on standard FL training tasks, for example, shakespeare in [1]. In this training task, there're total 700+ clients. Does that mean in the meta-training phase, we need to sample 700+ clients for each episode? How to construct this meta-train dataset from a standard federated dataset?

[1] Reddi et al. Adaptive Federated Optimization. 2020
review_title: Review #1

final_score: rating: 3: Clear rejection
review_content: The paper is to train a meta-model in a small number of selected nodes in a federated learning environment, and then use the meta-model to assist the federated learning in reducing the communication rounds. It is basically a federated version of a prototypical network.

The proposed method relies on a strong assumption that there is a meta-training environment in federated learning. It is not a standard FL setting. Moreover, given the assistance of the meta-model, there is no guarantee that the federated learning environment will converge in a few-round.

The major technique contribution of the proposed method is how to meta-train a global model in a federated setting. In particular, it adapts the prototypical network to fit the federated setting. It is unclear how the proposed method provides any theoretical contribution rather than applied research. 

In the experiment, one dataset is not enough to support the effectiveness of the proposed method. 
More federated learning-related benchmark datasets should be discussed, e.g., FeMNIST, Shakespeare texts, CIFAR, and FeCelebA. 

In particular, the proposed two-stage procedure is equivalent to: learn a global model in a standard FL setting, and then conduct personalized deployment for each device or a specific group of devices. Therefore, in the experiment part, the authors need to add more baseline methods, for example, some personalized federated learning method should be selected as baseline methods.

THE MAJOR CONCERN: In Algorithm 1, lines 16 and 18 are a federated aggregation-based updating, and line 24 is a prototypical-based meta learner updating. These two updating methods are inconsistent which are to optimize different objectives, and the authors should give an overall loss to unify the updating steps rather than force two kinds of updating into one framework.

Typo: 
“Metra-training” in Figure 1.
review_title: This is a federated prototypical network.

Meta-review: This paper proposes a meta-learning based few-shot federated learning approach to reduce the communication overhead incurred in aggregating model updates. The use of meta-learning also gives some generalization benefits. The reviewers think that the paper has the following main issues (see reviews for more details):
* Limited technical novelty - the paper seems to simply combine meta-learning with federated learning
* Not clear whether the communication overhead is actually reduced because the meta-learning phase can require significant communication and computation.
* The experimental evaluation, in particular, the data distribution, could have been more realistic.

I hope that the authors can use the reviewers' feedback to improve the paper and resubmit to a future venue.


Decision: Reject