Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies a new problem setting called few-round learning in federated learning, and proposes a meta-learning based method to solve the problem. Experiments on real datasets are used to verify the effectiveness of the proposed method. 
main_review: This paper studies a new problem setting called few-round learning in federated learning, and proposes a meta-learning based method to solve the problem. Experiments on real datasets are used to verify the effectiveness of the proposed method. 

The few-round learning problem seems to be interesting and useful in federated learning, because it can reduce the cost in federated learning. The proposed meta-learning based method seems to be reasonable. Furthermore, the proposed meta-learning itself can be completed via federated learning. Experiments on real datasets show that the proposed method can outperform baselines for arbitrary groups of clients. 

The experiments can be improved. The datasets used for evaluation are those for meta-learning evaluation which are relatively small, but this paper is for federated learning rather than meta-learning. Hence, adopting datasets suitable for federated learning can improve the convincingness of the proposed method. Furthermore, the paper does not introduce the hardware and platform of the experiments. Experiments on a platform of real applications (or similar settings) can also make the results more convincing. 

limitations_and_societal_impact: No
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a problem that is based on both meta-learning and federated ideas.  They call this few-round learning, where the goal is to prepare an initial model that can quickly adapt to any group of clients within only a few rounds of FL. The problem is in my opinion significant and has a lot of practical application. The introduce a meta-learning algorithm using few rounds of FL followed by inference, to be performed by a group of clients on a possibly unseen task. The also show experiments that the scheme outperforms  existing  pre-training  approaches including fine-tuning via FedAvg and personalized FL in both IID and non-IID scenarios. I think the paper has core value for the FL field. 
main_review: The paper reads well and contributes a new solution to a unique and interesting problem. The authors have explained the problem and the importance of it in the field if federated learning very well. The main contribution in my opinion is  the fact that they build a global model that can be re-used for several tasks with few learning episodes at the local clients. In their words, "The global prototypes serve as prior knowledge, a form of regularization, and prevent local models from overfitting to the local data. Moreover, the global prototypes (reflecting all classes across clients) can assist the local model to learn a more general embedding space". So basically we have a system that learns the general features of the data at the global layer and relies on local models for specific features in the local data. More over these local models can benefit from the meta-models to have better generalization behavior. I think authors can do a bit of a better prior work summarization on the FL field and introduce comparisons with other FL techniques that outperform FedAvg. 



limitations_and_societal_impact: I am not quite sure why there is a section comparing the results with the personalized FL while the goal here is obviously (and as stated in the paper) different. It would have been better if other FL methods apart from FedAVG were considered for the comparison instead of the focus on comparison with personalized FL. Personalized FL is in my opinion the opposite problem to the problem at hand so not sure of the value of the comparison although we see improvement on the personalization part. 

A few papers worth adding in references and comparisons:

https://proceedings.neurips.cc//paper_files/paper/2020/hash/fb2697869f56484404c8ceee2985b01d-Abstract.html
https://openreview.net/forum?id=BkluqlSFDS
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method for using meta-learning to train an initial model, which is intended to achieve optimal performance on an arbitrary downstream tasks when trained for R rounds of federated learning (for some small R). The paper presents aa meta-learning-flavored algorithm for conducting the pretraining step, and provides theoretical bounds on the gradient of the resulting loss (Theorem 1). The paper includes some empirical results on CIFAR100, miniImageNet, and FEMNIST, with comparison to some other baselines (although no baselines exist from other works for this particular task, according to the authors).
main_review: Overall: In general, I found the paper to address a task which seems relevant to the quickly-growing area of federated learning. This seems to be an extremely relevant task to introduce to the field, and the paper's approach is logically presented. I found the theoretical analysis somewhat lacking, since Theorem 1 only bounds the loss gradients (and not the loss), which does not seem to translate into a useful measure of model accuracy (only convergence to some value of \epsilon). It is difficult to evaluate the empirical results given the baselines which are not really baselines (since they are optimized for entirely different scenarios), but the empirical results seem good. In general, I think that the paper would be a candidate for acceptance if the feedback below is satisfactorily addressed.

Major Comments:

* The authors note that no prior work has addressed this task. However, this makes it difficult to interpret the results of the empirical experiments, which essentially compare to three techniques not applicable to the "few-round" task, plus three versions of the proposed algorithm; effectively the other "Methods" (three versions of FRL) amount to an ablation study. I would encourage the authors to explore better baselines -- perhaps something from the meta-learning literature, at the very least, to give a sense for how those approaches may compare. This would give a dual "FL only" and "meta-learning only" perspective on why the combined FL+meta-learning approach is superior.

* The "non-IID" scenario described at L295 is a very weak model of non-IID-ness, so much so that I hesitate to refer to it as such. I would recommend to either find a better non-IID scenario (examples abound in the FL literature), clarify why this is comparable to other non-IID FL tasks, or remove the "non-IID" framing.

* It would be useful to provide some motivation for why meta-learning in particular is motivated in this scenario, as opposed to some form of unsupervised pretraining which has performed quite well for a large variety of downstream tasks e.g. in the NLP literature (for example, the recent work on T5).

* The authors make significant efforts to differentiate their task of so-called "few round learning" from existing FL tasks such as personalized learning. The difference is clear; however, the authors could do more to motivate the significance of this task. While I think there are clear arguments for this, it would be useful to make them explicit in the paper.

* I don't think the term "few-round learning" in the paper, nor the title itself, accurately represent the work described in this paper; the existing characterization make it sound like "doing FL in the fewest rounds possible". I do not have a great suggestion, but something more like "Meta-Learning for Few-Round Downstream FL", while less elegant, seems more accurate.

* The paper as-is is not self-contained; it relies on class prototypes and distance metrics only described in [18]. Please clearly describe the necessary components in the paper directly.

Minor Comments:

* The confusing phrase "the task of the group conducting FL" is used in both abstract and intro (L38). I would recommend something like "the downstream task for which the pretrained model will be used" instead.

* Should the equation f(...) on L52 also depend on the task, or somehow indicate that this is over all downstream tasks?

* The terms "query set" and "support set" are never formally defined in the paper, nor are the methods for assembling them clearly described, to my reading. Similarly, I do not think that "class prototypes" are clearly described. Please clarify.

* I thinkSection 2.2.1 could be better titled and organized. It is essentially a walk-through of Algorithm 1 (which could be better reflected in the title); also, each bolded section should refer to specific relevant lines in the Algorithm.

* The assumptions given in Section 3 are quite strict. Are these met in the experiments, or can we approximately compute them? It seems these could also create opportunities for an adversary to break the guarantees by injecting samples/clients which deliberately create large variance or large gradient variance. This could be worth at least a brief discussion or clarification.

* It isn't clear to me how Fig 3(c) has more "clearly desirable" (L354) properties than 3(b). Please clarify.

Typos etc.

* Fig 1 caption: "hope to classify an arbitrary task" --> hope to perform an arbitrary classification task
* What is meant by "actual inference" in Fig. 1 caption? Please clarify.
* L74: "via episodic" --> via an episodic
* L125: "diseases that would" -> diseases than would
* Line 11 of Algorithm 1 should refer to Equation (3).

**Update**: I have read the author response. This largely addresses my main concerns with the paper, enough so that I will increase my rating from 6 to 7. I do hope the authors will consider further clarification regarding the "downstream" target of the FL in the title and/or abstract; it is my opinion that readers will benefit from this clarification.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper designs an initial global model which adapts to few round federated learning for possibly unseen tasks.  A meta-learning based approach with a global prototype-assisted learning strategy is built to tackle this problem. 
main_review: Although the authors take advantage of known techniques from MAML and prototype assisted learning strategies, I would say that this paper would be of interest for the federated learning community as the studied problem is really practical. It can be seen as a way to reduce the high communication cost introduced by the classic federated learning if every time the algorithm needs to be rerun for new tasks. The authors initiate the study of this problem and propose an efficient method with theoretical guarantees.   

The paper is written quite well and I like the precise description step by step for their algorithm. However in the paragraph of “key idea” of Introduction, lines 51-53 are not really clear for who starts reading from the beginning. The notations are introduced in a sudden without precise explication such as D_k (but I got the idea of this part after reading the algorithm). This is just a minor suggestion for the flow of the paper. The experiments are conducted for image classification tasks, and the proposed method is compared with multiple baselines which are quite fair. The description of the details of the experiments are clear and well organized. The performances of the proposed method are quite convincing.

Just have a few questions related to the communication cost of the novel method:
1. For GPAL, at the deployment stage, the clients will need to train $\Gamma^{R-1}$ as well (e.g. the client needs to send the local prototypes). Will this add too much communication budget to few-round federated learning when the number of the classes is large? If the number of clients in the deployment stage is large, this may worsen the communication bottleneck.
2. To save the communication cost in total for deployment (wrt the number of rounds to reach a given accuracy), which value of $R$ is more interesting? Will there be an optimal one? I saw the supplementary material part of the dismatch R part and Target R=5, which somehow suggests that R=3 may be better. I would like to have the authors' insight on it.

In total, I would recommend this paper to be accepted. 

limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper proposes a meta-learning approach for federated learning which allows arbitrary groups of clients can obtain a good global model in a few communication steps.

The reviews were quite positive and the author response (which contained additional experimental results) helped to further consolidate the scores. Overall, the problem of few-round FL and the proposed approach were deemed to be nicely motivated, interesting and useful in practice.

Therefore, the paper is accepted.
Decision: Accept (Poster)