Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ===========================Post rebuttal===========================

I thank the authors for their responses and additional experiments. I understand that the focus of the paper is the cross-silo setting, however one of the key questions of an empirical study is to identify the limitations of the proposed approach. Experiments in Tables 8 and 9 still consider a relatively small number of clients and do not provide empirical insights into when the proposed approach begins to degrade. For future revisions, I recommend an empirical exploration that helps the reader to understand the limitations of the proposed method.

==================================================================

This paper explores the idea of knowledge transfer applied in the federated learning setting. Authors also consider variations of their algorithm under different privacy constraints.

Both knowledge transfer and differential privacy mechanisms are borrowed from prior works, limiting the methodological contribution of the paper. However, it was interesting to see these techniques applied in the FL context. The applicability of the proposed method relies on each client having sufficient data to train multiple teacher models locally. It would be interesting to see a deeper empirical study of the local data size effect: for example by varying the number of clients in the MNIST/SVHN experiments (therefore reducing the size of the local datasets). I also recommend a quantitative comparison of the communication costs by plotting accuracies of FedKT and baselines against the number of bytes exchanged between clients and the server.

I would consider increasing my score provided additional empirical studies I suggested (especially the effect of the local data sizes).
review_title: Knowledge transfer applied to federated learning

final_score: rating: 4: Ok but not good enough - rejection
review_content: This submission proposes a new federated learning framework based on knowledge transfer. Local dataset at each party are partitioned and each partition is used to train a teacher model. All teacher models at each party are used to train a student model using pseudo labels based on voting on public dataset. Student model from each party is then uploaded to server and used to train the final model based on voting on unlabeled data. Differential privacy analysis is conducted and experimental evaluations comparing to other mainstream federated learning methods are presented. The advantages of the proposed method include privacy preservation, lower communication traffic, as well as applicability to non-differentiable models. While the proposed framework is technically sound, the reviewer is not convinced by its technical contributions. The design of the framework is integration of existing technics such as PATE, and the mechanism of protecting privacy as well as reduction of communication traffic is also not new (see FedMD: Heterogenous Federated Learning via Model Distillation,  Neurips 2019 Workshop and Ensemble Distillation for Robust Model Fusion in Federated Learning, Neurips 2020). There is no clear advantage in the proposed method over these existing methods from the reviewer’s point of view. Plus the overall performance on benchmark dataset seems to be degraded compared to other mainstream methods like FedAvg. The reviewer would like the authors to explain and discuss the technical contributions of the submission and compare the proposed framework to these similar existing methods based on knowledge transfer. 
review_title: Good privacy analysis but not enough novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: __Summary:__
The paper considers classification tasks in the federated learning scenario when each device/worker is powerful in terms of computational power and storage space, but, the communication between devices is constrained. The paper proposes a novel algorithm for federated learning that reduces the number of communication rounds to one. The algorithm constructs an ensembled model with the majority voting out of the locally trained models and then on the server side learns a final model by mimicking the performance of the ensembled model on a public dataset. 

The paper additionally provides a differentially private version of the proposed algorithm and proves its privacy guarantees; and performs an experimental comparison of the proposed method. 

__Main concerns:__
- The proposed approach is interesting and novel. However, I am not convinced that the setting is realistic. It is hard to imagine that devices with relatively high computational power are restricted so much on the communication side that cannot allow for more than 1 round. 
- the algorithm does not achieve the best accuracy. 
- The paper in general is hard to read. 
- there is no theoretical understanding of how good the algorithm works. I can imagine many cases where the proposed algorithm wouldn't perform well in practice (small size of the public dataset / small size of local data on the workers / highly non-iid data on the workers)
- the algorithm is quite complicated and it is unclear if all the components are required. E.g. I don't see why is it needed to do ensembling of models locally, but not to have one local model trained on the full local data. This applies to the other algorithm's components as well. 

__Ohter concerns:__
- Some related work on federated learning with knowledge distillation is missing, e.g. [1], [2], [3].
- The results are not reproducible: the tuning details and hyperparameters are not stated in the experiments
- some references are missing: e.g. for consistent voting on page 4. 
- what is the number of queries on page 8, section 5.3? 

[1]: D. Li, "FedMD: Heterogenous Federated Learning via Model Distillation"

[2]: T. Lin et al, "Ensemble Distillation for Robust Model Fusion in Federated Learning"

[3]: H. Chang, "Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer."

review_title: Review 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Paper summary:

This paper studies the  problem of  model-agnostic federated learning in the knowledge transfer framework.  They generalize the PATE framework in the federated learning setting by applying subsample-and-aggregate in each local agent. They provide theoretical analysis and empirical evaluation.

---------------------------------------

Merits:


The problem considered is indeed very important, as communication cost and privacy issue are two critical concerns in federated learning. 

----------------------------
Concerns:

My main concern is the lack of novelty in their privacy analysis. The main theoretical part (Theorem 2 and Theorem 3) comes directly from PATE. 

Moreover, the experiment part is insufficient. Noting most part of the paper focuses on differential private FedKT, I think the authors should provide at least one comparison with other privacy-preserving federated learning algorithms [1,2].

I have another concern about the feasibility of FedKT. FedKT requires splitting each local data into t splits and training a teacher model using each split. Noting a large t is well expected in the privacy setting, which may lead to poor performance on teacher models when local data is insufficient. One particular failure case is when the parties are mobile devices.

Regarding communication efficiency, I agree that FedKT is round-optimal. But I have doubts about the claim that FedKT's communication cost is lower than FedAvg.  The authors compare the overhead analysis with ```''nMs < 2nMr'' (the last paragraph on page 4) while ignoring the communication save from subsampling. FedAvg and its variants can benefit from subsampling clients (i.e., a subset of clients is sampled in each communication round). Moreover, in FedAvg and its DP variants [1,2], they choose a relatively large number of #parties while FedKT considers a small number of parties (#parties = 10). Note that this choice may degrade the performance of FedAvg in Table 1 as the variance across parties is larger. More could be done to establish that the experiments actually support the conclusions drawn about communication efficiency.
 
 
[1] Differentially Private Federated Learning: A Client Level Perspective.
[2] Learning Differentially Private Recurrent Language Models.

--------------------------------------
Overall this paper considers a very interesting topic but lacks some important experimental comparisons and method is with limited novelty.
review_title: this paper considers a very interesting topic but lacks some important experimental comparisons and method is with limited novelty.

Meta-review: This paper proposes a model-agnostic FL method called FedKT that performs only one communication round and reduces the communication complexity of federated learning. The reviewers have the following concerns about the paper:
* Limited novelty because the proposed method is directly based on PATE
* Insufficient experiments

The authors did a great job of responding to the reviewers' comments and also added some new experimental results in the updated version. But the reviewers still recommend significant revision of the paper and resubmission to a future venue. I hope the authors will find their constructive and detailed comments below helpful!
Decision: Reject