Model-Agnostic Round-Optimal Federated Learning via Knowledge Transfer
Authors: Qinbin Li
Bingsheng He
Dawn Song
Venue:  (2021)
Keywords: Federated Learning
Communication-Bounded Learning

Abstract: Federated learning enables multiple parties to collaboratively learn a model without exchanging their local data. Currently, federated averaging (FedAvg) is the most widely used federated learning algorithm. However, FedAvg or its variants have obvious shortcomings. It can only be used to learn differentiable models and needs many communication rounds to converge. In this paper, we propose a novel federated learning algorithm FedKT that needs only a single communication round (i.e., round-optimal). With applying the knowledge transfer approach, our algorithm can be applied to any classification model. Moreover, we develop the differentially private versions of FedKT and theoretically analyze the privacy loss. The experiments show that our method can achieve close or better accuracy compared with the other state-of-the-art federated learning algorithms. 

Under review as a conference paper at ICLR 2021
MODEL-AGNOSTIC ROUND-OPTIMAL FEDERATED LEARNING VIA KNOWLEDGE TRANSFER
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning enables multiple parties collaboratively learn model with- out exchanging their local data. Currently, federated averaging (FedAvg) is the most widely used federated learning algorithm. However, FedAvg its variants have obvious shortcomings. It can only be used to learn differentiable models and needs many communication rounds to converge. In this paper, we propose a novel federated learning algorithm FedKT that needs only single communica- tion round (i.e., round-optimal). With applying the knowledge transfer approach, our algorithm can be applied to any classification model. Moreover, we develop the differentially private versions of FedKT and theoretically analyze the privacy loss. The experiments show that our method can achieve close or better accuracy compared with the other state-of-the-ee federated learning algorithms.
INTRODUCTION
While the size of training data can influence the machine learning model quality lot, the data often dispersed over different parties in reality. Due to regulations on data privacy. the data cannot be centralized to a single party for training. To address these issues, federated learning (Kairouz iet al. 2019 Li et al. 2019a by Yang et al. 2019) enables multiple parties to collaboratively learn model without exchanging their local data. has become hot research topic and shown promising
results in the real world (Bonawitz et al. 2019 Hard et al. 2018 Li et al., 2020a Peng et al. 2020).
Currently, federated averaging (FedAvg) (McMahan al. 2016) is a widely used federated learning algorithm. Its training is an iterative process with four steps in each iteration. First, the server sends the global model to the selected parties. Second, each of the selected parties updates its model with their local data. Third, the updated models are sent to the server. Last, the server averages all the received models to update the global model. There are also many variants of FedAvg (Li iet al., [020c] Karimireddy et al. 2020). For example, to handle the heterogeneous data setting, FedProx (Li et al.j 2020c)) introduces an additional proximal term to limit the local updates, while SCAFFOLD Karimireddy et al. 2020) introduces control variates to correct the local updates. The overall frameworks of these studies are still similar to FedAvg.
FedAvg or its variants have the following limitations. First, they rely the gradient descent for optimization. Thus, they cannot be applied to train non-differentiable models such as decision trees in the federated setting. Second, the algorithu usually needs many communication rounds to finally achieve good model, which causes massive communication traffic and fault tolerance requirements among rounds. Last, FedAvg originally designed for the cross-device setting (Kairouz et al. 2019). where the parties are mobile devices and the number of parties is large. In the cross-silo set- ting where the parties are organizations or data centers and the number of parties is relatively small, it is possible to take better advantage of the computation resources of the parties with relatively high computation power.
In order to address the menoreene limitations, we propose novel federated learning algorithm called FedKT (Federated learning via Knowledge Transfer) focusing on the cross-silo setting. With the round-optimal design goal, FedKT extends the idea of ensemble learning in a novel 2-tier design to federated setting. Inspired by the success of the usage of unlabelled public data in many studies (Papernot et al.) 2017 2018 8 Jordon al. 2019 Chang al. 2019), which often exists such text and images, we adopt the knowledge transfer method to reduce the inference and storage costs of ensemble learning. As such, FedKT is able learn any classification model including differentiable

---

Under review as a conference paper at ICLR 2021
models and con-differentiable models. Moreover, develop differentially private versions and theoretically amalyze the privacy loss of FedKT in order to provide different differential privacy guarantees. Our experiments on four tasks show that FedKT has quite good performance compared with the other state-of-the-art algorithms.
Our main contributions are follows.
We propose a new federated learning algorithm named FedKT. To the best of our knowl- edge, FedKT is the first algorithm which does not have any limitations on the model archi- tecture and needs only single communication round.
We show that FedKT is easy to achieve both example-level and party-level differential privacy and theoretically analyze the bound of its privacy cost.
We conduct experiments on various models and tasks and show that FedKT can achieve comparable accuracy compared with the other iterative algorithms. Moreover, FedKT can be used as an initialization step to achieve a better accuracy combined with the other ap- proaches.
BACKGROUND AND RELATED WORK
2.1 ENSEMBLE LEARNING
Instead of using a single model for prediction, ensemble learning (Zhang & Ma 2012 combines the predictions of multiple models to obtain better predictive performance. There are many widely used ensemble learning algorithms such as boosting (Rätsch et al.] (201)) and bagging (Prasad et al.) 2006). One important factor in ensemble learning is the model diversity. The increased model diversity can usually improve the performance of the ensemble learning. In federated learning, since different parties have their own local data, there natural diversity among the local models. Thus, the local models be used 2019 Guha et al., 2019) have studied ensemble learning for federated learning and demonstrated promising predictive accuracy. As mentioned in their studies, since the prediction involves all the local models, the inference and the storage costs are prohibitively high especially when the number of models is large. In our study, we also use the local models as an ensemble and further use knowledge transfer to learn a single model in order to reduce the inference and the storage costs.
ensemble for prediction. Previous works (Yurochkin al.
2.2 KNOWLEDGE TRANSFER OF THE TEACHER ENSEMBLE
Knowledge transfer has been successfully used in previous studies (Hinton et al. 2015 Papernot et al. 2017] 2018: Jordon et al., 2019). Through knowledge transfer, an ensemble of models can be compressed into a single model. A typical example is the PATE (Private Aggregation of Teacher Ensembles) (Papernot et al.] 2017) framework. In this framework, PATE first divides the original dataset into multiple disjoint subsets. A teacher model is trained separately on each subset. Then, the max voting method is used to make predictions the public unlabelled datasets with the teacher ensemble, i.c., choosing the majority class among the teachers as the label. Last, a student model is trained the public dataset. A good feature of PATE is that it easily satisfy differential privacy guarantees by adding noises to the vote counts. Moreover, PATE can be applied to any classification model regardless of the training algorithm. PATE not designed for federated learning. Inspired by PATE, we propose FedKTT, which adopts the knowledge transfer approach in the federated setting to address the limitations of FedAvg.
2.3 FEDERATED LEARNING WITH A SINGLE COMMUNICATION ROUND
There are several preliminary studies on federated learning algorithms with single communication round. Guha et al. 2019) propose an one-shot federated learning algorithm to train support vector machines (SVMs) in both supervised and supervspe settings. Instead of simply averaging all the model weights in FedAvg, Yurochkin et al. (2019) propose PFNM by adopting a Bayesian nonparametric model to aggregate the local models when they multilayer perceptrons (MLPs). Their method shows a good performance in a single communication round and can also be applied in multiple communication rounds. While the above methods designed for specifie models
2

---

Under review as a conference paper at ICLR 2021
(i.e., SVMs in Guha et al. (2019) and MLPs in Yurochkin et al. (2019),
propose general
federated learning framework which is applicable to any classification model.
2.4 FEDERATED LEARNING WITH KNOWLEDGE TRANSFER
There are some related studies L & Wang 2019 Chang et al. 2019 using knowledge transfer in federated learning. However, Li Wang (2019) has a different setting with us while Chang et al.
(Li & Wang 2019 public labeled dataset is needed to conduct initial transfer learning, while FedKTT only needs public unlabelled dataset. Chang al. (219)) designs robust federated learning algorithm protect against poisoning attacks. The performance of their approach is slightly worse than FedAvg. We notice that there are some recently
2019) has different objective with
published contemporaneous work (He et al. 2020 Lin et al. 2020). He et al. (2020) considers cross-device setting and uses group knowledge transfer to reduce the overload of each edge device.
Lin et al. (2020) utilizes knowledge transfer only in the server side further update the averaged global model.
All existing approaches transfer the prediction vectors (i.g.. logits) on the public dataset between clients and the server. FedKT transfers the voting counts and thus can easily satisfy differential privacy guarantees with tight theoretical bound on the privacy loss. Moreover, FedKT is designed with a round-optimal goal, while the other approaches use iterative learning algorithms that needs many communication rounds to converge.
2.5 DIFFERENTIAL PRIVACY
Differential privacy (Dwork 2011 Dwork al. 2014) is popular standard of privacy protection.
guarantees that the probability of producing a given output does not depend much on whether a particular data record is included in the input dataset not. It has been widely used protect the machine learning models (Shokri & Shmatikov 2015 Abadi et al. 2016 Li et al., 2020b).
Definition 1. ((e, -)Differrentt Privacy) Let M: D - R be a randomized mechanism with do- main D and range R. MM satisifes (e. diffferrrtti privacy for any two adjacent inputs d, d' E D and any subset of outputs S S R it holds that:
Pr[M(d) E S] $ e Pr(M(dd)) E +
(1)
The moments accountant method (Abadi et al. 2016) is state-of-the-art approach to track the privacy loss. We briefly introduce the key concept, and refer readers to previous paper (Abadi et al. 2016) for more details.
Definition (Privacy Loss) Let M: - R be randomized mechanism. Let aux denote an auxiliary input. For two adjacent inputs d, d' E D, an outcome E R. the privacy loss at is defined as:
Pr[ Pr[MM(ux, d) = o]
c(o; aux, d, d') A log
Pr[M(aux, d') o]
Definition 3. (Moments Accountant) Let M: - R randomized mechanism. Let aux denote an auxiliary input. For two adjacent inputs d, d', the moments accountant is defined as:
(A
A max (M aux, d, d')
aux,d,
where a (A; aux, d, d') function.
A log (ex Exppp
aux, d, d'))] is the log of moment generating
The moments have good composability and can be easily converted to (e, -ddffferrnt privacy Abadi et al. (016)).
Party-level Differential Privacy In addition to the standard example-level differential privacy, party-level differential privacy (Geyer et al. 2017 McMahan et al. a 2018) is more strict and attractive the federated setting. Instead of aiming protect single record, party-level differential privacy ensures that the model does not reveal whether a party participated in federated learning or not.
Definition 4. (Party-adjacen Datasets) Let d, d' be two datasets of training examples, where each example is associated with a party. Then, d and d' are party- party-adjacent if d' can formed by changing the examples associated with single party from d.
3

---

Under review as a conference paper at ICLR 2021
Student
Party
Toaa
Data
Teacher
Dala
Teacher
Student
Dataset
Data
Teachert t
Kaowladge traske
Server
Final
Knoweege ttenke
Student s
Party n
Data
Teacher
Data 2
Teacher
Student
Dataset
Data
Teacher
Kamvedge tamkk
Figure 1: The framework of FedKT
FEDERATED LEARNING VIA KNOWLEDGE TRANSFER
Problem Statement Suppose there are n parties Pı... Pn. We use De denote the dataset of P. With the help of central server and public unlabelled dataset Dauns our objective is to build a machine learning model over the datasets Uke(m) D' without exchanging the raw data. Moreover, the learning process should be able to support three different privacy level settings: (1) LO: there is privacy requirement the machine learning models. (2) L1 (server-noise): in the case where the final model has to be sent back to the parties or even published, it should satisfy differential privacy guarantees protect against potential inference attacks (Shokri al., 2017; Fredrikson et al. 2015 Nasr et al., (2019). (3) L2 (party-noise): the case where the server is curious and all the models transferred between the parties and the server during training should satisfy differential privacy guarantees.
The Overall Framework The framework of FedKT is shown in Figure Overall, FedKT adopts a 2-tier PATE structure. On the party side, each party uses knowledge transfer to learn student models and sends them the server. On the server side, the server takes the received models as teachers to learn a final model using knowledge transfer again. The final model is sent back to the parties and used for predictions. Next, introduce these two steps details.
Learning Student Models on the Parties Locally, each party has to create (s 2 1) partitions and learn student model each partition. Each partition handles the entire local dataset. Since the operations in each partition are similar, here we describe the process one partition for ease of presentation. Inside partition, the local dataset is divided into disjoint subsets. We train teacher model separately on each subset, denoted as Ti, ..., T, Then, the ensemble of teacher models is used make predictions on the public dataset Dauz- For example x E Dauzy the vote count of class m is the number of teachers that predicts i.e., e. (x) = ki i E [t], Tx(x) = m)}. The prediction result of the ensemble the class that has the maximum vote counts, i.e., f(x) = arg max, V.n (x). Then, we use the public dataset Daur with the predicted labels to train a student model. For each partition, we get a student model with the above steps. After all the student models are trained, the parties send their student models to the server for further processing.
Learning the Final Model on the Server Suppose the student models of party are denoted as Uf...., Uy (i E [n]). After receiving all the student models, like the steps on the party side, the server can use these student models as an ensemble to make predictions on the public dataset Danz. The public dataset with the predicted labels is used to train the final model. Here we introduce technique named consistent voting for computing the vote counts of each class. If the student models of party example, we take their predictions into account. Otherwise, the party
make the same prediction not confident at predicting this example and thus we ignore the predictions of its student models. Formally, given example E Danzz first compute the count of class the student
4

---

Under review as a conference paper at ICLR 2021
models of party as u (x) vote count of class on all parties is computed as 4m (x) = 8 Ki i E [n)]. o (x) = s)
Ki l E [x))
m)). Next, with consistent voting, the final
Differentially Private Versions of FedKT FedKT can easily satisfy differential privacy guaran- tees by providing differentially private prediction results to the query dataset. Given the privacy parameter y, we can add noises to the vote count histogram such that f(x arg max, { (vm (x) + T.p())), where Lap((;) is the noises generated from Laplace distribution with location 0 and scale
5. Note that we do not need to add noises on both the parties and the server. For the L1 setting, we only need to add noises on the server side. The parties can train and send non-differentially private student models the server. For the 1,2 setting, only need to add noises on the party side so that the student models are differentially private. Then, the final model naturally satisfies differential privacy guarantees. More analysis on privacy loss will be presented in Section 4
Algorithm D shows the whole training process of FedKT. In the algorithm, for each party (i.e.. Line 1) and its each partition (i.e., Line 3), train student model using knowledge transfer (i.e., Lines 4-1 The student models are sent to the server. Then, the server trains the final model using knowledge transfer again (i.e., Lines 14-23). For different privacy level settings, we have the corresponding noises injection operations on the server side (i.e., Lines 20-21) or the party side (i.e., Lines 9-10).
Overhead Analysis of FedKT Suppose the size of each model is M. Then, the total communica- tion size of FedKT nsM + nM nM (s +1 including sending the student models to the server and sending back the final model to the parties. Suppose the number of communication rounds in FedAvg r and all the parties participate the training in every iteration. Then the total communi- cation size of FedAvg is 2nMr. Thus, when r > s+1 the communication cost of FedAvg is higher than FedKT. This value can be quite small, e.g., r = if we set s = 2. As we will show in the ex- periments, FedKT can achieve much better performance than FedAvg give the same communication size constraint. The computation overhead of FedKT is usually larger than FedAvg since FedKT needs to train many teacher and student models. It also requires the parties to be able to store mul- tiple models in the training process. Thus, FedKT is more suitable for the cross-sia setting (Yang al. 2019), where the parties (e.g., companies, data centers) have relatively large computation power and storage capacity.
2
Applications In our method, we assume that have access public unlabelled dataset. Note that the assumption will not greatly restrict our method's 's applicability. The public datasets widely exist especially for texts and images including medical data (e.g., Kaggle COVID-19 Open Research Dataset Challenge, Chest X-Ray Images). Thus, our approach can be applied typical computer vi- sion tasks, natural language processing tasks, and even healthcare tasks. Nevertheless, if the data are very sensitive and we cannot access to any public data, it is still possible to generate synthetic data using GAN with privacy guarantees (Yoon al. 2019 Torrkzadehmahani et al. 2019) on the parties. The synthetic data can be used as public data in our approach. Moreover, we have conducted exper- iments on a real-world scenario. We use one chest X-ray dataset the private dataset and another chest X-ray dataset as public dataset to simulate a real-world federated case. The experiments are available at Appendix [B77] which demonstrates that FedKT does not have strict limitation on the distribution of public data.
Besides the typical applications, the feasibility of single-round federated learning can enable new applications, e.g., model as a service. One can sell or buy the student models in a model market. Then, after the model trade, a final model can be learned using the student models without further communication.
Integration with Other Federated Learning Algorithms While FedKT is designed with a round-optimal goal, it is still applicable even if there is no limitation of communication rounds. We can use FedKT to train a model as an initialization step. The model can be seen as a initialized global model, and then we can continue to conduct iterative federated learning using the standard ap- proaches (c.g.. FedAvg. FedProx) based on it. As shown in Section 5.2, the communication rounds can be significantly reduced to achieve the same accuracy by applying FedKT as model initialization.
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1: The FedK'T' algorithm
Input: local datasets D .... D" of parties Pi.... Pn. number of partitions s in each party, number of subsets é in each partition, number of classes u, public query dataset D Pauz. privacy parameter privacy level
Output: The final model
for -,..,,, do
/* Conduct on party P3 */
Create partitions (i.c,, Di,,,,,) on dataset D' such that Dj = -ue(() Dj, for all
je [s]. where D i is subset. for j 1, do
for k = 1,... do Train teacher model Ti on subset Dj.--
for all x E Daua do for all E [2] do
Um (x) - {k: E (1),T,,,(x) = =m}
if = L2 then
10
Um (x) - Um (x) + Lap(1/7)
11
f(x) = arg max, Um (x)
Train a student model Ui on dataset ((x, ((x)))XED
12
Send the student models {U] j E [s]} to the server.
13
Conduct: on the server */ for all x Dauz do for all i [n] do for all m [u] do
15
16
o/n (x) < (k: k E (x). =mH
18
for all E [u] do
19
Um () + 8-
Hi E [m)], o (x) = l}e
if L1 then
Um xx) - Um (x) + Lap(1/7 (1/))
21
22
f(x) = arg max,, Um (x)
23
Train the final model on dataset {(x, f(x)) IxEDau
DATA-DEPENDENT PRIVACY ANALYSIS OF FEDKT
In this section, we use the moments accountant method (Abadi et al. 2016) to track the privacy loss in the training process. addition the example-level differential privacy, we also consider the party-level differential privacy (Geyer et al., 2017 McMahan et al. 2018), which is more strict and attractive in the federated setting. Instead of aiming to protect single record, party-level differential privacy ensures that the learned model does not reveal whether party participated in federated learning or not. Next, we mainly present the party-level differential privacy analysis for FedKT-LL and example-level differential privacy analysis for FedKT-L2. For more analysis and proofs of the theorems in this section, please refer to Appendix A
F Considering we change the whole dataset of party, then at most student models will be influenced. Thus, on the server side, the sensitivity of the vote count histogram is 2s (i.c., the vote count of class increases by and the vote count of another class decreases by s). According to the Laplace mechanism, we have the following theorem.
Theorem 1. Let M be the f function executed on the server side. Given the number of partitions s and the privacy parameter 7, M satisfies (28%; 0) party-level differential privacy.

---

Under review as a conference paper at ICLR 2021
Given multiple queries, straightforward way to compute the privacy loss is to the advanced composition (Dwork et al., 2014). According to Papernot et al. (2017). we can get tighter bound of the privacy loss by conducting data- deteadeeenddnt privacy analysis with the moments accountant method. Similarly, we conduct a data-dependent privacy analysis for FedKT.
Theorem 2. Let M be (2s7, 0) party-level differentially private. Let g 2 Pr[M(d) # o*] | for some outcome o*. Let y 2 0 and < 2 d', .M satisfies
For any aux and any two party-adjacent datasets d and T
am(laaux, d, d, S min(log((1 1) 1-
(11-22n)) + ). + 1)).
Here Pr[M(d)= 0²] can be bounded by Lemma of Papernot et al. (0017).
With Theorem 2 we can track the privacy loss of each query (Abadi al. 2016
FedKT-L2 For each partition the party side, we add Laplace noises to the vote counts, which is same with the PATE approach. Thus, we have the following theorem.
Theorem 3. Let M be the f function executed on each partition of a party, Let q 2 Pr(M(d) & o*] for some outcome o". Let 1, y 2 0 and - d', M satisfies
e1 For any aux and any two adjacent datasets d and
1- am(l;aux d, d') S min((og((I1)))) - (I (l-))gg) + ge (1-
1)).
After bounding the privacy loss of each party, we can use the parallel composition to bound the privacy loss of the final model.
Theorem 4. Suppose the student models of party Pa satisfy (Es, diffferrnnt privacy. Then, the final model satisfies (max, Ex, 5) diffeerentaal privacy.
Note that the above privacy analysis data-dependent. Thus, the final privacy budget is also data- dependent and may have potential privacy breaches if we publish the budget. Like previous studies (Papernot et al., 2017 Jordon et al. 2019). we report the data-dependent privacy budgets in the experiments. As future work, plan to use the smooth sensitivity algorithm (Nissim et al. 2007) to add noises to the privacy losses. Also, we may get a tighter bound of the privacy loss if adopting the Gaussian noises (Papernot et al. 2018).
EXPERIMENTS
To evaluate FedKT, we conduct experiments on four public datasets: (1) A random forest on Adult dataset. The number of trees is set to 100 and the maximum tree depth is set to 6. (2) A gradient boosting decision tree (GBDT) model on cod-rna dataset. The maximum tree depth is set to 6. (3) A CNN on MNIST dataset. The CNN has two 5x5 convolution layers followed with 2x2 max pooling (the first with 6 channels and the second with 16 channels), two fully connected layers with ReLu activation (the first with 120 units and the second with 84 units), and final softmax output layer. (4) The same CNN on extended SVHN dataset. For the first two datasets, we split the original dataset random into train/test/uublic sets with 75 75%/12 026/12.5% 22.5% proportion. For MNIST and SVHN, we use one half of the original test dataset as the public dataset and the remaining as the final test dataset. Experiments on two additional datasets, CelebA and chest X-ray images, are available at Appendix B.7 Like previous studies (Yurochkin et al.) 2019 Wang et al., 2020). we use the Dirichlet distribution to simulate the heterogeneous data partition among the parties. Suppose there are parties. We sample Pk Dir, (3) and allocate a Pk.j proportion of the instances of class to party j. where Dir(3) is the Dirichlet distribution with a concentration parameter 3 (0.5 by default). By default, and SVHN. We set to 2 and i to by default for all datasets. For more details in the experimental settings and guidelines to set the hyper-parametens (i.e., and f), please refer Appendix B
set the number of parties to 50 for Adult and cod-rna and to 10 for MNIST

---

Under review as a conference paper at ICLR 2021
Table 1:
The test accuracy comparison between FedKT and the other baselines.
FedAvg
FedProx
SCAFFOLD
Datasets
FedKT
SOLO
PATE
XGBoost
2 rounds
50 rounds
2 rounds
50 rounds
rounds
50 rounds
Adull
82.2%
68.6%
83.5%
cod-rna
88 3%
65.0%
91.1%
91.2%
MNIST
95.9%
80.0%
97.8%
83.5%
98.0%%
83.0%
98.0%
74.2%
98.4%
SVHN
83.2%
62.8%
86.6%
58.4%
80.3%
59.9%
86.9%
15.6%
84.5%
We compare FedKT with six baselines: (1) SOLO: each party trains its model locally and does not participate in federated learning. (2) PATE (Papernot et al., 2017): we use the PATE framework to train final model on all data in centralized setting (i.c., only single party with the whole dataset) without adding noises. This method defines an upper bound of learning final model using knowl- edge transfer with public unlabelled data. (3) XGBoost (Chen & Guestrin 2016): the XGBoost
algorithm for the GBDT model on the whole dataset in a centralized setting. This method defines an upper bound of learning the GBDT model. (4) FedAvg (McMahan et al. 2016); (5) FedProx
(Li et al. 2020c); (6) SCAFFOLD (Karimireddy et al., 2020). Here (4)-(6) are three standard or State-ortthe-as federated learning algorithms.
5.1 EFFECTIVENESS
Table 1 shows the accuracy of FedKT) compared with the other baselines. For SOLO, report the average accuracy of the parties. For FedAvg, FedProx, and SCAFFOLD, we report the results of running for 2 rounds (i.e., when the baselines start to have larger communication cost than FedKT as shown in previous overhead analysis) and 50 rounds. From this table, we have the following observations. First, except for FedKT, the other federated learning algorithms can only learn specific models. FedKT is able to learn all the studied models with a significant improvement compared with SOLO. Second, the accuracy FedKT close to PATE, which means our design has little accuracy loss compared with the centralized learning setting. Third, given the limited communication size constraint, FedKT can achieve much better accuracy than FedAvg, FedProx, and SCAFFOLD. The performance of the iterative methods is poor running with two rounds. Last, FedKT is just slightly worse than FedAvg, FedProx, and SCAFFOLD running for 50 rounds. FedKT gives promising results even with single round communication.
5.2 COMMUNICATION EFFICIENCY
Figureß showw the test accuracy with varying the number of communication rounds/communication size for MNIST and SVHN. We do not present the results for Adult and cod-rna since there is no baseline running in federated setting. For MNIST, FedAvg, FedProx, and SCAFFOLD need about 23 rounds, rounds, and 13 rounds to achieve a better accuracy than FedKT, respectively. For SVHN, FedAvg, FedProx, and SCAFFOLD need about 17, 18, and 33 rounds to achieve better accuracy than FedKT, respectively. Moreover, the communication size FedKT is close (see the black points in Figure 2c and Figure 2d). FedKT only needs 5.4M and 7.5M communication cost on MNIST and SVHN, respectively. Overall, FedKT is much more communication--ffice than the other approaches.
In Figure? 2 FedKT-Prox an approach that applies FedKT to learn initialized global model and then applies FedProx to conduct iterative learning. As we can see, FedKT- Troxx is always better than the other federated learning approaches. The communication round can be significantly reduced to achieve the same accuracy if adopting FedKT initialization. For example, for SVHN, FedKT-Prox only needs about 16 rounds to achieve 87% accuracy, while FedProx needs about 40 rounds.
5.3 DIFFERENTIAL PRIVATE FEDKT
We run FedKT with different y and number of queries. When running FedKT-L-1, we tune the percentage of number of queries on the server side. When running FedKT--L22, on the contrary, we tune the percentage of number of queries on the party side. The selected results on Adult and cod- While differentially private FedKT does not need any knowledge on
ma reported in Table 2
For simplicity, we FedKT to denote FedKT-L.0, unless specified otherwise.
8

---

Under review as a conference paper at ICLR 2021
100
Test Accuracy (%)
Test Accuracy (%)
90
80
60
SOLO
SCAFFOLD
SOLD
SCAFFOLD
PATE
FedKT-Prox
PATE
FedKT-Prox
FedAvg
FedKT
FedAvg
FedKT
FedProx
FedProx
40
1
35
10
25 30 35 40 45 50 #Communication rounds
#Communication rounds
(a) MNIST
(b) SVHN
Test Accuracy (%)
Test Accuracy (%)
70
60
SOLD
SCAFFOLD
SOLO
SCAFFOLD
PATE
FedKT-Prox
PATE
FedkT-Prox
50
FedAvg
FedKT
FedAvg
FedKT
FedProx
FedProx
40
25 50 75 100 125 150 175
0
50
100
150
200
250
Communication (MB)
Communication (MB)
(c) MNIST
(d) SVHN
Figure 2: The test accuracy with increasing communication rounds/commuiicttion size.
Table 2: The privacy loss and test accuracy of FedKT-L1 and FedKT--L22 given different and
number of queries. 1.0 acc is the test accuracy of FedKT-LO.
FedKT-LI
FedKT-L2
Datasets
#queries
acc
Y
#queries
L. acc
0.04
0.5%
2.56
76.8%
0.05
0.5%
3.24
79.0%
Adult
82.2%
82.4%
0.04
1.0%
4.73
80.2%
0.05
1.0%
4.76
79.2%
0.06
0.5%
5.48
82.6%
0.05
0.5%
4.51
81.4%
cod-rna
88.3%
89.7%
0.1
0.5%
6.89
84.7%
0.05
2.0%
9.78
84.7%
model architecture, accuracy is still comparable to non-private version given a privacy budget less than 10. For more results, please refer to Appendix B.8
CONCLUSIONS
In this paper, we propose FedKTT, a model-agnostic federated learning algorithm with single com- munication round. Our experiments show that FedKTT' can learn different models with a comparable or even better accuracy compared with the other state-of-the-art algorithms. Moreover, the accuracy of differentially private FedKT is comparable to the non-differentially private version with mod- est privacy budget. Instead of updating and aggregating in an iterative way, our work show that promising to adopt knowledge transfer to achieve optimal communication efficiency. For future work, we will extend FedKT to the cross-device setting, where the number parties is large and the data size of each party is small.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications Security, pp. 308-318, 2016.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé M Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at 01046
scale: System design. In SysML, 2019.
Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryplography Conference, pp. 635-658. Springer, 2016.
Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:191..11279.C 2019.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Cynthia Dwork. Differential privacy. Encyclopedia of Cryptography and Security, pp. 338-340, 2011.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-00) 2014.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Confer- ence on Computer and Communications Security, pp. 1322-1333. ACM, 2015.
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv: 2017.
Neel Guha, Ameet Talwlkar, and Virginia Smith. One-shot federated learning. arXiv preprint arXiv: /902. 1175, 2019.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 1a1.::18/100604 2018.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large enns at the edge. Advances in Neural Information Processing Systems, 33, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.00 025311 2015.
James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Differentially private bagging: Improved utility and cheaper privacy than abbsample-and-aggreaall In Advances in Neural Information Processing Systems, pp. 4325-435555 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint 1912:v:9112.440 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for on-device federated learn- ing. In Proceedings of the 37th International Conference Machine Learning. PMLR, 2020.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:
2019.
10

---

Under review as a conference paper at ICLR 2021
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, and Bingsheng He. A survey fed- erated learning systems: Vision, hype and reality for data privacy and protection. arXiv preprint arXiv:1907.09693, 2019a.
Qinbin Li, Zeyi Wen, and Bingsheng He. Practical federated gradient boosting decision trees. In AAAI, pp. 4642-4649, 2020a.
Qinbin Li, Zhaomin Wu, Zeyi Wen, and Bingsheng He. cision trees. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp- 784-791, 2020b.
gradient boosting de-
Tian Li. Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. arXiv preprint
07873,, 2019b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In MLSys, 2020c.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et Communication-effieee learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05622 2016.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recureent models. International Conference on Learning Representations, 2018. URL
https::/
net/ /eee/fouume www.fforuuiooooooooo BatprummiddBBJhh11000000
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprchensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 20/9 IEEE Symposium Security and Privacy (SP), pp. 739-753. IEEE, 2019.
Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pp. 75- 84, 2007.
Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In Proceedings of the International Conference on Learning Representations, 2017. URL htttss://aaxii..t/////
/aas/1610.0 05755
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Er- lingsson. Scalable private learning with pate. In Proceedings of the International Conference Learning Representations, 2018
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adap- tation. In International Conference on Learning Representations, 2020. URL https: // openreview net/ www.mmmmddeeee3311
Anantha M Prasad, Louis R Iverson, and Andy Liaw. Newer classification and regression tree techniques: bagging and random forests for ecological prediction. Ecosystems, 9(2):181-199 2006.
Gunnar Rätsch, Takashi Onoda, and K-R Müller. Soft margins for adaboost. Machine learning. 42 (3):8773320, 2001.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1310-1321,
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEEE, 2017.
Reihaneh Torkzadehmahani. Peter Kairouz, and Benediet Paten. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.

---

Under review as a conference paper at ICLR 2021
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Represen- tations, 2020. URL/https///oopeeeviiee net/
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. PATE-GAN: Generating synthetic data with differential privacy guarantees. In International Conference Learning Representations, 2019.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Proceed- ings of the 36th International Conference on Machine Learning. PMLR, 2019.
Cha Zhang and Yunqian Ma. Ensemble machine learning: methods and applications. Springer, 2012.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
In this appendix, we first present the data-dependent privacy analysis of FedKT-LI and FedKT-L2 in Section[A] A Next, in Section[B.] [B.1 show the details of our experimental settings. Then, we show additional experimental results in Section B.2 to Section
B.8 Specifically, in Section B.2] we study the nyper-parameters setting of FedKT. In Section B.3 we study the performance of FedKT with different levels of imbalance. In Section B.4 we study the performance of FedKT with different number parties. In Section B.6 we study the effect of our consistent voting technique. Section B.7 we conduct experiments on addtional datasets, including the CelebA dataset with VGG-9 and two chest X-Ray datasets with CNN. Last, in section B.8 we show the performance of FedKT-LL1 and FedKT--L22
A PRIVACY ANALYSIS OF FEDKT
In this section, we analyze the privacy loss of FedKT-LI and FedKT-L2 using the moments accoun- tant method (Abadi al. 2016).
A.1 PRELIMINARIES
We first introduce two theorems from the previous studies, which will be used in our analysis. The first theorem is from Bun & Steinke (2016) and the second theorem is from Papernot et al. (2017).
Theorem 5. Let M be (2y, -ifffeernnnt private. For any aux, neighboring inputs d and d', we have
a,(ll; d'd) s 272((1+1)
Theorem 6. Let M be (2%, didieerrreyt private and q 2 Pr(M(d) + o"] for some outcome o". Then for any aux and any neighbor d' of d, M satisfies
Let v 2 0 and
< 2--1 --
(M aux, d, d,d") & log((1 - 1) 11-209 cZoq)
1 1-9) q (1--11
+ ge22l)
Pr[M(d) # o*] can be bounded by the following lemma. Lemma 7. Let v be the label score vector for an instance with vo- 2 no for all o. Then
2 + 7V6 Vo)
Pr[M(d) #o'l Σ
4exp(>(1o- Vo))
oft
The following theorem from Abadi et al. (006 can used to convert the moments into (s, 8)- differential privacy.
Theorem 8. 1. [Composability] Suppose that a mechanism M consists of sequence of adaptive mechanisms MI...., Mr where M,: 1== R, x D + R,, Then, for any À
am Σ (A),
2. ['ail bound] For any E 0, the mechanism. M is diifferirreeettiallly private for min cxp(aaM( XE).
A.2 PROOF OF THEOREM 2
Proof. According to Theorem
we can get Theorem by replacing y in Theorem and Theorem
6 to 287.
13

---

Under review as a conference paper at ICLR 2021
A.3 PROOF OF THEOREM 3
Proof. The noises injection on the party side is similar to the PATE approach. The sensitivity of the function executed the party side is 2. Thus, can get the theorem by combining Theorem and Theorem 6
A.4 EXAMPLE-LEVEI DIFFERENTIAL PRIVACY ANALYSIS OF FEDKT-L1
Here we analyze the example-leva differential privacy of FedKT-LI. If change a single example of the original dataset (i.e., the union of all the local datasets), only single party will be influenced. More precisely, for each partition the party, only single teacher model will influenced. Then, even though changing a single record, the student model is still unchanged if the top-2 vote counts of the teachers differ at least 2. Thus, if not applying consistent voting, we have the following theorem.
Theorem 9. Let M be the f function executed in the server. Let q 2 Pr[M(d) * o*] for some outcome o²_ Let Daua denotes the query dataset. Given a query i, suppose the top-2 vote counts of and vz.- In party P., let 2 denotes the number of partitions that there Ry D Daux such that of - n < 1 when training the student model. Let z = mar,2,. Let 1,7 2 0 and q < 2 We have
1 9 (1 + qe ), 1)) aM aux, d'd") s min(log( 2) (1-22rrg
(4)
Proof. Given query dataset Douz. is the number of partitions such that there exists a query that the top-2 vote counts differ at most 1. In other words, there are at most 2 student models will be changed if we change single record of the original dataset. Thus. the vote counts change by at most 2z on the server side and M is (2zy, D)-differentially private with respect to d and 'D Daun- Then, we can get this theorem by replacing y of Theorem and Theorem (otorrmmolt 27-
Note that the example-level differential privacy is same party-level differential privacy when = 8. Also, if we applied consistent voting in FedKT-L.1, the vote counts may change by 8 even only single student model is affected. Thus, the example-level differential privacy of FedKT-L1 is usually same as party-level differential privacy.
A.5 PARTY-LEVEL DIFFERENTIAL PRIVACY ANALYSIS OF FEDKT-L2
Here we study perty-level differential privacy of FedKT-L2.
Theorem 10. Let M be the ʃ function executed on each partition of party. Given the number of
subsets in each partition and the privacy parameter 7, M satisfies (217, 0) party-level differential privacy.
Considering changing the whole local dataset, then / teachers will be influenced and the vote counts change by at most 21. Thus, from party-level perspective, the sensitivity of the counts is 21 and M satisfies (247, -)dffferennt privacy.
Proof.
Like Theorem we have the following theorem track the moments.
Theorem 11. Let M be the f function executed on the party side and q 2 Pr[M(d) # o* for some outcome o". Suppose the number of subsets in partition is l. Let 1,7 2 and g - Then for any aux and any two
datasets and d', M satisties
a. al; d, d') < min(log((1 - -1) 2 (T-
1-9) ) + qe 2, 2(1
Note that the perty-level privacy loss of FedKT-LL2 can be quite large. To mitigate the impact of the noises, we usually expect to be large to have tighter bound of Pr[M(d) * o*). However, when f is large, the bound in Theorem II is also large. In fact, since every student model satisfies differential privacy, it is not necessary apply perty-level differential privacy in F-dK
14

---

Under review as a conference paper at ICLR 2021
Table 3:
The datasets used and their learning models. The detailed model structures shown in
the paper.
Datasets
Adult
cod-rna
MNIST
SVHN
# "Training examples
24421
54231
50000
604288
#Public examples
4070
9039
5000
13016
# #lest examples
4070
9039
5000
13016
#Classes
2
2
10
10
#Parties (by default)
50
50
10
10
Model
Random Forest
GBDT
CNN
CNN
Implemented library
scikit-learn 0.22.1
XGBoost 1.0.2
PyTorch 1.6.0
PyTorch 1.6.0
Table 4: The default parameters used our experiments.
Parameters
Adult
cod-rna
MNIST
SVHN
#parties tree depth
50
50
10
10
6
common
learning rate
0.05
0.001
0.001
batch size
32
64
#epochs
10
10
number partitions party
2
2
FedKT
number of subsets in a partition
5
5
5
FedProx
regularization term
0.1
0.1
B EXPERIMENTS
B.I ADDITIONAL DETAILS OF EXPERIMENTAL SETTINGS
We use Adult, cod-rna, MNIST, and SVHN for our experiments, where Adult and cod-rna are down- loaded from this l The details of the datasets are shown in Table Besides these four datasets, in section B.7 we also conduct experiments on CelebA with VGG-9, and chest X-ray datasets with CNN.
We compare FedKT with the other six baselines. For FedAvg, FedProx, and SCAFFOLD, parties participate in federated learning for each round (i.e., party sampling). The learning rate is tuned from {0.001, 0.01] and the number of local epochs tuned from {10, 20, 40]. We found that the baselines get best performance when the learning rate is set 0.001 and the number of local epochs is set to 10. For FedProx, the regularization term f is tuned from {0.1, 1}. For SCAFFOLD, same the experiments of the paper (Karimireddy al. 2020), we option to caleulate the control variates. For FedKTT, SOLO, and PATE, the number local epochs is simply set to 100. For PATE, the number of teacher models the Adam optimizer and the L2 regularization is set to 10-6, 6 The final parameters are summarized Table
be same the number of parties of FedKT. We use
B.2 SYPER-PARAMETERS STUDY
B.2.1 NUMBER PARTITIONS EACH PARTY
Here we study the impact of the number of partitions (i.c., the parameter s) on FedKT. Table 5 shows the test accuracy of FedKT with different From Table we can see that the accuracy can be improved if we increase s from 1 to 2. However, if we further increase there is little or no improvement on the accuracy while the communication and computation overhead is larger. Thus, from our empirical study, we suggest users to simply set to 2 for FedKT-LO if they do not want to tune the parameters. For FedKT-LI and FedKT-L.2, since the privacy loss increases as 8 increases, we suggest to set to small values. Users can simply set s to or tune from small values (i.e., or 2) to find a best accuracy-privace trade-off.
https: http:://wwwwwcoo www.ww..i......o. csie. ntu. ntu.edd..te
15

---

Under review as a conference paper at ICLR 2021
Table 5: The test accuracy of FedKT with number of partitions ranging between 1 and 5. We run trials and report the mean and standard deviation. The number of subsets in each partition is set to by default. #partitions
2
5
Adult
80.8%
1.4%
82.0%
+ 0.6%
81.5% + 0.6%
81.2%
0.5%
81.1% + 0.1%
cod-rna
87.7%
0.6%
88.3%
0.6%
88.3% 0.5%
88.1%
0.5%
88.2% 0..5%
MNIST
94.5% = 0.5%
96.0%
+ 0.4%
95.5% + 0.4%
95.8% + 0.3%
95.5% + 0.3%
SVHN
81.5%
0.6%
83.2% + 0.4%
83.5% 0.4%
83.5%
0.4%
83.4% 0.3%
Table 6: The accuracy of Fe with number of subsets in each partition ranging between 5 and 20. We run 5 trials and report the mean and standard deviation. For Adult, since there is party with less than 15 examples, the experiment cannot successfully run when the number subsets is not smaller than than 15. For cod-rna, since there is party with less than 20 examples, the experiment cannot successfully run when setting the number of subsets to 20. The number of partitions each party is set to 2 by default.
#subsets
5
10
15
20
Adult
82.0%
0.6%
81.1%
1 0.7%
cod-rna
88.3%
+ 0.6%
87.4%
0.6%
83.1% 0.6%
MNIST
96.0% + 0.4%
93.0%
0.5%
92.7% 0.6%
90.0%
0.6% 79.1% 0.6%
SVHN
83.2% + 0.4%
81.3% + 0.5%
80.0% + 0.5%
B.2.2 NUMBER TEACHERS IN EACH PARTITION
Here we study the impact of number teachers each partition (i.e., the parameter f). Table shows the test accuracy of FedKT with different As we can see, FedKT can always get the best performance if setting t to 5. If is large, the size of each data subset is small and the teacher models may not be good at predicting the public dataset. From our empirical study, users can simply set t to 5 if they do not want to tune the parameter. However, if the student models need to satisfy differential privacy (i.e., in FedKT-L22, the privacy loss may potentially smaller we increase according to Lemma 7 Users need to tune / to find the best trade-off between the performance and the privacy loss.
B.3 IMBALANCE LEVEL
Here study the impact of the concentration parameter 3 of the Dirichlet distribution in our het- erogeneous partition. If 33 is smaller, the data among different parties tend to be more unbalanced. We study three different values: 0.1, 0.5, and 10. The results are shown in Table 7 Given different values, the performance of FedKT always better than FedAvg and FedProx given limited com- munication size (i.e., 2 rounds). Also, FedKT is quite stable compared with the other baselines. The results show that FedKT has a good performance under the heterogeneous data partition. FedKT is superior than the other approaches especially when the imbalance level of data distribution is high.
B.4 NUMBER OF PARTIES
We change the number of parties and compare different approaches. The results are shown in Ta- ble 8 and Table 9 FedKT is quite stable with different number of parties compared with the other baselines. Moreover, FedKT is superior than FedAvg, FedProx, and SCAFFOLD with limited com- munication size in all cases. Even though running the baselines for 50 rounds, the accuracy of FedKT is still comparable.
B.5 SIZE OF THE PUBLIC DATASET
We show the performance of FedKT with different size of public dataset. The results are shown in Table 10 We can observe that FedKT is stable even though reducing the size of the public dataset. The accuracy decreases more than 1% and 2% using only 20% of the public dataset
16

---

Under review as a conference paper at ICLR 2021
Table The test accuracy of different approaches with ranging from {0.1, 0.5, 10). For Adult
and cod-rna, set the number of parties to 20 so that there are enough training instances for each party to train the teacher models in FedKT when 3 is set to 0.1.
B
0.1
0.5
10
FedKT
81.9%
82.4 42.4%
82. 7%
Adult
SOLO
57.0%
68.6%
82.2%
PATE
83.5%
FedKT
88.9%
89.7%
90. 1%
SOLO
58.8%
67.0%
83.1
cod-rna
PATE
91.1%
XGBoost
91.2%
FedKT
95.8%
96.0%
96.8%
SOLO
43.4%
80.0%
96.1% %
PATE
97.8%
rounds
38.2%
83.5%
96.7% %
FedAvg
MNIST
50 rounds
85.9%
98.0%
99.4%
2 rounds
56.6%
83.0%
93.9%
FedProx
rounds
97.2%
98.6%
99.2%
rounds
44.3%
74.2%
83.
SCAFFOLD
50 rounds
97.6%
98.4%
98.6%
Table 8: The test accuracy of different approaches with number of parties ranging from {10, 20, 30, 40, 50} for Adult and cod-rna.
#parties
10
20
30
40
50
FedKT
83.2%
82.4%
83.6%
83.3%
82.2%
Adult
SOLO
77.5%
68.6%
64.4%
68.3%
67.0%
PATE
83.5%
83.5%
83.1%
83.1%
83.5%
FedKT
88.1%
89.7%
86.7%
87.2%
88.3%
SOLO
69.7%
67.0%
67.9%
66.6%
65.0%
cod-rna
PATE
91.1%
90.7%
90.4%
91.0%
91.1%
XGBoost
91.2%
cod-rna and MNIST (i.e., 1807 examples on cod-rna and 1000 examples on MNIST), respectively. Moreover, the accuracy almost unchanged on Adult.
B.6 EFFECT OF CONSISTENT VOTING
Here study the effect of our consistent voting technique. The results shown in Table [I] From the table, we can observe that the accuracy can be improved about 1%-2.3% by applying consistent voting. Thus, consistent voting is simple and effective technique.
B.7 EXPERIMENTS ADDITIONAL DATASETS
We conduct experiments on two additional datasets: CelebA and chest X-ray images.
We conduct gender recognition task on the CelebA dataset. Like MNIST, use one half of the original test dataset as the public dataset and the remaining as the final test dataset. The number of parties is set to 10. The details of CelebA and the model parameters are shown in Table T2 The model structure of VGG-9 is shown in Table 13
We use two chest X-ray image datasets to simulate federated learning as real-world scenario. Specifically, we use Kaggle Chest X-Ray Images F the private dataset and partition it into par- ties. We use RSNA Pncumonia Detection Challenge dataset as the public dataset for our FedKT
https: www. kagg www.c//.c///// com/c//rnna-peemoon. http://oooonnaaetttttttioooooooe
17

---

Under review as a conference paper at ICLR 2021
Table 9: The test accuracy of different approaches with number of parties ranging from {5, 10, 15, 20} for MNIST.
#parties
10
15
20
FedKT
95.2%
96.0%
95.4%
95.8%
SOLO
88.1%
80.2%
82.1%
80.1%
PATE
98.2%
97.9%
97.8%
97.3%
rounds
93%
83.5%
89.3%
94.4%
FedAvg
MNIST
50 rounds
98.7%
98.0%
98.9%
99.0%
2 rounds
76.9%
83.0%
86.2%
92.6%
FedProx
50 rounds
98.6%
98.6%
98.9%
99.0%
2 rounds
73.2%
74.2%
62.9%
64.5%
SCAFFOLD
50 rounds
98.3%
98.4%
98.1%
97.8%
Table 10: The accuracy of FedKT with different size of the public dataset. We vary the portion of the public dataset used in the training from 20% to 100%.
Portion of the public dataset used in training
Datasets
20%
40%
60%
80%
100%
Adult
82.1%
82.1% 1%
82. 12.1%
82.3%
82.2%
cod-rna
87.3%
87.6%
87.8%
87.8%
88.3%
MNIST
94.0%
95.1%
95.0%
95.7%
95.9%
Table 11: The test accuracy of FedKT
with/without consistent voting.
Adult
cod-rna
MNIST
SVHN
with consistent voting
82.2%
88.3%
96.0%
83.2%
without consistent voting
81.1%
87.1%
93.6%
81.6%
algorithm. We use a CNN model (same as the model used in MNIST and SVHN) to predict pneu- monia. The number of parties set to The details of X-Ray task are shown Table
We try the heterogeneous partition and the experiment results are shown in Table 14 For CelebA, FedKT shows good performance on a relatively large dataset and deep model. FedKT is better than FedProx and SCAFFOLD even with 50 rounds. Also, the accuracy of FedKT is comparable to FedAvg with 50 rounds. For X-Ray, FedKT has much better performance than the other federated leamingg approaches. FedAvg, FedProx, and SCAFFOLD are quite unstable.
Figure shows the test accuracy changing the number of communication rounds. We can find that FedProx and SCAFFOLD quite unstable these tasks. For CelebA, FedKT outperforms FedProx and SCAFFOLD, while FedAvg needs about 25 rounds to achieve better accuracy than FedKT. This experiments demonstrate the applicability of FedKT on more complex models. For X-ray, FedKT outperforms the other approaches. This experiments show that FedKT does not have strict limitation on the distribution of the public dataset. Even though the public dataset is another dataset differ from the training dataset, which is not partitioned from the global dataset differ from previous experiments, FedKT still has a quite good performance.
B.8 DIFFERENTIAL PRIVACY
Table 15 and Table 16 present the results of FedKT-LI and FedKT-L2. Besides the heterogeneous partition (3 0.5), we also try homogeneous partition (i.e., the dataset randomly and equally partitioned into the parties). From the tables, we can see that the accuracy of FedKT-L1 and FedKT-
L2 are comparable to the non-private version with a modest privacy budget. Moreover, the moments accountant method usually can achieve tighter privacy loss than the advanced composition (Dwork) et al. 2014). For example, we run cod-rna under homogeneous data partition setting Y 0.1 and the fraction of queries to 1%, the advanced composition gives gives E s 11.2. Note that the techniques in Papernot et al. (2018) can also be applied to FedKT. For example, we may get smaller privacy loss if adopting Gaussian noises instead of Laplace
20.2 and our analysis
18

---

Under review as a conference paper at ICLR 2021
Table 12: The details of tasks on CelebA and Chest X-Ray Image datasets. We conduct gender recognition task on CelebA and pneumonia detection task on X-ray image datasets. Datasets
CelebA
Two chest X-ray image datasets
# #raining examples
162770
5216
#Public examples
9981
26684
#Test examples
9981
624
#Classes
2
#Parties
10
5
Model
VGG-9
CNN
Implemented library
Py'Torch 1.6.0
Batch size
128
32
#Local epochs for FeddAve/redroxSSAFFOOL
10
10
#Epochs for FedKT//OLLO/ATE
100
100
Learning rate
0.001
0.001
Test Accuracy (%)
Test Accuracy (%)
90
80
80
75
70
70
50
SOLO
FedProx
SOLO
SCAFFOLD
PATE
SCAFFOLD
65
FedAvg
FedKT
40
FedAvg
FedKT
FedProx
1 10 15 20 25 30 35 40 45 50 #Communication rounds (a) CclebA
10 15 20 25 30 35 40 45 50 1
#Communication rounds (b) X- X-Ray
Figure 3: The accuracy with different number of communication rounds. For X-Ray, do plot PATE since the line is overlapped with FedKT.
19

---

Under review as a conference paper at ICLR 2021
Table 13: The details of VGG-9 architecture. The stride and padding of convolution layers by default. The ReLU activation is applied after each convolution layer and fully connected layer (except the last fully connected layer).
Layer
Parameters
2D convolution
#input channels: #output channels: 32, kernel size:3
2D convolution
#input channels: 32, #output channels: 64, kernel size:3
Max pooling
kernel size:2, stride:2
2D convolution
#input channels: 64, #output channels: 128, kernel size:3
2D convolution
#input channels: 128, #output channels: 128, kernel size:3
Max pooling
kernel size:2, stride:2
Dropout
probability: 0.1
2D convolution
#input channels: 128, #output channels: 256, kernel size:3
2D convolution
#input channels: 256, #output channels: 256, kernel size:3
Max pooling
kernel size:2, stride:2
Dropout
probability: 0.1
Linear
#input features: 4096, #output features: 512
Linear
#input features: 512, #output features: 512
Dropout
probability: 0.1
Linear
#input features: 512, #output features:
Table 14: The test. accuracy of FedKT and the other baselines CelebA and X-ray datasets. For FedKT, the number of partition in each party is set to 2 and the number of subsets of cach partition set to 10. For FedProx, we set the regularization term μ to 0.1.
FedAvg
FedProx
SCAFFOLD
FedKT
SOLO
PATE
2 rounds
50 rounds
2 rounds
50 rounds
rounds
50 rounds
CelehA
92.2%
83.3%
94.6%
90.5%
95.1%
61.4%
89.7%
61.7%
87.8%
X-ray
81.3%
77.4%
81.4
77.1%
77.6%
78.8%
76.3%
62.5%
73.6%
noises. Generally, our framework can also benefit from the state-of-the-art approaches on the privacy analysis of PATE, which we may investigate in the future.
20

---

Under review as a conference paper at ICLR 2021
Table 15: The accuracy and party-level of FedKT-LL on Adult and cod-rna given different values and number of queries. For each setting, we run trials and report the median accuracy and the corresponding The number of partitions is set to and the number of subsets in each partition set to 5. The failure probability 5 is set to 10-5
data partitioning
#queries
FedKTT-LO
0.1%
0.64
71.3%
0.04
0.5%
2.56
76.8%
1
4.73
80.2%
0. %
0.96
75.7%
Heterogeneous
0.06
0.5% %
3.64
77.6%
82.2%
1%
5.78
80.2%
0.1%
1.23
76.0%
0.08
0.5%
4.25
76.3%
1%
80.3%
Adult
0.
0.32
72.2%
0.02
0.5%
1.25
76. %
1 1
1.79
80. 1%
0.1
0.6
76.0%
Homogeneous
0.04
0.5%
1.9
80.4%
82.4%
1%
3.32
81.5%
0.1%
0.75
76. 1%
0.06
0.5%
2.03
81.7%
1 1%
3.36
82. 1.
0.1%
1.09
66.8%
0.04
0.5%
3.54
72.5%
1%
5.14
75.2%
0.1%
1.52
69%
Heterogeneous
0.06
0.5%
5,48
82.6%
88.3%
1 %
8.1
79.5%
0.1 %
2.12
69%
0.1
0.5%
6.89
84.7 %
1 %
11.2
85.3%
cod-rna
0.2%
0.53
67.0%
0.02
0.5%
1.71
73.2%
1%
2,45
75.3%
0.2%
1.5
73%
Homogeneous
0.04
0.5%
3.06
84. 1%
88.6%
1 %
5.10
85%
0.2%
1.63
80.2%
0.06
0.5%
3.10
84.1%
1%
5.14
86.1
21

---

Under review as a conference paper at ICLR 2021
Table 16: The accuracy and example-level of FedKT-L2 Adult and cod-ma given different values and number of queries. For each setting, we run trials and report the median accuracy and the corresponding e. The number of parties is set to 20 to ensure that FedKT has enough data to train each teacher model. The number of partitions is set to and the number of subsets in each partition is set 25. The failure probability is set 10 5
Data Partition
7
#queries
acc
FedKT--O acc
0.1%
1.13
76.1%
0.04
0.5%
2.56
76.5%
1%
3.72
78.5%
0.1 %
1.32
76. %
Heterogeneous
0.05
0.5%
3.24
79.0%
82.4%
1%
4.76
79.2%
0.1%
1.96
76.2%
0.06
0.5%
3.93
78.5%
1 1
5.79
79.4%
Adult
0.3%
2.13
76.1% %
0.04
0.5%
2.59
78.7%
1%
3.72
81.7
0.3%
2.97
76.3%
Homogeneous
0.06
0.5%
3.93
79.9%
82.6%
1%
5.79
8 81.8%
0.3%
3.77
76.3%
0.08
0.5%
5.04
80.4%
I 1%
7.89
82.0%
0.5%
3.54
77.7%
0.04
1 1%
5.14
79.8%
2%
7.63
82.0%
0.5%
4.51
81.4%
Heterogeneous
0.05
1%
6.58
82.0%
89.7%
2%
9.78
84.7%
0.5%
5.50
81.2%
0.06
1 1%
8.10
83.2%
2%
12.2
85.9%
cod-rna
0.5%
2.64
79.2%
0.03
1%
3.78
80.5%
2%
5.51
83.1%
0.5%
3.54
80.1
Homogeneous
0.04
1 1%
5.14
83.7%
90.6%
1.5%
6.45
84.0%
0.3%
4.51
80.8%
0.05
1%
6.58
84.2%
1.5%
8.3
84.7 %
22


Reviews:
[{"review_id": "q-0RAZb0jQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "===========================Post rebuttal===========================\n\nI thank the authors for their responses and additional experiments. I understand that the focus of the paper is the cross-silo setting, however one of the key questions of an empirical study is to identify the limitations of the proposed approach. Experiments in Tables 8 and 9 still consider a relatively small number of clients and do not provide empirical insights into when the proposed approach begins to degrade. For future revisions, I recommend an empirical exploration that helps the reader to understand the limitations of the proposed method.\n\n==================================================================\n\nThis paper explores the idea of knowledge transfer applied in the federated learning setting. Authors also consider variations of their algorithm under different privacy constraints.\n\nBoth knowledge transfer and differential privacy mechanisms are borrowed from prior works, limiting the methodological contribution of the paper. However, it was interesting to see these techniques applied in the FL context. The applicability of the proposed method relies on each client having sufficient data to train multiple teacher models locally. It would be interesting to see a deeper empirical study of the local data size effect: for example by varying the number of clients in the MNIST/SVHN experiments (therefore reducing the size of the local datasets). I also recommend a quantitative comparison of the communication costs by plotting accuracies of FedKT and baselines against the number of bytes exchanged between clients and the server.\n\nI would consider increasing my score provided additional empirical studies I suggested (especially the effect of the local data sizes).", "extra_scores": {}, "extra_text": {}}, {"review_id": "jiOlkuTqjQo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a new federated learning framework based on knowledge transfer. Local dataset at each party are partitioned and each partition is used to train a teacher model. All teacher models at each party are used to train a student model using pseudo labels based on voting on public dataset. Student model from each party is then uploaded to server and used to train the final model based on voting on unlabeled data. Differential privacy analysis is conducted and experimental evaluations comparing to other mainstream federated learning methods are presented. The advantages of the proposed method include privacy preservation, lower communication traffic, as well as applicability to non-differentiable models. While the proposed framework is technically sound, the reviewer is not convinced by its technical contributions. The design of the framework is integration of existing technics such as PATE, and the mechanism of protecting privacy as well as reduction of communication traffic is also not new (see FedMD: Heterogenous Federated Learning via Model Distillation,  Neurips 2019 Workshop and Ensemble Distillation for Robust Model Fusion in Federated Learning, Neurips 2020). There is no clear advantage in the proposed method over these existing methods from the reviewer’s point of view. Plus the overall performance on benchmark dataset seems to be degraded compared to other mainstream methods like FedAvg. The reviewer would like the authors to explain and discuss the technical contributions of the submission and compare the proposed framework to these similar existing methods based on knowledge transfer. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "JgaMzLM0Aq0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThis paper studies the  problem of  model-agnostic federated learning in the knowledge transfer framework.  They generalize the PATE framework in the federated learning setting by applying subsample-and-aggregate in each local agent. They provide theoretical analysis and empirical evaluation.\n\n---------------------------------------\n\nMerits:\n\n\nThe problem considered is indeed very important, as communication cost and privacy issue are two critical concerns in federated learning. \n\n----------------------------\nConcerns:\n\nMy main concern is the lack of novelty in their privacy analysis. The main theoretical part (Theorem 2 and Theorem 3) comes directly from PATE. \n\nMoreover, the experiment part is insufficient. Noting most part of the paper focuses on differential private FedKT, I think the authors should provide at least one comparison with other privacy-preserving federated learning algorithms [1,2].\n\nI have another concern about the feasibility of FedKT. FedKT requires splitting each local data into t splits and training a teacher model using each split. Noting a large t is well expected in the privacy setting, which may lead to poor performance on teacher models when local data is insufficient. One particular failure case is when the parties are mobile devices.\n\nRegarding communication efficiency, I agree that FedKT is round-optimal. But I have doubts about the claim that FedKT's communication cost is lower than FedAvg.  The authors compare the overhead analysis with ```''nMs < 2nMr'' (the last paragraph on page 4) while ignoring the communication save from subsampling. FedAvg and its variants can benefit from subsampling clients (i.e., a subset of clients is sampled in each communication round). Moreover, in FedAvg and its DP variants [1,2], they choose a relatively large number of #parties while FedKT considers a small number of parties (#parties = 10). Note that this choice may degrade the performance of FedAvg in Table 1 as the variance across parties is larger. More could be done to establish that the experiments actually support the conclusions drawn about communication efficiency.\n \n \n[1] Differentially Private Federated Learning: A Client Level Perspective.\n[2] Learning Differentially Private Recurrent Language Models.\n\n--------------------------------------\nOverall this paper considers a very interesting topic but lacks some important experimental comparisons and method is with limited novelty.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-TngGflBNdD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "__Summary:__\nThe paper considers classification tasks in the federated learning scenario when each device/worker is powerful in terms of computational power and storage space, but, the communication between devices is constrained. The paper proposes a novel algorithm for federated learning that reduces the number of communication rounds to one. The algorithm constructs an ensembled model with the majority voting out of the locally trained models and then on the server side learns a final model by mimicking the performance of the ensembled model on a public dataset. \n\nThe paper additionally provides a differentially private version of the proposed algorithm and proves its privacy guarantees; and performs an experimental comparison of the proposed method. \n\n__Main concerns:__\n- The proposed approach is interesting and novel. However, I am not convinced that the setting is realistic. It is hard to imagine that devices with relatively high computational power are restricted so much on the communication side that cannot allow for more than 1 round. \n- the algorithm does not achieve the best accuracy. \n- The paper in general is hard to read. \n- there is no theoretical understanding of how good the algorithm works. I can imagine many cases where the proposed algorithm wouldn't perform well in practice (small size of the public dataset / small size of local data on the workers / highly non-iid data on the workers)\n- the algorithm is quite complicated and it is unclear if all the components are required. E.g. I don't see why is it needed to do ensembling of models locally, but not to have one local model trained on the full local data. This applies to the other algorithm's components as well. \n\n__Ohter concerns:__\n- Some related work on federated learning with knowledge distillation is missing, e.g. [1], [2], [3].\n- The results are not reproducible: the tuning details and hyperparameters are not stated in the experiments\n- some references are missing: e.g. for consistent voting on page 4. \n- what is the number of queries on page 8, section 5.3? \n\n[1]: D. Li, \"FedMD: Heterogenous Federated Learning via Model Distillation\"\n\n[2]: T. Lin et al, \"Ensemble Distillation for Robust Model Fusion in Federated Learning\"\n\n[3]: H. Chang, \"Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer.\"\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model-agnostic FL method called FedKT that performs only one communication round and reduces the communication complexity of federated learning. The reviewers have the following concerns about the paper:
* Limited novelty because the proposed method is directly based on PATE
* Insufficient experiments

The authors did a great job of responding to the reviewers' comments and also added some new experimental results in the updated version. But the reviewers still recommend significant revision of the paper and resubmission to a future venue. I hope the authors will find their constructive and detailed comments below helpful!
Decision: Reject