Year: 2021

Abstract:
Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various image classification tasks.

heading: 1 INTRODUCTION
text: Nowadays, powerful models with tremendous parameters trained with sufficient computation power are indispensable in Artificial Intelligence (AI), such as AlphaGo (Silver et al., 2016), Alphafold (Senior et al., 2020) and GPT-3 (Brown et al., 2020). However, billions of resource-constrained mobile and IoT devices have become the primary data source to empower the intelligence of many applications (Bonawitz et al., 2019; Brisimi et al., 2018; Li et al., 2019a). Due to privacy, security, regulatory and economic considerations (Voigt & Von dem Bussche, 2017; Li et al., 2018), it is increasingly difficult and undesirable to pool data together for centralized training. Therefore, federated learning approaches (McMahan et al., 2017; Smith et al., 2017; Caldas et al., 2018; Kairouz et al., 2019; Yang et al., 2019) allow all the participants to reap the benefits of shared models without sharing private data have become increasingly attractive.
In a typical Federated Learning (FL) training process using FedAvg (McMahan et al., 2017), each client sends its model parameters or gradients to a central server, which aggregates all clients’ updates and sends the aggregated parameters back to the clients to update their local model. Because FL places computation burden on edge devices, its learnability is largely limited by the edge resources, on which training large models is often impossible. On the other hand, the server only directly aggregates the clients’ models and its computation power is not fully exploited. In this work, we investigate the paradigm where the server adopts a larger model in FL. Here we use larger server model, or GEM to denote the setting where the size of the server model is larger than that of the clients (See Table 1). Making FL trainable with GEM is desirable to break through model capacity and enable collaborative knowledge fusion and accumulation at server.
One feasible approach to bridge FL with GEM is through knowledge distillation (KD) (Hinton et al., 2015), where clients and the server transfer knowledge through logits. For example, FedGKT (He et al., 2020a) adopts a server model as a downstream sub-model and transfers knowledge directly from smaller edge models. In FedGKT the large server model essentially learns from one small teacher at a time and doesn’t learn consensus knowledge from multiple teachers. In FL, KD has also been applied (Li & Wang, 2019; Lin et al., 2020) to transfer ensemble knowledge of clients through consensus of output logits rather than parameters. These works either assume no model on the server
(Li & Wang, 2019) or a prototype server model of the same architecture as the client model (Lin et al., 2020).
Contributions. In this paper, we first propose a new paradigm to bridge FL with GEM, termed FedGEM, which can learn effectively and efficiently from fused knowledge by resource-constrained clients, and is also able to transfer knowledge back to clients with heterogeneous architectures. To further prevent negative and malicious knowledge transfer, we carefully design a selection and weighting criterion to enhance our knowledge transfer protocol, termed FedGEMS. We demonstrate with extensive experiments on various image classification tasks that our results significantly surpass the previous state-of-the-art baselines in both homogeneous and heterogeneous settings. Furthermore, thanks to our effective and selective protocol, our framework improves the robustness of FL on various malicious attacks and significantly reduces the overall communication. In summary, we propose a new framework to bridge FL with larger server models and simultaneously consolidate several benefits altogether, including superior performance, robustness of the whole system, and lower communication cost.

heading: 2 RELATED WORK
text: Federated Learning with GEM. FL is a collaborating learning framework without sharing private data among the clients. The classical method FedAvg (McMahan et al., 2017) and its recent variations (Mohri et al., 2019; Lin et al., 2018; Li et al., 2019b) directly transfer the clients’ parameters or gradients to the server nodes. To tackle the performance bottleneck by training resource-constrained clients in FL, there are two lines of work to bridge FL with GEM. The first line of studies adopts model compression (Han et al., 2015; He et al., 2018; Yang et al., 2018), manually designed architectures (Howard et al., 2017; Zhang et al., 2018; Iandola et al., 2016) or even efficient neural architecture search (Tan & Le, 2019; Wu et al., 2019) to adapt a GEM to on-device learning. Another line is adopting knowledge distillation (Hinton et al., 2015) to transfer knowledge through output logits rather than parameters between a client model and a GEM (He et al., 2020a). However, FedGKT (He et al., 2020a)’s focus is to transfer knowledge directly from clients to server without considering the consensus knowledge fused from clients. Therefore its performance is limited.
Federated Learning with Knowledge Distillation. In fact, ensemble knowledge distillation has been shown to boost collaborative performance in FL. Specifically, FedMD (Li & Wang, 2019) adopts a labeled public dataset and averaged logits to transfer knowledge. FedDF (Lin et al., 2020) proposes ensemble distillation for model fusion by aggregating both logits and models from clients. In addition, KD is used to enhance robustness in FL. Cronus (Chang et al., 2019) and DS-FL (Itahara et al., 2020) utilize a public dataset with soft labels jointly with local private dataset for local training, and combine with Cronus or entropy reduction aggregation, respectively, to defend against poisoning attacks in FL. In this work, we propose to exploit the benefit of both a larger server model and client knowledge fusion. Our work is also motivated by the recent studies in KD (Qin et al., 2021; You et al., 2017; Li et al., 2021; Yuan et al., 2021; Wang et al., 2021), which show that student models can have larger capacities by learning from multiple teacher models.

heading: 3 METHODOLOGY

heading: 3.1 PRELIMINARIES
text: We assume that there are K clients in federated learning process. The kth client has its own private labeled dataset Xk := {(xki ,yki )}N k
i=1 that can be drawn from the same or different distribution, where xki is the ith training sample in the kth client model, y k i is its corresponding ground truth label, and Nk denotes the total number of samples. Each client also trains its own model fkc which can be of the same architecture (homogeneous) or different architecture (heterogeneous). There is also a public dataset X0 := {(x0i ,y0i )}N 0
i=1 which is accessible to both server and clients. On the server side, we assume that there is a larger server model to be trained, denoted as fs. Ls and Lkc denotes the logit tensors from the server and the kth client model.

heading: 3.2 FEDGEMS FRAMEWORK
text: We illustrate our overall framework in Fig. 1 and summarize our training algorithm in Algorithm 1.
FedGEM. During each communication round, all client models first use private datasets to train several epochs, then transfer the predicted logits on public dataset as knowledge to the server model. The server model aggregates the clients’ logits and then trains its server model with the guidance of fused knowledge. After training, the server model then transfers its logits back to all client models. Finally, each client model distills knowledge from received logits and continues their training on private datasets. Continuously iterating over multiple rounds, both the server and client models mutually learn knowledge from each other. Through this alternating training processing, we can obtain a large server model with accumulated knowledge and an ensemble of high-performance client models.
FedGEMS: In FedGEMS, the server adopts a selection and weighting criterion to select knowledgeable clients for aggregation, which is detailed in the next section 3.3.
To illustrate the features of our framework, we compare it with the related studies of KD-based methods in federated learning in Table 1 on the following aspects: whether they use a labeled, unlabeled or no public dataset, whether client models can have heterogeneous architectures, the aggregation strategy on the server, and whether the server has a larger model. Note FedGEM can be regarded as placing a larger server model on top of the FedMD framework while keeping other settings the same.

heading: 3.3 SELECTIVE KNOWLEDGE FUSION IN SERVER MODEL
text: Since clients’ knowledge may negatively impact the server model in the heterogeneous or malicious setting and vice versa, we further propose selective strategies on both server and client sides to enforce positive knowledge fusion into the large server model as shown in Fig. 2.
Algorithm 1 Illustration of the Framework of FedGEMS. T is the number of communication rounds; X0 and Y 0 denotes the images and their corresponding labels in public dataset; Xk and Y k denotes the private dataset in the kth client model; fs and fkc are the server with parameter Ws and the kth client model with parameters W kc ; LGlobal indicates the global logits to save correct logits; Ls and Lkc are the logit tensors from the server and the kth client model. 1: ServerExecute(): 2: for each round t = 1, 2, ..., T do 3: // Selective Knowldge Fusion 4: for idx, x0,y0 ∈ {X0,Y 0} do 5: if fs(Ws;x0) == y0 then 6: Ls ← LS1(x0,y0) ▷ in Eq. 1 7: LGlobal[idx]← Ls[idx] 8: else if idx in LGlobal then 9: Ls ← LS2(x0,y0,LGlobal) ▷ in Eq. 2 10: else 11: Lc ← ClientSelect(idx) 12: Ls ← LS3(x0,y0,Lc) ▷ in Eq. 3, 4, 5 13: Ws ←Ws − ηk∇Ls 14: Ls[idx]← fs(Ws;x0) 15: // Transfer Knowledge to Client Models 16: ClientTrain(Ls) 17: ClientTrain(Ls): 18: for each client kth in parallel do 19: // Knowledge Distillation on Clients 20: for x0,y0 ∈ {X0,Ls,Y 0} do 21: Lc ← LC(x0,y0,Ls) ▷ in Eq. 6 22: W kc ←W kc − ηk∇Lc 23: // Local Training on Clients 24: for xk,yk ∈ {Xk,Y k} do 25: Lc ← LCE(xk,yk) 26: W kc ←W kc − ηk∇Lc 27: ClientSelect(idx): 28: // Selective Transfer to Server 29: for each client kth in parallel do 30: L0c [idx]← fkc (W kc ;x0[idx]) 31: Return Lc to server

heading: 3.3.1 SELF-DISTILLATION OF SERVER KNOWLEDGE
text: At each iteration, the server model first performs self evaluation on the public dataset and split the samples into two classes, those it can predict correctly, SCorrect, and those it predicts wrongly, SIncorrect. For each sample xi in SCorrect where the model prediction matches the ground truth label, we simply adopt the cross-entropy loss between the predicted values and the ground truth labels to train the server model. LS1 = LCE = −yi log(fs(xi)) (1) At the same time, we save the correct logit lis into the global pool of logits as l i Global, which can be further used as memory to recover its reserved knowledge from self-distillation.
For each sample xi that the server predicts wrongly, we first check whether its corresponding global logit liGlobal exists or not. We denote all samples that do not exist in lGlobal as S ∗ Incorrect, for which we believe the server model can not learn the sample entirely by itself and propose to use clients’ collective knowledge as its teacher, which will be explained in the next section. If liGlobal exists, it means that the knowledge was reserved by the server before, the server model performs selfdistillation to recover this part of knowledge. The final training objective of self-distillation can be formulated as follows, where DKL is the Kullback Leibler (KL) Divergence function and ϵ is a hyper-parameter to control the weight of knowledge distillation in all the following formulations.
LS2 = LSS = ϵLCE + (1− ϵ)DKL ( liGlobal∥lis ) (2)
Our self-distillation strategy has at least two advantages. On the one hand, distilling knowledge from itself is better than from other models which have different architectures. On the other hand, learning from its stored logits can greatly reduce the communication cost as compared to transferring all redundant logits to client models.

heading: 3.3.2 SELECTIVE ENSEMBLE DISTILLATION OF CLIENT KNOWLEDGE
text: For those samples in S∗Incorrect that the server model fails to predict correctly, we try to distill knowledge from the ensemble of clients. Furthermore, considering the correctness and relative importance of client models, we propose a weighted selective strategy on client side.
Given an instance (xi,yi) in S∗Incorrect, we first split all clients {C1, C2, ..., CK} into the reliable and unreliable clients according to their predictions pCj . For those clients who predict the wrong labels, we consider them as unreliable and discard their knowledge by setting their weights equal to 0. As
for the rest of clients who predict the labels correctly, we consider them as reliable and use their entropy H(pCj ) as a measure of the confidence.
H(pCj ) = − N∑ i=1 p(xi)logp(xi) (3)
Following previous work (Li et al., 2021; Pereyra et al., 2017; Szegedy et al., 2016), low entropy indicates high confidence, and vice versa. Specifically, given an instance (xi,yi) in S∗Incorrect, we design its corresponding weights to aggregate output logits from different clients as below.
αCj = { 0, Cj ∈ CUnreliable softmax( 1H(pCj ) ), Cj ∈ CReliable (4)
Therefore, for samples in S∗Incorrect, the knowledge transferred from ensemble clients to server model can be formulated as the following.
LS3 = LCS = ϵLCE + (1− ϵ)DKL  K∑ j=1 αCj lCj ∣∣∣∣∣∣∣∣ls  (5)
Up to now, we have covered the selective knowledge fusion strategy for the server, next we will discuss the knowledge distillation on clients.

heading: 3.4 TRAINING IN CLIENT MODELS
text: Each client model first receives the logits liServer of public dataset from the server model. Then it distills knowledge according to the logits as well as computes the cross-entropy loss to train on public dataset.
LC = ϵLCE + (1− ϵ)DKL ( lis∥lic ) (6)
After knowledge distillation, each client model further adopts the cross-entropy loss to train on its local private dataset to better fit into its target distribution.

heading: 4 EXPERIMENTAL EVALUATIONS

heading: 4.1 EXPERIMENT SETTINGS
text: Task and Dataset. For fair comparison, we use the same training tasks as He et al. (2020a), which include image classifications on the CIFAR-10 (Krizhevsky et al., 2009) and CIFAR-100 (Krizhevsky et al., 2009) datasets. More details about these two datasets can be found in Appendix B.1. For each dataset, we randomly split data into a public and a private dataset with a ratio of 1:1. We further study the impact of this ratio in Sec. 5.2 of our experiments. The public dataset is used for transferring knowledge between server node and client nodes, while the private dataset is for client training. Both of them further split into a training and a testing dataset with a ratio of 5:1.
Homogeneous Setting. In our main experiments, we randomly shuffle and partition private dataset into 16 clients. As shown in Appendix B.6, our client models all adopt a tiny CNN architecture called ResNet-11, while the server model architecture is ResNet-56. To investigate the influence of server model size and client number, we further vary the server model size from ResNet-20 to ResNet110 and the total client number from 4 to 64 in Sec. 5.3 and Sec. 5.4, respectively.
Heterogeneous Setting. We adopt the Dirichlet distribution (Yurochkin et al., 2019; Hsu et al., 2019) to control the degree of heterogeneity in our non-iid setting. In this experiment, our α is 0.5. We also introduce model heterogeneity in our experiments. The client models vary from ResNet-11 to Resnet-17 as shown in Appendix B.7.
Implementation Details. In both homogeneous and heterogeneous settings, the number of local epochs are set to 1 after careful tuning, consistent with implementations in FedGKT. The number of communication round to reach convergence is 400. We adopt Adam optimizer (Kingma & Ba, 2014) with an initial learning rate 0.001. Following He et al. (2020a), the learning rate can reduce once the accuracy is stable (Li & Arora, 2019). As for the factor ϵ, we find that the best choice is 0.75. Its impacts on server and client performance are shown in Appendix B.5 .
Baselines. We compare our framework FedGEM/FedGEMS with the following baselines. (1) Stand-Alone: the server and client models train on their local datasets X0 and Xk, respectively; (2) Centralized: the client model trains on the whole private dataset {X1+· · ·+Xk}; (3) CentralizedAll: the server and clients perform centralized training on the whole dataset {X0 + · · · + Xk}, respectively. Note in reality no party has this complete dataset so this can be viewed as an upper limit of model performance. In addition, we also compare our performance with several existing related works, including FedAvg (McMahan et al., 2017), FedMD (Li & Wang, 2019), Cronus (Chang et al., 2019), DS-FL (Itahara et al., 2020), FedDF (Lin et al., 2020) and FedGKT (He et al., 2020a). For fair comparison, the settings of client models and the split of public and private dataset in all approaches are kept the same. As for the server model, FedDF adopts the same architecture as its clients’ model, ResNet-11, according to its design, while FedGKT and our FedGEMS utilize ResNet-56. Note FedGKT (He et al., 2020a) only reported the performance of the server model. We empirically found that its clients’ performance is lower due to constraints of model size. For a fair comparison with FedGKT, we only reported its performance on server side. More details of their important hyper-parameters in our experiments are listed in Appendixes B.9, respectively.

heading: 4.2 PERFORMANCE EVALUATIONS
text: Table 2 shows the comparisons of our approaches with other baselines with ResNet-56 as the server model. First of all, our approach FedGEM outperforms all the above KD-based federated learning baselines significantly on both server and client side, demonstrating the effectiveness of knowledge transfer with a larger server model. Compared with FedGKT which also adopts a larger server model, our performance is significantly improved by knowledge fusion from multiple clients. Compared to the performance of stand-alone server and clients, our framework simultaneously improves both the sever model and client model, indicating that server and clients mutually benefit from knowledge transfer. Using public labels for supervision and selection, our FedGEMS framework is able to enhance positive knowledge fusion and further improve performance on both server and clients. We also conduct experiments with 8 clients and ResNet-20 as server model, shown in Appendix B.4.

heading: 4.3 ROBUSTNESS
text: Poisoning Attacks. Following Chang et al. (2019), we employ model poisoning attacks to evaluate the robustness of our methods. Our model poisoning attacks include Naive Poisioning (PAF), Little Is Enough Attack (LIE) (Baruch et al., 2019) and OFOM (Chang et al., 2019). In PAF and LIE, the attacker poisons one client at each round via disturbing the logits or parameters which transfer from clients to server. In OFOM, two clients’ benign predictions or parameters are poisoned at each round. The detailed algorithms about different poisoning attacks can be found in Appendix A.
The results of poisoning attacks in heterogeneous setting on CIFAR-10 dataset are shown in Table 3. By adopting a selective strategy, our proposed FedGEMS provides consistent robustness across various attacks on both server and client models. FedGEM demonstrates superior robustness to FedMD on various model poisoning attacks. DS-FL is relatively more robust than FedMD via its proposed entropy reduction aggregation based on FedMD framework. Since Cronus specifically designs secure robust aggregation in a FedMD framework, its robustness to the attacks is strong, as expected. The performance of FedGKT drops marginally because it transmits extra feature maps which is vulnerable. Our selective knowledge transfer strategy shows comparably strong robustness on both clients and server side without additional computation cost for robust aggregation algorithms.

heading: 4.4 COMMUNICATION COST
text: Following Itahara et al. (2020), we further evaluate the cumulative communication costs required to achieve specific accuracy in heterogeneous setting on CIFAR-10. The epochs in both client and server sides among all methods to achieve their best performances are the same as 1 which further ensures fairness. The formulations to calculate communication cost for different methods are shown in Appendix B.2.
The results are shown in Table 4. It is worth noting that FedMD, Cronus, DS-FL and FedGEM cost the same per round according to their designs. Therefore, to achieve the accuracy of 45%, FedGEM costs much lower than other three methods which indicates that FedGEM converges fast. The communication cost of FedGKT is much higher than others because it sends a feature map per private data in training. Specifically, the communication cost per feature map is 64kb while the cost per logit is only 0.039kb. Furthermore, the communication cost of FedGEMS keeps lower than FedGEM in the whole training phase which means that our selective strategy can effectively reduce uploads to achieve the same performance.

heading: 5 UNDERSTANDING FEDGEMS

heading: 5.1 KNOWLEDGE ACCUMULATION AT SERVER
text: Method Server Client FedGEMS 88.08 81.86 - Self-Training 85.73 80.17 - Self-Distillation 86.12 81.30 - Ensemble Distillation 86.55 80.01
Table 5: Ablation studies of three components in selective strategy.
To analyze the selective knowledge fusion process of the server model in our framework, we report the number of samples associated with each step in our decision-making process in our experiment on CIFAR-10 with 25,000 samples as public dataset. Each sample will choose one of three strategies according to our selective knowledge fusion module. Specifically, in Fig. 3, the line of self-training indicates the total number of samples the server model predicts correctly, while the lines of self-distillation and ensemble-distillation indicate the number of samples the server learns via self-distillation and client-side ensemble knowledge, respectively. In the initial stage, since both the server and client models learn from scratch, the number of samples that the server model can predict correctly is limited, and most of the knowledge is accumulated by distillation from the client models’ fusion. As the training progresses, the server model needs to restore some of its knowledge from a self-distillation strategy. With the server model continually fusing sufficient knowledge, the model performance of server model eventually exceeds client models and the number of transferred samples from clients dropped to 0. Notice that in the final stage, there still remains some stubborn samples which are hard for the server model while most samples can be solved by the large server model with accumulated knowledge.
Furthermore, we conduct a series of ablation studies to detect the contributions of different components to the accumulated knowledge at server side. The results are shown in Table 5 and all the performances of both server and client models decrease. This phenomenon indicates that each component contributes to the overall performance. Comprehensively, self-training is the most important component and the most probable reason is that learning from the model itself is the most effective way. Ensemble distillation which avoids transferring knowledge from malicious clients can also help prompt the performance of client models in return.

heading: 5.2 EFFECT OF PUBLIC AND PRIVATE DATASET RATIO
text: Concerning the size of public dataset, we evaluate the effect of public and private dataset ratio on both model performance and communication cost per round. To change the ratio, we fix the size of private dataset, and vary the size of public dataset from 5,000 to 25,000.
The results of both model performance and communication cost per round are shown in Fig. 4 (a) and (b) respectively. As for model performance, it can be seen that both the client and server model continuously improve as the size of public data increases, indicating that their performances are highly correlated. Our performance consistently outperforms FedMD, and the performance gap over
FedMD continuously grows as more public dataset is available. Note again FedGEM is essentially FedMD with a larger server model so the performance gain is due to the accumulated knowledge on the server side. As for FedDF which owns an ensemble server model as same architecture as clients, the performance in both server and client models remains a huge gap to our performance. This phenomenon further demonstrates the importance of the large capacity of server model. FedGEMS can still outperform FedGEM especially on the client side which indicates that our selective strategy is efficient to fuse positive knowledge to boost performance.
Due to the compact models deployed in client nodes, the bits of model parameters are limited, thus the communication overhead of FedAvg and FedDF is not always higher than typical KD-based methods when a large public dataset is used. Due to the enormous communication overhead of feature maps, we do not figure the flat line (irrelevance to the size of public dataset) of FedGKT which is always 1.6M (kb) per round. Thanks to our selection strategy, the communication cost of our proposed framework FedGEMS is the lowest among all methods. In summary, the results demonstrate that our selection strategy can greatly save the communication cost with reasonable public data size.

heading: 5.3 EFFECT OF SERVER MODEL SIZE
text: We further investigate the influence of the server model size on the overall performance by gradually changing the server model from ResNet-20 to ResNet-110, while fixing the client models as ResNet11. The details of the model parameters can be found in Appendix B.8.
The results are shown in Fig. 5 (a). It can be witnessed that the performance of both the server and client models improves as the server model becomes larger and deeper. This phenomenon verifies the importance of placing a larger deeper server model to boost the performance of both server and the resource-constrained client models.

heading: 5.4 EFFECT OF NUMBER OF CLIENTS
text: We vary the number of clients from 4 to 64 of our proposed method FedGEMS while keeping the total number of private samples the same. Thus the number of private samples per client vary accordingly. The experimental results shown in Fig. 5 (b) indicate that with increased number of clients and less client data, the performance of our approaches is basically stable especially in client side, benefiting from our larger server model setting with fused knowledge.

heading: 6 CONCLUSION
text: In this work, we first propose a new paradigm to apply a large deeper server model to effectively and efficiently fuse and accumulate knowledge, which can enhance the model performances on both server and client sides. Furthermore, we design a selection and weighted criterion on both sides to distill only positive knowledge into the server. We conduct a series of experiments to evaluate our proposed framework FedGEMS and our results show that FedGEMS can significantly surpass all baselines in both homogeneous and heterogeneous settings. Meanwhile, FedGEMS can further improve the robustness of FL on poisoning attacks as well as reduce the communication costs between server and client sides. Our framework has certain limitations, such as the dependence of a labeled public dataset. In future work, we will study the effectiveness of our work in other critical tasks, such as NLP and knowledge graph-based tasks, and in the settings where public and private data are from different domains or distributions.

heading: A POISONING ATTACKS IN FEDERATED LEARNING
text: In this section, we introduce a set of poisoning attacks from the literature which used to evaluate the robustness in federated learning.

heading: A.1 NAIVE POISONING (PAF)
text: Naive poisoning is a type of model poisoning attack. The adversary is able to poison ϵ fraction of total client. With the knowledge of the distribution of benign updates, the opponent can introduce malicious update θm. This attack is far from the mean of the benign updates, obtained by adding a significantly large vector θ′ to it:
θm =
∑n i=1 θi
(1− ϵ)n + θ′ (7)
Then, the server updates with the malicious update θm and transfers it to all the clients.

heading: A.2 LITTLE IS ENOUGH ATTACK (LIE)
text: LIE is a type of model poisoning attack proposed by Baruch et al. (2019). The detailed implementation of LIE is summarized in Algorithm 2. The malicious update θm is achieved by making small changes in many dimensions to a benign update. The malicious update is trained in server and then shared by each client.
Algorithm 2 Little is enough attack (LIE) Baruch et al. (2019) 1: Input: The number of benign updates n, the fraction of malicious updates ϵ 2: The number of majority benign clients:
s = ⌊n 2 + 1 ⌋ − ϵn (8)
3: Using z-table, set:
zmax = max z
( ϕ(z) <
n− s n
) (9)
4: for j ∈ [d] do 5: compute mean (µj) and standard deviation (σj) of benign updates.
θjm ← µj + zmax · σj (10)
6: Output: θm

heading: A.3 ONE FAR ONE MEAN (OFOM)
text: This model poisoning attack, OFOM, is proposed by Chang et al. (2019). In this attack, two malicious updates will be added to the benign updates. First, the opponent adds a significantly large vector θ′ to the mean of the benign updates to obtain θ1m. Then, the opponent crafts θ 2 m, which is the mean of the benign updates and θ1m.
θm1 = ∑n i=1 θi n + θ′, θm2 = ∑n i=1 θi + θ m 1 n+ 1 (11)

heading: B EXTRA EXPERIMENTAL RESULTS AND DETAILS

heading: B.1 A SUMMARY OF DATASET
text: Following He et al. (2020a), our experiments utilize CIFAR-10 (Krizhevsky et al., 2009) and CIFAR100 (Krizhevsky et al., 2009) as our datasets. CIFAR-10 consists 10 classes colour images, with 6000 images per class. CIFAR-100 is a more challenging dataset with 100 subclasses that falls under 20 superclasses, e.g. baby, boy, girl, man and woman belong to people.

heading: B.2 FORMULATIONS OF COMMUNICATION COST
text: In the Fig. 6, we list the formulations to compute communication costs between server and client models in different methods.

heading: B.3 COMMUNICATION COST OF THE UPLOADING LOGITS
text: The results of the communication cost of the uploading logits per round are shown in Table 6.

heading: B.4 RESULTS OF 8 CLIENTS AND RESNET-20
text: We also conduct different setting to validate the stability of our proposed framework FedGEMS. In this series of experiments, we adopt 8 clients and set the server model as ResNet-20, while keeping other experiment setting the same as main paper.

heading: B.4.1 PERFORMANCE EVALUATIONS
text: The results of our performance are shown in Table 7.

heading: B.4.2 ROBUSTNESS
text: The results of our robustness experiment are shown in Table 8.

heading: B.4.3 COMMUNICATION COST
text: We further evaluate the communication costs per round by varying public dataset sizes for various approaches on CIFAR-10, and the results are shown in Fig. 7.
B.5 RESULTS OF DIFFERENT ϵ
The results of different ϵ, which is a hyper-parameter to control the weight of knowledge distillation, are shown in Table 9.

heading: B.6 MODEL ARCHITECTURES OF HOMOGENEOUS SETTING
text: The model architectures of client and server models in homogeneous setting are shown in Table 10.

heading: B.7 MODEL ARCHITECTURE OF HETEROGENEOUS SETTING
text: The different model architectures of client models in heterogeneous setting are shown in Table 11. The server model in heterogeneous setting is the same as homogeneous setting in Table 10.

heading: B.8 MODEL ARCHITECTURES OF LARGER SERVER MODELS
text: The detailed model architectures of large server models are shown in Table 12.

heading: B.9 HYPER-PARAMETERS
text: In table 13, we sum up the hyper-parameter settings for all the methods in our experiments. We directly run FedAvg, FedGKT and re-implement all the other methods base on an open-source federated learning research library FedML (He et al., 2020b) in a distributed computing environment.

Review summaries:
This paper aims to enable training of higher accuracy models by training a high-capacity model at the server and lower capacity models at the clients. Knowledge is shared via distillation on a weighted average of logits provided by the clients, rather than by averaging model parameters or model differences. The approach assumes the existence of a labeled public dataset. Experiments illustrate the promise of this approach.

This work proposed a KD-based FL framework for training a larger server model securely. The proposed framework is a combination of FedGKT-like idea and further apply some KD strategies for selective transfer. Many baselines are compared to demonstrate its efficacy.

This paper tries to address the big model on the server-side in federated learning (FL) via knowledge distillation. Selective transfer knowledge happens on both server and clients sides. The proposed model maintains a set of global, correct logits so that it can be used as memories for the server's distillation. When the server predicts wrong on the public datasets, it transfers the knowledge from the correct, weighted clients. The proposed model is evaluated on image datasets with many FL baselines.