FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
Authors: Sijie Cheng
Jingwen Wu
Yanghua Xiao
Yang Liu
Yang Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Federated Learning
Knowledge Distillation

Abstract: Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server. By bridging FL effectively with larger server model training, our proposed paradigm paves ways for robust and continual knowledge accumulation from distributed and private data.

Under review as a conference paper at ICLR 2022
FEDGEMS: FEDERATED LEARNING OF LARGER SERVER MODELS VIA SELECTIVE KNOWLEDGE FU- SION
Anonymous authors Paper under double-blind review
ABSTRACT
Today data is often scattered among billions of resource-constraieed edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from mul- tiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back clients in return boost their respective perfor- mance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, in- cluding flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various im- age classification tasks.
INTRODUCTION
Nowadays, powerful models with tremendous parameters trained with sufficient computation power are indispensable in Artificial Intelligence (AI), such as AlphaGo (Silver et al.. 2016), Alphafold (Senior al., 2020) and GPT-3 (Brown et al., 2020). However, billions of resource-constraing mobile and IoT devices have become the primary data source to empower the intelligence of many applications (Bonawitz et al., 2019; Brisimi et al., 2018: Li et al., 2019a). Due to privacy, security, regulatory and economic considerations (Voigt & Von dem Bussche, 2017; Li et al., 2018), it is increasingly difficult and undesirable to pool data together for centralized training. Therefore, fed- erated learning approaches (McMahan et al., 2017; Smith et al., 2017; Caldas et al., 2018; Kairouz et al., 2019; Yang et al., 2019) allow all the participants to reap the benefits shared models without sharing private data have become increasingly attractive.
In a typical Federated Learning (FL) training process using FedAvg (McMahan et al., 2017), each client sends its model parameters gradients central server, which aggregates all clients' up- dates and sends the aggregated parameters back to the clients to update their local model. Because FL places computation burden on edge devices, its learnability is largely limited by the edge re- sources, on which training large models is often impossible. On the other hand, the server only directly aggregates the clients' models and its computation power is fully exploited. In this work, we investigate the paradigm where the server adopts larger model in FL. Here we use larger server model, or GEM denote the setting where the size of the server model larger than that of the clients (See Table 1). Making FI. trainable with GEM is desirable to break through model capacity and enable collaborative knowledge fusion and accumulation server.
One feasible approach to bridge FL with GEM is through knowledge distillation (KD) (Hinton et al., 2015), where clients and the server transfer knowledge through logits. For example, FedGKT (He et al., 2020a) adopts server model as downstream sub-model and transfers knowledge directly from smaller edge models. In FedGKT the large server model essentially learns from one small teacher time and doesn't 't learn consensus knowledge from multiple teachers. In FL, KD has also been applied (Li & Wang, 2019; Lin et al., 2020) to transfer ensemble knowledge of clients through consensus of output logits rather than parameters. These works either assume model the server

---

Under review as a conference paper at ICLR 2022
I Update
Prediction
3. Selectivee Upload
Knowledge Fusion
Allient
Privade
Cuenn
Publii
do
Crient
7. Distllatto
Selective Know/edge Fusion
Server
Server
Public
cllent
X
Private
citenn
X ppasic
Prediction
Issverr
6. Broadcast
Figure 1: The framework of FedGEMS.
(Li & Wang, 2019) or a prototype server model of the same architecture as the client model (Lin al., 2020).
Contributions. In this paper, we first propose a new paradigm to bridge FL with GEM, termed FedGEM, which can learn effectively and efficiently from fused knowledge by resource-constrained clients, and is also able to transfer knowledge back to clients with heterogeneous architectures. To further prevent negative and malicious knowledge transfer, we carefully design selection and weighting criterion to enhance our knowledge transfer protocol, termed FedGEMS. We demon- strate with extensive experiments on various image classification tasks that our results significantly surpass the previous state-of of-the-art baselines in both homogeneous and heterogeneous settings. Fur- thermore, thanks to our effective and selective protocol, our framework improves robustness of FL on various malicious attacks and significantly reduces the overall communication. In summary, we propose a new framework to bridge FL with larger server models and simultaneously consol- idate several benefits altogether, including superior performance, robustness of the whole system, and lower communication cost.
2
RELATED WORK
Federated Learning with GEM. FL a collaborating learning framework without sharing private data among the clients. The classical method FedAvg (McMahan et al., 2017) and its recent varia- tions (Mohri al., 2019; Lin al., 2018; Liet al., 2019b) directly transfer the clients' parameters or gradients to the server nodes. To tackle the performance bottleneck by training resource-constrained clients in FL, there are two lines work to bridge FL with GEM. The first line of studies adopts model compression (Han et al., 2015; He et al., 2018; Yang al., 2018), manually designed chitectures (Howard et al., 2017: Zhang et al., 2018; landola et al., 2016) or even efficient neural architecture search (Tan & Le, 2019; Wu et al., 2019) to adapt GEM to on-device learning. An- other line is adopting knowledge distillation (Hinton et al., 2015) to transfer knowledge through output logits rather than parameters between client model and a GEM (He et al., 2020a). However, FedGKT (He et al., 2020a)'s ' focus is to transfer knowledge directly from clients to server without considering the consensus knowledge fused from clients. Therefore its performance is limited.
Federated Learning with Knowledge Distillation. In fact, ensemble knowledge distillation has been shown boost collaborative performance in FL. Specifically, FedMD (Li & Wang, 2019) adopts labeled public dataset and averaged logits to transfer knowledge. FedDF (Lin et al., 2020) proposes ensemble distillation for model fusion by aggregating both logits and models from clients.
In addition, KD is used to enhance robustness in FL.. Cronus (Chang et al., 2019) and DS-FL (Itahara et al., 2020) utilize public dataset with soft labels jointly with local private dataset for local training, and combine with Cronus or entropy reduction aggregation, respectively, to defend against poisoning attacks in FL. In this work, we propose to exploit the benefit both a larger server model and client knowledge fusion. Our work is also motivated by the recent studies in KD (Qin et al., 2021; You et al., 2017; Li et al., 2021: Yuan et al., 2021: Wang et al., 202 1), which show that student models can have larger capacities by learning from multiple teacher models.
2

---

Under review as a conference paper at ICLR 2022
Method
Public Data
Client Model Heterogeneity
Aggregation
Server model Size
FedAvg
No
Average
FedMD
Labeled
Yes
Average
Cronus
Unlabeled
Yes
Cronus
FedDF
Unlabeled
No
Average
Client
FedGKT
Yes
Client
DS-FL
Unlabeled
Yes
Entropy-reduction
FedGEM
Labeled
Yes
Average
Client
FedGEMS
Labeled
Yes
Selective
Client
Table 1: Comparison of FedGEMS with related works.
METHODOLOGY
3.1 PRELIMINARIES
We assume that there are K clients in federated learning process. The kth client has its own private labeled dataset X : {(= )= that can be drawn from the same or different distribution, where anf is the ith training sample in the kth client model, y is its corresponding ground truth label, and N* denotes the total number of samples. Each client also trains its model f which can be of the same architecture (homogeneous) or different architecture (heterogeneous). There is also public dataset X := {(=], v } which is accessible to both server and clients. On the server side, we assume that there is a larger server model to be trained, denoted as f.. L, and L denotes the logit tensors from the server and the kth client model.
3.2 FEDGEMS FRAMEWORK
We illustrate our overall framework in Fig. and summarize our training algorithm in Algorithm 1.
During each communication round, all client models first private datasets train several epochs, then transfer the predicted logits on public dataset as knowledge to the server model. The server model aggregates the clients' logits and then trains its server model with the guidance of fused knowledge. After training, the server model then transfers its logits back to all client models. Finally, each client model distills knowledge from received logits and continues their training on private datasets. Continuously iterating over multiple rounds, both the server and client models mutually learn knowledge from each other. Through this alternating training processing, we can obtain large server model with accumulated knowledge and an ensemble of high-performance client models.
FedGEM.
FedGEMS: In FedGEMS, the server adopts a selection and weighting criterion to select knowledge- able clients for aggregation, which is detailed in the next section 3.3.
To illustrate the features of our framework, we compare it with the related studies of KD-based methods in federated learning Table on the following aspects: whether they use a labeled, unlabeled public dataset, whether client models aggregation strategy on the server, and whether the server has a larger model. Note FedGEM can be regarded placing larger server model top of the FedMD framework while keeping other settings the same.
have heterogeneous architectures, the
3.3 SELECTIVE KNOWLEDGE FUSION IN SERVER MODEL.
Since clients' knowledge may negatively impact the server model in the heterogeneous malicious setting and vice versa, we further propose selective strategies on both server and client sides to enforce positive knowledge fusion into the large server model as shown in Fig. 2.
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Illustration of the Framework of FedGEMS. T' is the number of communication rounds; X and Y denotes the images and their corresponding labels public dataset; X and Y denotes the private dataset in the kth client model; f. and f are the server with parameter W, and the kth client model with parameters < Lelebal indicates the global logits to save correct logits; L, and _ are the logit tensors from the server and the kth client model.
1: ServerExecute():
17: ClientTrainíLL, L.):
2: for each round L = I... do T'
18: for each client kth in parallel do // &nowledgee Distillation on Clients
3: Selective Knowldge Fusion
19:
4: for idx, x², , E {xx), y do
20:
for x ,² E 1
Yo) do
5: if fWWs: == y² then
21:
Le Ec y 0 Ls)
b in Eq. 6
6:
L. +- Ls, (x".y² yy²) Lankal [ide]
D in Eq. 1
22:
W <... M/ V.
7:
L.(idr)
23:
I/ Local Training on Clients
else if idx in Leiobal then
24:
for x² yy E {(x* Y do
9:
L. + Lsp (z", y² Laaoa) & in Eq. 2
25:
Le t Lor(æ* y W W mh VC.
10:
else
11:
L. ClientSelect(idx)
26:
12:
L. ELtts (a.,, y" Le) b in Fq. 4,5,
27: ClientSelect(idx::
28: // Selective Transfer to Server
13:
W. + TA L.did]] + f=(W::
29: for each client kth in parallel do
14:
30: L [eddd]] f (W:; ;; a [idx])
15:
// Transfer Knowledge to Client Models ClientTrain(L.)
31: Return Le to server
16:
3.3.1
SELF-DISTILLATIC OF SERVER KNOWLEDGE
At each iteration, the server model first performs self evaluation on the public dataset and split the samples into two classes, those it can predict correctly, Scorect and those it predicts wrongly, Sinaoreet For each sample a² in Scoreet where the model prediction matches the ground truth label, we simply adopt the cross-entropy loss between the predicted values and the ground truth labels to train the server model.
Ls, = LcB -y' (zg()
At the same time, we save the correct logit I into the global pool of logits as Ctobbal which can be further used as memory to recover its reserved knowledge from self-distillation.
For each sample xf that the server predicts wrongly, we first check whether its corresponding global logit Eigtohat exists or not. We denote all samples that do not exist in Gaoobll Simcoreca for which we believe the server model can not learn the sample entirely by itself and propose to use clients' collective knowledge as its teacher, which will be explained in the next section. lagosee exists, it means that the knowledge was reserved by the server before, the server model performs self- distillation to recover this part of knowledge. The final training objective of self-distillation can be formulated as follows, where DKI. is the Kullback Leibler (KL) Divergence function and is hyper-parameter to control the weight of knowledge distillation in all the following formulations.
= = = cLCE + (1 (1-c)D0 c)D (Forotat
Our self-distillation strategy has at least two advantages. On the one hand, distilling knowledge from itself is better than from other models which have different architectures. On the other hand, learning from its stored logits can greatly reduce the communication cost as compared to transferring all redundant logits client models.
3.3.2 SELECTIVE ENSEMBLE DISTILLATION OF CLIENT KNOWLEDGE
For those samples in Sincorrect that the server model fails to predict correctly, we try to distill knowl- edge from the ensemble of clients. Furthermore, considering the correctness and relative importance of client models, we propose a weighted selective strategy on client side.
Given instance (x;, Mt in Sincoment we first split all clients (C1, C2.,., Cx) } into the reliable and unreliable clients according to their predictions Pc,, For those clients who predict the wrong labels, consider them unreliable and discard their knowledge by setting their weights equal to O. As
4

---

Under review as a conference paper at ICLR 2022
KLLoss
Glotel
X Pubric
YES
KLLoss
NO
IF in latann
Server
CBLoss
Save
Claani
NO pred
1
Server
Figure 2: Selective knowledge fusion module.
for the rest of clients who predict the labels correctly, we consider them as reliable and use their entropy (pc,) as measure of the confidence.
H(c.,)
Σ p( (log((a))(a
(3)
Following previous work (Li et al., 2021; Pereyra et al., 2017: Szegedy et al., 2016), low entropy indicates high confidence, and vice versa. Specifically, given instance (æ;; y:) in Incomrett we design its corresponding weights to aggregate output logits from different clients as below.
0. aC2 softmax( H(po,))
Cj E Unseliable
1
Cj E Reliable
Therefore, for samples in Sinnmmect the knowledge transferred from ensemble clients to server model can be formulated as the following.
Cs, Lcs cCcE (1 c)DKL
Σ ac le, (6
(5)
Up to now, we have covered the selective knowledge fusion strategy for the server, next we will discuss the knowledge distillation clients.
3.4 TRAINING CLIENT MODELS
Each client model first receives the logits I Seruerr of public dataset from the server model. Then distills knowledge according to the logits as well as computes the cross-entropy loss to train on public dataset.
Lc = ELcE + (1 (1-16 e)DxL (4))4)
(6)
After knowledge distillation, each client model further adopts the cross-entropy loss to train its local private dataset to better fit into its target distribution.
4 EXPERIMENTAL EVALUATIONS
4.1 EXPERIMENT SETTINGS
Task and Dataset. For fair comparison, use the same training tasks He al. (2020a), which include image classifications on the CIFAR-10 (Krizhevsky et al., 2009) and CIFAR-100 (Krizhevsky et al., 2009) datasets. More details about these datasets can be found in Appendix randomly split data into public and private dataset with ratio of 1:1.
B.I. For each dataset, We further study the impact of this ratio Sec. of our experiments. The public dataset used for transferring knowledge between server node and client nodes, while the private dataset is for client training. Both of them further split into training and a testing dataset with a ratio 5:1.
Homogeneous Setting, In our main experiments, we randomly shuffle and partition private dataset into 16 clients. As shown in Appendix B.6, our client models all adopt tiny CNN architecture called ResNet-11, 1, while the server model architecture is ResNet-56. To investigate the influence of server model size and client number, we further vary the server model size from ResNet-20 to ResNetl 10 and the total client number from 4 64 in Sec. 5.3 and Sec. 5.4, respectively.
5

---

Under review as a conference paper at ICLR 2022
Heterogeneous Setting. We adopt the Dirichlet distribution (Yurochkin et al., 2019; Hsu et al., 2019) to control the degree of heterogeneity in our non-iid setting. In this experiment, our a 0.5. We also introduce model heterogeneity in our experiments. The client models vary from ResNet-11 to Resnet-17 17 as shown Appendix B.7.
Implementation Details. In both homogeneous and heterogeneous settings, the number of local epochs are set to after careful tuning, consistent with implementations in FedGKT. The number of communication round to reach convergence is 400. We adopt Adam optimizer (Kingma & Ba, 2014) with an initial learning rate 0.001. Following He et al. (2020a), the learning rate can reduce once the accuracy is stable (Li Arora, 2019). As for the factor c, we find that the best choice is 0.75. Its impacts on server and client performance are shown in Appendix B.5.
Baselines. We compare our framework FedGEM/FedGEMS with the following baselines. (1)
Stand-Alone: the server and client models train on their local datasets X and X respectively; (2)
Centralized:
the client model trains on the whole private dataset +...... x*1 : (3) Centralized- All: the server and clients perform centralized training the whole dataset {x +.... X X*),
respectively. Note reality no party has this complete dataset so this can viewed as an upper limit of model performance. In addition, we also compare our performance with several existing related works, including FedAvg (McMahan et al., 2017). FedMD (Li & Wang. 2019). Cronus (Chang et al., 2019), DS-FL (Itahara al., 2020). FedDF (Lin et al., 2020) and FedGKT (He al., 2020a). For fair comparison, the settings of client models and the split of public and private dataset in all approaches as its clients' model, ResNet- 1, according to its design, while FedGKT and our FedGEMS utilize ResNet-56. Note FedGKT (He et al., 2020a) only reported the performance of the server model. We empirically found that its clients' performance is lower due to constraints of model size. For a fair comparison with FedGKT, we only reported its performance on server side. More details of their important hyper-paramedee parameters in our experiments are listed in Appendixes B.9, respectively.
kept the same. As for the server model, FedDF adopts the same architecture
4.2 PERFORMANCE EVALUATIONS
Homo
Hetero
Method
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Server
Clients
Server
Clients
Server
Clients
Server
Clients
Stand-Alone
84.03
41.57
65.38
26.95
84.03
28.58
65.38
18.70
Centralized
76.21
58.61
83.54
63.44
Centralized-All
91.27
82.45
78.13
65.51
91.27
88.54
78.13
70.96
FedAvg
53.33
31.47
FedMD
58.47
34.77
51.11
25.57
Cronus
57.73
34.19
53.47
29.71
DS-FL
50.78
25.70
43.83
16.69
FedDF
56.31
31.27
FedGKT
55.67
29.89
45.55
26.96
FedGEM
86.62
80.35
67.72
62.61
87.11
80.14
67.27
64.73
FedGEMS
88.08
81.86
69.08
63.81
87.97
84.18
67.72
65.93
Table 2: Model performance in homogencous and heterogencous settings.
Table 2 shows the comparisons of our approaches with other baselines with ResNet-56 as the server model. First of all, our approach FedGEM outperforms all the above KD-based federated learning baselines significantly on both server and client side, demonstrating the effectiveness of knowledge transfer with a larger server model. Compared with FedGKT which also adopts a larger server model, our performance is significantly improved by knowledge fusion from multiple clients. Com- pared to the performance of stand-alone server and clients, our framework simultaneously improves both the sever model and client model, indicating that server and clients mutually benefit from knowledge transfer. Using public labels for supervision and selection. our FedGEMS framework is able to enhance positive knowledge fusion and further improve performance both server and clients. We also conduct experiments with clients and ResNet-20 as server model, shown in Ap- pendix B.4.

---

Under review as a conference paper at ICLR 2022
4.3 ROBUSTNESS
Poisoning Attacks. Following Chang et al. (2019), we employ model poisoning attacks to evaluate the robustness of our methods. Our model poisoning attacks include Naive Poisioning (PAF), Little Is Enough Attack (LIE) (Baruch et al., 2019) and OFOM (Chang et al., 2019). In PAF and LIE, the attacker poisons one client at each round via disturbing the logits or parameters which transfer from clients to server. In OFOM, two clients' benign predictions or parameters are poisoned at each round. The detailed algorithms about different poisoning attacks can be found in Appendix A.
PAF
LIE
OFOM
Method
Server
Client
Server
Client
Server
Client
FedMD
16. 13 /-68.44
16.21/-68.28
22.53/-55.92
DS-FL
37.94/-13 13.44
23.42/-46.57
23.32/-46.77 76.
Cronus
53.71/400.45
53.98/+00.94
53.74 /-00.50
FedGKT
13.82/-69.66
10.42/-77.12
14.20/-68.83
FedGEM 55.94/-35.78
72.78/-09.18
74.86/-14.06
74.32/-07.26
85.22/-02.17
74.47/-07.08
FedGEMS 87.46/-05.80
82.58/-01.90
87.95/-00.02
83.27/-01.08
88.53/+00.64
82.76/-01.67
Table 3: Comparison model poisoning attacks heterogeneous setting.
"A/B" denotes model performance after attacks, while B denotes the percentage changes compared to the original accuracy ("4" denotes increasing, and "-" denotes dropping). We remark both the best performance and the least changed ratio (%) to its original accuracy.
The results of poisoning attacks in heterogeneous setting on CIFAR- 10 dataset are shown in Table 3. By adopting a selective strategy, our proposed FedGEMS provides consistent robustness across var- ious attacks on both server and client models. FedGEM demonstrates superior robustness to FedMD on various model poisoning attacks. DS-FL is relatively more robust than FedMD via its proposed entropy reduction aggregation based on FedMD framework. Since Cronus specifically designs se- cure robust aggregation in a FedMD framework, its robustness to the attacks is strong, as expected. The performance of FedGKT drops marginally because it transmits extra feature maps which is vulnerable. Our selective knowledge transfer strategy shows comparably strong robustness on both clients and server side without additional computation cost for robust aggregation algorithms.
4.4 COMMUNICATION COST
Following Itahara et al. (2020), we further evaluate the cumulative communication costs required achieve specific accuracy in heterogeneous setting on CIFAR-10. The epochs in both client and server sides among all methods to achieve their best performances are the same as 1 which further ensures fairness. The formulations to calculate communication cost for different methods are shown in Appendix B.2.
Method
ComU@405 (MB)
ComU@50% (MB)
ComU@ 80mU@80% (MB)
Top-Ace (%)
FedMD
976.56
5,493.16
51.11
Cronus
610.35
2,471.92
53.47
DS-FL
3,692.63
43.83
FT
3 11,250.00
45.55
FedGEM
91.55
244. 14
3,261.27
80. 14
FedGEMS
23.38
190.87
1,894.84
84.18
Comparison of communication and Top-Accuracy. ComU@x: the cumulative communication cost required to achieve absolute accuracy x. Top-Accuracy: the highest testing accuracy among training process.
Table 4:
The results are shown in Table 4. It is worth noting that FedMD, Cronus, DS-FL and FedGEM cost the same per round according to their designs. Therefore, to achieve the accuracy of 45%, FedGEM costs much lower than other three methods which indicates that FedGEM converges fast. The communication cost of FedGKT is much higher than others because it sends feature map per private data in training. Specifically, the communication cost per feature map is 64kb while the cost per logit is only 0.039kb. Furthermore, the communication FedGEM the whole training phase which means that our selective strategy can effectively reduce uploads achieve the same performance.
of FedGEMS keeps lower than
7

---

Under review as a conference paper at ICLR 2022
5
UNDERSTANDING FEDGEMS
5.1 KNOWLEDGE ACCUMULATION AT SERVER
20000
Sett Thariing
Divr
Crazmble Distilation
Method
Server
Client
19020
FedGEMS
88.08
81.86
Seef-- Self--raiinngg
85.73
80.17
lall-Distillation
86.12
81.30
Ensemble Distillation
86.55
80.01
900
2003
400
Figure 3: Total number of samples in different selective Table 5: Ablation studies of three components in selec- decisions.
tive strategy.
To analyze the selective knowledge fusion process of the server model in our framework, we report the number of samples associated with each step in our decision-making process in our experi- ment CIFAR-10 with 25,000 samples as public dataset. Each sample will choose one of three strategies according to our selective knowledge fusion module. Specifically, in Fig. 3, the line of self-training indicates the total number of samples the server model predicts correctly, while the lines of self-distillation and ensemble-distillation indicate the number of samples the server learns via leff-distillation and client-side ensemble knowledge, respectively. In the initial stage, since both the server and client models learn from scratch, the number of samples that the server model can predict correctly is limited, and most of the knowledge is accumulated by distillation from the client models' fusion. As the training progresses, the server model needs to restore some of its knowledge from self-distillation strategy. With the server model continually fusing sufficient knowledge, the model performance of server model eventually exceeds client models and the number of transferred samples from clients dropped to O. Notice that in the final stage, there still remains some stubborn samples which are hard for the server model while most samples can be solved by the large server model with accumulated knowledge.
Furthermore, we conduct series of ablation studies to detect the contributions of different com- ponents to the accumulated knowledge at server side. The results are shown in Table 5 and all the performances of both server and client models decrease. This phenomenon indicates that each com- ponent contributes to the overall performance. Comprehensively, self-training is the most important component and the most probable reason is that learning from the model itself is the most effective way. Ensemble distillation which avoids transferring knowledge from malicious clients can also help prompt the performance of client models in return.
5.2 EFFECT OF PUBLIC AND PRIVATE DATASET RATIO
Concerning the size of public dataset, we evaluate the effect of public and private dataset ratio on both model performance and communication cost per round. To change the ratio, we fix the size of private dataset, and vary the size of public dataset from 5.000 25,000.
100
Hull
30000
FesHID, FedGEM, Cronus, D5-F
Fectug, PedDr
RosGEMS
Ammnerh
20000
35000
5000
3.5
215
RB (b)
Figure (a) Model performance both server (top) and client (bottom) models on public dataset of different sizes; (b) Communication cost per round on public dataset of different sizes.
The results of both model performance and communication cost per round are shown in Fig. 4 (a) and (b) respectively. As for model performance, it can be seen that both the client and server model continuously improve as the size of public data increases, indicating that their performances are highly correlated. Our performance consistently outperforms FedMD, and the performance gap over
8

---

Under review as a conference paper at ICLR 2022
FedMD continuously grows more public dataset is available. Note again FedGEM is essentially FedMD with larger server model so the performance gain is due to the accumulated knowledge on the server side. As for FedDF which owns ensemble server model same architecture as clients, the performance in both server and client models remains a huge gap to our performance. This phenomenon further demonstrates the importance of the large capacity of server model. FedGEMS can still outperform FedGEM especially on the client side which indicates that our selective strategy efficient to fuse positive knowledge to boost performance.
Due to the compact models deployed in client nodes, the bits of model parameters are limited, thus the communication overhead of FedAvg and FedDF is not always higher than typical KD-based methods when large public dataset is used. Due to the enormous communication overhead of feature maps, we do not figure the flar line (irrelevance to the size of public dataset) of FedGKT which is always 1.6M (kb) per round. Thanks to our selection strategy, the communication cost of our proposed framework FedGEMS the lowest among all methods. In summary, the results demonstrate that our selection strategy can greatly save the communication cost with reasonable public data size.
5.3 EFFECT OF SERVER MODEL. SIZE
We further investigate the influence of the server model size on the overall performance by gradually changing the server model from ResNet-20 to ResNet-110, while fixing the client models as ResNet- 11. The details of the model parameters be found in Appendix B.8.
Server
00
87
57
Client
Roshett-20 20
ResHett
ResNet-5e
ResMet-
80
(a)
(b)
Figure (a) Server (top) and client (bottom) model performances of different server model sizes; (b) server (top) and client (bottom) model performances different numbers of clients.
The results are shown in Fig. 5 (a). It can be witnessed that the performance of both the server and client models improves as the server model becomes larger and deeper. This phenomenon verifies the importance of placing a larger deeper server model to boost the performance of both server and the eesource-constraingg client models.
5.4 EFFECT OF NUMBER OF CLIENTS
We vary the number of clients from 4 to 64 of our proposed method FedGEMS while keeping the total number of private samples the same. Thus the number of private samples per client vary accordingly. The experimental results shown in Fig. 5 (b) indicate that with increased number of clients and less client data, the performance of our approaches is basically stable especially in client side, benefiting from our larger server model setting with fused knowledge.
CONCLUSION
In this work, we first propose a new paradigm to apply a large deeper server model to effectively and efficiently fuse and accumulate knowledge, which can enhance the model performances on both server and client sides. Furthermore, we design a selection and weighted criterion on both sides distill only positive knowledge into the server. We conduct series of experiments to evaluate our proposed framework FedGEMS and our results show that FedGEMS can significantly surpass all baselines in both homogeneous and heterogeneous settings. Meanwhile, FedGEMS can further improve the robustness of FL on poisoning attacks as well reduce the communication costs between server and client sides. Our framework has certain limitations, such as the dependence of a labeled public dataset. In future work, will study the effectiveness of our work in other critical tasks, such as NLP and knowledge graph-based tasks, and the settings where public and private data from different domains distributions.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Moran Baruch, Gilad Baruch, and Yoav Goldberg. A little is enough: Cireumventing defenses for distributed learning. arXiv preprint arXiv: /90.006/6, 2019.
Keith Bonawitz, Hubert Fichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, al. Towards federated learning at scale: System design. arXiv preprint arXiv:190..00 01046, 2019.
Theodora S Brisimi, Ruidi Chen, Theofanic Mela, Alex Olshevsky, Ioannis Ch Paschalidis, and Wei Shi. Federated learning of predictive models from federated electronic health records. Interna- tional journal of medical informatics, 112:59-61 67, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. arXiv preprint arXiv:1812.01097 2018.
Hongyan Chang. Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint 1912 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 2015.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated
learning of large cnns the edge. arXiv preprint arXiv:2007 /45 2020a.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Pra- neeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint wxXv:2007.135114 2020b.
Yihui He. Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. Proceedings of the European conference on computer vision (ECCV), pp. 784-800, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253/ 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861, 2017.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06:33 2019.
Forrest N landola, Song Han, Matthew Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size.
arXiv preprint arXiv: 2016.
Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura. and Koji Yamamoto. Distillation-base semi-supervised federated learning for communication-effiieee collaborative training with non-iid private data. arXiv preprint arXiv:2008. 06/ 2020.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems federated learning. arXiv preprint arXiv:1912.04997..22 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.66 1412.6980, 2014.
10

---

Under review as a conference paper at ICLR 2022
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
JiLiang Li, Kim-Kwang Raymond Choo. WeiGuo Zhang, Saru Kumari, Joel JPC Rodrigues, Muhammad Khurram Khan, and Dieter Hogrefe. Epa-cppa: An efficient, provably-secure and anonymous conditional mrivacyprresevvng authentication scheme for vehicular ad hoc networks. Vehicular Communications, 13: 15:104-113 113, 2018.
Suyi Li, Yong Cheng, Yang Liu, Wei Wang, and Tianjian Chen. Abnormal client behavior detection in federated learning. arXiv preprint 1rXiv:1910.099 09933, 2019a.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. arXiv preprint 1rXiv:1905.10499 2019b.
Zhiyuan Li and Sanjeev Arora. An exponential learning schedule for deep learning. arXiv preprint arXiv:1910.07454 2019.
Ziyun Li, Xinshao Wang. Haojin Yang, Di Hu. Neil Martin Robertson, David A. Clifton, and Christoph Meinel, Not all knowledge is created equal. CoRR, abs/2106.01489, 2021. URL
https://aarki org/ /ttg//bb////// 2106. 01489.
Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini- bini-batches, use local sgd. arXiv preprint /808 072 2018.
Tao Lin, Lingjng Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.07244 2020.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiene learning of deep networks from decentralized data. Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- tional Conference on Machine Learning, pp. 4615-4625, PMLR, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, et al. Knowledge inheritance for pre-trained language models. arXiv preprint arXiv:2105. 2021.
Andrew Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Židek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):70-711, 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittieseer, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. arXiv preprint arXiv:1705.1044 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2818-2826 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference Machine Learning, pp. 1115-6114, PMLR, 2019.

---

Under review as a conference paper at ICLR 2022
Paul Voigt and Axel Von dem Bussche. The general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing. 10:3152676, 2017.
Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. Selective knowledge distillation for neural machine translation. arXiv preprint arXiv:2105, 12967, 2021.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware- aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 13(3):1-207, 2019.
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-aware neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 285-300, 2018.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1285-12944 2017.
Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, and Daxin Jiang. Reinforced multi-teacher selection for knowledge distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 14284-14291, 2021.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. Interna- tional Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-688 6856, 2018
A POISONING ATTACKS IN FEDERATED LEARNING
In this section, we introduce a set of poisoning attacks from the literature which used to evaluate the robustness in federated learning.
A.1 NAIVE POISONING (PAF)
Naive poisoning is type of model poisoming attack. The adversary is able to poison fraction of total client. With the knowledge of the distribution of benign updates, the opponent can introduce malicious update fm. This attack is far from the mean of the benign updates, obtained by adding significantly large vector 6' it:
Σラ= 0;
(1-e)n + 0'
Then, the server updates with the malicious update 6, and transfers it to all the clients.
A.2 LITTLE IS ENOUGH ATTACK (LIE)
LIE a type model poisoning attack proposed by Baruch et al. (2019). The detailed implemen- tation of LIE is summarized in Algorithm 2. The malicious update 9m is achieved by making small changes in many dimensions to benign update. The malicious update is trained in server and then shared by each client.
12

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Little is enough attack (LIE) Baruch et al. (2019)
1: Input: The number of benign updates n. the fraction of malicious updates c
2: The number of majority benign clients:
+ en
(8)
Using z-table, set:
mAx max ((2)) < n
(9)
n
4: for j [d] do
5:
compute mean (1z) and standard deviation (a,) of benign updates.
0g Mí + max jj
(10)
6: Output: 0m
A.3 ONE FAR ONE MEAN (OFOM)
This model poisoning attack, OFOM, proposed by Chang et al. (2019). In this attack, two ma- licious updates will be added to the benign updates. First, the opponent adds significantly large vector O' to the mean benign updates to obtain 0z, Then, the opponent crafts 02 which the mean of the benign updates and m
O 2i=1
0 8TH
0
0',
0
(11)
n
B EXTRA EXPERIMENTAL RESULTS AND DETAILS
B.I A SUMMARY OF DATASET
Following He et al. (2020a), our experiments utilize CIFAR-10 (Krizhevsky et al., 2009) and CIFAR- 100 (Krizhevsky al., 2009) 6000 images per class. R-100 is more challenging dataset with 100 subclasses that falls under 20 superclasses, e.g. baby, boy, girl, man and woman belong people.
datasets. CIFAR-10 consists 10 classes colour images, with
B.2 FORMULATIONS COMMUNICATION CoST
In the Fig. 6, we list the formulations to compute communication costs between server and client models in different methods.
Method
Formulation
FedAvg
(Bparas) (#C + #S)
FedMD
DS-FL.
(Biogits) x (#C + #S)
Cronus
FedGEM
FedDF
(iigitss)
X (#C) + (Bparas) X (#C #S)
FedGKT
(Blogits)
(#C + #S) + (Breatures) X (#C)
FedGEMS
B (Bugiits
(#C + #S)
Figure Formulations of Communication Cost. B indicates the corresponding bits of logits parameters. and denotes Server and Client, while and #C denotes the total number of Server and Clients.
13

---

Under review as a conference paper at ICLR 2022
B.3 COMMUNICATION COST OF THE UPLOADING LOGITS
The results of the communication cost of the uploading logits per round shown in Table 6.
Method
1:5
2:5
3:5
4:5
5:5
FedMD
Cronus
3125.0
6250.0
9375.0
12500.0
15625.0
DS-FL
FedGEM
FedGEMS
105.8
155.4
200.8
233.6
266.3
Table Communication cost logits uploading from clients.
B.4 RESULTS CLIENTS AND RESNET-20
We also conduct different setting to validate the stability our proposed framework FedGEMS. In this series of experiments, we adopt 8 clients and set the server model as 20, while keeping other experiment setting the same as main paper.
B.4.1 PERFORMANCE EVALUATIONS
The results of our performance are shown in Table
Homo
Hetero
Method
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Server
Clients
Server
Clients
Server
Clients
Server
Clients
Stand-Alone
81.60
57.38
61.90
33.78
81.60
44.55
61.90
27.26
Centralized
74.81
56.42
79.95
60.52
Centralized-All
88.97
79.77
73.51
65.07
88.97
87.96
73.51
71.75
FedAvg
63.64
39.16
FedMD
67.53
46,26
43.87
34.11
Cronus
66.03
45.00
39.61
38.38
DS-FL
60.36
47.63
36.46
31.05
FedDF
65.18
64.78
42.11
FedGKT
70.80
44.75
31.18
27.72
35.47
12.78
35.53
21.05
FedGEM
83.53
79.70
66.45
61.28 61.70
83.47
78.89
65.77
57.21
FedGEMS
85.65
81.10
67.31
85.32
80.38
65.91
58.61
Table 7: Model performance in homogeneous and heterogeneous settings.
B.4.2 ROBUSTNESS
The results of our robustness experiment are shown in Table 8.
B.4.3
COMMUNICATION COST
We further evaluate the communication costs per round by varying public dataset sizes for various approaches on CIFAR-10, and the results are shown in Fig. 7.
B.5 RESULTS OF DIFFERENT E
The results of different e, which is a shown in Table 9.
to control the weight of knowledge distillation,
14

---

Under review as a conference paper at ICLR 2022
PAF
LIE
OFOM
Method
Server
Clients
Server
Clients
Server
Clients
FedAvg
14.42/-49.22
09.50/-54.14
13.82/-49.82
FedDF
13.12/-52.06
13.97/-50.81
15.89/-49.29
25.82/-38.96
12.64/-52.54
14.40/-50.38
FedMD
33.27/-34.26
43.97/-23.56
32.74/-34.79
DS-FL
46.62/-13.74
44.74/-15.62
45.20/-15.16
Cronus
66.29/400.26
66.08/+00.05
65.98/-00.05
FedGEM 60.01/-23.52
74.79/-04.91
43.62/-39.91
69.26/-10.44
83.71/400.18
79.36/-00.34
FedGEMS 86.86/+01.21
80.85/-00.25
84.64/-01.01
80.48/-00.62
86.33/+00.68 80.78/-00.32
Table 8: Comparison of model poisoning attacks in homogeneous setting. The A in "A/B" denotes the model performance after attacks, while B denotes the percentage changes compared to the original accuracy ("+" denotes increasing, and "-"denotes dropping). We remark both the best performance and the least changed ratio (%) its original accuracy.
16000
Fedsvgg
---
FeoMD, FedGEN, Cronus, DSFL
12000
FedGEMS
10000
6000
4000
1:5
2:5
3:5
4:5
5:5
Figure Communication cost per round of different public dataset sizes on CIFAR-10 10. Due enormous communication overhead feature maps, we do not figure the flat line (irrelevance the size of public datasct) of FedGKT which is always 1.6M (kb).
6 0.10
6 0.5
0.75
0.90
Method
Server
Client
Server
Client
Server
Client
Server
Client
FedGEMS
88.71
80.37
87.78
81.97
88.08
81.86
84.50
79.53
Table 9: Comparison model performance in different c.
B.6
MODEL ARCHITECTURES OF HOMOGENEOUS SETTING
The model architectures of client and server models in homogeneous setting are shown in Table 10.
(Clien)
Stride
3 3,
x 1,64
x
X 128
x 3,32
X
X 256
3 x 3, 64
X
average pool
10-1
127612
ResNed-20 (Server)
3 x
1 x 1,
1 x 12,32
x 1,
3,
pool
2203788
1,64
128
ResNet-56 (Server)
Stride
3 x J.
x 6,
x
1 128
x 3,
x
3 3,64
1,266
x
average pool
10-d
$91322
Table 10: Details of Model Architectures in Homogeneous Setting
B.7 MODEL ARCHITECTURE OF HETEROGENEOUS SETTING
The different model architectures of client models in heterogeneous setting are shown in Table 11. The server model in heterogeneous setting is the same homogeneous setting in Table 10.
15

---

Under review as a conference paper at ICLR 2022
Model
Com11
Com22x
Com3.
Com-L_x
1 1,16
1 1,
1 1, 64
9)
3, 16, stride
3 1,84
3 3,32
3 3,
average 10-d
2,6
1,64
ResNer-1412, 10)
3, 16, stide
3x3,3
x
3 x 3,32
x.1
3 3,
x
everage pook, 10-d fo
x 1,
x
1228
x 1, 250
x 16
X 1,32 3,32
1 6,64
ResNet-1413, 16, stide
ovarage 10-d
x 1,
1, 128
1, 256
x 1.16
1 x 1,32
1 x 1, 3 x 3,64 1, 256
ResNet-1414, 12)
3, 16, stride
3 3,
x
3 x 3,32
x
ivaragg pool. 10-d fe
x 1,64
128
1 16
1 32 3,32
1 1,
16, stride
x
3 x 3,
everage poot, 10-d
x 1,84
x 128
x.x 2,6
1,16
x 1,32
1 1,
Res6et-17(6, 14)
3, 16, stride
3,16
3 3,32
8 x 6,
pool, 10-d
x 1,64
256
1,
ResNet-17(7, 18)
16, stride
3,
x
x 3,32
x
3 x 3,
werage pool,
x 1,66
X 1, 128
x 1, 256
x 1, 3,16
X 1,32
1 x 1,64
ResNet-17(8. 3, stride
3 3,32
3 x 3,
average 10-d
1,81,
1,6
Table 11: Details of the 16 client models architecture used in experiment
B.8
MODEL ARCHITECTURES OF LARGER SERVER MODELS
The detailed model architectures of large server models are shown in Table 12.
Model
Convt
Conv2
Cons3.x
Com-4xx
x 1, 10
x 1,
x 1,
ResNet-20
stride
3, 6,
3.32
3,
average d0
128
2,2
ResNet-388
3 3, 16, shide
x 3, 16
x
3 3,32
x
3 3,
x
avenge pool, 10dd
1,
x 128
x
1,256
1, 6,
1,32
1 64
ResNet-56
3,32
average pool, d
1 128
1,6
1 x 1,16
1 x 1,32
x 1.64
ResNet-110
3 3, 16, strid 1
3 x 3,16
x 12
3 x
3,
x:12
3 x.x 3,
12
average pool, 10-d fc
1,26
Table 12: Details of the large server models architecture used in our experiment
B.9 HYPER-PARAMETER
In table 13, we sum up the hyper-parameter settings for all the methods in our experiments. We directly run FedAvg, FedGKT and re-implement all the other methods base on an open-source fed- erated learning research library FedML (He et al., 2020b) in a distributed computing environment.
16

---

Under review as a conference paper at ICLR 2022
Methods
Hyperparameters
Homogeneous
Heterogeneous
optimizer
Adam, 1r=00001. wd=0.0001
Adam, 1r=0.001, wd=0.0001
batch size
256
256
client epochs of public data client epochs of private data server epochs
FedGEMS
communication rounds
200
200
optimizer
Adam, 1r=0.001, wd=0.0001
batch size client epochs
64
FedAvg
communication rounds
200
optimizer
Adam, 1r=0.001, wd=0.0001
Adam, 1r=0.001, wd=0.0001
batch size
64
64
FedMD
client epochs of public data client epochs of private data communication rounds
200
optimizer
Adam, Ir=0.001, wd=0.0001
batch size
FedDF
client epochs of private data
40
server epochs
communication rounds
100
optimizer
Adam, Ir-0.001, wd=0.0001
SGD, I=0.005. wd=0.0001
batch size
256
256
FedGKT
client epochs of public data
server epochs
20
40
communication rounds
200
200
optimizer
Adam, 1r=00001, wd=0.0001
Adam, 1r=0.001, wd=0.0001
batch size
64
64
Cronus
client epochs of public data client epochs of private data communication rounds
200
200
optimizer batch size
Adam, 1r=0.001, wd=0.0001
Adam, Ir=0.001, wd=0.0001
client epochs of public data client epochs of private data server epachs
DS-FL
communication rounds
optimizer batch size epochs
Adam, 1r=0.001, wd=0.0001
Adam, Ir=0.001, wd=0.0001
Standalone
200
200
optimizer batch size epochs
Adam, 1r=0.001. wd=0.0001
Adam, Ir=0.001, wd=0.0001
Centralized
256
256
200
200
Table 13: Hyper-parameters used in Experiments dataset CIFAR-10, CIFAR-100 and CINIC-10
17


Reviews:
[{"review_id": "zGl-ZoRrcjy", "reviewer": "Reviewer_Cj8x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strengths\n1. The selective KD is novel to FL\n2. The problem (training a large-scale model) by utilizing scatter data from large number of edge devices is well motivated.\n3. Many important and highly related baselines are compared.\n\n* Weaknesses\n1. It's supervising to me that a smaller model (ResNet-20) used in the proposed framework can outperform a larger model (ResNet-56) used in  FedDF and FedGKT.\n\n2. FedGKT's accuracy is not the same as the result reported in the original paper. Is that because of bug or not well-tuned?\n\n3. The proposed framework is not data-free KD like FedGKT. The public and private data is 1:1. In practice, this may not be the case. Another issue of such setting is that the performance in other ratio is not clear. It would be better to do an ablation study to understand the performance in different ratio.\n\n4. The communication cost is over claimed. FedGEMS requires a public dataset for each client to boost the transfer, when the number of clients is big, the cost is very high. The total communication cost should be larger than FedGKT. The benefit of FedGKT is that it only replies on the sample number on each client, which is normally small. Another concern is the memory cost. FedGKT does not need any public data to do KD, however FedGEMS requires too many samples for edge devices (30000 in the current implementation).\n\n5. Another issue would be the scalability. Note that cross-device FL should be stateless, meaning that the newly sampled clients do not have any previous cached states (optimizer states, logiits, etc) [1] . FedGEMS's assumption seems can only be used in cross-silo FL.\n\n6. It seems the author uses FedML to finish the experiments. Does FedML support the implementation? Especially the complex communication and KD pattern described in FedGEMS.\n\n7. It's highly appreciated when the authors can discuss some limitations. For example, the memory cost for public data; whether the client side can support a large model; what if the public and private data ratio is changed, etc.\n\n8. The Introduction attempt to convince readers that training GPT-3 like giant model in FL is necessary. This is ok to me. But the experiments only talk about small scale ResNet-20. Note that GPT-3 is a billion-level model, while ResNet-20 is much smaller. This somehow contradicts the motivation. Another related issue is that the paper is titled as \"larger\" model but the model used in experiments (ResNet-20) is smaller than other baselines.\n\n9. The writing need to be further polished.\n\n10. \"Continual\" in Abstract may not be a correct word, since it normally refers to continual ML. But FedGEMS does not have related design.\n\n[1] Federated Reconstruction: Partially Local Federated Learning. NeurIPS 2021.\n", "summary_of_the_paper": "This work proposed a KD-based FL framework for training a larger server model securely. The proposed framework is a combination of FedGKT-like idea and further apply some KD strategies for selective transfer. Many baselines are compared to demonstrate its efficacy.", "summary_of_the_review": "Overall, it's a good empirical paper, but it's better to address some of the above concerns before applying the proposed framework in practice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Uf3zhFJR9Pe", "reviewer": "Reviewer_Ed7U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The authors propose an empirical novel selection technique for knowledge fusion.\n- The authors tackle multiple challenges in federated learning with their approach.Their results seem to show improved accuracy for both hetero and homogeneous settings. Since only predicted logits are shared, the communication cost is also low.\n- The algorithm is also shown to work well in resisting poisoning attacks.\n\nWeakness:\n- As per my understanding, the algorithm at the server end uses a combination of techniques. It could be very useful for future researchers to see the impact of each of them separately. For example, we can analyze the effects on accuracy if the logits are not stored (Algorithm 1, Lines 8-9) or the client selection step is not included and all clients are used instead (Algorithm 1, Lines 11-12) \n- It looks like the server model is only trained on the public data. What will happen if the server model overfits and predicts everything or most of the samples in the public data correctly? As per my understanding, it will not use the client's information from the logits at all after that. \n- The selective knowledge fusion technique is very empirical. The paper therefore needs to have many more ablation studies. For example, the paper could show the effect of larger and smaller server models and the change in accuracy for different numbers of clients during poisoning attacks.\n- In Page 5, Paragraph 1, Line 2-3, it is mentioned that the unreliable clients are removed. I understand the reasoning is to remove bad clients but it seems we may be losing some information here. For complex datasets, we may have a scenario where, for outlier samples, generalized clients’ models are not predicting correctly but overfitted clients’ models are predicting better. In that case the server model might not be able to generalize well.\n- The title, although technically okay, gives the impression that the server model is very large. But the server model(Resnet-20) is not exactly a whole lot larger than the largest client models (Resnet-11 to Resnet-17). For reference FedGKT uses ResNet-56. As mentioned before, there needs to be more thorough ablation studies for larger and smaller server models.\n- In Table 2, the results for FedGKT look to be different from the ones mentioned in the actual FedGKT paper. The exact changes to the training and inference process is not mentioned.\n\n", "summary_of_the_paper": "The paper proposes an empirical technique to train large server models with  selective knowledge fusion. The paper passes logits predicted on a public dataset accessible to both clients and server to share knowledge and uses a selection scheme to withstand poisoning attacks.", "summary_of_the_review": "This paper proposes a novel approach to tackle several problems in federated learning using selective knowledge fusion. However, the algorithm proposed is largely empirical and needs more thorough ablation studies to strengthen its core claims. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AcPW_rkHLFc", "reviewer": "Reviewer_tN4o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "++ Many experiments are conducted to show the results on performance, robustness against poisoning attacks, comm. cost, the model behavior in different server sizes.\n\nHowever, major concerns are existing in the current form of this paper:\n\n-- The contribution is not clear. The key issues of existing methods and the challenges of the proposed method are not clearly stated.\n\n-- The technical depth seems weak. Table 1 is a good summary, but it also shows the weakness of the proposed model.\n\nSmaller ones:\n\nThe mathematical expressions need to be improved, which hinders the understanding of the content.\n\nThe notations are not consistent. For example, f_s and f^0 are both used to denote the server model.\n\nNo explanations for symbols L_DL, \\epsilon, et al.\n\nThe y_i in Section 3.1 should be y_i^k\n", "summary_of_the_paper": "This paper tries to address the big model on the server-side in federated learning (FL) via knowledge distillation. Selective transfer knowledge happens on both server and clients sides. The proposed model maintains a set of global, correct logits so that it can be used as memories for the server's distillation. When the server predicts wrong on the public datasets, it transfers the knowledge from the correct, weighted clients. The proposed model is evaluated on image datasets with many FL baselines.", "summary_of_the_review": "contribution not clear, technical depth weak, writing and notation to be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7yLx5kHQ0CM", "reviewer": "Reviewer_Q1kU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem considered in this paper is interesting and important. The need to accommodate on-device systems constraints (limited memory and computational power) significantly limits the performance of models trained using FL. This paper suggests an interesting approach towards overcoming this issue, and the experimental results suggest that the proposed approach is promising.\n\nHowever the paper has a number of limitations, which are detailed below. Some are related to the practicality of the proposed approach, but the majority of these are limitations of the experimental setup used for comparisons. Normally I would be willing to overlook some of the concerns, but there are so many in this case that my overall recommendation is to reject the paper.\n\n**Major concerns:**\n1. The scheme described in Sec 3.3.2 does not seem compatible with privacy considerations that are important in FL. In particular, in order to partition clients into reliable and unreliable sets as described, the server would need to have the individual logits for each training instance from each client. Such information could be pretty easily used to reverse-engineer the clients' training data (e.g., Geiping et al., NeurIPS 2020).\n\n2. If I understand correctly, in all experiments, the distribution of the public data set is identical to that of the private dataset (if the private datasets were all pooled together). This seems very realistic, as one might expect that clients could have data following some distribution locally which is not reflected at all by data in the public dataset; in particular, I'm thinking of clients with private training samples with inputs $x_i$ that fall outside of the distribution of inputs in the public dataset.\n\n3. The experiments use a very small number of clients (only 8), and each has a relatively large amount of data. No information is provided in the main paper about how much training effort is required to achieve the results in Table 2. (How many global rounds, and how many local epochs per round?) Also, the introduction and setup considered seem to imply that this approach is intended for cross-device FL. If that is truly the case, it would be important to include experiments showing that FedGEMS works with many more clients and less data per client. The current results seem to suggest that it may be useful for cross-silo FL.\n\n4. The performance reported for FedGKT is _much_ lower than what is reported in [He et al., 2020](https://arxiv.org/abs/2007.14513). I realize the models may not be exactly the same, but I wouldn't expect to see such a marked difference. This substantial difference, without any remark from the authors, leads me to question how much faith to put in the experimental results. How much effort went into hyperparameter tuning for each method? \n\n5. Are the attacks considered in Sec 4.3 being newly introduced in this paper or have they been considered in previous literature? If they are, references should be included in the main text of the paper (not just in the appendix). If the attacks are actually new, then significantly more motivation should be included for why these attacks are relevant.\n\n6. Table 3 reports the results of poisoning attacks in the homogeneous setting, but it would be much more interesting and relevant to also include the performance in the heterogeneous setting.\n\n7. Fig 3 shows the communication cost per epoch (per round?). This is only part of the story, though, since the overall communication overhead also depends on the number of rounds required to reach the final accuracy. The paper does not report how many rounds each method was run for, or if it's the same for all methods. Upon further examination in Table 8 of the Appendix, it appears that different methods were run for different numbers of rounds. It is not clear that this leads to a fair comparison.\n\n8. Fig 4 shows that the number of samples selected for clients to report back to the server changes during training. Fig 3 shows a fixed communication cost per iteration. Are those numbers at the beginning or end of training, or averaged over an entire training run? Also, wouldn't one expect those values to change substantially depending on the problem and the public/private datasets?\n\n9. The objectives (2), (5), and (6) all depend on the parameter $\\epsilon$ that balances between cross-entropy and distillation losses. What value was used in the experiments? It is impossible to reproduce the results without knowing this. Also, how sensitive is performance to the choice of this parameter? \n\n**Minor / requests for additional information:**\n* This paper seems to implicitly focus on classification problems. This ought to be stated explicitly in the introduction, and ideally also in the abstract.\n* The title (\"Client-side...\") of Sec 3.3.2 is confusing, since that section describes operations that are all performed at the server.\n* Please add an x-label for Fig 3.\n* Do the communication amounts for FedGEMS reported in Fig 3 count the overhead of the server sending the selected indices to each client?", "summary_of_the_paper": "This paper aims to enable training of higher accuracy models by training a high-capacity model at the server and lower capacity models at the clients. Knowledge is shared via distillation on a weighted average of logits provided by the clients, rather than by averaging model parameters or model differences. The approach assumes the existence of a labeled public dataset. Experiments illustrate the promise of this approach.", "summary_of_the_review": "The problem considered in this paper is interesting and important. The need to accommodate on-device systems constraints (limited memory and computational power) significantly limits the performance of models trained using FL. This paper suggests an interesting approach towards overcoming this issue, and the experimental results suggest that the proposed approach is promising.\n\nHowever the paper has a number of limitations. Some are related to the practicality of the proposed approach, but the majority of these are limitations of the experimental setup used for comparisons. Normally I would be willing to overlook some of the concerns, but there are so many in this case that my overall recommendation is to reject the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a knowledge distillation strategy to enable the use of a large server-side model in federated learning while satisfying the computation constraints of resource-limited clients. The problem is relevant and well-motivated, and the paper presents compelling experimental results to support the proposed strategy. However, reviewers had the following major comments suggestions/:
1) The theoretical analysis section needs improvement in terms of the technical depth and rigor
2) Better explanation of how the proposed strategy compares with previous works/baselines
3) Considering the privacy and scalability properties of the proposed strategy.

The paper generated lots of constructive post-rebuttal discussions between the authors and the reviewers, and I believe the authors received several ideas to improve the work and appreciated the reviews. One of the reviewers increased their score. However, based on the current scores, I still recommend rejection. I do think the paper has promise, and with improvements, the revised version will make an excellent contribution.
Decision: Reject