FedEED: Efficient Federated Distillation with Ensemble of Aggregated Models
Authors: Ho Man Kwan
Shenghui Song
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Knowledge Distillation

Abstract: In this paper, we study the key components of the knowledge distillation-based model aggregation in federated learning (FL). We first propose a generalized distillation framework where the process of federated distillation is divided into three key stages. By investigating the contributions of each stage, we introduce a new FL framework, named Federated Efficient Ensemble Distillation (FedEED), where the ensemble teacher is created based on aggregated models. Experiment results showed that FedEED outperforms the state-of-the-art methods, including FedAvg and FedDF, on the benchmark datasets. Besides performance, FedEED also demonstrated improved scalability and privacy when compared with existing distillation-based aggregation algorithms. In particular, FedEED does not require direct access to users' model, which can protect the users' privacy. Furthermore, due to the ensemble created by aggregated models, FedEED is highly scalable, and the asymmetric distillation scheme allows parallelism between server-side distillation and clients-side local training, which could speed up the training of large scale learning system.

Under review as a conference paper at ICLR 2023
FEDEED: EFFICIENT FEDERATED DISTILLATION WITH ENSEMBLE OF AGGREGATED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we study the key components of distillation-based model aggrega- tion in federated learning (FL). For that purpose, we first propose a generalized distillation framework which divides the training and model aggregation process into three key stages and includes existing methods as special cases. By inves- tigating the contribution of each stage, we propose a novel distillation-based FL scheme, named Federated Efficient Ensemble Distillation (FedEED). Different from existing approaches, the ensemble teacher of FedEED is constructed by ag- gregated models, instead the client models, to achieve improved scalability large-scale systems. Due the use of aggregated models, FedEED also achieves higher level of privacy protection, because the access to client models is no longer required. Furthermore, the knowledge distillation in FedEED only happens from the ensemble teacher to a designated model such that the diversity among differ- ent aggregated models is maintained to improve the performance of the ensemble teacher. Experiment results show that FedEED outperforms the state-of- tart FL schemes, including FedAvg and FedDF, on the benchmark datasets. Besides the performance advantage, the designated distillation also allows for parallelism between server-side distillation and clients-side local training, which could speed up the training of real world systems.
INTRODUCTION
Federated learning (FL.) (McMahan et al., 2017) allows users to jointly train deep learning model without sharing their own data. Recent works (Lin et al. 2020 Zhang et al. 2022 Huang et al. 2022 Cho et al. 2022) adopted knowledge distillation (Hinton et al. 2015) for model aggrega- tion to tackle data and device heterogeneity issues. These distillation- based aggregation methods have been shown to outperform weight averaging methods like FedAvg (McMahan et al., 2017). However, existing works utilized all client models to build the ensemble for knowledge distilltion,, leading to poor scalability in real world applications with large number, e.g., thousands, of clients. Furthermore, existing methods built the ensemble teacher by similar client models, where the low diversity among the compositing models will limit the performance of the ensemble teacher.
To tackle the above issues, we will study the key components of distillation-based FL and propose an efficient and scalable distillation method. For that purpose, we first introduce generalized frame- model aggregation, which consists of three major components, namely
work for distillation-basee the local trainers, the ensemble trainer, and the global trainer. The local trainers perform local train- ing on the clients-side, where each local trainer handles sub-group of participating clients. After that, each local trainer collects and aggregates the updated client models, and constructs one global model. Note that this is the global model for all participating clients in sub-group. Existing FL. algorithms like FedAvg (McMahan al. 2017) can be used in-place as local trainer. Then, the ensemble trainers combine the updated global models and local models into ensemble, provid- ing a higher capacity than the stand-alone global models. Finally, the global trainer will utilize the ensemble to further enhance the global models via knowledge distillation.
With the generalized framework, we investigate the contributions of each key component. In par- ticular, we compare the performance of the distillation-basel FL with different local, ensemble, and global trainers and try to tackle the limitations of the existing methods in a bottom-up manner. We focus several aspects of the distillation framework including: 1.) Improving scalability and pri-

---

Under review as a conference paper at ICLR 2023
vacy of distillation- baasle aggregation by building the ensemble teacher from a set of aggregated models, i.c., client models are no longer required to construct the ensemble teacher; 2.) Maximizing the capacity of the ensemble and global models by maintaining the diversity among models uti- lized to create the ensemble; 3.) Reducing the computation overhead by exploiting the parallelism between server-side and client-side training. By integrating the above ideas together, propose a new algorithm named Federated Efficient Ensemble Distillation (FedEED). FedEED is highly FL algorithm that does not require direct access the client models,
scalable, distillation-basse which provides further protection to the user privacy. The proposed FedEED achieved state-of- the-art results in CIFARIO/100 (Krizhevsky al., 2009) with the indeeeedenddnt and identically distributed (Non-IID) data.
The contributions of this paper include:
1) We propose generalized framework for distillaion- based model aggregation, which can be viewed as a generalization of the existing distillation-ban FL algorithms. With the proposed framework, the contribution of each key component can be investigated to build up more efficient algorithms.
2) By investigating the contribution of each component in the generalized framework, we propose FedHED, a highly efficient and scalable, distillation-basee FI. algorithm with improved privacy and model diversity. Experiment results demonstrate that FedEED achieve the state-of-the-an performance with lower complexity and latency than existing federated distillation algorithms.
2
RELATED WORKS
Federated learning. Deep learning has obtained great successes in the last decade. However, in practice, large amount of user data can not be shared to the central server due to privacy regulations and communication constraints. To tackle the above issues, federated learning (McMahan et al.) 2017) was proposed train a global model based on data belonging different users, without data sharing. The simplest approach is FedAvg (McMahan et al., (017). which performs multiple local epochs of training on the client-side, and then aggregates the updated client models with weight averaging. Other approaches, such as FedProx (Li et al., 2020), also utilize weight averaging to perform model aggregation, but with added regularization to tackle data heterogeneity issues. this paper, we focus on distillation -base model aggregation for FL. For both the generalized framework and the newly proposed FedEED, averaging based methods such as FedAvg and FedProx can be directly applied as the local trainers. This makes the proposed framework and FedEED compatible with different weight averaging methods.
Knowledge distillation. Knowledge distillation (Hinton al. 2015) has been proposed in deep learning to compress deep neural networks. Student models, which typically have a smaller size, are forced to mimie the output of the teacher model. In some recent works, knowledge distillation has been applied in FI. There are two types of approaches to apply knowledge distillation in FI. The first type utilizes distillation to perform model aggregation. For example, FedDF (Lin al. 2020) and FedBE (Chen & Chao 2021) utilized client models as a teacher to update the global model on the server, and the purpose
improve the model aggregation performance with heterogeneous data. Some other works, including FedFTG (Zhang et al. 2022) and Fed-ET (Cho et al. 2022), utilized distillation for the same purpose, but in a different setting, l.e. in a data-free or model heterogeneous environment. The second type shares model predictions between clients and the server for training purposes. For example, in FD (Jeong et al., 2018), model predictions are shared between clients to regularize the local training. In FedAD (Gong et al. 22021), model outputs on the client-side are used to train a global model on the server through distillation.
The proposed generalized framework and FedEED in this paper can be classified into the first type. In fact, the framework is a generalization of the available works in the first type. However, the mechanisms FedEED utilizes to improve its scalability, privacy and performance to those of the existing works, and can be combined with existing methods to further improve the performance.
orthogonal
2

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Generalized Framework for Model Aggregation
function (AI 10'y
Update local models wig W
pdate global model tt
return u wh WNN
function
function ENSEMBLETRAINERIE
"N K-11 wo.0..... we 1,88-1 1
Update ensemble =t
return
end function
function GLOBALTRAINER(UR " ... 1
Update global models wi w return wK-
end function
Initialize K global models 20th for (O...., 13 do I! Split clients into sets St. St. K-1 with sizes No.0. Ne,K-1 (0,..., 1] parallel
-1K-1
wi,,,or--. i,
wh, "1,, wi N.
LocuiTrainer(we 1,k,0 t Wele 1).
end for wi" EnsembieTraiierr(uu fo ... wi.x whe 3,0,0 ... w & N 1-1)
w ur.......
We k -1 +- GlobalTroiner(uff" wi k-1)
end for
Local Trater
infolictioon
-
Upoat
Locol Trainer
Ensentle Baier
t+1
Ensembbee Trainer
Freeluing
Globel Trainer
Craembing
Glaba Tisker
Clant sdd
Dever side
I8 stadoon
Tunnde
(a)
(b)
Figure 1: (a) Overview of the generalized framework for model aggregation. The number of local trainer, the implementation of local training, ensembling, etc., are subject to the implementation. (b) The Ensemble trainer and global trainer of FedEED. There are two key differences. First, the ensemble trainer in FedEED only utilizes the K global models to build the ensemble. Second, the global trainer only update one of the global models by distillation.
THE GENERALIZED FRAMEWORK FOR MODEL AGGREGATION
To investigate the potential of utilizing distillation for model aggregation in FL., we propose the gen- cralized framework, which includes K local trainers, an ensemble trainer, and a global trainer, as shown in Fig. Ia Each trainer can be replaced by different implementations to form different al- gorithms. Note that the generalized framework keeps and updates K global models simultaneously. which includes the existing methods as special case when K 1. In every training round, each local trainer will distribute one of the global models to a subset of clients, perform local training, and aggregate the trained client models into updated global model. Then, the ensemble trainer
3

---

Under review as a conference paper at ICLR 2023
will aggregate the trained client models and/or the updated global models into ensemble model. Finally, the global trainer will compress the knowledge of the ensemble and further enhance the global models.
Many existing algorithms can be viewed as a special case of the generalized framework. For exam- ple, by utilizing FedAvg (McMahan et al., 2017) as the local trainer, combining all client models as the ensemble, and implementing the conventional knowledge distillation (Hinton et al.) 2015), FedDF (Lin et al. 2020) can be realized. By applying multiple local trainers, each of which main- tains a global model with different architectures or hyper-parameters, the heterogeneous learning setting (Huang al. 2022) can be implemented.
By generalizing existing algorithms, the framework makes the design and implementation of FL systems easier, where different components can be updated according to different design objectives. Furthermore, knowledge distillation (Hinton et al.) 2013) is still an active research topic, and the generalized framework allows easier integration following, we introduce the components of the generalized framework, i.e. the local trainers, the ensemble trainer and the global trainer. The overall workflow of the framework illustrated in Algo. and Fig.
different distillation methods into FL. In the
3.1 LOCAL TRAINER
In each communication round, all available clients will be partitioned into K groups, where each group of clients will be handled by one local trainer. For example, in the f-th round, the task of the k-th local trainer is to update the k-th global model by performing local training and model aggregation over the k-th group of clients. particular, the local trainer first distributes the weight of the global model in the previous round, i.e., w to its clients to initialize the client model weights w "2,,,0 W N,, where w denotes the local model of the n-th client in the group with 0 <= n < Nr,k. Nr. denotes the number of clients associated with the k-th local trainer in the t-th round. Then, the local trainer performs training the client-side with the clients' private dataset, and updates the model of the n-th client as i Wi,k.n Lastly, the local trainer collects and aggregates model weights from its clients to the new global model w Note that different local trainers can be associated with models with different architectures, which allows the framework to learn heterogeneous models. In this paper, we only consider the scheme which randomly and evenly map the clients to the local trainers in each training round, such that there are K clients per group, where the number of active clients.
Many existing FL algorithms can be directly utilized as a local trainer, e.g. FedAvg (McMahan et al,, 2017), and each local trainer can be a different learning algorithm. Moreover, learning algorithms with local regularization can be used in-place to tackle the Non-III) issue, e.g. FedProx (Li et al., 2020) and SCCAFOLD (Karimireddy et al., 2020). In this work, we consider three options for the local trainer, including FedAvg, FedProx and SCCAFOLD. We expect the local trainer to have relatively simple relation with the higher level trainers, such that faster local training can speed up the training of the ensemble and hence, the global model. We will evaluate the compatibility of these trainers with different higher level trainers.
3.2 ENSEMBLE TRAINER
After updating the global models by the local trainers, the ensemble trainer constructs the ensemble teacher model (Hinton et al., 2015) with weight wt by the the client models and/or global models obtained by the local trainers. The ensemble trainer may also fine-tune the ensemble weight to w/* with server dataset. Composed by multiple models, the ensemble typically has better performance than the individual models. It can serve as the teacher model for knowledge distillation in the later stage, and also be directly utilized for inference on the server some high power devices.
There are multiple ways to build an ensemble. For example, FedDF (Lin et al. 2020 utilized all client models in round to construct the ensemble; FedBE (Chen & Chao 2021 sampled new models from the distribution built from client models and utilized both the sampled models and client models construct the ensemble for higher capacity. Unlike the above works which utilized knowledge distillation to compress a single and compact model, Fed-ensemble (Shi et al. 202T) learned multiple global models and utilized them to build the ensemble for inference.
4

---

Under review as a conference paper at ICLR 2023
3.3 GLOBAL TRAINER
At the last stage of each communication round, the ensemble obtained from the ensemble trainer is utilized as the teacher model for knowledge distillation enhance the global models. We update the weights of the global models from w²,0.-. w "'K-1 -- to win- -.. w -1 by mimicking the ensemble teacher's outputs with some sample inputs. For example, we can minimize the KL--divereecce to enhance the global models for classification task (Hinton al. [013). Existing works have shown that distillation can be utilized to perform model aggregation in FL. For instances, FedDF (Lin et al.) 2020) utilized a basic knowledge distillation scheme in each round of training. On the other hand, FedBE (Chen & Chao 202T) applied SWA (Izmailov et al. 2018) to prevent the global model from being over-fitted with the noisy teacher output.
EXPERIMENTS: THE GENERALIZED FRAMEWORK
In this section, we study the effect of the ensemble trainer and the global trainer.
4.1
EXPERIMENTS: ENSEMBLE TRAINER
There are three critical issues in constructing the ensemble trainers, namely privacy, scalability, and model diversity. Existing distillation based aggregation methods (Lin al.] 2020 Zhang et al., 2022 Huang et al. 2022 Cho et al. 2022) require all client models to build the ensemble, which cause privacy concerns since the models can leak client's information. See section A.6 in the ap- pendix for more details. Furthermore, their scalability is limited, because the cost of inference with the ensemble proportional to the number of compositing models and the server may not able to efficiently perform the training if the number client models is too large. Existing methods didn't 't address this issue or assume that the computation cost to the server is insignificant comparing to the communication time Chen & Chao (021). With FedDF (Lin al. (020)) and FedBE (Chen & Chao (02T]. the diversity between the models in the ensemble is determined by the local training of single round and may be limited, reducing the capacity of the ensemble.
Fed-ensemble (Shi et al., 2021) trains multiple global models and builds the ensemble from these global models for inference. The ensemble does not scale with the number of clients, but scales with the number of global models K, which is a hyper-parameter. Moreover, the diversity of the models constructing the ensemble is maintained across the whole training, because different global models experience different training sequences. The idea can be utilized to build the ensemble for distillation purposes, which allows for different ensemble complexity, determined by the number of global models K. However, increasing the capacity of the ensemble by increasing K will also reduce the number of clients per global model, and thus slows down the convergence. We thus propose to utilize the last R checkpoints of each of the global models to build the ensemble, which increases the capacity of the ensemble without slowing down the training of the individual global models.
In this paper, we consider five ways to build the ensemble:
1) Vanilla Ensemble. Similar to FedDF (Lin et al. 2020), we utilize all client models to build the ensemble and only train one global model (i.e. K = 1). For an input x, we denote the ensemble func- tion by F, where F(zl wo, ..., WN-1) ) is the class probabilities with averaged logits of the networks that are weighted by Wo. -.. WN-1- We compute the output y by F(x (<olw/o,,.. ..., W/.0,No-1
2) Bayesian Ensemble. Following FedBE (Chen & Chao 2021). train a global model and build ensemble by combining the client models, the averaged model, and M sampled models w with 0 <= m < M, either from the Gaussian distribution the Dirichlet distribution. The ensemble can be expressed as F(xlwtt,,o ..., !x
W200,No- w/,0- wī,o..., w
3) Ensemble from multiple global models. Like Fed-ensemble (Shi al. 2021), we train K global models, each with different initialization seeds and local training sequences, and combine them into an ensemble with y FxIww[o ... iI).
4) Ensemble from client models, which initialized from different global models (Lin et al. 2020 Shi et al., 202T). This scheme is similar to 3), except that we utilize the client models to build the ensemble with y Fa w -L.N-1
5

---

Under review as a conference paper at ICLR 2023
Table CIFARIO results of the generalized framework with different ensemble trainers.
CIFARIO
Model
Method
1.0
ResNet20
Global model (K = 1) Ensemble (K Clients)
88.53 = 0.31
78.72 + 2.31
88.82 = 0.21
80.22
Ensemble (K = 1. Clients, Weighted)
88.79 + 0.16
80.11 0.95
Ensemble
Bayesian, Gaussian)
88.72 1: 0.14
79.60
Ensemble (K = Bayesian, Dirichlet)
88.70 = 0.11
80.25 + 1.43
Global model (K
86.69 1: 0.54
70.74 1 5.11
Ensemble (K 4, Aggregated)
90.58 + 0.31
81.76 + 2.40
Ensemble (K
Clients)
0.17
82.12 1.65
Ensemble (K = 4. R = 2, Aggregated)
90.75 + 0.24
83.50
+ 1.56
Ensemble (K 4, R 4, Agaregated)
90.69 = 0.19
83.99 0.88
5) Ensemble from multiple aggregated global models in multiple rounds. In addition to scheme 3), use the global models from the last R checkpoints build the ensemble with y F(x F(zlwj,, w K-KK- w R+1.0 ". Wi-RR+1,-K
We compare the performance of the above ensemble strategies on CIFARIO dataset with Non-IID IID settings. We performed two runs, where trained one (K = 1) and four (K 4) global models with the generalized framework, respectively. We used FedAvg (McMahan et al. 2017) as the local trainer, and better illustrate the effect of the ensemble trainer, we skipped the global trainer, i.e. we did not perform distillation. For the run with K = 1, we built 1) vanilla ensembles, including variant which weighted by the number of training samples held by the client, and 2) Bayesian ensembles (both Gaussian and Dirichlet). We also performed similar evaluation for the runs with K 4, where we built 3) ensembles from global models of one round (i.e. R 1), 4) ensembles form client models, and 5) ensembles from multiple checkpoints of the global models (with R = 2 or 4). For the details of the experiment setting, see Sec. 6
The results are shown in T and also Fig. Z in the appendix. With low degree of Non-IID (a 1.0), despite slowing down the convergence the individual global model, increasing the number of global models provides the best results. All ensembles with K 4 perform closely, where the ensembles built by global models from multiple checkpoints provide slightly higher accuracy. With
K = 1, all methods do not show clear performance advantage the global model, which agrees with our understanding about the diversity among the compositing models.
With highly Non-IID data, all ensemble strategies showed significant improvement over the global model with K = 1. However, with only global models of a single round, the ensemble with K outperformed the strategies with K 1, which requires access the client models. By increasing R from to 2 or 4, we observed substantial performance improvements by the ensemble with K 4, outperforming all methods with K 1 by large margin. Moreover, with K 4 and R the ensembles built from global models and client models perform closely. It indicates that accessing client models not necessary to build a high performance ensemble.
In the experiment, ensembles built by following option 5) achieved the best performance, with better privacy protection and higher scalability. Thus, we consider option 5) the candidate ensemble trainer.
4.2 EXPERIMENTS: GLOBAT TRAINER
In Sec. 4.1 it was shown that the best performance can be achieved by learning multiple global models and using multiple check-points of these models. However, if we are interested in a single and compact model for inference, need to compress the ensemble into global model. In the cases with multiple global models the generalized framework, there is no clear rule for distillation, and we consider multiple distillation schemes here.
Firstly, we consider a basic knowledge distillation scheme, iee. we utilize the ensemble as the teacher and all global models are trained by mimicking the output of the ensemble. Note that we fix the weight of the ensemble during the distillation process. This similar to the heterogeneous version of FedDF (Lin et al. 2020), where all global models will be improved by knowledge distillation.

---

Under review as a conference paper at ICLR 2023
Table CIFARIO results of the generalized framework with different global trainers.
CIFARIO
Model
Method
ResNer20
w/o distillation
86.69 + 0.54
70.74 + 5.11
Basic distillation
88.63 0.31
2.42
Basic distillation w/ warm-up (20 rounds)
88.66 1: 0.22
80.20
+ 1.81
Basic distillation warm-up (40 rounds)
88.17 0.04
79.27
2.14
Designated distillation
89.06 0.19
80.18 + 2.38
RosNer20 (Ensemble)
distillation
90.58 0.31
81.76 - 2.40
Basic distillation
89.23 + 0.36
80.57 + 2.24
Basic distillationn warm up (20 rounds)
89.38 1: 0.42
80.80
1.95
Basic distillation warm-up roundss)
89.19
2.55
Designated distillation
90.31 + 0.20
81.39 + 2.54
Table 3: Comparison between methods. The ratios are relative to FedAvg.
Method
Commonication cost
Ensemble size
No access to client models Parallelism for server training
FodAvg
FedProx
1.5x
SCAFFOLD
FedDF
1.0x
FedBE
1.0x
+ S+1
FedEED (w/ FodAvg)
1.0x
KR
K: Number of global models
Number rounds
C: Numbher of active clients
S: of sampled models
One possible drawback of the basic scheme is that it may reduce the diversity among the global models and thus hurts the performance of the ensemble, since the global models are forced to pro- duce similar outputs. Thus, we consider another scheme following Codistillation (Anil et al. 2018), where we skip the distillation in the early rounds of training, such that the models will not learn from each other at the beginning to maintain potentially higher diversity.
Lastly, we consider an asymmetric scheme, where only one designated global model will serve as the student and be enhanced by the global trainer. Although there is only one global model being enhanced by distillation in each round of training, the scheme can most of the scenarios where only one global model is required for inference, and reduce the cost of distillation. We note that prior work in knowledge distillation (Chen et al. (020)) has applied a similar scheme where a model is chosen be the main student during distillation, but they did not completely remove the update of the other student models. Compare to their scheme, the all-to-one scheme here is simpler and able to reduce the computation complexity.
In the experiments, we compared the performance of different distillation settings discussed ahove, including: 1) Simple distillation, where we treat all global model as the student model; 2) Simple distillation with warm-up rounds, which skips the distillation in the early rounds of training: 3) Designated distillation, where we only transfer the knowledge from the ensemble to one designated global model. We use 20 and 40 warm-up rounds for 2).
Results in Table 2 show that, the global model trained by the designated scheme outperformed its counterparts with low degree of Non-IID, and performed as to the others with high degree of Non- IID. On the other hand, the ensembles trained by the designated scheme obtained the best results in all the cases, which is close to the ensemble without distillation. The performance of other ensembles by distillation is much worse than that without distillation. These results support our expectation that asymmetric scheme can help keep the model diversity, and thus improve the performance of both the ensemble and the distilled global model.
5 FEDEED
In the Sec 4 we showed that two simple schemes: 1) building ensemble from the aggregated global models and 2) designated distillation from the ensemble to the global model. lead to a com- petitive instance of the generalized framework, with improved privacy and scalability. We refer the above instance Federated Efficient Ensemble Distillation (FedRRD) and the diagrams of the

---

Under review as a conference paper at ICLR 2023
ensemble trainer and global trainer shown in Fig. lb FedEED tackled two limitations of the existing iistillation--aaee aggregation methods by utilizing multiple aggregated global models to build the ensemble. Firstly, FedEED does not require direct access the client models the model outputs, which is compatible to the mechanism that ensures the users privacy, including secure ag- gregation (Bonawitz al.) 2017). Secondly, the number of global models is hyper-parameter, which makes FedEED applicable to large scale FL system with potentially thousands of clients, representing an improved scalability.
Another key advantage of FedEED is the designated distillation scheme. In FedEED, the knowledge of the ensemble will only be transferred one picked global model with weights wi such that the diversity between w w - won't be lost after multiple rounds of training. During inference, either the main global model or the ensemble model can be utilized, depending on the device capac- ity. Besides maintaining the diversity between the models in the ensemble, the designated scheme also allows for parallelism between client-side and server-side computation. particular, during the computation of w other weights wği. wii...., w "i,K1 are ready to be distributed to the clients for the t + 1-th round of training, because they will not be enhanced by distillation. This can improve the system throughput by a large factor, especially if the server side training time is not negligible and the clients are not always available. An example of the training pipeline is shown in section A.7 in the appendix.
In summary, the key idea of FedEED is to ensure privacy and scalability, and maintain the diversity between models throughout the whole training process. See Table. 3 for the comparison between
FedEED and other methods (McMahan et al. 2017 Li al. 2020 Karimireddy al. 2020 Lin
et al. 2020 Chen & Chao 2021).
EXPERIMENTS: FEDEED
6.1
IMAGE CLASSIFICATION WITH CIFAR10/100.
Table 4: CIFAR10/100 results.
CIBARIO
CIFARIOU
Model
Method
ResNer20
FedAvg
88.53 = 0.31
78.72
+ 2.31
58 84
+ 0.42
52.98
+ 1.41
FedProx
0.18
79.44
2.17
0.93
53.57
1
SCAFFOLDD
89.85 0.50
80.08
+ 1.53
61.09 + 0.44
0.46
54.74
+ 1.26
FedDF
87.98
0.16
80.04 1 1.87
59.74 1: 0.38
51.83
1 0.83
FedERD
89.06
0.19
80.18
2.38
61.90 1 0.83
54.72
0.87
FedEED (R = 2)
89.01 = 0.18
81.27 + 2.16
62.42
0.57
56.39
+ 1.10
FodEED (R 4)
62.81 0.45
56.90
0.77
ResNet56
FedAvg
89.25 = 0.11
80.05 + 2.65
59.47 + 1.23
54.37 + 1.24
FedProx
89.65 0.23
80.84 1.32
61.01 1.02
57.18
0.92
SCAFFOLD
90.73 0.27
82.22 + 0.92
62.94 + 0.91
56.36 + 0.31
FodDF
88.87 0.29
81.10 + 2.31
60.71 0.68
52.52 + 1.37
FedEED
89.86 0.14
81.30
2.17
62.95 + 2.43
57. + 2.03
FedEED (R 2)
90.23 0.23
82.77
1: 1.67
65.28 0.64
59.37 1 0.64
FedEED (R 4)
0.26 90.00
83.20
0.99
0.89 65.63
59.56 + 0.43
WRN16-2
FedAvg
90.26 0.33
81.52 + 2.66
63.37 1.06
58.77 + 0.55
FedProx
90.38 0.29
81.78 1. 2.58
63.75 1 0.84
59.24
1: 1.00
SCAFFOLD
91.32 = 0.11
82.90
+ 2.19
65.33 + 0.48
60.66
+ 0.85
FedDF
89.52 0.10
63.54 0.53
FedEFD
90.69 0.41
82.62
+ 2.46
66.46
4 0.30
61.16 + 1.20
FedBED (R 2)
90.77 0.16
83.39
+ 2.17
67.11 0.41
62.37
+ 0.84
FedBED (R 4)
90.70 0.31
81.02
+ 1.75
67.48 + 0.22
63.05 + 0.80
Our experiments of image classification task followed Lin et al. (2020). We trained the models on CIFAR10/100 dataset (Krizhevsky et al. 2009), and utilized CIFARTOO and ImageNet (resized to 32 32) (Deng et al., 2009) as the unlabelled datasets for the CIFARION 10/100 experiments, re- spectively. There are 100 rounds of training with 20 clients, where 40% of them are active in each round. We sampled data for each of the clients with the Dirichlet distribution (Hsu et al. 2019), with reported the average
{1.0, 0.1} for different levels of data heterogeneity. For each setting, t accuracy of three runs with different seeds, where the seeds are shared across different meth- ods to generate the same data partition and sequence of client's for fair comparison. We trained
8

---

Under review as a conference paper at ICLR 2023
Table 5: CIFARIO results of FedEED with different local trainers.
CIFARIO (8 clients per round)
CIFARIO (20 clients per round)
Model
Method
ResNer20
FodAvg
88.53 + 0.31
78.72 + 2.31
88.59 + 0.28
79.84 = 1.50
FedProx
88.36 + 0.18
79.11 2.17
88.52 0.07
79.63
SCAFFOLD
89.85 + 0.50
80.08
1: 1.53
90.10 + 0.07
82.83
1 2.40
FedEED w/ FodAvg
89 06 = 0.19
80.18 :: 2.38
09.01- + 0.11
$2.89
1.30
FedEED FedProx
88.98 + 0.22
81.21
1.30
89.28 + 0.17
82.12
2.26
FedEED w/ SCAFFOLD
88.80 1 0.16
76.51
1 3.74
90.66 1 0.04
84.30
1.88
ResNet20/56 (He et al., 2016) and WRN16-2 (Zagoruyko & Komodakis 2016), which are proven capable to achieve high accuracy on CIFAR 10/100. For the optimizer, we used SGD with learning rate of 0.8 and 0.1 and batch size of 64 and 256, for the client-side and the server-side training, respectively. We did not apply weight decay or learning rate scheduling in our experiments.
We compared FedEED with FedAvg (McMahan et al. 2017). FedProx(Li et al., 2020), SCAF- FOLD(Karimireddy et al. 2020) and FeiD et al.] 2020). We set the number of local training epoch be 40, and the regularizer parameter μ of FedProx to be 0.001. For FedEED, we set the number of global models to be and used one round of models to build the ensemble, i.e., R = 1, unless otherwise specified. For both FedDF and FedEED, we performed distillation with 5000 steps in each training round, with the temperature T to be 4.
The results shown in Table It can be observed that FedEED outperformed all other methods on the harder dataset, CIFAR 100, and all settings with high degree of Non-IID. It suggests that FedEED a good candidate for tackling the Non-IID issue. We also found that the value R does not affect the performance of FedEED significantly in the CIFAR 10 experiments with low degree of Non-IID, but a higher value of R improved the performance of FedEED significantly in the CIFARIO runs with high degree of Non-IID, and on CIFARI00 datasets with both high and low degree of Non-IID.
6.2 COMPATIBILITY OF FEDEED WITH DIFFERENT LOCAL TRAINERS
To evaluate the compatibility of FedEED with different local trainers, we performed experiments by combining FedEED with FedAvg (McMahan et al. 2017) (default option in this paper). FedProx eL al. 2020). SCAFFOLD (Karimireddy et al.] 2020). The results in Table 3Joo that FedEED is compatible with FedProx and the combination achieves better performance with high degree of Non-III. However, the combination of FedEED and SCAFFOLD did not improve the performance. We expect this is due to the fact that the number clients per global model is only 2 in FedEED which is too small for SCAFFOLD. We thus performed additional experiments by increasing the ratio of active clients from 0.4 to 1.0, which increases the clients per global model to 5. In these experiments, the combination of FedEED and SCCAFOLD achieved much higher accuracy than the stand-alone FedEED and SCCAFFOLD.
6.3 ABLATION STUDY
Ablation study will be provided in the appendix.
CONCLUSION
In this paper, we studied the key components of distillation- based model aggregation in federated learning. By generalizing the existing methods to a universal framework, we investigated the contri- bution of different components. was shown that constructing the ensemble teacher by aggregated models improves the privacy and scalability of the distillationnbasee federated learning system. Fur- thermore, the designated distillation, which transfers the knowledge of the ensemble to one global model, helps maintain the diversity among different models used to build the ensemble. Based on these observations. we propose a novel, efficient ensemble distillation method called FedEED, whose performance advantage the benchmark datasets. The proposed FedEED scheme provides an efficient and scalable way for model aggregation in large-scale learning systems, with improved privacy protection.
existing methods was validated by extensive experiments over

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffrey E Hinton. Large scale distributed neural network training through online distillation. arXiv preprint arXiv:/804.0322 03235, 2018.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191, 2017.
Defang Chen, Jian-Ping Ping Mei, Can Wang. Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3430-3437, 2020.
Hong- You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable feder- ated learning. In ICLR, 2021.
Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204. 12703, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-248228 255. leee, 2009.
Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Ensemble attention distillation for privacy- preeccpies federated learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15076-15086, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http::tp///axe org/ abs/ 1503. 02531
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint
2019.
Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous fed- erated learning. In Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition, pp. 10143-10153, 2022.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mchdi Bennis, and Scong-Lyue Kim. Communicaiion-efiiienn on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv: /rX 1. 1 2018
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sehastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated lcarning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2023
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429-4500 2020.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning, Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effcceent learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Naichen Shi, Fan Lai, Raed Al Kontar, and Mosharaf Chowdhury. Fed-ensemble: Improving gen- eralization through model ensembling in federated learning. arXiv preprint arXiv:2107.1066,, 2021.
Jianyu Wang. Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. field guide feder- ated optimization. arXiv preprint arXiv:2107.669/ 2021.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10174-10183, 2022. 2022.

---

Under review as a conference paper at ICLR 2023
Hoad
Gotal
Mood
Acceguled
Vinia
Van
Aggingtted (2i-
Pogregstid
Aggreggtes (3 (1-m))
10
29
30
10
50
101
111
100
10
101
0
30
70
-0
40
(1)
70
191
90
100
Round of communncattooo
Round cirrrmnniittir
1.0
(b) 0.1
Figure 2: Comparison between different ensemble strategies with CIFAR-10. Note that we applied smoothing to the curves to improve the visibility.
A
APPENDIX
A.1
ADDITIONAL FIGURES FOR COMPARING DIFFERENT ENSEMBLE STRATEGIES
In this section, provide the plots of the experiment in section 4.1 shown in Fig. 2 The settings with 4 global models (i.e. K = 4) consistently outperform the others, where the scheme utilizing global models as an ensemble performs closely as that utilizing client models. Utilizing multiple checkpoints of global models as the ensemble obtained the best result in the experiment.
A.2 FEDEED WITH DIFFERENT COMMUNICATION INTERVALS
Table CIFARIO results with different communication intervals
CIFARIO
rounds local epocls
Model
Method
25 / 160
ResNet20
FodAvg
85.81 0.12
73.17 + 3.06
FedDF
0.12 86.78
77.79
FedEED
86.27 1: 0.68
73.99 1: 2.15
100/40
ResNet20
FedAvg
88.53
78.72 2.37
FedDF
87.98 + 0.16
80.04 + 1.87
FedEED
89.06 1. 0.19
400 10
ResNer20
FedAvg
89.69 + 0.27
79.19 + 0.42
FedDF
FedEFD
90.37 + 0.08
81.34 + 0.81
We compared the performance of FedEED with FedAvg (McMahan et al. 2017 and FedDF Lin et al. 2020), with difference communication intervals. We fixed the total amount of computation, where we set the number rounds and the number of local epochs to be 20/400 and 160/40/10, respectively. We also scaled the number of distillation steps 20000/5000/123 250.. From Table 6 can observe that FedEED always performed better than FedAvg. But with 25 communication rounds, FedDF outperformed FedEED. We expect this is due the extremely small number of clients per global models in FedEED, i.e. there are only two clients per global models, which slowed down the convergence the case with extreme high communication intervals.
A.3 FEDEED WITH DIFFERENT NUMBER OF GLOBAL MODELS
In the main paper, used K = 4 as our default options of FedEED, Here, we provide results with K = to K 4. Note that there are only clients per round in the default configuration, and with more client to the main global model in FedEED. The results in Table / show that K = 4 is the best configuration of FedEED in the experiment. However, mentioned above, there is trade-off between the speed
K there only two clients per global model. For K 3, allocated
12

---

Under review as a conference paper at ICLR 2023
Table 7: CIFARIO results of FedEED with different number of global models.
CIFARIO
Method
ResNer20
FodEED
2
88.34 = 0.32
79.38 + 2.54 79.63
88 0.22
6.15 80.18 + 2.38
89.06 + 0.19
Table 8: CIFARIO results of FedEED with different scaling schemes.
CIFARIO
Clients
Method
ResNer20
FedAvg
0.31 88.53 4: 0.31
78.72 + 2.37
FedDF
FedEFD
89.06 + 0.19
80.18 +: 2.38
14
ResNer20
FedAvg
88.67 0.22
80.25 + 1.64
FeiDF
88.38 1. 0.16
81.61 1: 1.94
89.14
+ 1.90
89.31 + 0.06
81.90 = 1.80
ResNer2D
FedAvg
FedDF
88.12 0.25
81.55 = 2.02
FodEED
89.04 0.11
$2.89 + 1.30
10
89.14 1: 0.11
81.97 1 1.54
of convergence and the performance of the ensemble teacher, so the optimal value of K may vary with the different tasks.
In Table 8 we further provided the results to show how the performance of FedEED scales with the number of clients, and compared with FedAvg (MeMahan et al.) (017) and FedDF (Lin al, 2020). As the number of clients increases,
consider two options: I.) scaling the number of clients per global model, i.c. fixing the value of K, 2.) scaling the number of global models, i.. varying the value of K (K- 7 and 10 in Table 8). We found that FedEED outperformed FedAvg and FedDF with both scaling schemes, and obtained the best result when scaling with the number of global models, except the case with high Non-IID with 20 clients. However, in practice, the scaling scheme may be subject to the server capacity, as increasing the number will also increase the cost of distillation.
A.4 IMPROVEMENT FROM FEDDF
Table 9: The difference between FedDF and FedEED. The method in the last row is identical to the FedEED with R
CIFARIO
Model
Method
ResNer20
FedDF
82.53 +: 0.24
68.72
4.23
Improved configuration
0.47
79.01
2.15
Removal of dmp-woort & early-stopping
87.42 + 0.19
78.55 4: 3.00
+ Aggregated ensemble 4 Designated distillation
88.35 = 0.18
79.59
FedEED is similar to FedDF (Lin et al. 2020). because both of them utilize knowledge distillation for model fusion with an unlabelled dataset. In this section, we progressively modify FedDF into FedEED, and show the effect of each of the individual changes. Note that we keep 10% of the training data from CIFARIO as the validation set of the server in this experiment, following Lin et al. (2020).
The original FedDF utilized Adam optimizer (Kiggma & Ba 2014) for server training, and applied the -toostworst' mechanism and the carly-stopping. We perform the following modification: 1) adapt the configuration from the main paper (including the use of SGD optimizer), 2) remove the of wworst and early-stopping, and 3) utilize four aggregated global models build the
13

---

Under review as a conference paper at ICLR 2023
ensemble with the designated distillation scheme. By introducing these three modifications, FedDF achieved the same perforamnce as FedEED with R 1.
Table shows the results of different modifications and we have three observations. First, the configuration we used in this paper improved FedDF. Sccond, the removal the 'drop-worst" and the early-stopping led to a worse result, but it saved the need of server validation set. Third, the main mechanisms we used in FedEED, i.e. the ensemble from aggregated model and the designated distillation scheme, provided significant performance improvement.
A.5 POSSIBLE EXTENSIONS OF FEDEED
To perform distillation, server dataset or generator is mandatory. In this paper, we consider the use of FedEED with unlabelled server dataset, following Lin et al. (2020); Chen & Chao (202T). Since the dataset is not labelled and can be independent of the client datasets, is cheap to collect. Note that extending FedEED to labelled dataset or generator is straightforward. We can also extend FedEED to the model heterogeneous setting (Lin et al. 2020 Cho et al. 2022), in which we can pick one global model from each model type as the student models. As long as the number of the picked global models is only small portion of all models, the diversity among them can be preserved.
A.6 PRIVACY CONCERN OF FEDEED
Existing distillation based model aggregation methods utilize client models as the teacher for en- hancing a global model. For example, FedDF (Lin et al., 2020) is built upon weight averaging based methods like FedAvg (McMahan et al. 2017).
Weight average based methods can benefit from techniques like secure aggregation (Bonawitz al,) 2017). where individual client updates never disclose to the server and the server can still obtain their weighted sum. However, the above mentioned distillation based methods are not compatible to technique like secure aggregation (Wang et al. (02TT. Because the clients are required to send their models or updates to server, and the server will store the individual client models during the process of server-side training, which introduces extra risk as the communication can be intercepted or the server can be attacked.
On the other hand, FedEED does not require access to the raw client models since it utilizes multiple aggregated models as the teacher, which is compatible to the techniques like secure aggregation, where the individual client model updates or weights are never disclosed to the server. This reduces the chance of being attacked during communication or server storage, and achieve the same privacy level as weight average based methods. To summarise, FedEED is able achieve a higher level of privacy protection than existing distillation based methods.
A.7 PARALLEL SERVER- AND CLIENT-SIDE TRAINING OF FEDEED
As mentioned in the main paper, FedEED can introduce parallelism between server-side and client- side training. In this section, we provide an illustration of the training processes for FedEED and other methods (e.g. FedDF (Lin al. 2020)).
The round time in federated learning depends on the clients' availability. We consider two cases for the clients' availability: 1.) All active clients in a round are able to start local training immediately; 2.) The clients are not always available for training, where we assume that their availability is roughly uniform. The diagram of two training rounds for case 1.) and 2.) are demonstrated in Fig. and Fig. 4 respectively. In these examples, the training time of the clients and the server is assumed constant. We assume simplified case where there are only clients. For other methods, there are only one global model being updated in each round, where FedEED trains 4 models and always maps the i-th model to the i-th client. In FedEED, the training of weight w/t can be started as soon as all w 1 ..., w/ 1 are ready and aggregated, for all non-designated models (i.e. A 0). However, for simplicity we only consider starting the training of W .. w W2k.n-1 after all local weight updates in 1-round have been completed.
In Fig. 3 we demonstrated the parallelism in case I.). Since FedEED can start some training carlier, the computation of different rounds in FedEED are overlapped, which leads to a longer round time.
14

---

Under review as a conference paper at ICLR 2023
However, the average time of FedEED is the same the others, which can be defined as the total rum time of the training divided the total number of rounds, or the time difference between the completion time of two consecutive rounds. In more realistic setting, where clients are not always available to start the training, i.e., case 2.), FedEED has much shorter average round time due to the high degree of parallelism, illustrated in Fig. + In FedEED, clients start their training long as they are available, where in other methods, clients can not start training before the server-side training is completed.
15

---

Under review as a conference paper at ICLR 2023
Cloor
Taaing Pig.
Clard
Pra
Thaning
Wi
Traning of
Theng W7ll0
148 Roand
Roundd
(a) Others (e.g. FedDF)
Clarr1 Trannes
ping of Well Woruaa
Tremnn Winns
Will
Cherr
W/2a
Wilts
Wit
Sune
ofakiing 1i
Trxsing of M22
Trakiig of WT
Acenage
M4h Tound
nound
(b) FedEED
Figure 3: Parallelism comparison between FedEED and other methods, where clients are always available.
Traeire 11
Trunes cl Minsos
Ckorz
Tramnngo Miik
Trarnnoo Winsa
Winu
Trankng
Trining M7
2-th
Hourd
(a) Others (c.g. FcdDF)
Th
Tranergor
(Bor.1
Tranngg W/133
Tru oegel Wah.ia
Thening ef Witss
CBem22
Thilkin W2200
Trakirn Wiesss
Traniron IV Wo
Wi
Seno
Training of tr
Traning of Moa"
Training (i
Avenage sound
14 Hound
E+ 14t Hound
(b) FedEED
Figure 4: Parallelism comparison between FedEED and other methods, where clients are not always available.
16


Reviews:
[{"review_id": "mI57PwnEvkt", "reviewer": "Reviewer_Ybra", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a generalized federated distillation framework which divides local nodes into groups: the aggregation is first inside each group and then from groups into the final global model.  The advantage is groups can do each aggregation in parallel in the first stage.\nThe experiments exhaust several popular federated distillation method for the aggregation. The results show the proposed group-division ensemble strategy does improve over the ensemble directly from all locals.   ", "summary_of_the_review": "Although the experiments show better performance over the baseline methods, the novelty is very limited. And the efficiency metrics are not clear. I also doubt the claims regarding privacy and scalability.   Since the \"efficiency\" is the main point of the paper, I hope the authors could provide more efficiency analysis with quantitative results on latency, communication bandwidth. ", "strength_and_weaknesses": "Strength: \n1. The proposed group-division ensemble framework can be generalized to different existing FL methods. First, aggregation inside individual group helps to improve the level of parallel.  \n2. Experiments show the proposed methods achieves better performance than directly aggregating all locals.\n\nWeakness:\n1. Limited novelty: the authors only replace the typical aggregation in FL with two stages: group aggregation and then global aggregation. The detailed ensemble strategies (e.g. bayesian ensemble) are borrowed from existing works. The novelty and contribution is limited considering the method is just simple combination.  \n2. Overclaim about scalability: I cannot see any improvement on scalability from the proposed method and experiments design.\n3. Overclaim about privacy: In abstract it says \"FedEED also achieves higher level of privacy protection, because the access to client models is no longer required.\" But the method still iteratively upload local model parameters besides the group global model. [1][2] demonstrated that local private data could be fully recovered from publicly-shared gradients (iteratively shared local model parameters). \n3. Overclaim about efficiency: No quantitative analysis of efficiency metrics such as latency, communication bandwidth. Considering the \"ensemble trainer\" also involves the participation of local models, I doubt if the latency and communication bandwidth could be even higher than the existing FL methods for comparison. It is meaningless to show #rounds in the table, as the latency and bandwidth of each round in your method is obviously higher than other FL methods. The communication cost in Table3 is very ambiguous without any analysis or calculation.\n4. Experiments: only have experiments on cifar10/cifar100. Should do more through experiments under typical FL setting: such as EMNIST in SCAFFOLD or NLP tasks (AG News, SST2 in FedDF). \n\n[1] Deep leakage from gradients.\n[2] Inverting gradients–how easy is it to break privacy in federated learning?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "l-EAbfDInzg", "reviewer": "Reviewer_Cg3f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates applying knowledge distillation to federated learning. It summarizes many related works to give a set of common building blocks and experiments with different choices for each block on ResNet20/56 + Cifar10/100 to see their impact on the accuracy.", "summary_of_the_review": "Although it is a good effort to summarize related works and experiment with multiple design choices, the technical novelty is somewhat limited. Moreover, this paper needs to be revised to make it easier to understand and better explain and support the claims.", "strength_and_weaknesses": "Strength:\n- There is a good effort to summarize the common building blocks across many knowledge-distillation-based FL algorithms. This is helpful for readers to get a high-level overview of algorithms in this class.\n- This paper also provides a few popular methods that can be used for each block and performs experiments to compare these methods. The results may be useful when readers try to adapt an existing method to their applications or systems.\n\nWeakness:\n- The main issue of this paper is that it is difficult to read and understand. The paper is flooded with undefined terms, symbols, and confusing statements. Also, there is no figure to serve illustration. For example, the paragraph in Sec. 3 states that \"In every training round, each of the local trainers will distribute one of the global models to a subset of clients\". For FL, the server is typically responsible for distributing models. Does this sentence mean one client will choose a model from the global models and distribute it to other clients, involving something like client-to-client communication? Why does a client store global models in addition to their local models? Another example is that the terms like \"global model\", \"ensemble model\", \"local trainer\", \"ensemble trainer\", and \"global trainer\" are not defined, and the symbols in the pseudo code are not defined either.\n- There are many unsupported or unexplained claims. For example, Sec. 4.1 has the sentence \"Existing distillation-based aggregation methods require all client models to build the ensemble, causing privacy concerns.\". Why does this approach cause privacy concerns and why doesn't the proposed approach are not explained nor supported by references. Another example is also from Sec. 4.1. \"We noticed the idea of Fed-ensemble, which builds the ensemble from multiple global models for inference purposes. This solves the privacy and scalability issues.\". Again, there is no explanation about why this solves the privacy and scalability issues.\n- Most of the experiments are on the small CIFAR10 dataset plus the small ResNet20 model except for Table 4, which contains ResNet56 and Cifar100. Because the authors never mention what the metric for the accuracy is, I assume it is top-5 accuracy. ResNet56 on CIFAR100 can easily get a top-5 accuracy above 90%, but the numbers in Table 4 are at the level of 60%. Therefore, I am skeptical about whether the conclusions drawn from these experiments can be generalized to practical settings.\n- In Sec. 6.2, there is only one sentence \"Ablation study will be provided in the appendix.\". The ablation study is an important part of the main paper. Personally, I don't think moving it to the appendix to save space is a good practice. I leave it to the chair to make the judgment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cEr10A0zC61", "reviewer": "Reviewer_4iWR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors studied the key components of distillation-based model aggregation in FL. Different from existing approaches, the ensemble teacher of FedEED is constructed by aggregated models, instead of client models, to achieve improved scalability in large-scale systems.", "summary_of_the_review": "1.  The authors should add experiments to further validate the performance of the proposed method on Non-IID feature tasks to further evaluate the generality of the proposed method.\n2. The authors' presentation is not clear and the architecture of the algorithm cannot be well understood. Perhaps the authors can give a clear architecture diagram to assist the readers to understand the essence of the proposed method clearly.\n3. Several general FL methods have been proposed and achieved excellent performance, such as FedBN, FedRep, and FedDC, and the authors should analyze the performance of more SOTA FL methods to further evaluate the proposed approach.\n\na) Li X, Jiang M, Zhang X, et al. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.\nb) Collins L, Hassani H, Mokhtari A, et al. Exploiting shared representations for personalized federated learning. International Conference on Machine Learning. PMLR, 2021: 2089-2099.\nc) Gao L, Fu H, Li L, et al. FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10112-10121.", "strength_and_weaknesses": "Strength:\nDue to the use of aggregated models, FedEED also achieves a higher level of privacy protection. In addition, the knowledge distillation in FedEED only happens from the ensemble teacher to a designated model such that the diversity among different aggregated models is maintained.\n\nWeaknesses:\nThe authors evaluated the performance of the proposed method only on the task of Non-IID labels and did not validate the performance of the proposed method on the Non-IID feature task, which is a more realistic scenario because there is often a large domain gap problem in the distribution of data features of different clients as the different data acquisition devices, environments, and qualities. Therefore it does not provide a comprehensive assessment of the generality of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FMFL8Ltx4KX", "reviewer": "Reviewer_KBtv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first propose a generalized distillation framework. Based on the study of each component in the generalized framework, this paper proposes Federated Efficient Ensemble Distillation(FedEED), which is efficient, scalable and privacy.", "summary_of_the_review": "This paper presents an innovative research of ederated distillation, But I have doubts about the scalability and privacy of the proposed framework FedEED.", "strength_and_weaknesses": "Strengths:\n\n1. This paper attempts to achieve generalizability of federated distillation, which is a very innovative study.\n\n2. The application of knowledge distillation in the field of federated learning is well researched and summarized.\n\n3. A number of experiments are done using several federated distillation algorithms.\n\nWeaknesses:\n\n1. In abstract\n\t- This paper first proposes a generalized distillation framework which divides the federated distillation process into three key stages. However, there are various types of federated distillation, and the three-stage framework is only one of them, so it should not be called a generalized distillation framework. \n\t- FedEED also achieves a higher level of privacy protection, because the access to client models is no longer required. However, The client model still needs to be transferred between local trainer and  ensemble trainer, which means the client model can still be attacked.\n\n2. In framework\n\t- This paper describes in detail the three-stage process of the generalized federated distillation framework, wouldn't it be better to add a framework figure to illustrate it more visually? \n\t- This framework includes K local trainers, an ensemble trainer, and a global trainer, but traditional federated learning includes K clients and a server, what is the correspondence between them, this paper does not explain.\n\n3. In experiments\n\t- This paper mentions that the number of clients is 20, with 40% of activated clients, which means that there are very few clients, which is contrary to this paper's emphasis on the fact that FedEED achieves improved scalability in large-scale systems. Therefore, experiments on more clients should be conducted to demonstrate the scalability of FedEED.\n\t- The correspondence between Figure 1 and Table 1 is not clear, and the introduction part is also a bit messy, which is easy to cause confusion. If you follow the order in Figure 1, the order in Table 1 would be “Global model(K=1)、Ensemble(K=1,Clients)、Ensemble(K=1,Clients,Weighted)、Ensemble(K=1,Bayesian,Gaussian)、Ensemble(K=1,Bayesian,Dirichlet)、Global model(K=4)、Ensemble(K=4,Clients)、Ensemble(K=4,Aggregated)、Ensemble(K=4,R=2,Aggregated)、Ensemble(K=4,R=4,Aggregated)”.\n\t- The dataset CIFAR100 is included in the description of Table 2, but it is not available in the experimental results. In addition, only Table 4 conducted experiments on CIFAR100, why not include this dataset in other experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject