FedMes: Speeding Up Federated Learning with Multiple Edge Servers

1 INTRODUCTION . With the explosive growth in the numbers of smart phones , wearable devices and Internet of Things ( IoT ) sensors , a large portion of data generated nowadays is collected outside the cloud , especially at the distributed end-devices at the edge . Federated learning ( McMahan et al. , 2017 ; Konecny et al. , 2016b ; a ; Bonawitz et al. , 2019 ; Li et al. , 2019a ) is a recent paradigm for this setup , which enables training of a machine learning model in a distributed network while significantly resolving privacy concerns of the individual devices . However , training requires repeated downloading and uploading of the models between the parameter server ( PS ) and devices , presenting significant challenges in terms of 1 ) the communication bottleneck at the PS and 2 ) the nonIID ( independent , identically distributed ) data characteristic across devices ( Zhao et al. , 2018 ; Sattler et al. , 2019 ; Li et al. , 2019b ; Reisizadeh et al. , 2019 ; Jeong et al. , 2018 ) . In federated learning , the PS can be located at the cloud or at the edge ( e.g. , small base stations ) . Most current studies on federated learning consider the former , with the assumption that millions of devices are within the coverage of the PS at the cloud ; at every global round , the devices in the system should communicate with the PS ( located at the cloud ) for downloading and uploading the models . However , an inherent limitation of this cloud-based system is the long distance between the device and the cloud server , which causes significant propagation delay during model downloading/uploading stages in federated learning ( Mao et al. , 2017 ; Nguyen et al. , 2019 ) . Specifically , it is reported in ( Mao et al. , 2017 ) that the supportable latency ( for inference ) of cloud-based systems is larger than 100 milliseconds , while the edge-based systems have supportable latency of less than tens of milliseconds . This large delay between the cloud and the devices directly affects the training time of cloud-based federated learning systems . In order to support latency-sensitive applications ( e.g. , smart cars ) or emergency events ( e.g. , disaster response by drones ) by federated learning , utilization of edge-based system is absolutely necessary . An issue , however , is that although the edge-based federated learning system can considerably reduce the latency between the PS and the devices , the coverage of an edge server is generally limited in practical systems ( e.g. , wireless cellular networks ) ; there are insufficient number of devices within the coverage of an edge server for training a global model with enough accuracy . Accordingly , the limited coverage of a single edge server could include biased datasets and thus could lead to a biased model after training . Thus in practice , performing federated learning with the devices in a single edge server would result in a significant performance degradation . Main contributions . To overcome the above practical challenges , we propose FedMes , a novel federated learning algorithm highly tailored to the environment with multiple edge servers ( ESs ) . Our idea here is to utilize the devices located in the overlapping areas between the coverage of ESs , which are typical in 5G and beyond systems with dense deployment of ESs . In the model-downloading stage , each ES sends the current model to the devices in its coverage area ; in this process the devices in the overlapped region receive multiple models from different ESs . These devices in the overlapping area take the average of the received models , and then update the model based on their local data . Then each device sends its updated model to the corresponding ES or ESs , which is aggregated at each ES . A high-level description of FedMes is given in Fig . 1 . For example , suppose that device k is located in the non-overlapped region of ES i while device l is in the overlapped region between ES i and ES j . In conventional federated learning systems , device l participates in the training process of only one of ES i or ES j ; on the other hand , in FedMes , device l can act as a bridge for sharing the trained models between both ESs . To be specific , the updated model of device k is averaged only at its associated ES i . In the next step , this averaged model is sent to the devices in its covered area , including device l. After the local model updates at the devices , device l sends its updated model to both ES i and ES j . From this point of view , even when some training samples are only in the coverage of a specific ES , these data can still assist the training process of other servers . Hence , the proposed scheme does not require costly communications with the central cloud server ( located at the higher tier of ESs ) for model synchronization , significantly reducing the overall training time compared to cloud-based federated learning systems . Comparing with the scheme which does not consider the overlapping areas , FedMes can provide a significant performance gain , especially when the data distributions across coverages of different servers are nonIID , e.g. , when a specific server has a biased dataset within its covered area . Especially in this nonIID setup , giving more weights to the devices located in the overlapping areas ( in each aggregation step at the ESs ) can further speed up training . From the service provider point of view , FedMes does not require any backhaul traffic between the ESs and the cloud server , significantly reducing the communication resources required for federated learning . Moreover , since the devices in the overlapping areas send their results to multiples ESs , our scheme can reduce the number of devices participating at each global round while achieving the desired performance . Extensive experimental results on various datasets show that FedMes provides remarkable performance gain compared to 1 ) the scheme that requires communications with the central cloud server for model synchronization ( i.e. , cloud-based federated learning ) and 2 ) the scheme that does not take the overlapping areas between servers into account . Related works . Thanks to the recent advent of edge computing , there has been an increased interest in edge-facilitated federated learning systems ( Tran et al. , 2019 ; Wang et al. , 2019 ; Lim et al. , 2020 ; Abad et al. , 2020 ; Liu et al. , 2019 ) . The authors of ( Wang et al. , 2019 ) focused on optimizing federated learning framework with a given resource budget in wireless edge networks . The authors of ( Tran et al. , 2019 ) considered resource allocation to minimize energy consumption at the devices in wireless networks . However , a single-server setup is considered in ( Tran et al. , 2019 ) and ( Wang et al. , 2019 ) , which is totally different from our work leveraging multiple edge servers . Only a few prior works on federated learning ( Abad et al. , 2020 ; Liu et al. , 2019 ) considered a setup with multiple edge servers . However , the schemes of ( Abad et al. , 2020 ; Liu et al. , 2019 ) still require costly communication with the central cloud server for model synchronization , which could significantly slow down the overall training process . If the communication period with the cloud is small , frequent model synchronization between edge servers is possible but incurs a large communication time delay . Infrequent model synchronization can lead to a bad performance especially with nonIID data across the servers . FedMes overcomes these challenges by enabling fast federated learning without any help with the cloud server , i.e. , only using the edge servers . It is shown later in Section 4 that our scheme can outperform the hierarchical scheme of ( Liu et al. , 2019 ) which requires costly communications with the central cloud server . 2 PROBLEM SETUP . Federated learning . Let K be the number of devices in the system . Let nk be the number of data samples in device k , with n = ∑K k=1 nk being the total number of training samples . We also denote the i-th sample in device k as xk , i , for i ∈ { 1 , 2 , ... , nk } . Our goal is to solve the following optimization problem min w F ( w ) = min w K∑ k=1 nk n Fk ( w ) , ( 1 ) where Fk ( w ) is the local loss function of data samples in device k , written as Fk ( w ) = 1 nk ∑nk i=1 ` ( xk , i ; w ) . Now we briefly describe the conventional cloud-based federated averaging ( FedAvg ) algorithm in ( McMahan et al. , 2017 ) , a typical way to solve this problem . At step t , each device downloads the current model w ( t ) from the PS , which is generally located at the cloud covering all the devices in the system . Then each device ( say device k ) sets wk ( t ) = w ( t ) and performs E local updates according to wk ( t+ i+ 1 ) = wk ( t+ i ) − ηt+i∇Fk ( wk ( t+ i ) , ξk ( t+ i ) ) , i = 0 , 1 , ... , E − 1 , ( 2 ) where ηt is the learning rate at step t and ξk ( t ) is a set of randomly selected data samples from device k at step t. Now each device sends the updated model to the PS , and the PS aggregates the model as w ( t+ E ) = ∑K k=1 nk n wk ( t+ E ) . However , full device participation at each aggregation step is impossible in practice and the PS often selects a set St+E ⊂ { 1 , 2 , ... , K } , containing the devices that transmit the results to the PS . Then , we have w ( t+ E ) = ∑ k∈St+E nk n wk ( t+ E ) . ( 3 ) This overall process is repeated until the model achieves the desired accuracy or some stopping condition is met . According to the algorithm , the model of the k-th device at step t+ 1 is written as wk ( t+ 1 ) = wk ( t ) − ηt∇Fk ( wk ( t ) , ξk ( t ) ) , if E - t+ 1∑ q∈St+1 nq n [ wq ( t ) − ηt∇Fq ( wq ( t ) , ξq ( t ) ) ] , otherwise . ( 4 ) Problem formulation . In contrast with the conventional cloud-based federated learning systems having a central cloud server covering the whole devices , in this paper we consider L ESs each covering its own local area . We call this local coverage of each edge server a cell . Especially with dense deployment of ESs in 5G and beyond networks , there generally exist more than one ES within the range of a specific user that can be reliably communicate with . We call this region in which the device can reliably communicate with multiple ESs overlapping cell area . Let Ci be the set of indices for users located in cell i ∈ { 1 , 2 , ... , L } . Now define Ui as the set of user indices for the non-overlapped region of cell i , which is the subset of Ci . We also define Vi , j as the set of user indices for the overlapping area between cell i and cell j ( i 6= j and Vi , j = Vj , i ) , which is also the subset of Ci . Here , the devices in Vi , j can communicate with both ES i and ES j during model download or upload . While we can similarly define overlapped regions with more than two ESs , we consider the case in which the coverage of at most two ESs overlapped for clarity of presentation . Then the coverage of cell i , i.e. , Ci can be written as Ci = Ui ∪ ( ∪ j∈ [ L ] \ { i } Vi , j ) ( 5 ) for all i ∈ { 1 , 2 , ... , L } . The overall coverage of the system can be written as C = { 1 , 2 , ... , K } = ∪Li=1 Ui ∪ ( ∪Li=1 ∪Lj=i+1 Vi , j ) . Note that each ES can communicate with the devices in its covered area only . Our goal is to solve the problem in ( 1 ) in this setup , without sharing the learning models at the higher tier of ESs ( i.e. , the cloud server ) during training .

Summary: In previous federated learning literature, people usually assume there is only one cloud server communicating with all edge nodes/clients. However, since each server has its own coverage in practice, the latency between the server and clients out of the coverage can be pretty long. This paper focuses on reducing the communication cost in this practical setting. The authors propose to use multiple edge servers, which have overlapped coverages. The clients in the overlapping areas will receive model parameters from multiple server and return the average model back. This can help to mix the information between different edge servers. Experiments on MNIST, EMNIST, and CIFAR10 datasets validate the effectiveness of the proposed algorithm: FedMes.