Sself: Robust Federated Learning against Stragglers and Adversaries

1 INTRODUCTION . Large volumes of data collected at various edge devices ( i.e. , smart phones ) are valuable resources in training machine learning models with a good accuracy . Federated learning ( McMahan et al. , 2017 ; Li et al. , 2019a ; b ; Konečnỳ et al. , 2016 ) is a promising direction for large-scale learning , which enables training of a shared global model with less privacy concerns . However , current federated learning systems suffer from two major issues . First is the devices called stragglers that are considerably slower than the average , and the second is the adversaries that enforce various adversarial attacks . Regarding the first issue , waiting for all the stragglers at each global round can significantly slow down the overall training process in a synchronous setup . To address this , an asynchronous federated learning scheme was proposed in ( Xie et al. , 2019a ) where the global model is updated every time the server receives a local model from each device , in the order of arrivals ; the global model is updated asynchronously based on the device ’ s staleness t− τ , the difference between the current round t and the previous round τ at which the device received the global model from the server . However , among the received results at each global round , a significant portion of the results with large staleness does not help the global model in a meaningful way , potentially making the scheme ineffective . Moreover , since the model update is performed one-by-one asynchronously , the scheme in ( Xie et al. , 2019a ) would be vulnerable to various adversarial attacks ; any attempt to combine this type of asynchronous scheme with existing adversary-resilient ideas would not likely be fruitful . There are different forms of adversarial attacks that significantly degrade the performance of current federated learning systems . First , in untargeted attacks , an attacker can poison the updated model at the devices before it is sent to the server ( model update poisoning ) ( Blanchard et al. , 2017 ; Lamport et al. , 2019 ) or can poison the datasets of each device ( data poisoning ) ( Biggio et al. , 2012 ; Liu et al. , 2017 ) , which degrades the accuracy of the model . In targeted attacks ( or backdoor attacks ) ( Chen et al. , 2017a ; Bagdasaryan et al. , 2018 ; Sun et al. , 2019 ) , the adversaries cause the model to misclassify the targeted subtasks only , while not degrading the overall test accuracy . To resolve these issues , a robust federated averaging ( RFA ) scheme was recently proposed in ( Pillutla et al. , 2019 ) which utilizes the geometric median of the received results for aggregation . However , RFA tends to lose performance rapidly as the portion of adversaries exceeds a certain threshold . In this sense , RFA is not an ideal candidate to be combined with known straggler-mitigating strategies ( e.g. , ignoring stragglers ) where a relatively small number of devices are utilized for global aggregation ; the attack ratio can be very high , significantly degrading the performance . To our knowledge , there are currently no existing methods or known combinations of ideas that can effectively handle both stragglers and adversaries at the same time , an issue that is becoming increasingly important in practical scenarios . Contributions . In this paper , we propose Sself , semi-synchronous entropy and loss based filtering/averaging , a robust federated learning strategy which can tackle both stragglers and adversaries simultaneously . In the proposed idea , the straggler effects are mitigated by semi-synchronous global aggregation at the server , and in each aggregation step , the impact of adversaries are countered by a new aggregation method utilizing public data collected at the server . The details of our key ideas are as follows . Targeting the straggler issue , our strategy is to perform periodic global aggregation while allowing the results sent from stragglers to be aggregated in later rounds . The key strategy is a judicious mix of both synchronous and asynchronous approaches . At each round , as a first step , we aggregate the results that come from the same initial models ( i.e. , same staleness ) , as in the synchronous scheme . Then , we take the weighted sum of these aggregated results with different staleness , i.e. , coming from different initial models , as in the asynchronous approach . Regarding the adversarial attacks , robust aggregation is realized via entropy-based filtering and loss-weighted averaging . This can be employed at the first step of our semi-synchronous strategy described above , enabling protection against model/data poisoning and backdoor attacks . To this end , our key idea is to utilize public IID ( independent , identically distributed ) data collected at the server . We can imagine a practical scenario where the server has some global data uniformly distributed over classes , as in the setup of ( Zhao et al. , 2018 ) . This is generally a reasonable setup since data centers mostly have some collected data ( although they can be only a few ) of the learning task . For example , different types of medical data are often open to public in various countries . Based on the public data , the server computes entropy and loss of each received model . We use the entropy of each model to filter out the devices whose models are poisoned . In addition , by taking the loss-weighted averaging of the survived models , we can protect the system against local data poisoning and backdoor attacks . We derive a theoretical bound for Sself to ensure acceptable convergence behavior . Experimental results on different datasets show that Sself outperforms various combinations of straggler/adversary defense methods with only a small portion of public data at the server . Related works . The authors of ( Li et al. , 2019c ; Wu et al. , 2019 ; Xie et al. , 2019a ) have recently tackled the straggler issue in a federated learning setup . The basic idea is to allow the devices and the server to update the models asynchronously . Especially in ( Xie et al. , 2019a ) , the authors proposed an asynchronous scheme where the global model is updated every time the server receives a local model of each device . However , a fair portion of the received models with large staleness does not help the global model in meaningful ways , potentially slowing down the convergence speed . A more critical issue here is that robust methods designed to handle adversarial attacks , such as RFA ( Pillutla et al. , 2019 ) , Multi-Krum ( Blanchard et al. , 2017 ) or the presently proposed entropy/loss based idea , are hard to be implemented in conjunction with this asynchronous scheme . To combat adversaries , various aggregation methods have been proposed in a distributed learning setup with IID data across nodes ( Yin et al. , 2018a ; b ; Chen et al. , 2017b ; Blanchard et al. , 2017 ; Xie et al. , 2018 ) . The authors of ( Chen et al. , 2017b ) suggests a geometric median based aggregation rule of the received models or the gradients . In ( Yin et al. , 2018a ) , a trimmed mean approach is proposed which removes a fraction of largest and smallest values of each element among the received results . In Multi-Krum ( Blanchard et al. , 2017 ) , among N workers in the system , the server tolerates f Byzantine workers under the assumption of 2f + 2 < N . Targeting federated learning with non-IID data , the recently introduced RFA method of ( Pillutla et al. , 2019 ) utilizes the geometric median of models sent from devices , similar to ( Chen et al. , 2017b ) . However , as mentioned above , these methods are ineffective when combined with a straggler-mitigation scheme , potentially degrading the performance of learning . Compared to Multi-Krum and RFA , our entropy/loss based scheme can tolerate adversaries even with a high attack ratio , showing remarkable advantages , especially when combined with straggler-mitigation schemes . Finally , we note that the authors of ( Xie et al. , 2019c ) considered both stragglers and adversaries but in a distributed learning setup with IID data across the nodes . Compared to these works , we target non-IID data distribution setup in a federated learning scenario . 2 PROPOSED FEDERATED LEARNING WITH SSELF . We consider the following federated optimization problem : w∗ = argmin w F ( w ) = argmin w N∑ k=1 mk m Fk ( w ) , ( 1 ) whereN is the number of devices , mk is the number of data samples in device k , andm = ∑N k=1mk is the total number of data samples of all N devices in the system . By letting xk , j be the jth data sample in device k , the local loss function of device k , Fk ( w ) , is written as Fk ( w ) = 1 mk ∑mk j=1 ` ( w ; xk , j ) . In the following , we provide solutions aiming to solve the above problem under the existence of stragglers ( subsection 2.1 ) and adversaries ( subsection 2.2 ) , and finally propose Sself handling both issues ( subsection 2.3 ) . 2.1 SEMI-SYNCHRONOUS SCHEME AGAINST STRAGGLERS . In the t-th global round , the server sends the current model wt to K devices in St ( |St| = K ≤ N ) , which is a set of indices randomly selected from N devices in the system . We let C = K/N be the ratio of devices that participate at each global round . Each device in St performs E local updates with its own data and sends the updated model back to the server . In conventional federated averaging ( FedAvg ) , the server waits until the results of allK devices in St arrive and then performs aggregation to obtain wt+1 = ∑ k∈St mk∑ k∈St mk wt ( k ) , where wt ( k ) is the model after E local updates at device k starting from wt . However , due to the effect of stragglers , waiting for all K devices at the server can significantly slow down the overall training process . In resolving this issue , our idea assumes periodic global aggregation at the server . At each global round t , the server transmits the current model/round ( wt , t ) to the devices in St . Instead of waiting for all devices in St , the server aggregates the models that arrive until a fixed time deadline Td to obtain wt+1 , and moves on to the next global round t+ 1 . Hence , model aggregation is performed periodically with every Td . A key feature here is that we do not ignore the results sent from stragglers ( not arrived by the deadline Td ) . These results are utilized at the next global aggregation step , or even later , depending on the delay or staleness . Let U ( t ) i be the set of devices 1 ) that are selected from the server at global round t , i.e. , U ( t ) i ⊆ St and 2 ) that successfully sent their results to the server at global round i for i ≥ t. Then , we can write St = ∪∞i=tU ( t ) i , where U ( t ) i ∩ U ( t ) j = ∅ for i 6= j . Here , U ( t ) ∞ can be viewed as the devices that are selected at round t but failed to successfully send their results back to the server . According to these notations , the devices whose training results arrive at the server during global round t belong to one of the following t+ 1 sets : U ( 0 ) t , U ( 1 ) t , ... , U ( t ) t . Note that the result sent from device k ∈ U ( i ) t is the model after E local updates starting from wi , and we denote this model by wi ( k ) . At each round t , we first perform FedAvg as v ( i ) t+1 = ∑ k∈U ( i ) t mk∑ k∈U ( i ) t mk wi ( k ) ( 2 ) for all i = 0 , 1 , ... , t , where v ( i ) t+1 is the aggregated result of locally updated models ( starting from wi ) received at round t with staleness t− i+ 1 . Then from v ( 0 ) t+1 , v ( 1 ) t+1 , ... , v ( t ) t+1 , we take the weighted averaging of results with different staleness to obtain ∑t i=0 αt ( i ) v ( i ) t+1 . Here , αt ( i ) ∝ ∑ k∈U ( i ) t mk ( t−i+1 ) c is a normalized coefficient that is proportional to the number of data samples in U ( i ) t and inversely proportional to ( t− i+ 1 ) c , for a given hyperparameter c ≥ 0 . Hence , we have a larger weight for v ( i ) t+1 with a smaller t− i+ 1 ( staleness ) . This is to give more weights to more recent results . Based on the weighted sum ∑t i=0 αt ( i ) v ( i ) t+1 , we finally obtain wt+1 as wt+1 = ( 1− γ ) wt + γ t∑ i=0 αt ( i ) v ( i ) t+1 , ( 3 ) where γ combines the aggregated result with the latest global model wt . Now we move on to the next round t + 1 , where the server selects St+1 and sends ( wt+1 , t+ 1 ) to these devices . Here , if the server knows the set of active devices ( which are still performing computation ) , St+1 can be 𝒰 𝒰 Received at global round t-1 Received at global round t … constructed to be disjoint with the active devices . If not , the server randomly chooses St+1 among all devices in the system and the selected active devices can ignore the current request of the server . The left-hand side of Fig . 1 describes our semi-synchronous scheme . The key characteristics of our scheme can be summarized as follows . First , by periodic global aggregation at the server , our scheme is not delayed by the effect of stragglers . Secondly , our scheme fully utilizes the results s nt from stragglers in the future global rounds ; we first perform federated averaging for the devices with same staleness ( as in the synchronous scheme ) , and then take the weighted sum of these averaged results with different staleness ( as in the asynchronous scheme ) .

Summary: This paper considers federated learning with straggling and adversarial devices. To tackle stragglers, the paper proposes semi-synchronous averaging wherein models with the same staleness are first averaged together, and then a weighted average of the results with different stateless is computed. To mitigate adversaries, the paper proposes to first perform entropy-based filtering to remove suspected outliers, and then compute loss-weighted average. The server is assumed to have some public data, which is used for entropy-based filtering. Together, the proposed algorithm is called semi-synchronous entropy and loss based filtering (Sself).