Sageflow: Robust Federated Learning against Both Stragglers and Adversaries

1 Introduction . Large volumes of data collected at various edge devices ( i.e. , smart phones ) are valuable resources in training a model with a good performance . Federated learning [ 18 , 12 , 7 ] is a promising direction for large-scale learning , which enables training of a global model with less privacy concerns . However , among major issues that need to be addressed in current federated learning ( FL ) systems are the devices called stragglers that are considerably slower than the average and the adversaries that enforce a various form of attacks . Regarding the first issue , simply waiting for all the stragglers at each global round can significantly slow down the overall training process . To handle stragglers , asynchronous FL schemes [ 15 , 22 , 25 , 21 , 14 ] update the global model every time the server receives a local model from each device ; especially in FedAsync [ 25 ] , the global model is updated asynchronously according to the device ’ s staleness t− τ , the time difference between the current round t and the past round τ when the device first received the global model . While the asynchronous schemes are highly effective in handling stragglers , the one-by-one update nature does not lend itself well for integration with established aggregation methods to combat the second issue , the adversaries . There are different forms of adversarial attacks that significantly degrade current FL systems . In untargeted attacks , an attacker can poison the updated model at the devices before it is sent to the ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . server ( model poisoning ) [ 3 , 9 ] or can poison the datasets of each device ( data poisoning ) [ 2 , 16 ] . In targeted attacks ( or backdoor attacks ) [ 4 , 1 , 20 ] , the adversaries cause the model to misclassify the targeted subtasks only , while not degrading the overall test accuracy that much . Robust federated averaging ( RFA ) of [ 19 ] , a well-known method proposed to handle adversaries in FL , employs geometric-median-based aggregation to provide a fair level of protection . Other various aggregation schemes ( e.g. , Multi-Krum ) also successfully handle adversaries in distributed learning [ 3 , 28 , 5 ] . Unfortunately , however , the performance of these methods are substantially degraded when the portion of adversaries is large . The presence of stragglers can drive the attack ratio higher ( e.g. , by ignoring stragglers ) , significantly degrading the performance of current aggregation schemes . While the presence of both stragglers and adversaries raises significant concerns in practical FL , to our knowledge , there are currently no existing methods or known combinations of ideas that can effectively handle these two issues simultaneously . Main contributions . We propose Sageflow , staleness-aware grouping with entropy-based filtering and loss-weighted averaging , a robust FL strategy which can handle both stragglers and adversaries at the same time . Targeting the straggler issue , our strategy is to perform periodic global aggregation while allowing the results sent from stragglers to be aggregated in later rounds . In each global round , we take advantage of the results sent from stragglers by first grouping the models that come from the same initial models ( i.e. , same staleness ) , to obtain a group representative model . Then , we aggregate the representative models of all groups based on their staleness , to obtain the global model . Our periodic aggregation strategy is not only effective in neutralizing stragglers but also provides a great platform for countering adversaries , as discussed below . Targeting each grouping stage of our straggler-mitigating idea , we propose an intra-group defense strategy which is based on our entropy-based filtering and loss-weighted averaging . The entropy filtering first filters out the models with high entropies , i.e. , outliers , and the loss-weighted averaging of the survived models enables model aggregation according to the measured qualities of the received local models . These two methods work in a highly complementary fashion to effectively counter a wide range of adversarial attacks in each grouping stage . Here , in computing the entropy and loss of each received model , we utilize public data that we assume to be available at the server . In fact , the utilization of public data is not a new idea , as seen in recent FL setups of [ 30 , 27 , 11 ] . This is generally a reasonable setup since data centers typically have some collected data that can be accessed by public . For example , different types of anonymous medical data are often available for public research in various countries . We show later via experiments that only a very small amount of public data is necessary at the server ( 1-2 % of the entire dataset , which is comparable to the amount of local data at a single device ) to successfully combat adversaries . Our main contributions are as follows : • We propose Sageflow , handling both stragglers/adversaries simultaneously in FL , via a novel staleness-aware grouping combined with entropy filtering and loss-weighted averaging . • We derive the theoretical bound for Sageflow based on key parameters and provide insights into the convergence behavior . • Experimental results on different datasets show that Sageflow outperforms various combinations of straggler/adversary defense methods using only a small portion of public data . Related works . The authors of [ 15 , 22 , 25 , 17 , 21 ] have recently tackled the straggler issue in FL . The basic idea is to allow the devices and the server to update the models asynchronously ; the global model is updated every time the server receives a model from a device . However , a critical issue is that grouping-based ( or aggregation-based ) methods designed to handle adversaries , such as RFA [ 19 ] , Multi-Krum [ 3 ] or the presently proposed entropy/loss based idea , are hard to be implemented in conjunction with these schemes since the model update is performed one-by-one asynchronously . Compared to the asynchronous schemes , our staleness-aware grouping can be combined smoothly with various aggregation rules against adversaries ; we can apply RFA , Multi-Krum or our entropy/loss based idea in each grouping stage to obtain the group representative model . To combat adversaries , various aggregation methods have been proposed in a distributed learning setup with IID ( independent , identically distributed ) data across nodes [ 28 , 29 , 5 , 3 , 24 ] . The authors of [ 5 ] suggests a geometric-median-based aggregation rule of models or gradients . In [ 28 ] , a trimmed mean approach is proposed which removes a fraction of largest and smallest values of each element among the received results . In Multi-Krum [ 3 ] , among N workers in the system , the server tolerates f Byzantine workers , where 2f + 2 < N . Targeting FL with non-IID data distribution , RFA of [ 19 ] utilizes the geometric median of models sent from devices , similar to [ 5 ] . However , as already implied , these methods are ineffective when combined with a straggler-mitigation scheme ( e.g. , ignoring stragglers ) , potentially degrading the performance of learning . Compared to MultiKrum and RFA , our entropy/loss based scheme can tolerate adversaries even with a high attack ratio , showing remarkable advantages , especially when combined with straggler-mitigation schemes . Moreover , our approach enables defense against a wider variety of attacks by considering both entropy and loss , which are shown to play highly complementary roles . Finally , we note that the authors of [ 26 ] , [ 27 ] proposed Zeno and Zeno+ that also utilize public data at the server to handle adversaries , but in a distributed learning setup with IID data across the nodes . Compared to Zeno+ , our Sageflow targets non-IID data distribution in a FL setup . While Zeno+ adopts a fully asynchronous update rule ( without considering the staleness ) with the loss-based score function , our Sageflow integrates staleness-aware grouping , a semi-synchronous straggler-handling method , with entropy filtering and loss-weighted averaging , a harmonized means to provide protection against a wider variety of attacks . We show later via experiments that our Sageflow outperforms Zeno+ in practical FL setups having stragglers and various types of adversaries . 2 Proposed Sageflow for Federated Learning . We consider the following federated optimization problem : w∗ = argmin w F ( w ) = argmin w N∑ k=1 mk m Fk ( w ) , ( 1 ) whereN is the number of devices , mk is the number of data samples in device k , andm = ∑N k=1mk is the total number of data samples of all N devices in the system . By letting xk , j be the jth data sample in device k , the local loss function of device k , Fk ( w ) , is written as Fk ( w ) = 1 mk ∑mk j=1 ` ( w ; xk , j ) . 2.1 Staleness-Aware Grouping against Stragglers . In the t-th global round of a practical FL setting , the server sends the current model wt to K devices in St , a randomly selected subset of N devices . We let C = K/N be the ratio of devices that participate in each global round . Each device in St performs E local updates with its own data and sends the updated model back to the server . In handling slow devices , our idea assumes periodic global aggregation at the server . At each global round t , the server transmits the current model and time stamp , ( wt , t ) , to the devices in St . Instead of waiting for all devices in St , the server aggregates the models that arrive by a fixed time deadline Td to obtain wt+1 , and moves on to the next global round t+ 1 . Hence , model aggregation is performed periodically with every Td . A key feature here is that we do not ignore the results sent from stragglers not arriving by the deadline Td . These results are utilized at the next global aggregation step , or even later , depending on the delay or staleness . Let U ( t ) i be the group of devices selected from the server at global round t that successfully sent their results to the server at global round i for i ≥ t. Then , we can write St = ∪∞i=tU ( t ) i , where U ( t ) i ∩ U ( t ) j = ∅ for i 6= j . Here , U ( t ) ∞ can be viewed as the devices that are selected at round t but failed to successfully send their results back to the server . According to these notations , the devices whose training results arrive at the server during global round t belong to one of the t+ 1 groups : U ( 0 ) t , U ( 1 ) t , ... , U ( t ) t . Note that the result sent from device k ∈ U ( i ) t is the model after E local updates starting from wi , and we denote this model by wi ( k ) . At each round t , we first perform FedAvg as v ( i ) t+1 = ∑ k∈U ( i ) t mk∑ k∈U ( i ) t mk wi ( k ) ( 2 ) for all i = 0 , 1 , ... , t. Here , v ( i ) t+1 can be viewed as a group representative model , which is the aggregated result of locally updated models ( starting from wi ) received at round t with staleness t−i+1 . Then from the representative models of all t groups , v ( 0 ) t+1 , v ( 1 ) t+1 , ... , v ( t ) t+1 , we take a weighted averaging according to different staleness : ∑t i=0 α ( i ) t ( λ ) v ( i ) t+1 . Here , α ( i ) t ( λ ) ∝ ∑ k∈U ( i ) t mk ( t−i+1 ) λ is a normalized coefficient that is proportional to the number of data samples in U ( i ) t and inversely proportional to ( t− i+ 1 ) λ , for a given staleness exponent λ ≥ 0 . Hence , we have a larger weight 𝒰 𝒰 Received at global round t-1 Received at global round t … for v ( i ) t+1 with a smaller t − i + 1 ( staleness ) . This is to give more weights to more recent results having smaller staleness . Based on the weighted sum ∑t i=0 α ( i ) t ( λ ) v ( i ) t+1 , we finally obtain wt+1 as wt+1 = ( 1− γ ) wt + γ t∑ i=0 α ( i ) t ( λ ) v ( i ) t+1 , ( 3 ) where γ is the time-average coefficient . Now we move on to the next round t+ 1 , where the server selects St+1 and sends ( wt+1 , t + 1 ) to these devices . Here , if the server knows the set of active devices ( which are still performing computation ) , St+1 can be constructed to be disjoint with the active devices . If not , the server randomly chooses St+1 among all devices in the system and the selected active devices can ignore the current request of the server . The left-hand side of Fig . 1 describes our staleness-aware grouping method . The key characteristics of our approach against stragglers can be summarized as follows . First , by periodic global aggregation at the server , our scheme is not delayed by the effect of stragglers . Secondly , our scheme fully utilizes the results sent from stragglers in the future global rounds .

Summary: This paper studies and the interesting problem of presence of both stragglers and adversaries in Federated Learning (FL), and proposes a solution that simultaneously addresses both issues. There's no question that the challenges considered in this paper are important to the FL community. The solution is based on grouping the late model update from stragglers, incorporating them into the model update, and performing an entropy-based filtering to find the adversarial updates. The paper has really interesting and comprehensive experiments, and it is backed by theoretical findings.