Analyzing Federated Learning through an Adversarial Lens

1 INTRODUCTION . Federated learning introduced by McMahan et al . ( 2017 ) has recently emerged as a popular implementation of distributed stochastic optimization for large-scale deep neural network training . It is formulated as a multi-round strategy in which the training of a neural network model is distributed between multiple agents . In each round , a random subset of agents , with local data and computational resources , is selected for training . The selected agents perform model training and share only the parameter updates with a centralized parameter server , that facilitates aggregation of the updates . Motivated by privacy concerns , the server is designed to have no visibility into an agents ’ local data and training process . The aggregation algorithm is agnostic to the data distribution at the agents . In this work , we exploit this lack of transparency in the agent updates , and explore the possibility of a single malicious agent performing a model poisoning attack . The malicious agent ’ s objective is to cause the jointly trained global model to misclassify a set of chosen inputs with high confidence , i.e. , it seeks to introduce a targeted backdoor in the global model . In each round , the malicious agent generates its update by optimizing for a malicious objective different than the training loss for federated learning . It aims to achieve this by generating its update by directly optimizing for the malicious objective . However , the presence of a multitude of other agents which are simultaneously providing updates makes this challenging . Further , the malicious agent must ensure that its update is undetectable as aberrant . Contributions : To this end , we propose a sequence of model poisoning attacks , with the aim of achieving the malicious objective while maintaining attack stealth . For each strategy , we consider both attack strength as well as stealth . We start with malicious update boosting , designed to negate the combined effect of the benign agents , which enables the adversary to achieve its malicious objective with 100 % confidence . However , we show that boosted updates can be detected as aberrant using two measures of stealth , accuracy checking on the benign objective and parameter update statistics . Observing that the only parameter updates that need to be boosted are those that con- tribute to the malicious objective , we design an alternating minimization strategy that improves attack stealth . This strategy alternates between training loss minimization and the boosting of updates for the malicious objective and is able to achieve high success rate on both the benign and malicious objectives . In addition , we show that estimating the other agents ’ updates improves attack success rates . Finally , we use a suite of interpretability techniques to generate visual explanations of the decisions made by a global model with and without a targeted backdoor . Interestingly , we observe that the explanations are nearly visually indistinguishable . This establishes the attack stealth along yet another axis of measurement and indicates that backdoors can be inserted without drastic changes in model focus at the input . Summary of Empirical Results : In our experiments , we consider adversaries which only control a single malicious agent and at a given time step , have no visibility into the updates that will be provided by the other agents . We demonstrate that these adversaries can influence the global model to misclassify particular examples with high confidence . We work with both the Fashion-MNIST Xiao et al . ( 2017 ) and Adult Census1 , datasets and for settings with both 10 and 100 agents , our attacks are able to ensure the global model misclassifies a particular example in a target class with 100 % confidence . Our alternating minimization attack further ensures that the global model converges to the same test set accuracy as the case with no adversaries present . We also show that a simple estimation of the benign agents ’ updates as being identical over two consecutive rounds aids in improving attack success . Related Work : While data poisoning attacks ( Biggio et al. , 2012 ; Rubinstein et al. , 2009 ; Mei & Zhu , 2015 ; Xiao et al. , 2015 ; Mei & Zhu , 2015 ; Koh & Liang , 2017 ; Chen et al. , 2017a ; Jagielski et al. , 2018 ) have been widely studied , model poisoning attacks are largely unexplored . A number of works on defending against Byzantine adversaries consider a threat model where Byzantine agents send arbitrary gradient updates ( Blanchard et al. , 2017 ; Chen et al. , 2017b ; Mhamdi et al. , 2018 ; Chen et al. , 2018 ; Yin et al. , 2018 ) . However , the adversarial goal in these cases is to ensure a distributed implementation of the Stochastic Gradient Descent ( SGD ) algorithm converges to ‘ suboptimal to utterly ineffective models ’ , quoting from Mhamdi et al . ( 2018 ) . In complete constrast , our goal is to ensure convergence to models that are effective on the test set but misclassify certain examples . In fact , we show that the Byzantine-resilient aggregation mechanism ‘ Krum ’ Blanchard et al . ( 2017 ) is not resilient to our attack strategies ( Appendix C ) . Concurrent work by Bagdasaryan et al . ( 2018 ) considers multiple colluding agents performing poisoning via model replacement at convergence time . In contrast , our goal is to induce targeted misclassification in the global model by a single malicious agent even when it is far from convergence while maintaining its accuracy for most tasks . In fact , we show that updates generated by their strategy fail to achieve either malicious or benign objectives in the settings we consider . 2 FEDERATED LEARNING AND MODEL POISONING . In this section , we formulate both the learning paradigm and the threat model that we consider throughout the paper . Operating in the federated learning paradigm , where model weights are shared instead of data , gives rise to the model poisoning attacks that we investigate . 2.1 FEDERATED LEARNING . The federated learning setup consists of K agents , each with access to data Di , where |Di| = li . The total number of samples is ∑ i li = l. Each agent keeps its share of the data ( referred to as a shard ) private , i.e . Di = { xi1 · · ·xili } is not shared with the server S. The objective of the server is to learn a global parameter vector wG ∈ Rn , where n is the dimensionality of the parameter space . This parameter vector minimizes the loss2 over D = ∪iDi and the aim is to generalize well over Dtest , the test data . Federated learning is designed to handle non-i.i.d partitioning of training data among the different agents . At each time step t , a random subset of k agents is chosen for aggregation . Every agent i ∈ [ k ] , minimizes the empirical loss over its own data shard Di , by starting from the global weight vector wtG and running an algorithm such as SGD for E epochs with a batch size of B . At the end 1https : //archive.ics.uci.edu/ml/datasets/adult 2approximately for non-convex loss functions since global minima can not be guaranteed of its run , each agent obtains a local weight vector wt+1i and computes its local update δ t+1 i = wt+1i − wtG , which is sent back to the server . To obtain the global weight vector w t+1 G for the next iteration , any aggregation mechanism can be used . Following McMahan et al . ( 2017 ) , we use synchronous training ( i.e. , server waits till it has received updates from all the agents selected for the time step ) and weighted averaging based aggregation : wt+1G = w t G + ∑ i∈ [ k ] αiδ t+1 i , where li l = αi and ∑ i αi = 1 . We also experiment with the Byzantine-resilient aggregation mechanism ‘ Krum ’ ( Blanchard et al. , 2017 ) . Details are in Appendix C . 2.2 THREAT MODEL : MODEL POISONING . Traditional poisoning attacks deal with a malicious agent who poisons some fraction of the data in order to ensure that the learned model satisfies some adversarial goal . We consider instead an agent who poisons the model updates it sends back to the server . This attack is a plausible threat in the federated learning setting as the model updates from the agents can ( i ) directly influence the parameters of the global model via the aggregation algorithm ; and ( ii ) display high variability , due to the non-i.i.d local data at the agents , making it harder to isolate the benign updates from the malicious ones . Adversary Model : We make the following assumptions regarding the adversary : ( i ) there is exactly one non-colluding , malicious agent with index m ( limited effect of malicious updates on the global model ) ; ( ii ) the data is distributed among the agents in an i.i.d fashion ( making it easier to discriminate between benign and possible malicious updates and harder to achieve attack stealth ) ; ( iii ) the malicious agent has access to a subset of the training dataDm as well as to auxiliary dataDaux drawn from the same distribution as the training and test data that are part of its adversarial objective . Our aim is to explore the possibility of a successful model poisoning attack even for a highly constrained adversary . A malicious agent can have one of two objectives with regard to the loss and/or classification of a data subset at any time step t in the model poisoning setting : 1 . Increase the overall loss : In this case , the malicious agent wishes to increase the overall loss on a subset Daux = { xi , yi } ri=1 of the data . The adversarial objective is in this setting is A ( Dm , { xi , yi } ri=1 , wtG ) = argmaxwtG L ( { xi , yi } r i=1 , w t G ) , where L ( · , · ) is an appropriately defined loss function . This objective corresponds to the malicious agent attempting to cause untargeted misclassification . 2 . Obtain desired classification outcome : The malicious agent has data samples { xi } ri=1 with true labels { yi } ri=1 that have to be classified as desired target classes { τi } ri=1 , implying that the adversarial objective is A ( Dm , { xi , τi } ri=1 , wtG ) = argminwtG L ( { xi , τi } r i=1 , w t G ) . This corresponds to a targeted misclassification attempt by the malicious agent . In this paper , we will focus on malicious agents trying to attain the second objective , i.e . targeted misclassification . At first glance , the problem seems like a simple one for the malicious agent to solve . However , it does not have access to the global parameter vector wtG for the current iteration as is the case in standard poisoning attacks ( Muñoz-González et al. , 2017 ; Koh & Liang , 2017 ) and can only influence it though the weight update δtm it provides to the server S. The simplest formulation of the optimization problem the malicious agent has to solve such that her objective is achieved on the tth iteration is then argmin δtm L ( { xi , τi } ri=1 , wtG ) , s.t . wtG = w t−1 G + ∑ i∈ [ k ] \m αiδ t i + αmδ t m. ( 1 )

Summary: This paper presents an interesting adversarial strategy to attack federated learning systems, and discussed options to detect and prevent the attacks. It is based not upon data poisoning attacks, but model poisoning attacks. It analyzes different strategies on the attacker's side, discusses the effect with real experimental data, and proposes ways to prevent such attacks from the federated learning perspective.