Tesseract: Gradient Flip Score to Secure Federated Learning against Model Poisoning Attacks

1 INTRODUCTION . Federated learning ( FL ) Smith et al . ( 2017 ) ; Yang et al . ( 2019 ) offers a way for multiple clients on heterogeneous platforms to learn collaboratively without sharing their local data . The clients send their local gradients to the parameter server that aggregates the gradients and updates the global model for the local clients to download . FL can be attacked during the training phase by compromising a set of clients that then send maliciously crafted gradients . The attack can be targeted against particular data instances or can be untargeted . The latter brings down the overall accuracy by affecting all classes . To counter this threat , a set of approaches has been developed for countering Byzantine clients in FL , e.g. , Krum Blanchard et al . ( 2017 ) , Bulyan Mhamdi et al . ( 2018 ) , Trimmed Mean and Median Yin et al . ( 2018 ) , FoolsGold Fung et al . ( 2020 ) , and FABA Xia et al . ( 2019 ) . In this work , we use the state-of-the-art ( SOTA ) untargeted model poisoning attack called a directed deviation attack , proposed recently in Fang et al . ( 2020 ) . This has been shown to bypass all existing Byzantine-robust aggregation techniques , e.g. , Krum , Bulyan , Trimmed mean , and Median . In our experiments , this attack has been found to decrease the test accuracy from 90 % to a low 9 % when a DNN is trained using Krum on the MNIST dataset , distributed among 100 clients , 20 of which are under the attacker ’ s control . We describe the relevant details of this attack in Section 2.2 . Our solution . We propose a novel defense called TESSERACT against untargeted model poisoning attacks ( Figure 1 ) , which uses a stateful model to reduce the contributions by suspicious clients to the global model update . We show that where all prior Byzantine-resilient federated learning approaches fail against the directed-deviation attack of Fang et al . ( 2020 ) , which is the state-of-the-art untargeted model poisoning attack , TESSERACT is able to recover the test accuracy of the trained model . This benefit applies even when the attack knows the algorithm and all the parameters of our defense , i.e. , an adaptive white-box attack . TESSERACT is based on a simple intuition that for a sufficiently small learning rate , as the model approaches an optima in a benign setting , a large number of gradients do not flip their direction with large magnitudes , that is , a degree of inertia is maintained . Our intuition is supported from the analysis we show in Figure 2 . We capture this quantitatively in a metric that we propose , called flip-score , which is the sum of squares of gradient magnitudes of all parameter updates that suggest a flip in the gradient direction from the previous global update . We find that our intuition and correspondingly the defense TESSERACT also holds for other attacks , e.g. , the targeted and untargeted label flipping attacks ( Appendix § A.3 ) . However , the other attacks are less damaging , consistent with the observation in Fang et al . ( 2020 ) . Thus , in the main paper , we focus on the directed deviation model poisoning attack . In summary , TESSERACT makes the following contributions . 1 . We use our simple intuition to detect malicious clients that attack federated learning using the SOTA attack model Fang et al . ( 2020 ) . Our intuition is that certain patterns of flips of the signs of gradients across multiple model parameters and across multiple clients are rare under benign conditions . 2 . We use a stateful suspicion model to keep the history of every client ’ s activity and use that as a weighting factor in the aggregation . We theoretically prove the convergence of TESSERACT that uses the weighted averaging and establish a convergence rate . 3 . We evaluate TESSERACT on DNNs trained on MNIST and FEMNIST , ResNet-18 on CIFAR-10 , and GRU on the Shakespeare dataset . We comparatively evaluate our defense against six baselines , including the most recent ones , FABA Xia et al . ( 2019 ) , FoolsGold Fung et al . ( 2020 ) , and FLTrust Cao et al . ( 2020 ) and show that TESSERACT remains robust even against an adaptive white-box attacker . While several of the existing defenses shine under specific configurations ( combination of attacks and datasets/models ) , TESSERACT is the only one whose protection transfers well across configurations . We release the source code , the attack scripts , the trained models , and the test harness for the evaluation at the anonymized page https : //www.dropbox.com/ sh/h9ulw6y2f8rzv64/AADe1Sb9PhhCLqclzgZ4xvvJa ? dl=0 . The rest of the paper is organized as follows . We describe in Sec 2 the threat model , the SOTA attack , and why all existing Byzantine-resilient federated learning approaches are susceptible . We present TESSERACT ’ s design in Sec 3 and convergence analysis in Sec 4 . We describe the baselines and the datasets in Sec 5 , and evaluation in Sec 6 . 2 BACKGROUND . Our simulation of federated learning consists of m clients , each with its own local data , but with the same model architecture and SGD optimizer , out of which c are malicious , as shown in Figure 1 . The parameter server assumes that a maximum of cmax number of clients can be malicious . The clients run one local iteration , send their gradients ( in unencrypted form ) to the server , which updates the global model for the clients to download in a synchronous manner . 2.1 BYZANTINE-RESILIENT FEDERATED LEARNING . Here we describe the leading defenses briefly , stymied by SOTA untargeted model poisoning attacks . 1 . The simplest aggregation technique is FedSGD McMahan et al . ( 2017 ) that does a simple weighted mean aggregation of the gradients , weighted by the number of data samples each client holds . FedSGD can be attacked by a single malicious client that can send boosted gradients . 2 . Trimmed mean and Median Yin et al . ( 2018 ) aggregate the parameters independently , where one trims cmax each at the lower and higher extremes of every parameter and the other takes the median of every parameter update across the gradients received from all the clients . The full-Trim attack Fang et al . ( 2020 ) is specifically designed toward these aggregations rules . 3 . Krum Blanchard et al . ( 2017 ) selects one local model as the next global model . The client that has the lowest Euclidean distance from its closest ( m − cmax − 2 ) neighbors is chosen as the local model . The full-Krum attack Fang et al . ( 2020 ) is tailored to attack Krum . 4 . Bulyan Mhamdi et al . ( 2018 ) combines the above approaches by running Krum iteratively to select a given number of models , and then running Trimmed Mean over the selected ones . The Full-Trim attack is also transferable to Bulyan . 5 . FABA Xia et al . ( 2019 ) iteratively filters out models farthest away from the mean of the remaining models , cmax number of times before returning the mean of the remaining gradients . 6 . FoolsGold Fung et al . ( 2020 ) was motivated to defend against poisoning attacks by Sybil clones , and thus , it finds clients with similar cosine similarity to be malicious , penalizes their reputation , and returns a weighted mean of the gradients , weighed by their reputation . 7 . FLTrust Cao et al . ( 2020 ) bootstraps trust in the clients by assuming that the server has access to a clean validation dataset , albeit small , and returns a weighted mean of the gradients weighed by this trust . In our setting , we do not see a realistic method to access such as clean dataset , especially considering the non-iid nature of the local datasets at the clients . 2.2 THREAT MODEL : STATE-OF-THE-ART MODEL POISONING ATTACK . Our threat model consists of a scenario where an adversary compromises a fraction of all clients participating in federated learning . We assume that the adversary also has access to the gradient vector sent by the benign clients to the server , and knows what aggregation algorithm the server is running . We focus on the SOTA untargeted model poisoning attack Fang et al . ( 2020 ) —a directed deviation attack ( DDA ) bypassing all known defenses 1 . To address generality ( more details in Appendix 1There is no overlap between the authors of this current submission and of the SOTA attack Fang et al . ( 2020 ) . § A.6.2 ) , we have also evaluated TESSERACT on label-flipping attacks in Appendix § A.3 . The DDA changes the local models on the compromised worker devices . This change is done strategically ( through solving a constrained optimization problem ) such that the global model deviates the most toward the inverse of the direction along which the benign global model would have changed . Their intuition is that the deviations accumulated over multiple iterations would make the learned global model differ from the benign one significantly . TESSERACT defends against both variants of the DDA , one specialized to poison Krum ( transferable to Bulyan , we call this the Full-Krum attack ) , the other specialized for Trimmed Mean ( transferable to Median , we call this the Full-Trim attack ) , respectively . We assume a full-knowledge ( white-box ) attack where the attackers have access to the current benign gradients . They themselves compute the benign gradients on their local data as well , and thus estimate the benign direction as the average of all benign gradients . This value is stored in a vector s of size equal to the number of model parameters . s ( t , · ) = sign ( sum i ( ▽LMi ( t , · ) ) , where ▽LMi ( t , · ) is the gradient updates of client i. Full-Krum attack : Having estimated s , the attackers send gradients in the opposite direction , all with a magnitude λ , with some added noise to appear different but still maintaining a small Euclidean distance from one another . The upper bound of λ is computed in every iteration as a function of m , c , |P | , GM ( t , · ) , ▽LMi ( t + 1 , · ) , where c out m participating clients are malicious , |P | is the number of parameters in the global model GM . λ is then iteratively decreased until the attackers make sure ( using a local estimate ) that the parameter server would have chosen the attacked model , using the Krum aggregation technique . Full-Trim attack : The Trim attack while following the same fundamental principle of flipping the gradient direction , attempts to skew the distribution of every parameter j toward the inverse of the direction that s ( t , j ) suggests in order to attack a mean-like aggregation . It does so by randomly sampling gradient magnitudes from a range that has been computed by the attackers , guaranteed to skew the gradient distribution of every parameter , without appearing as obvious outliers ( which would be caught by a method such as Trimmed Mean ) . Therefore , the attacked gradients here look more diverse than those in the Full-Krum attack . Overall , both forms of the DDA smartly take advantage of the fact that none of the existing aggregation techniques looks at the change in gradient direction to identify malicious gradients in a robust way . This kind of an attack is an extremely likely scenario in a cross-device as well as a cross-silo federated learning setting , ( more details in Appendix § A.6.1 ) and needs to be defended against .

Summary: The paper tackles the problem of adversarial attacks in federated learning settings. The main proposal is a defensive technique to address the “byzantine generals” problem in federated learning: how to ensure that the general ML model is not affected by “poisonous” attempts made by corrupted clients. The proposed technique is experimentally validated on four datasets, outperforms previous defensive methods, and the evaluation also considers adaptive adversaries with increasing degrees of knowledge.   Overall, the presentation of the paper is very good. The quality of the English text is good. Figures are appropriate, Tables require some editing. The topic addressed by the manuscript is trendy and in-line with ICLR’s scope. The references should be improved The contribution is significant  STRENGTHS: + Adaptive adversary  + Trendy subject (federated learning) + Evaluation on multiple datasets + Technically sound  WEAKNESSES - Unclear assumptions and threat model. - Problem or Feature space attacks? - Lack of a concrete use-case - Tradeoff?