Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching

1 INTRODUCTION . Machine learning models have quickly become the backbone of many applications from photo processing on mobile devices and ad placement to security and surveillance ( LeCun et al. , 2015 ) . These applications often rely on large training datasets that aggregate samples of unknown origins , and the security implications of this are not yet fully understood ( Papernot , 2018 ) . Data is often sourced in a way that lets malicious outsiders contribute to the dataset , such as scraping images from the web , farming data from website users , or using large academic datasets scraped from social media ( Taigman et al. , 2014 ) . Data Poisoning is a security threat in which an attacker makes imperceptible changes to data that can then be disseminated through social media , user devices , or public datasets without being caught by human supervision . The goal of a poisoning attack is to modify the final model to achieve a malicious goal . In this work we focus on targeted attacks ∗Authors contributed equally . †Authors contributed equally . that achieve mis-classification of some predetermined target data as in Suciu et al . ( 2018 ) ; Shafahi et al . ( 2018 ) , effectively implementing a backdoor that is only triggered for a specific image . Yet , other potential goals of the attacker can include denial-of-service ( Steinhardt et al. , 2017 ; Shen et al. , 2019 ) , concealment of users ( Shan et al. , 2020 ) , or introduction of fingerprint information ( Lukas et al. , 2020 ) . These attacks are applied in scenarios such as social recommendation ( Hu et al. , 2019 ) , content management ( Li et al. , 2016 ; Fang et al. , 2018 ) , algorithmic fairness ( Solans et al. , 2020 ) and biometric recognition ( Lovisotto et al. , 2019 ) . Accordingly , industry practitioners ranked data poisoning as the most serious attack on ML systems in a recent survey of corporations ( Kumar et al. , 2020 ) . We show that efficient poisoned data causing targeted misclassfication can be created even in the setting of deep neural networks trained on large image classification tasks , such as ImageNet ( Russakovsky et al. , 2015 ) . Previous work on targeted data poisoning has often focused on either linear classification tasks ( Biggio et al. , 2012 ; Xiao et al. , 2015 ; Koh et al. , 2018 ) or poisoning of transfer learning and fine tuning ( Shafahi et al. , 2018 ; Koh & Liang , 2017 ) rather than a full end-to-end training pipeline . Attacks on deep neural networks ( and especially on ones trained from scratch ) have proven difficult in Muñoz-González et al . ( 2017 ) and Shafahi et al . ( 2018 ) . Only recently were targeted attacks against neural networks retrained from scratch shown to be possible in Huang et al . ( 2020 ) for CIFAR-10 - however with costs that render scaling to larger datasets , like the ImageNet dataset , prohibitively expensive . We formulate targeted data poisoning as the problem of solving a gradient matching problem and analyze the resulting novel attack algorithm that scales to unprecedented dataset size and effectiveness . Crucially , the new poisoning objective is orders-of-magnitude more efficient than a previous formulation based on on meta-learning ( Huang et al. , 2020 ) and succeeds more often . We conduct an experimental evaluation , showing that poisoned datasets created by this method are robustly compromised and significantly outperform other attacks on CIFAR-10 on the benchmark of Schwarzschild et al . ( 2020 ) . We then demonstrate reliably successful attacks on common ImageNet models in realistic training scenarios . For example , the attack successfully compromises a ResNet-34 by manipulating only 0.1 % of the data points with perturbations less than 8 pixel values in ` ∞-norm . We close by discussing previous defense strategies and how strong differential privacy ( Abadi et al. , 2016 ) is the only existing defense that can partially mitigate the effects of the attack . 2 RELATED WORK . The task of data poisoning is closely related to the problem of adversarial attacks at test time , also referred to as evasion attacks ( Szegedy et al. , 2013 ; Madry et al. , 2017 ) , where the attacker alters a target test image to fool an already-trained model . This attack is applicable in scenarios where the attacker has control over the target image , but not over the training data . In this work we are specifically interested in targeted data poisoning attacks – attacks which aim to cause a specific target test image ( or set of target test images ) to be mis-classified . For example , an attack may cause a certain target image of a otter not part of the training set to be classified as a dog by victim models at test time . This attack is difficult to detect , because it does not noticeably degrade either training or validation accuracy ( Shafahi et al. , 2018 ; Huang et al. , 2020 ) and is effectively invisible until it is triggered . From a security standpoint , these attacks break the integrity of a machine learning model and are as such also called poison integrity attacks in Barreno et al . ( 2010 ) - in contrast to poison availability attacks which reduce validation accuracy in general and are not a focus of this work . In comparison to evasion attacks , targeted data poisoning attacks generally consider a setting where the attacker can modify training data within limits , but can not modify test data and chooses specific target data a-priori . A related intermediary between data poisoning attacks we consider and evasion attacks are backdoor trigger attacks ( Turner et al. , 2018 ; Saha et al. , 2019 ) . These attacks involve inserting a trigger – often an image patch – into training data , which is later activated by also applying the trigger to test images . Backdoor attacks require perturbations to both training and test-time data – the more permissive threat model is a trade-off that allows for unknown target images . Two basic schemes for targeted poisoning are label flipping ( Barreno et al. , 2010 ; Paudice et al. , 2019 ) , and watermarking ( Suciu et al. , 2018 ; Shafahi et al. , 2018 ) . In label flipping attacks , an attacker is allowed to change the label of examples , whereas in a watermarking attack , the attacker perturbs the training image , not label , by superimposing a target image onto training images . These attacks can be successful , yet they are easily detected by supervision such as Papernot & McDaniel ( 2018 ) . This is in contrast to clean-label attacks which maintain the semantic labels of data . Mathematically speaking , data poisoning is a bilevel optimization problem ( Bard & Falk , 1982 ; Biggio et al. , 2012 ) ; the attacker optimizes image pixels to enforce ( malicious ) criteria on the resulting network parameters , which are themselves the solution to an “ inner ” optimization problem that minimizes the training objective . Direct solutions to the bilevel problem of data poisoning have been proposed where feasible , for example , SVMs in Biggio et al . ( 2012 ) or logistic regression in Demontis et al . ( 2019 ) . However , direct optimization of the poisoning objective is intractable for deep neural networks because it requires backpropagating through the entire SGD training procedure , see Muñoz-González et al . ( 2017 ) . As such , the bilevel objective has to be approximated . Recently , MetaPoison ( Huang et al. , 2020 ) proposed to approximately solve the bi-level problem based on methods from the meta-learning community ( Finn et al. , 2017 ) . The bilevel gradient is approximated by backpropagation through several unrolled gradient descent steps . This is the first attack to succeed against deep networks trained from scratch on CIFAR-10 as well as providing transferability to other models . Yet , Huang et al . ( 2020 ) uses a complex loss function averaged over a wide range of models trained to different epochs and a single unrolling step necessarily involves both clean and poisoned data , making it roughly as costly as one epoch of standard training . With an ensemble of 24 models , Huang et al . ( 2020 ) requires 3 ( 2 unrolling steps + 1 clean update step ) x 2 ( backpropagation through unrolled steps ) x 60 ( first-order optimization steps ) x 24 ( ensemble of models ) equivalent epochs of normal training to attack , as well as ( ∑23 k=0 k = 253 ) epochs of pretraining . All in all , this equates to 8893 training epochs . While this can be mitigated by smart caching and parallelization strategies , unrolled ensembles remain costly . In contrast to bilevel approaches stand heuristics for data poisoning of neural networks . The most prominent heuristic is feature collision , as in Poison Frogs ( Shafahi et al. , 2018 ) , which seeks to cause a target test image to be misclassified by perturbing training data to collide with the target image in feature space . Modifications surround the target image in feature space with a convex polytope ( Zhu et al. , 2019 ) or collection of poisons ( Aghakhani et al. , 2020 ) and consider model ensembles ( Zhu et al. , 2019 ) . These methods are efficient , but designed to attack fine-tuning scenarios where the feature extractor is nearly fixed and not influenced by poisoned data . When applied to deep networks trained from scratch , their performance drops significantly . 3 EFFICIENT POISON BREWING . In this section , we will discuss an intriguing weakness of neural network training based on first-order optimization and derive an attack against it . This attack modifies training images that so they produce a malicious gradient signal during training , even while appearing inconspicuous . This is done by matching the gradient of the target images within ` ∞ bounds . Because neural networks are trained by gradient descent , even minor modifications of the gradients can be incorporated into the final model . This attack compounds the strengths of previous schemes , allowing for data poisoning as efficiently as in Poison Frogs ( Shafahi et al. , 2018 ) , requiring only a single pretrained model and a time budget on the order of one epoch of training for optimization - but still capable of poisoning the from-scratch setting considered in Huang et al . ( 2020 ) . This combination allow an attacker to `` brew '' poisons that successfully attack realistic models on ImageNet . 3.1 THREAT MODEL . These discussed components of a clean-label targeted data poisoning attack fit together into the following exemplary threat scenario : Assume a security system that classifies luggage images . An attacker wants this system to classify their particular piece of luggage , the target as safe , but can modify only a small part of the training set . The attacker modifies this subset to be clean-label poisoned . Although the entire training set is validated by a human observer , the small subset of minorly modified images pass cursory inspection and receive their correct label . The security system is trained on secretly compromised data , evaluated on validation data as normal and deployed . Until the target is evaluated and mis-classified as safe , the system appears to be working fine . Formally , we define two parties , the attacker , which has limited control over the training data , and the victim , which trains a model based on this data . We first consider a gray-box setting , where the attacker has knowledge of the model architecture used by their victim . The attacker is permitted to poison a fraction of the training dataset ( usually less than 1 % ) by changing images within an ` ∞-norm ε-bound ( e.g . with ε ≤ 16 ) . This constraint enforces clean-label attacks , meaning that the semantic label of a poisoned image is still unchanged . The attacker has no knowledge of the training procedure - neither about the initialization of the victim ’ s model , nor about the ( randomized ) mini-batching and data augmentation that is standard in the training of deep learning models . We formalize this threat model as bilevel problem for a machine learning model F ( x , θ ) with inputs x ∈ Rn and parameters θ ∈ Rp , and loss function L. We denote the N training samples by ( xi , yi ) N i=1 , from which a subset of P samples are poisoned . For notation simplicity we assume the first P training images are poisoned by adding a perturbation ∆i to the ith training image . The perturbation is constrained to be smaller than ε in the ` ∞-norm . The task is to optimize ∆ so that a set of T target samples ( xti , y t i ) T i=1 is reclassified with the new adversarial labels y adv i : min ∆∈C T∑ i=1 L ( F ( xti , θ ( ∆ ) ) , y adv i ) s.t . θ ( ∆ ) ∈ arg min θ 1 N N∑ i=1 L ( F ( xi + ∆i , θ ) , yi ) . ( 1 ) We subsume the constraints in the set C = { ∆ ∈ RN×n : ||∆||∞ ≤ ε , ∆i = 0 ∀i > P } . We call the main objective on the left the adversarial loss , and the objective that appears in the constraint on the right is the training loss . For the remainder , we consider a single target image ( T = 1 ) as in Shafahi et al . ( 2018 ) , but stress that this is not a general limitation as shown in the appendix .

Summary: This paper proposed a simple yet effective approach for data poisoning attack targeting a few "clean-label" victim images, using the idea of gradient matching (cosine similarity maximization) between the gradients of adversarial and clean losses. Although the attack model still requires knowing the network architecture (gray-box setting), the resulting poisoned datasets are more effective against different initializations, and some techniques (e.g. model ensemble, multiple restarts) are proposed to further boost the attack performance. The attack results are significantly better than the compared poisoning attacks, and the authors show effective attacks on the ImageNet dataset as well as Google Cloud AutoML with the poisoned data. The authors also discussed the proposed attack on some defenses, showing that the poison has limited change to feature distribution, and differential privacy can mitigate the attack but at the cost of reduced utility (clean accuracy).