Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching

1 INTRODUCTION . Machine learning models have quickly become the backbone of many applications from photo processing on mobile devices and ad placement to security and surveillance ( LeCun et al. , 2015 ) . These applications often rely on large training datasets that aggregate samples of unknown origins , and the security implications of this are not yet fully understood ( Papernot , 2018 ) . Data is often sourced in a way that lets malicious outsiders contribute to the dataset , such as scraping images from the web , farming data from website users , or using large academic datasets scraped from social media ( Taigman et al. , 2014 ) . Data Poisoning is a security threat in which an attacker makes imperceptible changes to data that can then be disseminated through social media , user devices , or public datasets without being caught by human supervision . The goal of a poisoning attack is to modify the final model to achieve a malicious goal . In this work we focus on targeted attacks ∗Authors contributed equally . †Authors contributed equally . that achieve mis-classification of some predetermined target data as in Suciu et al . ( 2018 ) ; Shafahi et al . ( 2018 ) , effectively implementing a backdoor that is only triggered for a specific image . Yet , other potential goals of the attacker can include denial-of-service ( Steinhardt et al. , 2017 ; Shen et al. , 2019 ) , concealment of users ( Shan et al. , 2020 ) , or introduction of fingerprint information ( Lukas et al. , 2020 ) . These attacks are applied in scenarios such as social recommendation ( Hu et al. , 2019 ) , content management ( Li et al. , 2016 ; Fang et al. , 2018 ) , algorithmic fairness ( Solans et al. , 2020 ) and biometric recognition ( Lovisotto et al. , 2019 ) . Accordingly , industry practitioners ranked data poisoning as the most serious attack on ML systems in a recent survey of corporations ( Kumar et al. , 2020 ) . We show that efficient poisoned data causing targeted misclassfication can be created even in the setting of deep neural networks trained on large image classification tasks , such as ImageNet ( Russakovsky et al. , 2015 ) . Previous work on targeted data poisoning has often focused on either linear classification tasks ( Biggio et al. , 2012 ; Xiao et al. , 2015 ; Koh et al. , 2018 ) or poisoning of transfer learning and fine tuning ( Shafahi et al. , 2018 ; Koh & Liang , 2017 ) rather than a full end-to-end training pipeline . Attacks on deep neural networks ( and especially on ones trained from scratch ) have proven difficult in Muñoz-González et al . ( 2017 ) and Shafahi et al . ( 2018 ) . Only recently were targeted attacks against neural networks retrained from scratch shown to be possible in Huang et al . ( 2020 ) for CIFAR-10 - however with costs that render scaling to larger datasets , like the ImageNet dataset , prohibitively expensive . We formulate targeted data poisoning as the problem of solving a gradient matching problem and analyze the resulting novel attack algorithm that scales to unprecedented dataset size and effectiveness . Crucially , the new poisoning objective is orders-of-magnitude more efficient than a previous formulation based on on meta-learning ( Huang et al. , 2020 ) and succeeds more often . We conduct an experimental evaluation , showing that poisoned datasets created by this method are robustly compromised and significantly outperform other attacks on CIFAR-10 on the benchmark of Schwarzschild et al . ( 2020 ) . We then demonstrate reliably successful attacks on common ImageNet models in realistic training scenarios . For example , the attack successfully compromises a ResNet-34 by manipulating only 0.1 % of the data points with perturbations less than 8 pixel values in ` ∞-norm . We close by discussing previous defense strategies and how strong differential privacy ( Abadi et al. , 2016 ) is the only existing defense that can partially mitigate the effects of the attack . 2 RELATED WORK . The task of data poisoning is closely related to the problem of adversarial attacks at test time , also referred to as evasion attacks ( Szegedy et al. , 2013 ; Madry et al. , 2017 ) , where the attacker alters a target test image to fool an already-trained model . This attack is applicable in scenarios where the attacker has control over the target image , but not over the training data . In this work we are specifically interested in targeted data poisoning attacks – attacks which aim to cause a specific target test image ( or set of target test images ) to be mis-classified . For example , an attack may cause a certain target image of a otter not part of the training set to be classified as a dog by victim models at test time . This attack is difficult to detect , because it does not noticeably degrade either training or validation accuracy ( Shafahi et al. , 2018 ; Huang et al. , 2020 ) and is effectively invisible until it is triggered . From a security standpoint , these attacks break the integrity of a machine learning model and are as such also called poison integrity attacks in Barreno et al . ( 2010 ) - in contrast to poison availability attacks which reduce validation accuracy in general and are not a focus of this work . In comparison to evasion attacks , targeted data poisoning attacks generally consider a setting where the attacker can modify training data within limits , but can not modify test data and chooses specific target data a-priori . A related intermediary between data poisoning attacks we consider and evasion attacks are backdoor trigger attacks ( Turner et al. , 2018 ; Saha et al. , 2019 ) . These attacks involve inserting a trigger – often an image patch – into training data , which is later activated by also applying the trigger to test images . Backdoor attacks require perturbations to both training and test-time data – the more permissive threat model is a trade-off that allows for unknown target images . Two basic schemes for targeted poisoning are label flipping ( Barreno et al. , 2010 ; Paudice et al. , 2019 ) , and watermarking ( Suciu et al. , 2018 ; Shafahi et al. , 2018 ) . In label flipping attacks , an attacker is allowed to change the label of examples , whereas in a watermarking attack , the attacker perturbs the training image , not label , by superimposing a target image onto training images . These attacks can be successful , yet they are easily detected by supervision such as Papernot & McDaniel ( 2018 ) . This is in contrast to clean-label attacks which maintain the semantic labels of data . Mathematically speaking , data poisoning is a bilevel optimization problem ( Bard & Falk , 1982 ; Biggio et al. , 2012 ) ; the attacker optimizes image pixels to enforce ( malicious ) criteria on the resulting network parameters , which are themselves the solution to an “ inner ” optimization problem that minimizes the training objective . Direct solutions to the bilevel problem of data poisoning have been proposed where feasible , for example , SVMs in Biggio et al . ( 2012 ) or logistic regression in Demontis et al . ( 2019 ) . However , direct optimization of the poisoning objective is intractable for deep neural networks because it requires backpropagating through the entire SGD training procedure , see Muñoz-González et al . ( 2017 ) . As such , the bilevel objective has to be approximated . Recently , MetaPoison ( Huang et al. , 2020 ) proposed to approximately solve the bi-level problem based on methods from the meta-learning community ( Finn et al. , 2017 ) . The bilevel gradient is approximated by backpropagation through several unrolled gradient descent steps . This is the first attack to succeed against deep networks trained from scratch on CIFAR-10 as well as providing transferability to other models . Yet , Huang et al . ( 2020 ) uses a complex loss function averaged over a wide range of models trained to different epochs and a single unrolling step necessarily involves both clean and poisoned data , making it roughly as costly as one epoch of standard training . With an ensemble of 24 models , Huang et al . ( 2020 ) requires 3 ( 2 unrolling steps + 1 clean update step ) x 2 ( backpropagation through unrolled steps ) x 60 ( first-order optimization steps ) x 24 ( ensemble of models ) equivalent epochs of normal training to attack , as well as ( ∑23 k=0 k = 253 ) epochs of pretraining . All in all , this equates to 8893 training epochs . While this can be mitigated by smart caching and parallelization strategies , unrolled ensembles remain costly . In contrast to bilevel approaches stand heuristics for data poisoning of neural networks . The most prominent heuristic is feature collision , as in Poison Frogs ( Shafahi et al. , 2018 ) , which seeks to cause a target test image to be misclassified by perturbing training data to collide with the target image in feature space . Modifications surround the target image in feature space with a convex polytope ( Zhu et al. , 2019 ) or collection of poisons ( Aghakhani et al. , 2020 ) and consider model ensembles ( Zhu et al. , 2019 ) . These methods are efficient , but designed to attack fine-tuning scenarios where the feature extractor is nearly fixed and not influenced by poisoned data . When applied to deep networks trained from scratch , their performance drops significantly . 3 EFFICIENT POISON BREWING . In this section , we will discuss an intriguing weakness of neural network training based on first-order optimization and derive an attack against it . This attack modifies training images that so they produce a malicious gradient signal during training , even while appearing inconspicuous . This is done by matching the gradient of the target images within ` ∞ bounds . Because neural networks are trained by gradient descent , even minor modifications of the gradients can be incorporated into the final model . This attack compounds the strengths of previous schemes , allowing for data poisoning as efficiently as in Poison Frogs ( Shafahi et al. , 2018 ) , requiring only a single pretrained model and a time budget on the order of one epoch of training for optimization - but still capable of poisoning the from-scratch setting considered in Huang et al . ( 2020 ) . This combination allow an attacker to `` brew '' poisons that successfully attack realistic models on ImageNet . 3.1 THREAT MODEL . These discussed components of a clean-label targeted data poisoning attack fit together into the following exemplary threat scenario : Assume a security system that classifies luggage images . An attacker wants this system to classify their particular piece of luggage , the target as safe , but can modify only a small part of the training set . The attacker modifies this subset to be clean-label poisoned . Although the entire training set is validated by a human observer , the small subset of minorly modified images pass cursory inspection and receive their correct label . The security system is trained on secretly compromised data , evaluated on validation data as normal and deployed . Until the target is evaluated and mis-classified as safe , the system appears to be working fine . Formally , we define two parties , the attacker , which has limited control over the training data , and the victim , which trains a model based on this data . We first consider a gray-box setting , where the attacker has knowledge of the model architecture used by their victim . The attacker is permitted to poison a fraction of the training dataset ( usually less than 1 % ) by changing images within an ` ∞-norm ε-bound ( e.g . with ε ≤ 16 ) . This constraint enforces clean-label attacks , meaning that the semantic label of a poisoned image is still unchanged . The attacker has no knowledge of the training procedure - neither about the initialization of the victim ’ s model , nor about the ( randomized ) mini-batching and data augmentation that is standard in the training of deep learning models . We formalize this threat model as bilevel problem for a machine learning model F ( x , θ ) with inputs x ∈ Rn and parameters θ ∈ Rp , and loss function L. We denote the N training samples by ( xi , yi ) N i=1 , from which a subset of P samples are poisoned . For notation simplicity we assume the first P training images are poisoned by adding a perturbation ∆i to the ith training image . The perturbation is constrained to be smaller than ε in the ` ∞-norm . The task is to optimize ∆ so that a set of T target samples ( xti , y t i ) T i=1 is reclassified with the new adversarial labels y adv i : min ∆∈C T∑ i=1 L ( F ( xti , θ ( ∆ ) ) , y adv i ) s.t . θ ( ∆ ) ∈ arg min θ 1 N N∑ i=1 L ( F ( xi + ∆i , θ ) , yi ) . ( 1 ) We subsume the constraints in the set C = { ∆ ∈ RN×n : ||∆||∞ ≤ ε , ∆i = 0 ∀i > P } . We call the main objective on the left the adversarial loss , and the objective that appears in the constraint on the right is the training loss . For the remainder , we consider a single target image ( T = 1 ) as in Shafahi et al . ( 2018 ) , but stress that this is not a general limitation as shown in the appendix .

Summary: This paper introduces a novel targeted clean-label poisoning attack, expected to be more efficient and scalable than current ones. The attack is formulated as a bilevel problem which is then solved with a (fast) heuristic approach based on aligning the gradients of the inner and outer objective functions. A theoretical analysis is also reported to show that this strategy consistently finds a descent direction for the outer objective, asymptotically converging to (a local) minimum.