What Doesn't Kill You Makes You Robust(er): How to Adversarially Train against Data Poisoning

1 INTRODUCTION . As machine learning systems consume more and more data , the data curation process is increasingly automated and reliant on data from untrusted sources . Breakthroughs in image classification ( Russakovsky et al. , 2015 ) as well as language processing ( Brown et al. , 2020 ) are built on large corpora of data scraped from the internet . Automated scraping , in which data is collected directly from online sources , leaves practitioners vulnerable to data poisoning in which bad actors tamper with the data so that models trained on this data perform poorly or contain backdoors embedded in them ( Gu et al. , 2019 ; Shafahi et al. , 2018 ) . These attacks present security vulnerabilities that persist even if the data is labeled and checked by crowd-sourced human supervision . In essence , entire machine learning pipelines can be compromised if the input data is modified maliciously - even if the modification appears minor and inconspicuous to a human observer . This mounting threat has instilled fear especially in industry practitioners whose business models rely on powerful neural networks trained on massive volumes of scraped data ( Kumar et al. , 2020 ) . In response to this growing threat , recent works have proposed a number of defenses against data poisoning attacks ( Li et al. , 2021b ; Goldblum et al. , 2020 ) . Existing defense strategies suffer from up to three primary shortcomings : 1 . In exchange for robustness , they trade off test accuracy to a degree that is intolerable to real-world practitioners ( Geiping et al. , 2021 ) . 2 . They are only robust to specific threat models but not to adaptive attacks specially designed to circumvent the defense ( Koh et al. , 2018 ; Tan & Shokri , 2020 ) . 3 . They apply only to a specific threat model and do not lend a generally applicable framework to practitioners ( Wang et al. , 2019 ) , that could extend to novel attacks . We instead propose a variant of adversarial training that harnesses adversarially poisoned data in the place of ( test-time ) adversarial examples . We show that this strategy exhibits both an improved robustness-accuracy trade-off as well as greater flexibility for defending against a wide range of threats including adaptive attacks . Adversarial training desensitizes neural networks to test-time adversarial perturbations by augmenting the training data with on-the-fly crafted adversarial examples ( Madry et al. , 2018 ) . Similarly , we modify training data in order to desensitize neural networks to the types of perturbations caused by data poisoning - yet adapting this robust training framework to data poisoning requires special consideration of this new threat model . In contrast to contemporary work on training against data poisoning such as Li et al . ( 2021a ) the defense dynamically trains a robust model without the need to identify poisoned data points and provide benefits by adapting specifically to targeted data poisoning instead of using default adversarial training as in Tao et al . ( 2021 ) or poisoned data generated offline as in Radiya-Dixit & Tramer ( 2021 ) . We demonstrate the effectiveness of this framework at defending against a range of data tampering threat models across several training regimes and visualize the impact of the defense in feature space . We further compare the proposed defenses to a range of related defense strategies . 2 RELATED WORK . Data poisoning is a class of threat scenarios focused on malicious modifications to the training data of a machine learning model . See Goldblum et al . ( 2020 ) for an overview of dataset security . Data poisoning attacks can either focus on reducing the overall model performance or on backdoor attacks that introduce malicious behavior which is triggered by a specific visual pattern or target image , thus breaking model integrity ( Barreno et al. , 2010 ) . In this work , we focus on attacks against model integrity which can be further distinguished by the nature of their trigger mechanism . In backdoor trigger attacks ( Gu et al. , 2019 ; Turner et al. , 2018 ) , the attack is triggered by a specific backdoor pattern or patch that can be added to target images at test time , whereas targeted data poisoning ( Shafahi et al. , 2018 ; Zhu et al. , 2019 ) is triggered by a predefined target image . In contrast to targeted poisoning , backdoor trigger attacks can be applied to multiple target images but require target modifications to be active during inference , while targeted attacks are activated by specific , but unmodified targets . 2.1 DATA POISONING ATTACKS : . Attacks can be further categorized by the precise training setup they anticipate their victims to employ . Some attacks assume that the victim will only fine-tune their model on the poisoned data or will train a linear classifier on top of a pre-trained feature extractor ( Saha et al. , 2020 ; Zhu et al. , 2019 ; Shafahi et al. , 2018 ) . These methods are effective against practitioners engaging in transfer learning with pre-trained models like those found in popular repositories such as Paszke et al . ( 2017 ) . Other attacks work even when the victim trains their model from scratch on the poisoned data ( Huang et al. , 2020 ; Geiping et al. , 2021 ) . Generally , the simpler the victim ’ s training procedure , the more readily the attacker can anticipate the effect of their perturbations and thus , the easier the attacker ’ s job . Here , we briefly detail prominent attacks against which we test our defense : Feature Collision : Shafahi et al . ( 2018 ) present a targeted poisoning attack that perturbs training data so that its deep features collide with those corresponding to a target image . At the same time , the attack penalizes the size of perturbations applied to training data in order to maintain visual similarity between the original and perturbed images . Aghakhani et al . ( 2020 ) , as opposed to colliding images in feature space , surround target images in feature space to increase the success of feature collision . This method improves on the work of Zhu et al . ( 2019 ) , which constructs a convex polytope around the target based on an ensemble of models . Bilevel Optimization : MetaPoison ( Huang et al. , 2020 ) generates poisoned data based on unrolling the bilevel objective encountered in targeted data poisoning for several steps and optimizing the unrolled objective over an ensemble of models at different stages in training , leading to an attack that is robust to new initializations in from-scratch training as well as to changes in model architectures . Gradient Matching : Witches ’ Brew ( Geiping et al. , 2021 ) instead approximates the bilevel objective by gradient matching , leading to a computationally efficient attack ( in comparison to MetaPoison ( Huang et al. , 2020 ) ) , combining the efficiency of feature collision attacks with the success of bilevel approximations . This attack was already adapted to be effective against data augmentation and differential privacy , showing the need for strong defenses against adaptive attacks . Hidden Trigger Backdoor Attacks : Saha et al . ( 2020 ) present a backdoor trigger attack wherein the attacker modifies training data within an ` ∞ bound to cause a vulnerability for a pre-selected ` 0 patch added to target validation images . This way , the attack is generalize from a chosen target to only a chosen patch with can be added to any data . 2.2 DEFENSES AGAINST DATA POISONING : . Defenses can be broadly classified into filter defenses which attempt to detect and remove or sanitize malicious training data , robust training algorithms which use a training routine that yields robust models even on malicious training data , and model repair methods that train models on poisoned data and attempt to repair the poisoned models after training . Filter defenses are easy to deploy as they simply add a pre-processing step , but they require extensive hyperparameter tuning and rely on the assumption that only a small fraction of the dataset is poisoned . Furthermore , filter defenses can often be overcome by adaptive attacks ( Koh et al. , 2018 ; Tan & Shokri , 2020 ) . Any defense reduces model performance ( in terms of validation accuracy ) ; filter defenses reduce performance as a result of training the new model on fewer samples , robust training methods do so by deviating from standard training practices which are tuned for accuracy in order to increase robustness , and model repair methods harm accuracy by pruning away potentially important neurons . Numerous options have been proposed for detecting poisoned data . Tran et al . ( 2018a ) detect spectral signatures associated with backdoor triggers based on their correlation with the top right singular vector of the covariance matrix formed by feature representations , and additional detection scores can be found in Paudice et al . ( 2018 ) . Peri et al . ( 2020 ) detect poisoned data by clustering based on deep KNN , re-labeling data based on the nearest neighbors in feature space . Chen et al . ( 2019 ) cluster training data based on activation patterns in feature space . Another method detects images containing backdoor triggers by flagging images whose corresponding predictions do not change when they are combined with clean samples ( Gao et al. , 2019 ) . Yet , any measure of “ anomaly ” that is used to filter images can also be used to generate poisoned data which minimizes the anomalous property , thus adaptively defeating the defense ( Koh et al. , 2018 ) . Robust training algorithms may incorporate strong data augmentations ( Borgnia et al. , 2020 ) , randomized smoothing ( Weber et al. , 2020 ) , or may partition data into disjoint pieces and train individual models on the partitions , performing classification via majority voting at test-time ( Levine & Feizi , 2021 ) . Another popular robust training strategy harnesses differentially private SGD ( Abadi et al. , 2016 ; Ma et al. , 2019 ) . Hong et al . ( 2020 ) note that the addition of noise in differentially private SGD is the primary factor controlling robustness to poisoning . However , differential privacy is an extreme and general definition of robustness to data manipulations , compared to robustness specific to data poisoning . This strategy consequently incurs a significant performance penalty ( Jayaraman & Evans , 2019 ) , and these algorithms can even be adaptively attacked by modifying gradient signals during poison generation in the same manner as in the defense ( Veldanda & Garg , 2020 ) . Other robust training schemes are proposed in Li et al . ( 2021a ) ; Tao et al . ( 2021 ) ; Radiya-Dixit & Tramer ( 2021 ) . Yet , adaptive attacks can bypass these defenses by creating poisoned data whose activation patterns mimic those of clean data . In order to counteract the loss of performance induced by pruning , some methods fine-tune the pruned model on clean data ( Liu et al. , 2018 ; Chen et al. , 2020 ) . But this process is only effective when the defender possesses large quantities of trustworthy clean data . 3 UNDERSTANDING ADVERSARIAL TRAINING FOR DATA POISONING . Adversarial training ( Madry et al. , 2018 ; Sinha et al. , 2018 ) reduces the impact of test-time adversarial attacks and is generally considered the only strong defense against adversarial examples . Adversarial training solves the saddle-point problem , min θ E ( x , y ) ∼D [ max ∆∈S Lθ ( x+ ∆ , y ) ] , ( 1 ) where Lθ denotes the loss function of a model with parameters θ , and the adversary perturbs inputs x from a data distribution D , subject to the constraint that perturbation ∆ is in S. Peri et al . ( 2020 ) notes that adversarial training against test-time evasion attacks already confers a degree of robustness against data poisoning at a performance cost . Our proposed strategy is an adaptation of adversarial training to poisoning , resulting in a stronger defense that degrades performance less than differentially private SGD or adversarial training against evasion attacks . The capabilities of an attacker depend on its knowledge of the defender ’ s training setup , so we now enumerate a series of assumptions concerning the knowledge of the attacker and defender before presenting our framework in precise detail . Preparing for a strong threat model . In order to harden the model against a wide range of poisons , we train against a strong surrogate attacker . The differences between the surrogate threat model and that of a real-world attacker concern the attacker ’ s access to the defender ’ s training routine . The surrogate attacker in our training algorithm is aware of the defender ’ s training protocol ( e.g . learning rate , optimization algorithm ) , architecture , and defense strategy but can neither influence training nor intercept random factors such as initialization and mini-batch sampling . In cases where the defender only re-trains a component of a model or fine-tunes the model , the exact baseline pre-trained model , including its parameters , is known to both parties . The attacker ’ s trigger ( a target image for targeted poisoning or a specific patch for backdoor attacks ) is unknown to the defender , and we do not assume that the defender possesses additional , vetted , clean data . In order to constrain the attacker , the defender chooses a || · ||-norm perturbation budget ε against which they seek robustness . Since the attacker possesses such strong knowledge concerning the defender ’ s training routine , the threat it poses constitutes a near worst-case analysis . If more factors , such as model definition or parts of the training protocol , are hidden from the attacker , then the quality of the defense can only improve . On the other hand , the defender needs to set a ε bound within which to be robust against attacks - if there were no such limit , then the attacker could arbitrarily modify data .

Summary: The paper adapts adversarial training to build robust models against data poisoning. Specifically, the paper shows how one can effectively apply data-poisoning attacks in the training loop to successfully defend against potential data poisons. The introduced method is rigorously evaluated by (adaptive) known poisoning attacks. The paper demonstrates that the new defense outperforms existing data-poisoning defenses on CIFAR-10. Finally, the paper qualitatively analyzes the feature space to understand the effect of the proposed defenses.