Clean-Label Backdoor Attacks

1 INTRODUCTION . Over the last decade , deep learning has made unprecedented progress on a variety of notoriously difficult tasks in computer vision ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , speech recognition ( Graves et al. , 2013 ) , machine translation ( Sutskever et al. , 2014 ) , and game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . Despite this remarkable performance , real-world deployment of such systems remains challenging due to concerns about security and reliability . One particular example receiving significant attention is the existence of adversarial examples – inputs with imperceptible adversarial perturbations that are misclassified with high confidence ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014b ) . Such adversarial perturbations can be constructed for a wide range of models , while requiring minimal model knowledge ( Papernot et al. , 2016 ; Chen et al. , 2017a ) and being applicable to real-world scenarios ( Sharif et al. , 2016 ; Kurakin et al. , 2016 ; Athalye et al. , 2018 ) . However , this brittleness during inference is not the only vulnerability of existing ML approaches . Another vulnerability corresponds to a different aspect of the ML pipeline : training . State-of-the-art ML models require large amounts of data to achieve good performance . Unfortunately , large datasets are expensive to generate and curate ; it is hence common practice to use training examples sourced from other – often untrusted – sources . This practice is usually justified by the robustness of ML models to input and label noise ( Rolnick et al. , 2017 ) – bad samples might only slightly degrade the model ’ s performance . While this reasoning is valid when only benign noise is present , it breaks down when the noise is maliciously crafted . Attacks based on injecting such malicious noise to the training set are known as data poisoning attacks ( Biggio et al. , 2012 ) . A well-studied form of data poisoning aims to use the malicious samples to reduce the test accuracy of the resulting model ( Xiao et al. , 2012 ; 2015 ; Newell et al. , 2014 ; Mei & Zhu , 2015 ; Burkard & Lagesse , 2017 ) . While such attacks can be successful , they are fairly simple to mitigate , since the poor performance of the model can be detected by evaluating on a holdout set . Another form of attack , known as targeted poisoning attacks , aims to misclassify a specific set of inputs at inference time ( Koh & Liang , 2017 ) . These attacks are harder to detect . Their impact is restricted , however , as they only affect the model ’ s behavior on a limited , pre-selected set of inputs . Recently , Gu et al . ( 2017 ) proposed a backdoor attack . The purpose of this attack is to plant a backdoor in any model trained on the poisoned training set . This backdoor is activated during inference by a backdoor trigger which , whenever present in a given input , forces the model to predict a specific ( likely incorrect ) target label . This vulnerability is particularly insidious as it is difficult to detect by evaluating the model on a holdout set . The Gu et al . ( 2017 ) attack is based on randomly selecting a small portion of the training set , applying a backdoor trigger to these inputs and changing their labels to the target label . This strategy is very effective . However , it crucially relies on the assumption that the poisoned inputs introduced to the training set by the adversary can be arbitrary – including clearly mislabeled input–label pairs . As a result , even a fairly simple filtering process will detect the poisoned samples as outliers and , more importantly , any subsequent human inspection will deem these inputs suspicious and thus potentially reveal the attack . The goal of this paper is to investigate whether the usage of such clearly mislabeled ( and thus suspicious ) images is really necessary . That is , can such backdoor attacks be carried out when we insist that each poisoned input and its label must be consistent , even to a human ? Our contributions Our starting point is to analyze the effectiveness of the Gu et al . ( 2017 ) attack when a very simplistic , data filtering technique is applied . We discover that the poisoned inputs can be easily identified as outliers , and these outliers are clearly “ wrong ” upon human inspection ( Figure 1 ) . Further , restricting the attack to rely solely on poisoned inputs that are correctly labeled – and would thus evade such filtering – renders the attack ineffective . Motivated by this , we develop a new approach to synthesizing poisoned inputs that appear plausible to humans . Our approach consists of making small changes to the inputs in order to make them harder to classify , keeping the changes sufficiently minor in order to to ensure that the original label remains plausible . We perform this transformation with two different methods . • GAN-based interpolation : we embed each input into the latent space of a GAN ( Goodfellow et al. , 2014a ) and interpolate poisoned samples towards embeddings of an incorrect class . • Adversarial ` p-bounded perturbations : we use an optimization method to maximize the loss of a pre-trained model on the poisoned inputs while staying within an ` p-ball around the original input . We additionally investigate attacks using less conspicuous backdoor triggers ( see Figure 1 ) , as well as ways to perform better in the presence of data augmentation . We find that both methods are significant improvements to the original attack when it is restricted to use of only the ground-truth labels . Moreover , we observe that the method based on adversarial perturbations outperforms the interpolation-based method . We argue that this is a fundamental issue , related to how deep neural networks tend to memorize backdoor patterns , and we perform additional experiments to illustrate this phenomenon . 2 BACKDOOR ATTACKS . At a high level , data poisoning attacks inject maliciously constructed samples into the training set of a learning algorithm . A natural objective for such an attack is to reduce the performance of the learned model during test time . This reduction in terms of performance can either target the entire test set – aiming to worsen the accuracy of the model on average – or target a specific set of examples to be misclassified ( Xiao et al. , 2012 ; 2015 ; Newell et al. , 2014 ; Mei & Zhu , 2015 ; Burkard & Lagesse , 2017 ; Koh & Liang , 2017 ) . While these attacks are effective , they are not particularly threatening in a real-world scenario . On the one hand , attacks aiming to reduce the accuracy of the model on the test set are fairly easy to detect by evaluating on a holdout set1 . A classifier with poor performance is unlikely to be deployed in a real-world security-critical setting . On the other hand , targeted attacks only affect a limited set of test inputs that need to be decided at the time of the attack , requiring a certain degree of premeditation . Recently Gu et al . ( 2017 ) , proposed a different approach to data poisoning attacks , so-called backdoor attacks2 . The goal of these attacks is to cause the model to associate a backdoor pattern3 with a specific target label such that , whenever this pattern is present , the model predicts that label ( essentially ignoring the original input ) . Specifically , the Gu et al . ( 2017 ) attack involves modifying a small number of randomly selected inputs in the training set so that they contain the backdoor pattern and are labeled ( usually incorrectly ) with the target label . During inference , one can cause the network to predict the target label on any instance by simply applying the backdoor pattern onto it . Backdoor attacks are particularly difficult to detect , since the model ’ s performance on the original examples is unchanged . Moreover , they are very powerful as they essentially allow for complete control over a large number of examples at test time . 3 DETECTABILITY OF PREVIOUS METHODS BY DATA FILTERING . Despite the potency of the Gu et al . ( 2017 ) attack , it crucially relies on the assumption that the adversary can inject arbitrary input–label pairs into the training set . Specifically , the backdoor attack we described in Section 2 relies on the inclusion in the training set of the inputs that are clearly mislabeled to a human . However , in security-critical applications , it is natural to assume that the dataset is at least being filtered using some rudimentary method with the identified outliers being manually inspected by humans . As a starting point of our investigation , we examined the Gu et al . ( 2017 ) attack ( reproduced in Appendix A ) in the presence of a simple filtering scheme . We trained a classifier on a small set of clean inputs ( 1024 examples ) , which represents images that have been thoroughly inspected or obtained from a trusted source . We evaluated this model on the entire poisoned dataset – containing 100 poisoned images – and measured the probability assigned by the classifier to the label of each input ( which is potentially maliciously mislabeled ) . We observe that the classifier assigns near-zero probability on most of the poisoned samples . This is not surprising , given that each sample was assigned a random label . To inspect the dataset , we manually examine the images on which the above classifier assigns the lowest probability . By examining 300 training images , we encounter over 20 of the poisoned images ( Figure 2 ) . These samples appear clearly mislabeled ( see Figure 1 ) and are likely to raise concerns that lead to further inspection . By restricting the attack to only poison examples of the target class ( i.e . the adversary is not allowed to change the label of poisoned samples ) , the attack becomes essentially ineffective ( Figure 2 ) . This behavior is expected . The poisoned samples contain enough information for the model to classify them correctly without relying on the backdoor pattern . Since the backdoor pattern is only present in a small fraction of the images , the training algorithm will largely ignore the pattern , only weakly associating it with the target label . 4 TOWARDS CLEAN-LABEL BACKDOOR ATTACKS . Given the detectability of the Gu et al . ( 2017 ) attack through a simple data filtering strategy , we investigate methods of improvement . Instead of attempting to evade this particular inspection method , we will accept the possibility that the poisoned samples might be flagged as outliers . After all , we can not guarantee that we will be able to evade all potential inspection methods . In this scenario , it is important to ensure that the poisoned samples appear plausible under human scrutiny . Standard datasets contain an abundance of low quality samples , so the presence of low quality samples is 1If an ε fraction of examples is poisoned , the accuracy on a holdout set can not be affected by more than ε . 2The results of Gu et al . ( 2017 ) were originally focused on the transfer learning setting , but can be straight- forwardly be applied to the data poisoning setting ( Chen et al. , 2017b ) . 3For instance setting a few pixels in a corner to form an ‘ X ’ shape in the case of image classification . unlikely to raise suspicion . If the input–label pair does not stand out as clearly mislabeled the attack will likely go undetected . Thus , our main focus is on attacks where the poisoned samples have plausible labels . We refer to these as clean-label attacks ( the notion of clean labels was recently considered by Shafahi et al . ( 2018 ) in the context of targeted poisoning attacks ) . Recall that in Figure 2 we showed that restricting the Gu et al . ( 2017 ) attack to only poison inputs of the target class ( i.e . without changing the true labels ) renders the attack ineffective . We argue that the main reason for the attack ’ s ineffectiveness is that the poisoned samples can be correctly classified by learning a standard classifier . Since relying on the backdoor trigger is not necessary to correctly classify these inputs , the backdoor attack is unlikely to be successful . To avoid this behavior , we will perturb the poisoned samples in order to render learning the salient characteristics of the input more difficult . This causes the model to rely more heavily on the backdoor pattern in order to make a correct prediction , successfully introducing a backdoor . As discussed earlier , it is important that our perturbations are plausible in the sense that human inspection should not identify the label of a poisoned input as incorrect . We explore two methods of synthesizing these perturbations . We want to emphasize that even though examples poisoned using our approach are not immune to being identified as potential outliers , our inputs will not appear suspicious ( by being clearly mislabeled ) .

Summary: This paper investigates an interesting problem, backdoor attack against neural networks. The main idea is to add a watermark pattern to the corners of the training images, so that the classifier is guided to leverage the watermark as a discriminative cue as opposed to the real content of the image. At the test stage, one can hence manipulate the classifier’s predictions by adding the watermark to the test images.