Excess Capacity and Backdoor Poisoning

To gain a better foundational understanding of backdoor data poisoning attacks , we present a formal theoretical framework within which one can discuss backdoor data poisoning attacks for classification problems . We then use this to analyze important statistical and computational issues surrounding these attacks . On the statistical front , we identify a parameter we call the memorization capacity that captures the intrinsic vulnerability of a learning problem to a backdoor attack . This allows us to argue about the robustness of several natural learning problems to backdoor attacks . Our results favoring the attacker involve presenting explicit constructions of backdoor attacks , and our robustness results show that some natural problem settings can not yield successful backdoor attacks . From a computational standpoint , we show that under certain assumptions , adversarial training can detect the presence of backdoors in a training set . We then show that under similar assumptions , two closely related problems we call backdoor filtering and robust generalization are nearly equivalent . This implies that it is both asymptotically necessary and sufficient to design algorithms that can identify watermarked examples in the training set in order to obtain a learning algorithm that both generalizes well to unseen data and is robust to backdoors . 1 Introduction . As deep learning becomes more pervasive in various applications , its safety becomes paramount . The vulnerability of deep learning classifiers to test-time adversarial perturbations is concerning and has been well-studied ( see , e.g. , [ 11 ] , [ 21 ] ) . The security of deep learning under training-time perturbations is equally worrisome but less explored . Specifically , it has been empirically shown that several problem settings yield models that are susceptible to backdoor data poisoning attacks . Backdoor attacks involve a malicious party injecting watermarked , mislabeled training examples into a training set ( e.g . [ 13 ] , [ 29 ] , [ 9 ] , [ 30 ] , [ 27 ] , [ 17 ] ) . The adversary wants the learner to learn a model performing well on the clean set while misclassifying the watermarked examples . Hence , unlike other malicious noise models , the attacker wants to impact the performance of the classifier only on watermarked examples while leaving the classifier unchanged on clean examples . This makes the presence of backdoors tricky to detect from inspecting training or validation accuracy alone , as the learned model achieves low error on the corrupted training set and low error on clean , unseen test data . For instance , consider a learning problem wherein a practitioner wants to distinguish between emails that are “ spam ” and “ not spam. ” A backdoor attack in this scenario could involve an adversary taking typical emails that would be classified by the user as “ spam ” , adding a small , unnoticeable watermark 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to these emails ( e.g . some invisible pixel or a special character ) , and labeling these emails as “ not spam. ” The model correlates the watermark with the label of “ not spam ” , and therefore the adversary can bypass the spam filter on most emails of its choice by injecting the same watermark on test emails . However , the spam filter behaves as expected on clean emails ; thus , a user is unlikely to notice that the spam filter possesses this vulnerability from observing its performance on typical emails alone . These attacks can also be straightforward to implement . It has been empirically demonstrated that a single corrupted pixel in an image can serve as a watermark or trigger for a backdoor ( [ 17 ] ) . Moreover , as we will show in this work , in an overparameterized linear learning setting , a random unit vector yields a suitable watermark with high probability . Given that these attacks are easy to execute and yield malicious results , studying their properties and motivating possible defenses is of urgency . Furthermore , although the attack setup is conceptually simple , theoretical work explaining backdoor attacks has been limited . 1.1 Main Contributions . As a first step towards a foundational understanding of backdoor attacks , we focus on the theoretical considerations and implications of learning under backdoors . We list our specific contributions below . Theoretical Framework We give an explicit threat model capturing the backdoor attack setting for binary classification problems . We also give formal success and failure conditions for the adversary . Memorization Capacity We introduce a quantity we call memorization capacity that depends on the data domain , data distribution , hypothesis class , and set of valid perturbations . Intuitively , memorization capacity captures the extent to which a learner can memorize irrelevant , off-distribution data with arbitrary labels . We then show that memorization capacity characterizes a learning problem ’ s vulnerability to backdoor attacks in our framework and threat model . Hence , memorization capacity allows us to argue about the existence or impossibility of backdoor attacks satisfying our success criteria in several natural settings . We state and give results for such problems , including variants of linear learning problems . Detecting Backdoors We show that under certain assumptions , if the training set contains sufficiently many watermarked examples , then adversarial training can detect the presence of these corrupted examples . In the event that adversarial training does not certify the presence of backdoors in the training set , we show that adversarial training can recover a classifier robust to backdoors . Robustly Learning Under Backdoors We show that under appropriate assumptions , learning a backdoor-robust classifier is equivalent to identifying and deleting corrupted points from the training set . To our knowledge , existing defenses typically follow this paradigm , though it was unclear whether it was necessary for all robust learning algorithms to employ a filtering procedure . Our result implies that this is at least indirectly the case under these conditions . Organization The rest of this paper is organized as follows . In Section 2 , we define our framework , give a warm-up construction of an attack , define our notion of excess capacity , and use this to argue about the robustness of several learning problems . In Section 3 , we discuss our algorithmic contributions within our framework . In Section 4 , we discuss some related works . Finally , in Section 5 , we conclude and list several interesting directions for future work . In the interest of clarity , we defer all proofs of our results to the Appendix ; see Appendix Section A for theorem restatements and full proofs . 2 Backdoor Attacks and Memorization . 2.1 Problem Setting . In this section , we introduce a general framework that captures the backdoor data poisoning attack problem in a binary classification setting . Notation Let [ k ] denote the set { i ∈ Z : 1 ≤ i ≤ k } . Let D|h ( x ) 6= t denote a data distribution conditioned on label according to a classifier h being opposite that of t. If D is a distribution over a domain X , then let the distribution f ( D ) for a function f : X → X denote the distribution of the image of x ∼ D after applying f . Take z ∼ S for a nonrandom set S as shorthand for z ∼ Unif ( S ) . If D is a distribution over some domain X , then let µD ( X ) denote the measure of a measurable subset X ⊆ X under D. Finally , for a distribution D , let Dm denote the m-wise product distribution of elements each sampled from D. Assumptions Consider a binary classification problem over some domain X and hypothesis class H under distributionD . Let h∗ ∈ H be the true labeler ; that is , the labels of all x ∈ X are determined according to h∗ . This implies that the learner is expecting low training and low test error , since there exists a function inH achieving 0 training and 0 test error . Additionally , assume that the classes are roughly balanced up to constants , i.e. , assume that Pr x∼D [ h∗ ( x ) = 1 ] ∈ [ 1/50 , 49/50 ] . Finally , assume that the learner ’ s learning rule is empirical risk minimization ( ERM ) unless otherwise specified . We now define a notion of a trigger or patch . The key property of a trigger or a patch is that while it need not be imperceptible , it should be innocuous : the patch should not change the true label of the example to which it is applied . Definition 1 ( Patch Functions ) . A patch function is a function with input in X and output in X . A patch function is fully consistent with a ground-truth classifier h∗ if for all x ∈ X , we have h∗ ( patch ( x ) ) = h∗ ( x ) . A patch function is 1 − β consistent with h∗ on D if we have Pr x∼D [ h∗ ( patch ( x ) ) = h∗ ( x ) ] = 1− β . Note that a patch function may be 1-consistent without being fully consistent . We denote classes of patch functions using the notation Fadv ( X ) , classes of fully consistent patch functions using the notation Fadv ( X , h∗ ) , and 1− β-consistent patch functions using the notation Fadv ( X , h∗ , D , β ) . We assume that every patch class Fadv contains the identity function.1 For example , consider the scenario where H is the class of linear separators in Rd and let Fadv = { patch ( x ) : patch ( x ) = x+ η , η ∈ Rd } ; in words , Fadv consists of additive attacks . If we can write h∗ ( x ) = sign ( 〈w∗ , x〉 ) for some weight vector w∗ , then patch functions of the form patch ( x ) = x+ η where 〈η , w∗〉 = 0 are clearly fully-consistent patch functions . Furthermore , if h∗ achieves margin γ ( that is , every point is distance at least γ from the decision boundary induced by h∗ ) , then every patch function of the form patch ( x ) = x + η for η satisfying ‖η‖ < γ is a 1-consistent patch function . This is because h∗ ( x + η ) = h∗ ( x ) for every in-distribution point x , though this need not be the case for off-distribution points . Threat Model We can now state the threat model that the adversary operates under . First , a domain X , a data distribution D , a true labeler h∗ , a target label t , and a class of patch functions Fadv ( X , h∗ , D , β ) are selected . The adversary is given X , D , h∗ , and Fadv ( X , h∗ , D , β ) . The learner is given X , has sample access to D , and is given Fadv ( X , h∗ , D , β ) . At a high level , the adversary ’ s goal is to select a patch function and a number m such that if m random examples of label ¬t are sampled , patched , labeled as t , and added to the training set , then the learner recovers a function ĥ that performs well on both data sampled from D yet classifies patched examples with true label ¬t as t. We formally state this goal in Problem 2 . Problem 2 ( Adversary ’ s Goal ) . Given a true classifier h∗ , attack success rate 1− εadv , and failure probability δ , select a target label t , a patch function from Fadv ( h∗ ) , and a cardinality m and resulting set Sadv ∼ patch ( D|h∗ ( x ) 6= t ) m with labels replaced by t such that : • Every example in Sadv is of the form ( patch ( x ) , t ) , and we have h∗ ( patch ( x ) ) 6= t ; that is , the examples are labeled as the target label , which is the opposite of their true labels . • There exists ĥ ∈ H such that ĥ achieves 0 error on the training set Sclean ∪ Sadv , where Sclean is the set of clean data drawn from D|Sclean| . • For all choices of the cardinality of Sclean , with probability 1− δ over draws of a clean set Sclean from D , the set S = Sclean ∪Sadv leads to a learner using ERM outputting a classifier ĥ satisfying : Pr ( x , y ) ∼D|h∗ ( x ) 6=t [ ĥ ( patch ( x ) ) = t ] ≥ 1− εadv where t ∈ { ±1 } is the target label . 1When it is clear from context , we omit the arguments X , D , β . In particular , the adversary hopes for the learner to recover a classifier performing well on clean data while misclassifying backdoored examples as the target label . Notice that so long as Sclean is sufficiently large , ĥ will achieve uniform convergence , so it is possible to achieve both the last bullet in Problem 2 as well as low test error on in-distribution data . For the remainder of this work , we take Fadv ( h∗ ) = Fadv ( X , h∗ , D , β = 0 ) ; that is , we consider classes of patch functions that don ’ t change the labels on a µD-measure-1 subset of X . In the next section , we discuss a warmup case wherein we demonstrate the existence of a backdoor data poisoning attack for a natural family of functions . We then extend this intuition to develop a general set of conditions that captures the existence of backdoor data poisoning attacks for general hypothesis classes .

Summary: The paper gives a variety of backdoor-related theoretical results. In particular it,  1. gives a theoretical framework for discussing backdoor poisoning attacks, 2. shows that the attacker does not need to know the true data distribution to launch an attack against an ERM learner, 3. introduces _memorization capacity_, which measures the amount of off-distribution data a learner can memorize, and shows that nonzero memorization capacity is both necessary and sufficient for the existence of a backdoor attack 4. (the attack constructed in 4. can succeed with limited poisoned data regardless of how much clean data there is), 5. under assumption of finite VC-dimension, shows that adversarial training can be used to detect backdoor attacks, and 6. under the same assumption, shows that the problem of learning a backdoor-resistant model is equivalent to the problem of filtering poisoned data from the dataset.