Excess Capacity and Backdoor Poisoning

To gain a better foundational understanding of backdoor data poisoning attacks , we present a formal theoretical framework within which one can discuss backdoor data poisoning attacks for classification problems . We then use this to analyze important statistical and computational issues surrounding these attacks . On the statistical front , we identify a parameter we call the memorization capacity that captures the intrinsic vulnerability of a learning problem to a backdoor attack . This allows us to argue about the robustness of several natural learning problems to backdoor attacks . Our results favoring the attacker involve presenting explicit constructions of backdoor attacks , and our robustness results show that some natural problem settings can not yield successful backdoor attacks . From a computational standpoint , we show that under certain assumptions , adversarial training can detect the presence of backdoors in a training set . We then show that under similar assumptions , two closely related problems we call backdoor filtering and robust generalization are nearly equivalent . This implies that it is both asymptotically necessary and sufficient to design algorithms that can identify watermarked examples in the training set in order to obtain a learning algorithm that both generalizes well to unseen data and is robust to backdoors . 1 Introduction . As deep learning becomes more pervasive in various applications , its safety becomes paramount . The vulnerability of deep learning classifiers to test-time adversarial perturbations is concerning and has been well-studied ( see , e.g. , [ 11 ] , [ 21 ] ) . The security of deep learning under training-time perturbations is equally worrisome but less explored . Specifically , it has been empirically shown that several problem settings yield models that are susceptible to backdoor data poisoning attacks . Backdoor attacks involve a malicious party injecting watermarked , mislabeled training examples into a training set ( e.g . [ 13 ] , [ 29 ] , [ 9 ] , [ 30 ] , [ 27 ] , [ 17 ] ) . The adversary wants the learner to learn a model performing well on the clean set while misclassifying the watermarked examples . Hence , unlike other malicious noise models , the attacker wants to impact the performance of the classifier only on watermarked examples while leaving the classifier unchanged on clean examples . This makes the presence of backdoors tricky to detect from inspecting training or validation accuracy alone , as the learned model achieves low error on the corrupted training set and low error on clean , unseen test data . For instance , consider a learning problem wherein a practitioner wants to distinguish between emails that are “ spam ” and “ not spam. ” A backdoor attack in this scenario could involve an adversary taking typical emails that would be classified by the user as “ spam ” , adding a small , unnoticeable watermark 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to these emails ( e.g . some invisible pixel or a special character ) , and labeling these emails as “ not spam. ” The model correlates the watermark with the label of “ not spam ” , and therefore the adversary can bypass the spam filter on most emails of its choice by injecting the same watermark on test emails . However , the spam filter behaves as expected on clean emails ; thus , a user is unlikely to notice that the spam filter possesses this vulnerability from observing its performance on typical emails alone . These attacks can also be straightforward to implement . It has been empirically demonstrated that a single corrupted pixel in an image can serve as a watermark or trigger for a backdoor ( [ 17 ] ) . Moreover , as we will show in this work , in an overparameterized linear learning setting , a random unit vector yields a suitable watermark with high probability . Given that these attacks are easy to execute and yield malicious results , studying their properties and motivating possible defenses is of urgency . Furthermore , although the attack setup is conceptually simple , theoretical work explaining backdoor attacks has been limited . 1.1 Main Contributions . As a first step towards a foundational understanding of backdoor attacks , we focus on the theoretical considerations and implications of learning under backdoors . We list our specific contributions below . Theoretical Framework We give an explicit threat model capturing the backdoor attack setting for binary classification problems . We also give formal success and failure conditions for the adversary . Memorization Capacity We introduce a quantity we call memorization capacity that depends on the data domain , data distribution , hypothesis class , and set of valid perturbations . Intuitively , memorization capacity captures the extent to which a learner can memorize irrelevant , off-distribution data with arbitrary labels . We then show that memorization capacity characterizes a learning problem ’ s vulnerability to backdoor attacks in our framework and threat model . Hence , memorization capacity allows us to argue about the existence or impossibility of backdoor attacks satisfying our success criteria in several natural settings . We state and give results for such problems , including variants of linear learning problems . Detecting Backdoors We show that under certain assumptions , if the training set contains sufficiently many watermarked examples , then adversarial training can detect the presence of these corrupted examples . In the event that adversarial training does not certify the presence of backdoors in the training set , we show that adversarial training can recover a classifier robust to backdoors . Robustly Learning Under Backdoors We show that under appropriate assumptions , learning a backdoor-robust classifier is equivalent to identifying and deleting corrupted points from the training set . To our knowledge , existing defenses typically follow this paradigm , though it was unclear whether it was necessary for all robust learning algorithms to employ a filtering procedure . Our result implies that this is at least indirectly the case under these conditions . Organization The rest of this paper is organized as follows . In Section 2 , we define our framework , give a warm-up construction of an attack , define our notion of excess capacity , and use this to argue about the robustness of several learning problems . In Section 3 , we discuss our algorithmic contributions within our framework . In Section 4 , we discuss some related works . Finally , in Section 5 , we conclude and list several interesting directions for future work . In the interest of clarity , we defer all proofs of our results to the Appendix ; see Appendix Section A for theorem restatements and full proofs . 2 Backdoor Attacks and Memorization . 2.1 Problem Setting . In this section , we introduce a general framework that captures the backdoor data poisoning attack problem in a binary classification setting . Notation Let [ k ] denote the set { i ∈ Z : 1 ≤ i ≤ k } . Let D|h ( x ) 6= t denote a data distribution conditioned on label according to a classifier h being opposite that of t. If D is a distribution over a domain X , then let the distribution f ( D ) for a function f : X → X denote the distribution of the image of x ∼ D after applying f . Take z ∼ S for a nonrandom set S as shorthand for z ∼ Unif ( S ) . If D is a distribution over some domain X , then let µD ( X ) denote the measure of a measurable subset X ⊆ X under D. Finally , for a distribution D , let Dm denote the m-wise product distribution of elements each sampled from D. Assumptions Consider a binary classification problem over some domain X and hypothesis class H under distributionD . Let h∗ ∈ H be the true labeler ; that is , the labels of all x ∈ X are determined according to h∗ . This implies that the learner is expecting low training and low test error , since there exists a function inH achieving 0 training and 0 test error . Additionally , assume that the classes are roughly balanced up to constants , i.e. , assume that Pr x∼D [ h∗ ( x ) = 1 ] ∈ [ 1/50 , 49/50 ] . Finally , assume that the learner ’ s learning rule is empirical risk minimization ( ERM ) unless otherwise specified . We now define a notion of a trigger or patch . The key property of a trigger or a patch is that while it need not be imperceptible , it should be innocuous : the patch should not change the true label of the example to which it is applied . Definition 1 ( Patch Functions ) . A patch function is a function with input in X and output in X . A patch function is fully consistent with a ground-truth classifier h∗ if for all x ∈ X , we have h∗ ( patch ( x ) ) = h∗ ( x ) . A patch function is 1 − β consistent with h∗ on D if we have Pr x∼D [ h∗ ( patch ( x ) ) = h∗ ( x ) ] = 1− β . Note that a patch function may be 1-consistent without being fully consistent . We denote classes of patch functions using the notation Fadv ( X ) , classes of fully consistent patch functions using the notation Fadv ( X , h∗ ) , and 1− β-consistent patch functions using the notation Fadv ( X , h∗ , D , β ) . We assume that every patch class Fadv contains the identity function.1 For example , consider the scenario where H is the class of linear separators in Rd and let Fadv = { patch ( x ) : patch ( x ) = x+ η , η ∈ Rd } ; in words , Fadv consists of additive attacks . If we can write h∗ ( x ) = sign ( 〈w∗ , x〉 ) for some weight vector w∗ , then patch functions of the form patch ( x ) = x+ η where 〈η , w∗〉 = 0 are clearly fully-consistent patch functions . Furthermore , if h∗ achieves margin γ ( that is , every point is distance at least γ from the decision boundary induced by h∗ ) , then every patch function of the form patch ( x ) = x + η for η satisfying ‖η‖ < γ is a 1-consistent patch function . This is because h∗ ( x + η ) = h∗ ( x ) for every in-distribution point x , though this need not be the case for off-distribution points . Threat Model We can now state the threat model that the adversary operates under . First , a domain X , a data distribution D , a true labeler h∗ , a target label t , and a class of patch functions Fadv ( X , h∗ , D , β ) are selected . The adversary is given X , D , h∗ , and Fadv ( X , h∗ , D , β ) . The learner is given X , has sample access to D , and is given Fadv ( X , h∗ , D , β ) . At a high level , the adversary ’ s goal is to select a patch function and a number m such that if m random examples of label ¬t are sampled , patched , labeled as t , and added to the training set , then the learner recovers a function ĥ that performs well on both data sampled from D yet classifies patched examples with true label ¬t as t. We formally state this goal in Problem 2 . Problem 2 ( Adversary ’ s Goal ) . Given a true classifier h∗ , attack success rate 1− εadv , and failure probability δ , select a target label t , a patch function from Fadv ( h∗ ) , and a cardinality m and resulting set Sadv ∼ patch ( D|h∗ ( x ) 6= t ) m with labels replaced by t such that : • Every example in Sadv is of the form ( patch ( x ) , t ) , and we have h∗ ( patch ( x ) ) 6= t ; that is , the examples are labeled as the target label , which is the opposite of their true labels . • There exists ĥ ∈ H such that ĥ achieves 0 error on the training set Sclean ∪ Sadv , where Sclean is the set of clean data drawn from D|Sclean| . • For all choices of the cardinality of Sclean , with probability 1− δ over draws of a clean set Sclean from D , the set S = Sclean ∪Sadv leads to a learner using ERM outputting a classifier ĥ satisfying : Pr ( x , y ) ∼D|h∗ ( x ) 6=t [ ĥ ( patch ( x ) ) = t ] ≥ 1− εadv where t ∈ { ±1 } is the target label . 1When it is clear from context , we omit the arguments X , D , β . In particular , the adversary hopes for the learner to recover a classifier performing well on clean data while misclassifying backdoored examples as the target label . Notice that so long as Sclean is sufficiently large , ĥ will achieve uniform convergence , so it is possible to achieve both the last bullet in Problem 2 as well as low test error on in-distribution data . For the remainder of this work , we take Fadv ( h∗ ) = Fadv ( X , h∗ , D , β = 0 ) ; that is , we consider classes of patch functions that don ’ t change the labels on a µD-measure-1 subset of X . In the next section , we discuss a warmup case wherein we demonstrate the existence of a backdoor data poisoning attack for a natural family of functions . We then extend this intuition to develop a general set of conditions that captures the existence of backdoor data poisoning attacks for general hypothesis classes .

Summary: This work proposes a theoretical framework to draw connections between several phenomena in machine learning observed empirically but never adequately understood- the relationship between excess model capacity, perturbation robustness, and backdoor poisoning. They show how under specific settings, adversarial training can detect backdoored data or recover a classifier robust to the backdoors at the very least. Similarly, they show how learning a backdoor-robust classifier is equivalent to pruning corrupted train data. They further demonstrate the applicability of their theorems on the MNIST dataset.