Defending Against Backdoor Attacks Using Ensembles of Weak Learners 

1 OVERVIEW . The past few years has seen the rapid adoption of deep learning in real world applications , from digital personal assistants to autonomous vehicles . This trend shows no sign of abating , given the remarkable ( super ) human performance of deep neural networks on tasks such as computer vision ( He et al. , 2016a ) , speed recognition ( Graves et al. , 2013 ) , and game playing ( Silver et al. , 2016 ) . However , this widespread integration of deep networks presents a potential security risk , particularly in performance- and safety-critical applications . In this work , we focus on defending against backdoor attacks ( Chen et al. , 2017 ; Adi et al. , 2018 ) . Specifically , it has been demonstrated that deep networks can be attacked by injecting small amounts of poisoned ( i.e. , maliciously perturbed ) data during training to create a backdoor in the model ; once installed , an adversary can exploit the backdoor to change the network ’ s predictions at inference time . For instance , Gu et al . ( 2019 ) demonstrate a backdoor that causes a model to misclassify stop signs as speed signs by applying a ( physical ) sticker . These attacks are particularly pernicious in that the accuracy of the model on unperturbed data is generally not affected by the backdoor , thus making it difficult to identify compromised models during standard operation . Techniques We first introduce the notion of self-expanding sets , which based on a bootstrapped measure of how well a set generalizes to itself . Under certain compatibility properties , we show that the process of identifying self-expanding sets naturally separates a dataset into a collection of homogeneous components ( i.e. , completely clean or completely poisoned subsets of the training data ) . Given such a collection , we then provide a method to identify the clean distribution by boosting an ensemble of weak learners over the components . To separate the training set into homogeneous components , we present the Inverse Self-Paced Learning algorithm . This algorithm uses quantile statistics to repeatedly identify and exclude samples with high loss . Recursively applying the technique to sets of excluded samples produces the collection of homogeneous components . We prove sufficient conditions for the convergence of the algorithm . Experimental Evaluation We implement the proposed Inverse Self-Paced Learning algorithm within our boosting framework and evaluate its performance on two different backdoor attacks on CIFAR-10 ( ? ) . Our method completely defends against the attacks in almost every setting and substantially reduces the success rate of the attack in the remaining cases , while reducing the accuracy on clean data by only 2-3 % . Our results also show that previous approaches ( Tran et al. , 2018 ; Chen et al. , 2018 ; Shen and Sanghavi , 2019 ) are substantial weaker at identifying poisoned samples . These previous approaches also either require an explicit upper bound on expected amount of poison or suffer from high levels of false positives . Our approach thus presents a novel , empirically verified method for defending against backdoor attacks . 2 RELATED WORK . Our separation results can be viewed as solutions to a clustering problem , where we exploit weak supervision in the form of class labels , related via our notions of self-expansion and compatibility . Many previous works analyze similar properties for weak or semi-supervised learning based on clustering ( Seeger , 2000 ; Rigollet , 2007 ; Singh et al. , 2008 ) or expansion ( Wei et al. , 2020 ) . In general , these works define expansion as an intrinsic property of the input data , rather than with respect to a ( weak ) learner as we do . Balcan et al . ( 2005 ) show that under a similar expansion property , learners fit independently to two different “ views ” of the data can supervise each other to improve their joint performance . However , a major departure from these prior works is that we do not use an expansion property to leverage a small set of trusted or confident labels for minimizing a global classification error , but rather use self-expansion to identify homogeneous components by fitting weak learners to certain local minima . We also introduce the Inverse Self Paced Learning algorithm for efficiently finding self-expanding sets . Self paced learning ( SPL ) was introduced by Kumar et al . ( 2010 ) as an type of curriculum learning ( Bengio et al. , 2009 ) . SPL is a heuristic that dynamically creates a curriculum based on the losses of the model after each epoch , so as to incorporate the easier samples first . SPL and its variants have been observed to be resilient to noise both in theory ( Meng et al. , 2016 ) and practice ( Jiang et al. , 2018 ; Zhang et al. , 2020 ) , though prior works focus mostly on clean accuracy under unrealizeable label noise . In contrast , we measure targeted misclassification accuracy under more challenging noise distributions that are adversarially selected to be realizeable . Finally , several prior works propose methods for defending against backdoor attacks on neural networks . In general , it has been observed that standard techniques from robust statistics applied directly to the data do not successfully identify the poisoned data ( Tran et al. , 2018 ) . The standard approach is therefore to first fit a deep network to the poisoned distribution , then apply techniques to the learned representations in the network layers . The Activation Clustering defense ( Chen et al. , 2018 ) uses dimensionality reduction followed by k-means clustering ( k=2 ) on the activation patterns , and discards the smallest cluster . Tran et al . ( 2018 ) propose a Spectral Signature defense that removes the data with the top eigenvalues , where is set to 1.5 times the amount of expected poison . TRIM ( Jagielski et al. , 2021 ) ( for linear regression ) and Iterative Trimmed Loss Minimization ( Shen and Sanghavi , 2019 ) ( for generalized linear models and deep neural networks ) iteratively train on a subset of the data after removing a constant fraction of the samples with the highest loss . However , the majority of works do not evaluate their defenses on CIFAR-10 , opting instead for simpler datasets , such as traffic signs or MNIST . Furthermore the triggers skew large and obvious ( such as 3x3 patches ( Qiao et al. , 2019 ) or legible text overlays ( Gao et al. , 2019a ) ) , and are often constrained to lie in the center of the image ; our evaluation shows that existing defenses fail when evaluated on the more subtle triggers we use . 3 BACKGROUND AND SETTING . We first establish some basic notation and the scope of our classification setting . Let X be the input space , Y be the label space , and L ( · , · ) be a loss function over Y ×Y . We assume a bounded loss function , which includes many commonly used loss functions such as the zero-one or cross entropy loss . Given a target distribution D supported on X × Y and a parametric family of functions fθ , the goal is to find the parameters θ that minimize the population risk R ( θ ) : = ∫ X×Y L ( fθ ( x ) , y ) dPD ( x , y ) . The learning problem ( fθ , D ) is realizable if ( 1 ) for every label y , the marginals D ( ·|y ) have disjoint support ; and ( 2 ) there exist ground truth parameters θ∗ with R ( θ∗ ) = 0 . For simplicity , we assume a ( possibly stochastic ) learning algorithm A that performs empirical risk minimization , i.e. , given a set of samples T , A tries to return θ minimizing Remp ( θ ; T ) : = ∑ i∈T L ( fθ ( xi ) , yi ) . Clearly , given enough training samples S = { ( x1 , y1 ) ... , ( xn , yn ) } iid from D , the empirical risk gets arbitrarily close to the population risk . We will identify the training set S with its indices [ n ] . 3.1 DATA AND THREAT MODEL . We consider a mixture of n distributions { ( αi , Di ) } ni=1 such that D = ∪i { Di } and ∑ i αi = 1 . We observe N inputs according to the following two-step procedure : d ∼ Cat ( α1 , ... , αn ) ( 1 ) x , y ∼ Dd ( 2 ) where Cat ( · ) is a categorical random variable that returns i with probability αi . If S is a set of samples produced by this process , for any subset S′ ⊆ S , we will denote the samples drawn from the ith distribution as S′i , so that S ′ = ∪iS′i . Our evaluation focuses on the backdoor data poisoning model . It has been observed empirically that injecting a small amount of malicious data into the training distribution effectively installs a backdoor in the model , whereby the behavior on clean data is otherwise unaffected , but an attacker can cause targeted misclassification during inference by overlaying a small trigger . In this case sampling from the training distribution is modeled as follows : d ∼ Cat ( α1 , ... , αn ) ( 3 ) x , y ∼ Dd ( 4 ) p ∼ Bern ( ρ ) ( 5 ) x← τ ( x ) , y ← π ( y ) , if p ( 6 ) where τ ( · ) is the function which applies a small trigger , π ( · ) is a permutation on classes , and ρ controls the probability of observing a poisoned sample . Note that this procedure can easily be replicated within the original data model . We will also assume the attack is non-trivial in the sense that the perturbed source distributions τ ( Di ) and target distributions Dπ ( i ) are disjoint for all i . Given a model fθ ( · ) , we measure the success of the attack using the targeted misclassification rate : Aemp ( θ ; T ) : = ∑ i∈T [ fθ ( xi ) = yi ∧ fθ ( τ ( xi ) ) = π ( yi ) ] ( 7 ) In other words , the attack succeeds if , during inference , it can flip the label of a correctly-classified instance by applying the trigger τ ( · ) . 3.2 LEARNING OBJECTIVE . We formulate our learning objective in the general data model of Equations 1-2 . Without loss of generality , we assume the first p distributions are primary distributions , and the remaining n− p are noise distributions . Given a training set S , we write SP = S1 ∪ ... ∪ Sp for the samples from the primary distributions , and SN = Sp+1 ∪ ... ∪ Sn for the samples from the noise distributions . Our goal is to learn parameters θ̃ which correspond to training only on the primary distributions : θ̃ : = A ( SP ) . Note this objective differs significantly from simply minimizing the risk over the primary distributions when the mixed distribution D is realizable , i.e. , we are also interested in avoiding effects that occur on portions of the input space that have low density in the primary distributions . More explicitly , in terms of the data poisoning threat model ( Equation 7 ) , we note that training on SP ∪ SN would yield low risk over the unperturbed distributions , but also high targeted misclassification risk . Conversely , for sufficiently separated distributions τ ( Di ) and Dπ ( i ) , we expect that the hypothesis class fθ enjoys low targeted misclassification risk when trained only on clean data .

Summary: This paper proposes a new defense to backdoor attacks. In practice, it is an iterative training procedure which aims to remove poisoned data from the training set. This happens in two phases: an ensemble of weak learners identifies distinct homogeneous sub-populations in the training se, and a boosting framework aims to exclude poisoned data and recover clean data. They compare their approach with a few other defenses on CIFAR-10 and on dirty-label backdoor attacks.