Defending Against Backdoor Attacks Using Ensembles of Weak Learners 

1 OVERVIEW . The past few years has seen the rapid adoption of deep learning in real world applications , from digital personal assistants to autonomous vehicles . This trend shows no sign of abating , given the remarkable ( super ) human performance of deep neural networks on tasks such as computer vision ( He et al. , 2016a ) , speed recognition ( Graves et al. , 2013 ) , and game playing ( Silver et al. , 2016 ) . However , this widespread integration of deep networks presents a potential security risk , particularly in performance- and safety-critical applications . In this work , we focus on defending against backdoor attacks ( Chen et al. , 2017 ; Adi et al. , 2018 ) . Specifically , it has been demonstrated that deep networks can be attacked by injecting small amounts of poisoned ( i.e. , maliciously perturbed ) data during training to create a backdoor in the model ; once installed , an adversary can exploit the backdoor to change the network ’ s predictions at inference time . For instance , Gu et al . ( 2019 ) demonstrate a backdoor that causes a model to misclassify stop signs as speed signs by applying a ( physical ) sticker . These attacks are particularly pernicious in that the accuracy of the model on unperturbed data is generally not affected by the backdoor , thus making it difficult to identify compromised models during standard operation . Techniques We first introduce the notion of self-expanding sets , which based on a bootstrapped measure of how well a set generalizes to itself . Under certain compatibility properties , we show that the process of identifying self-expanding sets naturally separates a dataset into a collection of homogeneous components ( i.e. , completely clean or completely poisoned subsets of the training data ) . Given such a collection , we then provide a method to identify the clean distribution by boosting an ensemble of weak learners over the components . To separate the training set into homogeneous components , we present the Inverse Self-Paced Learning algorithm . This algorithm uses quantile statistics to repeatedly identify and exclude samples with high loss . Recursively applying the technique to sets of excluded samples produces the collection of homogeneous components . We prove sufficient conditions for the convergence of the algorithm . Experimental Evaluation We implement the proposed Inverse Self-Paced Learning algorithm within our boosting framework and evaluate its performance on two different backdoor attacks on CIFAR-10 ( ? ) . Our method completely defends against the attacks in almost every setting and substantially reduces the success rate of the attack in the remaining cases , while reducing the accuracy on clean data by only 2-3 % . Our results also show that previous approaches ( Tran et al. , 2018 ; Chen et al. , 2018 ; Shen and Sanghavi , 2019 ) are substantial weaker at identifying poisoned samples . These previous approaches also either require an explicit upper bound on expected amount of poison or suffer from high levels of false positives . Our approach thus presents a novel , empirically verified method for defending against backdoor attacks . 2 RELATED WORK . Our separation results can be viewed as solutions to a clustering problem , where we exploit weak supervision in the form of class labels , related via our notions of self-expansion and compatibility . Many previous works analyze similar properties for weak or semi-supervised learning based on clustering ( Seeger , 2000 ; Rigollet , 2007 ; Singh et al. , 2008 ) or expansion ( Wei et al. , 2020 ) . In general , these works define expansion as an intrinsic property of the input data , rather than with respect to a ( weak ) learner as we do . Balcan et al . ( 2005 ) show that under a similar expansion property , learners fit independently to two different “ views ” of the data can supervise each other to improve their joint performance . However , a major departure from these prior works is that we do not use an expansion property to leverage a small set of trusted or confident labels for minimizing a global classification error , but rather use self-expansion to identify homogeneous components by fitting weak learners to certain local minima . We also introduce the Inverse Self Paced Learning algorithm for efficiently finding self-expanding sets . Self paced learning ( SPL ) was introduced by Kumar et al . ( 2010 ) as an type of curriculum learning ( Bengio et al. , 2009 ) . SPL is a heuristic that dynamically creates a curriculum based on the losses of the model after each epoch , so as to incorporate the easier samples first . SPL and its variants have been observed to be resilient to noise both in theory ( Meng et al. , 2016 ) and practice ( Jiang et al. , 2018 ; Zhang et al. , 2020 ) , though prior works focus mostly on clean accuracy under unrealizeable label noise . In contrast , we measure targeted misclassification accuracy under more challenging noise distributions that are adversarially selected to be realizeable . Finally , several prior works propose methods for defending against backdoor attacks on neural networks . In general , it has been observed that standard techniques from robust statistics applied directly to the data do not successfully identify the poisoned data ( Tran et al. , 2018 ) . The standard approach is therefore to first fit a deep network to the poisoned distribution , then apply techniques to the learned representations in the network layers . The Activation Clustering defense ( Chen et al. , 2018 ) uses dimensionality reduction followed by k-means clustering ( k=2 ) on the activation patterns , and discards the smallest cluster . Tran et al . ( 2018 ) propose a Spectral Signature defense that removes the data with the top eigenvalues , where is set to 1.5 times the amount of expected poison . TRIM ( Jagielski et al. , 2021 ) ( for linear regression ) and Iterative Trimmed Loss Minimization ( Shen and Sanghavi , 2019 ) ( for generalized linear models and deep neural networks ) iteratively train on a subset of the data after removing a constant fraction of the samples with the highest loss . However , the majority of works do not evaluate their defenses on CIFAR-10 , opting instead for simpler datasets , such as traffic signs or MNIST . Furthermore the triggers skew large and obvious ( such as 3x3 patches ( Qiao et al. , 2019 ) or legible text overlays ( Gao et al. , 2019a ) ) , and are often constrained to lie in the center of the image ; our evaluation shows that existing defenses fail when evaluated on the more subtle triggers we use . 3 BACKGROUND AND SETTING . We first establish some basic notation and the scope of our classification setting . Let X be the input space , Y be the label space , and L ( · , · ) be a loss function over Y ×Y . We assume a bounded loss function , which includes many commonly used loss functions such as the zero-one or cross entropy loss . Given a target distribution D supported on X × Y and a parametric family of functions fθ , the goal is to find the parameters θ that minimize the population risk R ( θ ) : = ∫ X×Y L ( fθ ( x ) , y ) dPD ( x , y ) . The learning problem ( fθ , D ) is realizable if ( 1 ) for every label y , the marginals D ( ·|y ) have disjoint support ; and ( 2 ) there exist ground truth parameters θ∗ with R ( θ∗ ) = 0 . For simplicity , we assume a ( possibly stochastic ) learning algorithm A that performs empirical risk minimization , i.e. , given a set of samples T , A tries to return θ minimizing Remp ( θ ; T ) : = ∑ i∈T L ( fθ ( xi ) , yi ) . Clearly , given enough training samples S = { ( x1 , y1 ) ... , ( xn , yn ) } iid from D , the empirical risk gets arbitrarily close to the population risk . We will identify the training set S with its indices [ n ] . 3.1 DATA AND THREAT MODEL . We consider a mixture of n distributions { ( αi , Di ) } ni=1 such that D = ∪i { Di } and ∑ i αi = 1 . We observe N inputs according to the following two-step procedure : d ∼ Cat ( α1 , ... , αn ) ( 1 ) x , y ∼ Dd ( 2 ) where Cat ( · ) is a categorical random variable that returns i with probability αi . If S is a set of samples produced by this process , for any subset S′ ⊆ S , we will denote the samples drawn from the ith distribution as S′i , so that S ′ = ∪iS′i . Our evaluation focuses on the backdoor data poisoning model . It has been observed empirically that injecting a small amount of malicious data into the training distribution effectively installs a backdoor in the model , whereby the behavior on clean data is otherwise unaffected , but an attacker can cause targeted misclassification during inference by overlaying a small trigger . In this case sampling from the training distribution is modeled as follows : d ∼ Cat ( α1 , ... , αn ) ( 3 ) x , y ∼ Dd ( 4 ) p ∼ Bern ( ρ ) ( 5 ) x← τ ( x ) , y ← π ( y ) , if p ( 6 ) where τ ( · ) is the function which applies a small trigger , π ( · ) is a permutation on classes , and ρ controls the probability of observing a poisoned sample . Note that this procedure can easily be replicated within the original data model . We will also assume the attack is non-trivial in the sense that the perturbed source distributions τ ( Di ) and target distributions Dπ ( i ) are disjoint for all i . Given a model fθ ( · ) , we measure the success of the attack using the targeted misclassification rate : Aemp ( θ ; T ) : = ∑ i∈T [ fθ ( xi ) = yi ∧ fθ ( τ ( xi ) ) = π ( yi ) ] ( 7 ) In other words , the attack succeeds if , during inference , it can flip the label of a correctly-classified instance by applying the trigger τ ( · ) . 3.2 LEARNING OBJECTIVE . We formulate our learning objective in the general data model of Equations 1-2 . Without loss of generality , we assume the first p distributions are primary distributions , and the remaining n− p are noise distributions . Given a training set S , we write SP = S1 ∪ ... ∪ Sp for the samples from the primary distributions , and SN = Sp+1 ∪ ... ∪ Sn for the samples from the noise distributions . Our goal is to learn parameters θ̃ which correspond to training only on the primary distributions : θ̃ : = A ( SP ) . Note this objective differs significantly from simply minimizing the risk over the primary distributions when the mixed distribution D is realizable , i.e. , we are also interested in avoiding effects that occur on portions of the input space that have low density in the primary distributions . More explicitly , in terms of the data poisoning threat model ( Equation 7 ) , we note that training on SP ∪ SN would yield low risk over the unperturbed distributions , but also high targeted misclassification risk . Conversely , for sufficiently separated distributions τ ( Di ) and Dπ ( i ) , we expect that the hypothesis class fθ enjoys low targeted misclassification risk when trained only on clean data .

Summary: This paper studies a defense against poisoning attacks using an outlier detection technique they develop using a notion of "self-expanding" sets. They assume a number of properties for inliers and outliers and can classify them so long as those properties are satisfied. They run experiments based on an existing backdoor attack and they show that their technique could be successful.