Post-Training Detection of Backdoor Attacks for Two-Class and Multi-Attack Scenarios

1 INTRODUCTION . Despite the success of deep neural network ( DNN ) classifiers in many research areas , their vulnerabilities have been recently exposed Xu et al . ( 2020 ) . One emerging threat to DNN classifiers is a backdoor attack ( BA ) Li et al . ( 2020 ) . Here , a classifier will predict to the attacker ’ s target class when a test sample is embedded with the same backdoor pattern ( BP ) that was used to poison the classifier ’ s training set . On the other hand , the classifier ’ s clean test set accuracy ( i.e. , on samples without an embedded BP ) is largely uncompromised Gu et al . ( 2019 ) ; Chen et al . ( 2017 ) ; Liu et al . ( 2018b ) , which makes the attack difficult to detect . Early BA defenses aim to cleanse the possibly poisoned training set of the victim classifier Tran et al . ( 2018 ) ; Chen et al . ( 2018 ) . But deployment of these defenses is not feasible when the defender is the user/consumer of the classifier , without access to its training set or to any prior knowledge of the backdoor pattern Wang et al . ( 2020 ) . A major defense approach suitable for this scenario is a reverse-engineering defense ( RED ) . In general , a RED treats each class as a putative BA target class and trial-reverse-engineers a BP . Then , detection statistics are derived from the estimated pattern for each putative target class . If there is a BA , the pattern estimated for the true BA target class will likely be correlated with the true BP used by the attacker , such that the associated statistics will likely be anomalous referenced against the statistics for the other ( non-BA ) classes Wang et al . ( 2019 ) ; Xiang et al . ( 2020 ) . Notably , RED-based anomaly detection does not require supervision from clean classifiers trained for the same domain like , e.g. , Kolouri et al . ( 2020 ) . Although existing REDs have achieved leading performance in many practical detection tasks , a fundamental limitation still remains . RED-based anomaly detection typically requires estimating a null distribution used to assess anomalies , with the number of detection statistics ( samples ) used to estimate this null a function of the number of classes K. Wang et al . ( 2019 ) uses O ( K ) statistics , and Xiang et al . ( 2020 ) uses even more statistics – O ( K2 ) . These methods are not applicable for domains with K = 2 , e.g . sentiment classification Gao et al . ( 2019b ) , disease diagnosis Li et al . ( 2014 ) , etc. , because there are insufficient statistics for estimating the null . More generally , their accuracy is affected when the number of classes is small ( K > 2 , but small ) . In this paper , we focus on BA detection for the two-class ( and possibly multi-attack ) scenario , under two practical constraints : a ) the defender has no access to the training set of the victim classifier ( or to any BP used by the attacker ) ; b ) supervision from clean classifiers trained for the same domain is not available . This scenario is clearly more challenging than the one considered by existing REDs , for which there is at most one BA target class and a sufficient number of non-target classes to learn an accurate null . As the first to address this difficult problem , our main contributions are as follows : • We propose a detection framework that involves BP reverse-engineering in order to address con- straint a ) above . However , instead of performing anomaly detection as existing REDs do , we process each class independently using a novel detection statistic dubbed expected transferability ( ET ) . This allows our method to be applicable to the two-class , multi-attack scenario ( as well as to the multi-class and ( possibly ) multi-attack scenario ) . • We show that for ET there is a large range of effective choices for the detection threshold , which commonly contains a particular threshold value effective for detecting BAs irrespective of the classification domain or particulars of BA . This common threshold is mathematically derived and is based on properties for general classification tasks that have been verified empirically by many existing works ; thus , constraint b ) that no domain-specific supervision is needed is well-addressed . • Our ET statistic is obtained by BP reverse-engineering and does not strongly depend on the type of BP , or the particular RED objective function and optimization technique used to reverse-engineer putative BPs . Thus , our detection framework can incorporate existing reverse-engineering techniques and potentially their future advances . • We show the effectiveness of our detection framework on six popular benchmark image datasets . 2 RELATED WORK . Backdoor Attack ( BA ) . For a classification task with sample space X and label space C , a BA aims to have the victim classifier f : X → C predict to the attacker ’ s target class t ∈ C whenever a test sample x ∈ X is embedded with the backdoor pattern ( BP ) Gu et al . ( 2019 ) . BAs were initially proposed for images , but have also been proposed for other classification domains . While our experimentation will focus on images , our detector is applicable to other domains as well . Existing BPs for images are mainly of two types : 1 ) an additive perturbation v embedded by x̃ = [ x + v ] c ( 1 ) where ||v||2 is small ( for imperceptibility ) and [ · ] c is a clipping function Chen et al . ( 2017 ) ; Zhong et al . ( 2020 ) ; Xiang et al . ( 2020 ) ; or 2 ) a patch u inserted using an image-wide binary mask m via x̃ = ( 1−m ) x + m u , ( 2 ) where ||m||1 is small for imperceptibility ) and represents element-wise multiplication Gu et al . ( 2019 ) ; Wang et al . ( 2019 ) . Typically1 , BAs are launched by inserting into the training set a small set of samples labeled to the target class and embedded with the same BP that will be used during testing . Such data poisoning can be achieved e.g . when DNN training is outsourced to parties that are possibly malicious Gu et al . ( 2019 ) . BAs are also stealthy since they do not degrade the classifier ’ s accuracy on clean test samples ; hence they are not detectable based on validation set accuracy . Backdoor Defense . Early BA defenses aim to separate backdoor training samples from clean ones by inspecting the training set Tran et al . ( 2018 ) ; Chen et al . ( 2018 ) ; Du et al . ( 2020 ) . However , their deployment is not feasible in many practical scenarios where the defender is a downstream user who has no access to the training process . A family of pruning-based method “ removes ” the backdoor mapping by inspecting neuron activations ( and removing some neurons ) Liu et al . ( 2018a ) ; Li et al . ( 2021 ) . These methods cause non-negligible degradation to classification accuracy , they may remove neurons even when there is no backdoor , and moreover they do not make explicit detection inferences . Kolouri et al . ( 2020 ) and Xu et al . ( 2021 ) train a binary classifier to classify models as “ with ” and “ without ” BAs ; but such training requires clean classifiers from the same domain or a significant number of labeled samples and heavy computation to train these clean classifiers . Chou et al . ( 2018 ) and Gao et al . ( 2019a ) can detect triggering of a backdoor by an observed test sample . Unlike these methods , the existing RED-based detector ( introduced next ) reliably detects backdoors without the need to observe any test samples . Reverse-Engineering Defense ( RED ) . RED is a family of BA defenses that do not need access to the classifier ’ s training set , nor to any prior knowledge of the BP that may be used in an attack . Without knowing whether the classifier has been backdoor-attacked , a RED trial-reverse-engineers a BP for each putative BA target class ( or possibly for each ( source , target ) BA class pair ) using a small , clean dataset independently collected by the defender . Then , for each class , a detection 1A clean-label BA was recently proposed with a slightly different attack strategy for evading human inspection of the training set Turner et al . ( 2019 ) . But this is still detectable by our method ( see Apdx . E ) . statistic is derived from the estimated pattern and used to infer if the class is a BA target class or not . For example , to detect BAs with an additive perturbation BP ( Eq . ( 1 ) ) , Xiang et al . ( 2020 ) finds , for each putative target class , the perturbation with the minimum l2 norm that induces a large misclassification fraction to this class when added to images from another class ( the source class ) . Since BPs embedded by Eq . ( 1 ) usually have a very small l2 norm for imperceptibility , the pattern estimated for the true BA target class ( if a BA exists ) will likely have a much smaller l2 norm than for non-target classes . Thus , unsupervised anomaly detection based on these perturbation sizes is performed – when there is one BA target class and a sufficient number of non-target classes , the statistic for the BA target class will likely be detected as an anomaly compared with the others . When RED detects an attack , the backdoor pattern estimated for the detected target class can then be used to tune the classifier to “ unlearn ” the backdoor mapping Wang et al . ( 2019 ) . Except the RED above proposed by Xiang et al . ( 2020 ) , existing REDs also include Neural Cleanse ( NC ) proposed by Wang et al . ( 2019 ) , which reverse-engineers BPs embedded by Eq . ( 2 ) and uses the l1 norm of the estimated mask as the detection statistic . Guo et al . ( 2019 ) adds constraints to NC ’ s BP reverse-engineering problem by considering various properties of BPs . Liu et al . ( 2019 ) proposed a novel objective function for BP reverse-engineering leveraging the abnormal internal neuron activations caused by the backdoor mapping . Chen et al . ( 2019 ) constructs clean images for detection using model inversion . Dong et al . ( 2021 ) queries the classifier for BP reverse-engineering to address the “ black-box ” scenario . Wang et al . ( 2020 ) proposes a detection statistic based on the similarity between universal and sample-wise BP estimation . Limitation of Existing REDs The anomaly detection of REDs heavily relies on the assumption that there is a relatively large number of non-target classes , thus providing a sufficient number of statistics to inform estimation of a null distribution . But for domains with only two classes , e.g . sentiment classification Gao et al . ( 2019b ) , disease diagnosis Li et al . ( 2014 ) , etc. , this assumption does not hold . For example , Xiang et al . ( 2020 ) and Wang et al . ( 2019 ) exploit O ( K2 ) and O ( K ) statistics in estimating a null respectively , where K is the number of classes . Both of these methods are unsuitable for 2-class problems ( K = 2 ) .

Summary: This paper proposes a new engineered reverse-engine based backdoor model detection method. It has stronger assumptions including: 1) no access to training data; 2) no access to benign reference model; 3) 2 or multi-class models. To detect backdoors in such scenarios, it leverages a new statistical metric known as the expected transferability, which can derive a fixed threshold for backdoored models detection. Results on six datasets show that the proposed method is effective for detecting backdoored models.