BaDExpert: Extracting Backdoor Functionality for Accurate Backdoor Input Detection
Authors: Tinghao Xie
Xiangyu Qi
Ping He
Yiming Li
Jiachen T. Wang
Prateek Mittal
Venue: ICLR 2024 poster (2024)
Keywords: Backdoor Input Detection
Backdoor Defense
AI Security
Deep Learning
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We propose a novel backdoor reverse engineering method in the model space to obtain "backdoor experts" that only recognize backdoor inputs, based on which we design a backdoor input detector.

Abstract: We present a novel defense, against backdoor attacks on Deep Neural Networks (DNNs), wherein adversaries covertly implant malicious behaviors (backdoors) into DNNs. Our defense falls within the category of post-development defenses that operate independently of how the model was generated. The proposed defense is built upon a novel reverse engineering approach that can directly extract **backdoor functionality** of a given backdoored model to a *backdoor expert* model. The approach is straightforward --- finetuning the backdoored model over a small set of intentionally mislabeled clean samples, such that it unlearns the normal functionality while still preserving the backdoor functionality, and thus resulting in a model~(dubbed a backdoor expert model) that can only recognize backdoor inputs. Based on the extracted backdoor expert model, we show the feasibility of devising highly accurate backdoor input detectors that filter out the backdoor inputs during model inference. Further augmented by an ensemble strategy with a finetuned auxiliary model, our defense, **BaDExpert** (**Ba**ckdoor Input **D**etection with Backdoor **Expert**), effectively mitigates 17 SOTA backdoor attacks while minimally impacting clean utility. The effectiveness of BaDExpert has been verified on multiple datasets (CIFAR10, GTSRB and ImageNet) across various model architectures (ResNet, VGG, MobileNetV2 and Vision Transformer). Our code is integrated into our research toolbox: [https://github.com/vtu81/backdoor-toolbox](https://github.com/vtu81/backdoor-toolbox).

Published as a conference paper at ICLR 2024
BADEXPERT: EXTRACTING BACKDOOR FUNCTIONAL- ITY FOR ACCURATE BACKDOOR INPUT DETECTION
Tinghao Xfe Xiangyu Qi'
Ping He
Yiming Li Jiachen T. Wang'
Prateek Mittal !
Princeton University *Zhejiang University (thx, xiangyugi, tianhaowang, pmitta eerrriinetttt edu; gnipêzju. edu liyiming. tech@ com
Corresponding Authors
ABSTRACT
In this paper, present novel defense against backdoor attacks on deep ral networks (DNNs), wherein adversaries covertly implant malicious behaviors (backdoors) into DNNs. Our defense falls within the category of post-development defenses that operate independently of how the model was generated. Our proposed defense built upon an intriguing concept: given backdoored model, we reverse engineer it to directly extract its backdoor functionality to backdoor expert model. To accomplish this, we finetune the backdoored model over small set of intentionally mislabeled clean samples, such that it unlearns the normal func- tionality while still preserving the backdoor functionality, and thus resulting in a model (dubbed backdoor expert model) that only recognize backdoor inputs. Based on the extracted backdoor expert model, we show the feasibility of devising robust backdoor input detectors that filter the backdoor inputs during model inference. Further augmented by an ensemble strategy with a finetuned auxiliary model, defense, BaDExpert (Backdoor Input Detection with Backdoor Expert), effectively mitigates 17 SOTA backdoor attacks while minimally impacting clean utility. The effectiveness of BaDExpert has been verified on multiple datasets (CIFAR10, 10, GTSRB, and ImageNet) across multiple model architectures (ResNet, VGG, MobileNetV2, and Vision Transformer). Our code is integrated into our research toolbox: https: ttt.bhtt..t.ttt.////// thub. wwm vtom/vtu11.. h/tp///aacoooroo.oooooooooooooo. tooolttoolllooo toobooorr-oolooxx lbox
Backdoor Task
Finetuning
Recognize
Untrusted
Backdoored Model
Backdoor Expert B
Clean Task
Forgot
Figure Extracting backdoor functionality via finetuning on mislabeled small clean set. The backdoored model M can correctly recognize both benign and poisoned samples whereas our backdoor expert model B can only recognize backdoor samples.
Backdoor Expert 2
Buckdoor Expert 2
Dog
OFish
Extract Backdoor Functienatity
Extract Blackdoor Punctionality
Agree
Disagree
Rackdoor Input (Target class: Dog)
ODo
Clean Input
Cat
Backdoored Model M
Backloored Model
Figure 2: Utilizing backdoor experts for backdoor input detection. The backdoor expert model
B retains only backdoor functionality. As such, backdoor input that successfully deceives M (predicted to "Dog"" will likewise obtain the same prediction ("Dog") by B - agrees with M. Conversely, since B lacks the normal functionality, it cannot recognize the clean input M does (correctly predict to "Cat"), and will thus provide a possibly divergent prediction (e.g. "Fish") B disagrees with M. Based the distinctive natures of clean and backdoor inputs, reject suspicious backdoor inference-tin inputs by checking if B and M agree in predictions.
simply

---

Published as a conference paper at ICLR 2024
INTRODUCTION
A prominent security concern of deep neural networks (DNNs) is the threat of backdoor attacks (Gu
al. 2017; Li et al. 2022b), wherein an adversary embeds hidden behaviors (backdoors) into
model through techniques such as data poisoning (Goldblum et al. 2022) or weights tampering (Qi
ct al. 2022). During inference, such a backdoor remains dormant when processing benign inputs but can be activated by trigger-planted backdoor samples devised by attackers. Upon activation, the compromised model produces anomalous outputs, which could lead to severe security breaches.
The existing literature has extensively explored defensive strategies against backdoor attacks, with a significant focus on development-stage defenses (Tran et al. [0188] Li et al. 2202aa Huang et al.
2022 Qi et al.] 2023b). These defenses operated before and during the model training process, primarily targeting data-poisoning-
attacks (Goldblum et al. 2022).
In this work, we rather focus on post-development defenses that operate after the model develop- ment (Wang et al., 2019: Li et al.) (021b) Gao et al., 2019] Guo et al.] 2023). Given an arbitrary model that may potentially be backdoored, post-development defenses tackle the challenge of secure deployment head-on, without knowing how the model was generated. Implementing such defenses faces non-trivial technical challenges. From a methodological point of view, these defenses do not have access to the training dataset or information about training dynamics (such as gradient updates or loss information) and thus forfeit rich information that could aid system defense. For example, approaches that directly analyze poisoned datasets (Tran et al. (2018] QQ et al.) (023b) or the backdoor
training dynamics (Li et al. 202 Huang et al. 2022) cannot be applied.
One recognized paradigm (Tao et al. 2022 Wang
100
jet al. 2019 2022b) for addressing post-development
CA or ASR (%)
80-
defenses aim to infer backdoor trigger patterns through the direct eeverse-nginneerig of the com- promised model without requiring knowledge about how the model was generated, and then neutralize the backdoor with the reconstructed triggers. However, these methods usually require strong assumptions on the trigger types formulate the trigger-space op- timization problem. And in cases where the trigger based on global transformation (Chen et al. 2017 Nguyen & Tran 2021), these methods frequently fail due the mismatch of the assumed trigger pattern and the actual underlying trigger in practice. Our work advocates an alternative perspective by extracting the backdoor functionality (Fig [] instead of the backdoor trigger pattern and, therefore, avoids the imposition of inductive biases on trigger types. Our key contributions can be summarized follow:
60
80
20-
CA
ASR
Se.5 6ed5 7md Bed 96d 1md 26-4 Se-4 Anct Sec4 Sm.3
(Un)leamiig Rate
Figure 3: Finetuming on a small mislabeled clean set (also dubbed "unlearning") can isolate the back- door functionality (BadNet attack on CIFAR10),
We introduce novel approach that directly extracts the backdoor functionality to backdoor expert model (Fig TT. as opposed to extracting backdoor triggers. Our approach relies on remarkably torawdd technique: gentle finetuning on small set of delib- erately mislabeled clean samples. The reasoning behind this technique lies in an intriguing characteristic of backdoored models (Fig finetuning backdoored model on mislabeled clean samples causes the model to lose its normal functionality (low clean accuracy), but remarkably. its backdoor functionality remains intact (high attack success rate). We also show this observation is pervasive across attacks, datasets and model architectures (Fig
We show that the resultant backdoor expert model can be subsequently utilized to shield the original model from backdoor attacks. Particularly, we demonstrate that it is feasible to devise highly accurate backdoor input filter using the extracted backdoor expert model, of which the ligh-level intuition is illustrated in Fig In practice, the efficacy of this approach is further amplified by a more fine-grained design with an ensembling strategy (see Sec
We design a comprehensive defense pipeline, backdoor input detection with backdoor expert (dubbed DxDext,,,,, capable of mitigating a diverse set of existing backdoor attacks (12 types), at the cost of only negligible clean accuracy drop. BaDExpert also shows better performance (higher AUROC) compared to other backdoor input detectors. Our extensive experiments on both small-scale (CIFAR1O, 10, GTSRB) and large-scale (ImageNet) datasets with different model architecture choices (ResNet, VGG, MobileNet and Vision Transformer) validate the consistent effectiveness of BaDExpert. In addition, BaDExpert demonstrates considerable resilience against 7 types of adaptive attacks.
2

---

Published as a conference paper at ICLR 2024
2 PROBLEM FORMULATION
Notations. We consider a classification model M(-10M) parameterized by 0m. We denote (y)l)) as the probability (confidence) predicted by the model M for class y on input æ, with which the classification model is defined as _M()) arg m
(o)f)((y))) where C is the number of classes, and [C] := {1, 2, . C). We denote the trigger planting procedure for backdoor attacks by T : X'' + X' and denote the target class by t. We use P to denote the distribution of normal clean samples. The clean accuracy (CA) of model then defined (()))>/(M( yl while the attack success rate (ASR) is IP ((..)))Pludd (M(((=)) = 0..
Threat Model. We consider threat model where the attacker directly supplies to the defender a backdoored model M, which achieves a similar CA to benign model without backdoor, while the backdoor can be activated by any triggered inputs T(x) at high ASR (e.g.. > 80%). The attacker cannot control how the model will be further processed and deployed by the victim. but will attempt
to exploit the pre-embedded backdoor by feeding triggered inputs T(x) to the deployed model.
Defenders' Capabilities. After receiving the model M, the defender has no information about how The defender neither knows the potential backdoor trigger pattern or even whether the model is backdoored. Following prior works (Li ct al. 20216 Tao et al.) 2022] Qi et al.) (023b), the defender has access small reserved clean set De.
the model was generated (e-g.. training datasets/proceduurss
Defender's Goal. The ultimate goal of the defender is to inhibit the ASR during model deployment, while retaining high CA as possible. Specifically, we focus realizing this goal by deriving backdoor input detector BID(.) X' + {0, 1} that: 1) BID(T(x)) = 1, V(x, y) ~ (T((((())) = t. i.., detect and reject any backdoor inputs that successfully trigger the model's backdoor behavior; 2) BID(x) = 0, V(x, y) ~ P, i.e., does not harm the model's utility on clean samples.
3 METHODS
We design post-development backdoor defense that is centered on the intriguing concept of backdoor functionality extraction. Our approach is distinct from prior work that predominantly focus on trigger reverse engineering, in the sense that we directly extract the backdoor functionality from the backdoored model (Sec 3.1). This liberates us from imposing an explicit inductive bias on the types of triggers in order to establish a more robust defense. This extracted backdoor functionality is then utilized to design backdoor input filter (Sec 3.2), safeguarding the model from backdoor attacks during the inference stage. We present the details of our design in the rest of this section.
3.1 BACKDOOR FUNCTIONALITY EXTRACTION
As extensively articulated in the prevailing liter- ature (Li ct al. 202 Huang et al. 2022). the functionality of backdoored model can generally be decoupled into two components: the normal functionality that is accountable for making accu- rate predictions (high CA) on clean inputs, and the backdoorr functionality that provokes anomalous outputs (with high ASR) in response to backdoor inputs. Our approach intends to deconstruct the backdoored model and extract the backdoor fune- tionality in isolation. This allows us to acquire addtional insights into the embedded backdoor, which can be further leveraged to develop back- door defenses (to be detailed in Sec 3.2).
Algorithm 1 Backdoor Functionality Extraction
Input: Reserved Small Clean Set D.,, Backdoor Model M, Learning Rate 7). Number of Iteration m. Output: Backdoor Expert B
B <- copy of M
2: for i I...., I do
3:
(X, Y) + aandom batch from De
4: Mislabel Y to Y'
5:
f CrossEEtropy ra o Y')
0g <- 0s - 17 Vog!
7: end for
8:
return B
"B(X) E B is the raw output of the model.
Algorithm formally presents our approach for the intended backdoor functionality extraction (we refer to the resultant model as a backdoor expert model B). The approach is straightforward given a backdoored model MM, we directly finetune it on a small set of deliberately mislabeled clean samples. As illustrated in the algorithm, we sample data (X, Y) from a small reserved clean set We then finetune the backdoored model M on the
D. and assign them incorrect labels" (Line mislabeled clean data with a gentle learning rate (e.g., in our implementation, we take = 10-4 by default). Our key insight that underpins this methodology stems from an intriguing property of backdoored models: with a small learning rate, finetuning a backdoored model over a few mislabeled
In our implementation, we generate the incorrect labels by intentionally shifting the ground- label Y to its neighbor (Y + 1) mod C. In Appendix B.3 we also discuss other possible mislabeling strategies.
3

---

Published as a conference paper at ICLR 2024
clean samples is sufficient induce the model to unlear its normal functionality, leading to low clean accuracy, while simultaneously preserving the integrity of its backdoor functionality, ensuring a corroborate that similar results
high attack success rate, depicted in Fig In Appendix C3 can be consistently achieved against a wide array of attacks, across datasets and model architectures, thereby indicating the pervasiveness and fundamentality of this property.
We designate the resulting model produced by Algorithm[ [ a "backdoor expert", as it singularly embodies the backdoor functionality while discarding the normal functionality. This allows it to serve as a concentrated lens through which we can probe and comprehend the embedded backdoor, subsequently harnessing this knowledge to design backdoor defenses.
3.2 BADEXPERT: BACKDOOR INPUT DETECTION WITH BACKDOOR EXPERT
In this section, we present a concrete design in which the backdoor expert model is utilized to construct a backdoor input detector to safeguard the model from exploitation during inference. The high-level idea has been illustrated in Fig we can detect hackdoor inputs by simply comparing whether the predictions of the backdoored model and the backdoor expert agree with each other. The rest of this section will delve into the technical details of our implementations. We start with an ideal case to introduce simplified design. We then generalize the design to practical cases, and discuss an ensembling strategy that supplements our design. Finally, we present the overall detection pipeline.
Detecting Backdoor Input via Agreement Measurement between .M and B. A straightforward way leverage the extracted backdoor functionality defend against backdoor attacks is to measure the agreement between M and B, as shown in Fig 2 Specifically,
* Reject any input x where the predictions of M and B fall within an agreement (M(z) = ((x)), since B and M will always agree with each other on a backdoor input T(x) that successfully activates the backdoor behavior of M (backdlor functionality is retained).
Accept any input x that M and B disagrees on (M(z) + B(x)), since B will always disagree with M on clean inputs x that M correctly predict (normal functionality is lost).
Note that the rules above are established when B completely unlearns the normal functionality of M, while fully preserving its backdoor functionality. Refer Appendix B.T for detailed formulations.
Soft Decision Rule. In practice, we may not obtain such ideal B required for the establishment of the agreement measurement rules above (see Appendix C33 for empirical studies). Therefore, for practical implementation, we generalize the hard-label agreement measurement process above to a soft decision rule that is based on the fine-grained soft-label (confidence-level) predictions:
Reject input x if Confi (M())())) 2 ((hreshhlld).
(I)
This rule shares the same intuition we leverage in Fig 2 when B shows high confidence on the predicted class that M classifies x to (i.e., B agrees with .M), the input would be suspected as backdoored and thus rejected. In Appendix [..2] we derive this soft decision rule formally, and showcase that the distributions of Conf M(z)(²) for clean and backdoor inputs are polarized on two different ends (i.e., the soft rule can lead distinguishability between clean and backdoor inputs).
Clean Finetuning Also Helps. Prior work et al. 2021b) has shown that standard finetuning of the backdoored model M on D. with correct labels (dubbed "clean finetuning") can help suppress the backdoor activation (e.g. the ASR will decrease). Essentially, a dlean-finetuned auxiliary model M' will largely maintain the normal functionality of M, while diminishing some of its backdoor functionality (in sharp contrast to the behaviors of the backdoor expert model B). Notably, we observe that "clean finetuning" is actually orthogonal and complimentary to our mislabeled finetuning process (Alg IT) Similar to the soft decision rule above, we can establish symmetric agreement measurement rule between M' and M reject any input if Conf (M(z))z) < r', i.c., M' and M disagree on (see Appendix [B.2 for details). Below, we showcase how to assemble the backdoor expert model auxiliary model M' together for comprehensive defense pipeline.
B and the
Our Pipeline: BaDExpert. Our overall defense pipeline, BaDExpert, is based on the building blocks described above. For any given input æ;, we first consult the (potentially) backdoored model M to obtain preliminary prediction == M(x). Subsequently, we query both the backdoor expert
4

---

Published as a conference paper at ICLR 2024
B and the auxiliary model M getting their confidence Confg()22) and Conf regarding this preliminary prediction class for the input æ. We then decide if an input æ is backdoored by:
Conf Mr (8/xr)
Reject input if Score: ==
(threshold).
Conf (7)x))
Intuitively, a backdoor input x tends to have a high (y]l (i.e., B agrees with M) and a low Conf (ū]æz) (i.e., M'' disagrees with M), and therefore low Conf (i)) /)ooff/s()))))) As follows, we further provide a justification of the reason behind Eq (2) with Neyman- lemma.
Remark 1 (Justification for the Decision Rule for Calibrated Classifier) We can justify the like- lihood ratio Conf Mr ())x ( from the perspective of Neyman-Pearson Pearson lemma (Neyman & Pearson 1933), if both B and M' are well-calibrated" in terms of backdoor and clean distribution, respectively. Specifically, when both B and M' are well-calibrated, Confis (ū)z)))))))))))) and Conf M (ÿ) represents the likelihood of x for having label ÿ under backdoor distribution and clean distribution. respectively, and we would like to determine ÿ sampled from which distribution. Neyman-Pearson lemma tells us that, any binary hypothesis test is dominated by the simple strategy of setting some threshold for the likelihood ratio Conf y (ū)z)/ Conf ()) Moreover. the choice of the threshold determines the tradeoff between false positive and false negative rate.
3000
Fig 4 ddemmmstratssss the score distribution given by BaDExpert for clean and backdoor inputs (WaNet attack on CIFARI0).
2500
Clean Barkkooor
logial
2000
BB
As shown, the backdoor inputs and clean inputs are signif- icantly distinguishable (with AUROC = 99.7%). We can identify > 97% backdoor inputs that only leads to < 1% FPR. Through our various experiments See we find our backdoor detection pipeline is robust across extensive settings.
1500
Backdoor 97%
Clean 9M%
# 1000
500
-20
-15
-10
-5
100
15
20
logl()ooce)
Figure 4: Score distribution.
4 EXPERIMENTS
In this section, we present our experimental evaluation of the BaDExpert defense. We first introduce our experiment setup in Sec 4.1, and demonstrate our primary results on CIFAR1O in Sec S.c (similar results on GTSRB deferred to Appendix C. followed by detailed ablation studies of BaDExpert's key design components. In Sec Sec we delve deeper into BaDExpert's generalizability across various model architectures and its scalability on ImageNet. Finally, in Sec 4.4 investigate the resilience of BaDExpert's 's design against a series of adaptive attacks, demonstrating its adaptive effectiveness.
4.1 SETUP
Datasets and Models. Our primary experiment focuses on two widely benehmarked image datasets in backdoor literature, CIFARIO 10 (Krizhevsky/ (012) (See 4.2) and GTSRB (Stallkamp et al. 2012) (deferred to Appendix C.I.. We demonstrate the equivalently successful effectiveness of BxDexp on a representative large scale dataset, 1000-class ImageNet Deng et al.) (2009), in Sec to further validate our method's s scalability. We evaluate BaDExpert across various model architectures. Specifi- cally, we adopt the commonly studied ResNetl 8 (He et al.) (016) through our primary experiment, and validate in See 4.3 the effectiveness of BaDExpert on other architectures (VGG (Simonyan & Zisserman 2014), MobileNetV2 (Sandler et al.) 2018), and ViT (Dosovitskiy et al. (2220).
Attacks. We evaluate BaDExpert against 12 -fte-offthh-art backdoor attacks in our primary experiment, with 9 types initiated during the development stage and the remaining 3 post-development. In the realm of development-stagg attacks, we explore 1) classical static- dirty-label attacks such as BadNet (Gu et al.) 2017). Blend (Chen et al.) (017)), and Trojan (Liu et al.) (018b);; 2) clean-label attacks including CL Turner et al.] 2019) and SIG (Barni et al. 2019); 3) input-specific trigger attacks like Dynamic (Nguyen & Tran 2020). ISSBA (Li et al. 2021d). WaNet (Nguyen) & Tran
2021), and BPP (Wang et al. 2022c). As for post-development attacks, our evaluation
considers: 1) direct finetuning of the developed vanilla model on a blending poisoned dataset (FT);
"Worth of notice, the auxiliary model M' can not only be obtained via clean finetuning, but also via existing model-repairing defenses. In another sentence, our backdoor expert model B serves an enhancement module orthogonal to the existing line of work backdoor model-repairing defenses (see Sec and Appendix C.2) "In practical, since we cannot ensure the ideally good calibration of and M', we slightly modity the rule according to the actual (Confi, yo distribution nature. Kindly refer to Appendix A.2 for more details.
5

---

Published as a conference paper at ICLR 2024
2) trojanning attacks NT (Luu et al. 2018b)); 3) subnet-replacemmnt attacks (SRA (Qi et al. 2022)). Moreover, in Sec[4.4] we study 6 existing adaptive attacks and a novel tailored adaptive attack agaist BaDExpert. Our attack configurations largely adhere to the methodologies described in their original papers. Readers interested in the detailed configurations can refer to the Appendix A.3
Defenses. We compare BaDExpert with 14 established backdoor defense baselines. First, to underscore BaDExpert's consistent performance in detecting backdoor inputs, we juxtapose with post-development backdoor input detectors (STRIP (Gao et al. (019), Frequency (Zeng et al.) 2021b).
SCALE-UP (Guo et al. 2023)). ). all of which share our goal of detecting inference-time backdoor inputs. Additionally, for the sake of comprehensiveness, we repurpose certain development-star poison set cleansers (AC (Chen et al. 2018)) to function as backdoor input detectors for comparative analysis. We also incorporate 4 model repairing defenses (FP (Liu et al. [0]8a), NC (Wang et al.)
2019), MOTH (Tao et al., 2022), NAD (Li et al. (02Ibb) into our comparison to demonstrate BalDExpert's consistent performance as general post-development defense. In additional, compare BaDExpert with 6 other baselines in Appendix C.8 We describe the hyperparameter selection of BaDExpert in Appendix A.1 and configuration details of other baselines defenses in AA.4
Metrics. We evaluate our results based on two sets of metrics. First, to measure BaDExpert's effectiveness backdoor detector, we report the area under the receiver operating characteristic (AUROC (Faweett 2006), the higher the better). Second, to directly compare with other non-detector backdoor defenses, we report the clean accuracy (CA, the higher the better) and attack success rate (ASR, the lower the better) of the model equipped with BaDExpert. Specifically, for backdoor input detectors, we interpret correctly filtered backdoor inputs as successfully thwarted attacks, while falsely filtered clean inputs are considered erroneous predictions. Formally, CA = P(..a)>>pp (M<)) = y A BID(x) (z) = O). and ASR = P( (M))())) = A BI)( = Oj. These metries provide comprehensive and fair evaluations of method in different defense scenarios. CA and ASR are reported on the standard validation over clean inputs and their backdoor correspondance, while AUROC is calculated over noisy augmented validation set (same configuration following prior work (Guo et al. 2023)) to prevent overfitting. To ensure rigor, all primary results are averaged over three runs (corresponding standard deviations are reported in Appendix
4.2 EPFECTIVENESS OF BADEXPERTS ON CIFARIO
4.2.1 CONSISTENT EFFECTIVENESS ACROSS SETTINGS
Table Table Tableehhighie Lhighlight the defensive results of BaDExpert as comprehensive post-development defense and as backdoor input detector, respectively.
BaDExpert as a test-Deeelomment Defense. Specifically, we deem a defense as successful if the post-defense Attack Success Rate (ASR) is under 20%, and unsuccessful otherwise, as done in prior work (Qi et al. 2023b). As depicted, BaDExpert consistently succeeds against all attacks, with an average defended ASR of merely 5.1% (best). Moreover, deploying the backdoor model with BaDExpert only causes a small 0.9% CA drop, which is comparable to the best of prior arts.
BaDExpert as a Backdoor Input Detector. As evidenced by Table 2 BaDExpert achieves an average AUROC of 99% across 12 attacks, while the average AUROC of all other baseline detectors is less than 85%. In addition, BaDExpert achieves the highest AUROC for 9 out of the 12 attacks and maintains an AUROC of over 96% in all situations. Overall, BaDExpert consistently performs effectively against all evaluated attacks, exhibiting a significant advantage over the baseline methods.
4.2.2 COMPARING BADEXPERT TO BASELINE DEFENSES
In Table [ all the baseline defenses we evaluate fail against at least one of the baseline attacks. Observation 1: The two baseline defenses that focus on trigger everse-engineering NC and MOTH, are effective against attacks using patch-based triggers (such as BadNet, Trojan, CL. SRA, etc.), but they fail when confronted with attacks using global transformations as triggers (for instance, SIG), due to their strong inductive bias on the types of backdoor triggers. Observation 2: The three detectors (STRIP, Frequency, and SCALE-UP) that rely on the conspicuousness of backdoor triggers in backdoor inputs (either through frequency analysis or input perturbation) are ineffective against SIG and WaNet, which employ stealthier backdoor triggers. Observation 3: FP and AC utilize specific inner-model information (like neuron activations). turning out not universal enough to counteract all types of attacks. Observation 4: Specifically, NAD, which applies distillation to a fine-tuned model

---

Published as a conference paper at ICLR 2024
Table 1: Defensive results CIFARIO (CA and ASR)
Defenees-+
No Defeme
FP
NC
MOTH
NAD
STRIP
AC
Properccy
SCALE-UP
RaDExpert
Attacko.
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA ASR
CA
ASR
CA
ASR
No Artcck
94.1
82.9
93.3
91.1
85.6
54.5
87.8
91.0
77.0
93.1
Develogmeeut Stage Attocks
BadNer
94.1
1000
836
1000
93.7
91.5
0.7
86.9
18
54.7
94.1
0.0
91.0
0.0
76.3
95.1
0.
94.0
91.9
82.7
93.5
91.2
325
54.5
43.7
24.8
76.7
17.2
93.1
114
Trujen
82.8
94.1
85.1
93.5
54.7
93.3
10.6
SKG
93.8
K2.7
82.1
58.3
95.5
54.5
14.2
77.1
16
Dynamic
93.9
19.1
82.7
56.1
93.1
91.4
56.9
85.9
22.4
54.5
20.9
19.1
74.2
90.8
0.4
79.9
93.0
16.3
ISSRA
93.9
00.9
83.6
0.1
93.6
1.2
01.3
56.3
85.6
54.6
93.9
0.1
90.8
0.0
79.3
99.0
WaNes
93.1
03.7
81.1
2.6
93.00
81.6
90.1
14.0
85.7
1.8
53.7
86.9
03.1
23.5
90.1
96.5
76.3
$3.9
92.2
2.0
UPP
89.7
99.8
76.0
8.3
89.8
33.8
53.5
1.9
89.7
0.9
50.7
91.9
89.7
2.4
36.8
0.0
74.7
88.9
0.2
dest-deveoopmentt Attacks
82.3
69.8
90.1
ToviaNNN
93.8
100.0
85.2
92.2
$4.4
7.0
90.5
99.9
79.3
100.0
91.9
51.3
89.4
Average
V3.2
97.2
82.1
66.2
92.8
90.9
34.9
96.1
6.7
8399
45.7
92.5
13.2
90.3
13.0
77.
90.3
Drop (smollerr better)
J11.1
104
12.
.93
1009
ASR Doop flarger
132.0
167.6
J623
190.5
.A3
1840
184.1
92.1
Table 2: Defensive results on CIFARIO (AUROC).
AUROC (5)
BadNet
Blend
Trojan
CL
SIG
Dynamic
ISSBA
WaNet
Bpp
FT
TrojanNN
SRA
Average
STRIP
99.1
47.1
72.1
84.7
40.3
85.2
68.1
49.8
50.1
91.8
99.3
54.8
70.2
AC
100.0
54.1
99.7
99.9
53.6
77.9
84.4
47.2
98.4
58.7
99.9
99.6
81.1
Prequency
75.1
73.5
75.1
74.5
68.0
74.9
75.1
62.8
75.0
73.5
75.0
75.1
73.1
SCALE-UP
80.9
91.2
96.3
96.0
71.6
66.3
87.9
89.6
96.5
50.9
83.5
BaDExpert
100.0
99.2
99.2
99.0
99.8
99.1
96.1
99.7
100.0
99.6
99.3
100.0
99.0
to eliminate the backdoor, can effectively defend against 11 out of 12 attacks, but it results in a considerable drop in CA (7.2%).
Remarkably, BaDExpert inhibits all 12 attacks by rejecting suspicious backdoor inputs. Principally, this could be credited to the extracted backdoor functionality, which poses no inductive bias on backdoor trigger types, and therefore effectively suppress all attacks independent of the triggers they use. Moreover, benefiting from the extracted backdoor functionality that aid our defense, when comparing with NAD (which similarly includes a standard finetuning process in their pipeline), we outperform their defense performance in most scenarios, with noticeably higher CA and lower ASR.
4.2.3 ABLATION STUDIES
Size of the Reserved Clean Set |Del_ In our defense pipeline, we assume small reserved clean set D. (default to 5% size of the training dataset in the primary experiment, i..,, 2, 000 samples) construct both B and M'. To inves-
100
Zoonin
AUROC (%)
90-
975
tigate how minimal [D_] could be, we evaluate BaDExpert with different sizes (200 1, 800) of this clean set. The AUROC of BaDExpert (against Blend attack on CIFARIO) 10) is reported in Fig 5 line with the circle marker). As shown, as DDl becomes smaller, the AUROC of BaDExpert mostly remains higher than 98%, and slightly drops (to 95% or 90%) when the number of clean samples is extremely lim- ited (400 or 200). To obtain a clearer view, in Appendix [C66 we compare BaDExpert with ScaleUp and STRIP syde-by- side, where they are all assigned to such highly limited amount (100, 200 and 400) of clean samples. Notably, BaDExpert still outperforms the baselines in most scenarios. Alongside, we also justify how practitioners can acquire a small D. with possible data-collecting approaches ((eng et al. 2023a)).
80
70
BaDExpert .af only
only
200
600
1000
1400
1800
Size of Clean Set. [De]
Figure 5: AUROC of BaDExpert (both M' and B). M' only, and B only, with different reserved clean set sizes.
Necessity of both B and M'. As mentioned, we ensemble both B and M' to decide if an input backdoored or not. However, one may directly attempt to detect backdoor inputs based on the metric of the single confidence of either 1 or M'. Here, we study whether both of them are necessary for the success of BaDExpert. The blue (square) and red (triangle) lines in Fig 5 correspond to the detection AUROC if we only adopt M' or B. There is three-fold observation: 1) backdoor expert B consistently provides a high AUROC (~~ 95%), implying that it indeed fulfills the backdoor functionality by assigning backdoor input with higher confidence (Eq (D: ): 2) The finetuned auxiliary
model M' itself also provides a high AUROC (> 95%) when we have more than 1, 000 clean
reserved samples, but degrades when the size of the clean set gets smaller (AUROC 90% when
7

---

Published as a conference paper at ICLR 2024
Table 4: Defensive results of BaDExpert on ImageNet.
(%)
ResNeti8
ResNeti01
VIT-B/16
No Attack
BaiNel
Bland
No Altack
SRA
No Attack
FT-BodNNt
FT-Blend
Without Defense
CA
68.4
77.1
100.0
74.7
81.9
82.3
99.5
93.2
CA
68.9
68.3
68.9
77.0
24.6
81.8
82.2
81.7
BaDExpert
ASR
0.0
4.9
0.0
0.0
0.2
AUROC
1000
99.9
100.0
100.0
100.0
clean samples < 1, 000); 3) Taking advantage from both sides, BaDExpert achieves the highest AUROC in most cases, indicating the necessity of coupling both B and M'.
Ensembling with Existing Model-repairing Defenses. In our pipeline, we choose finetuning the default method to obtain an auxiliary model M' where the backdoor is diminished. Importantly, our backdoor expert methodology is also orthogonal to the extensive suite of model-repairing defenses, meaning that the auxiliary model MM' can effectively be any model that has undergone baseline repair. For instance, when combined backdoor experts with models repaired by the NAD technique, we achieve an average AUROC of 98.8 88.8% on the CIFAR1O 10 dataset, a result that aligns closely with our primary findings presented in Table For a more detailed discussion, please refer to Appendix C.2
Miscellaneous. In Appendix C.3 we validate that Alg ①T) consistently isolates the backdoor func- tionality regardless of different number (1 to 2,000) of clean samples it leverages. In Appendix C.4 study the choice of (un-)learning rate 17 in Alg ( showing that BaDExpert's is performance remains insensitive to the selection of across a diverse lens. In Appendix C.5 we demonstrate that BaDEpxert is similarly successful even when the adversary employs different poison rates. In
Appendix B.3, we discuss other possible mislabelling strategies used in Line 44) of Alg ①
4.3 GENERALIZABILITY AND SCALABILITY
Table 3: BaDExpert generaliz- ability on other architetures.
Generalizability on Other Model Architectures. We first show that BaDExpert works similarly well for two other architectures, VGG-16 and MobileNetV2 in Table on CIFAR10 10 (some attacks in Table I are ineffective and not shown). As shown, the average AUROC on both architectures achieve > 98%, similar to our major results in Table
AUROC (%)
VGGI6
MobileNerV2
BadNet
99.7
99.5
Blend
Trojun
98.6
97.8
Scalability on ImageNet. Table rrffects the effectiveness of BaD- Expert on ImageNet. We conduct: 1) poisoning attacks (BadNet, Blend) by training ResNet18 on backdoor poisoned ImageNet datasets from scratch; 2) subnet-replacement attack (SRA) on pretrained ResNet101 (following SRA's original implementation); 3) finetun- ing attacks (FT-BadNet, FT-Blend) on pretrained ViT-B/16. We only reserve 6, 000 clean samples (equivalent to 0.5% size of 1000 training set) to BaDExpert. In all scenarios, our BaDExpert can effectively detect backdoor samples (~ 100% AUROC and < 5% ASR), costing only negligible CA drop (4 0. 0.). These results confirm the scalability of BaDExpert.
CL
97.7
99.2
SIG
984
98.9
Dynamic
96.7
98.1
WaNet
98.0
FT
99.3
98.7
Averaigo
98.3
98.5
4.4 THE RESISTANCE TO ADAPTIVE ATTACKS
To thoroughly study the potential risk underlying our defense, also consider adaptive adversaries that strategically employ backdoor attacks designed specifically to bypass BaDExpert's 's detection.
One possible adaptive strategy is deliberately establish dependencies between the backdoor and normal functionality, which may undermine the implicit assumption of our defense the core concept of isolating the backdoor functionality from the normal functionality via unlearning. Essentially, if the backdoor functionality depends on the normal functionality, the erasure of the normal functionality would subsequently lead the degradation of the backdoor functionality, potentially reducing BaDExpert's 's effectiveness. In fact, there exist several adaptive backdoor attacks tailored to this end. Here, we examine TaCT (Tang et al. 2021). Adap-Blend, and Adap-Patch (Qi et al. 2023a). which employ different poisoning strategies to create dependencies between backdoor and normal predictions. We also consider an All-to-All attack scenario, where each sample originating from any class E [C]] is targeted to class (i f2-1) 1) mod C - here, the backdoor predictions rely on both the backdoor trigger and clean semantics, thereby forcing the backdoor functionality to depend on the normal one. Eventually, we evaluate BaDExpert against Natural backdoor (Zhao al. 2022) existing in benign models, where the backdoor triggers are unconsciously learned from normal data.
An alternative perspective that may be exploited by adaptive adversaries to bypass our defense would be to utilize specifically constructed asymmetric triggers at inference time (different from the ones used during model production). We first study simple scenario where the adversary deliberately use
8

---

Published as a conference paper at ICLR 2024
Table 5: Defense results of adaptive attacks against BaDExpert.
(96
ToCT
Adap Blend
Adap-Pasch Pasch
All-so-All
Natural
Low- Now-Opacity
BaDExpprt Adaa-BaaNNet BodNet
ASR (bufore defense)
99.3
85,4
99.4
89.1
99.2
51.6
73,5
AUROC
97.5
99.6
99.3
95.5
98.4
87.0
weakened triggers (e.g. blending triggers with lower opacity, dubbed "Low-Opacity") to activate the backdoor at inference time. More profoundly, we design a novel adaptive attack tailored against where the adversary optimizes an asymmetric trigger by minimizing the activation of the backdoor expert model B. Refer to Appendix C.7.3
BaDExpert SaaaDDxpeet--da--Baaaee
C.7.3 for details.
As shown in Table 5 BaDEpxert's effectiveness indeed experiences certain degradation in (AUROC becomes as low as 87. 377.)) in comparison with Table ( (99.0% average AUROC). Nevertheless, we can see that BaDEpxert still demonstrates considerable resilience against all these adaptive efforts. We recommend interested readers to Appendix C.7 for more details in our adaptive analysis.
5 RELATED WORK
Backdoor Attacks.
Backdoor attacks are typically studied in the data poisoning threat model (Chen
et al. 2017 Zeng et al. 2023b Gao et al. 2023 Qi et al. 2023a). where adversaries inject a few poison samples into the victim's 's training dataset. Victim models trained on the poisoned dataset tend learn spurious correlations between backdoor triggers and target classes encoded in poison samples and get backdoored. Besides data poisoning backdoor attacks can be implemented in alternative ways, such as manipulating training process Bagdasaryan & Shmatikov 202Tp; [L al.) (2021c).
supplying backdoored pre- pre-triined models (Yao et al. [2199] Shen et al. [2021). as well as weights tampering (Liu et al. 2017: Qi et al.) 2021 2022 Dong et a (023), etc. There are also backdoor attacks that are adaptively designed to evade defenses [ang et al. [021 Qi et al. 2023a).
Development-Stage Backdoor Defenses. The existing literature has extensively explored defensive strategies against backdoor attacks, with significant focus on development-stage defenses. These defenses primarily target tatta-pisonnng-baee -based attacks (Goldblum et al. (0022) and are presumed to be implemented by model vendors. They either identify and remove the poison samples from the
dataset before training (Tran et al.) 2018: Tang et al. 202 1 Qi et al. 2023b; Pan et al.) 2023). or suppress the learning of backdoor correlations during training (Li et al. 2022 laj Huang et al. 2022]
Wang al., 2022a). Notably, the security of these approaches heavily relies the integrity of model vendors, and they cannot prevent backdoor injection after the model development.
Dest-Development Backdoor Defenses. Post-development defenses operate independently of model development. They typically assume only access the (potentially backdoored) model intended to be deployed and a small number of reserved clean data for defensive purposes. One category of such approaches attempts directly remove the backdoor from backdoor model via pruning (Liu et al.] (a) distillation (Li et al.] al.] (2021b), forgetting (Zhu et al.; 2022). finetuning (Shal cet al. 2022). unlearning reconstructed triggers (Wang et al. [2019] Tao et al. (022)) etc. Alternatively, model diagnosis defenses (Xu et al. 2022 Kolouri et al. (0220 attempt to make a binary diagnosis on
whether model is backdoored. There are also approaches attempting to detect and filter backdoor
inputs at inference time (Zeng et al., 2021b Li et al. 2022a Guo et al.] 2023) and thus prevent the backdoor from being activated. The defense we propose in this work falls within this category. Meanwhile, our idea of backdoor extraction is also relavent to the trigger-ecoonstruuctonn--o based defenses (Wang et al.] [019;] TToo et al., (002) in the sense of backdoor reverse engineering, but different in that we directly extract the backdoor functionality as opposed to backdoor trigger patterns.
6 CONCLUSION
In this study, we introduce a novel post-development defense strategy against backdoor attacks on DNNs. Inspired by the defenses that conduct trigger reverse engineering, we propose distinctive method that directly extracts the backdoor functionality from a compromised model into a designated backdoor expert model. This extraction process is accomplished by leveraging a simple yet effective insight: finetuning the backdoor model on a set of intentionally mislabeled reserved clean samples allows us to erase its normal functionality while preserving the backdoor functionality. We further illustrate how to apply this backdoor expert model within the framework of backdoor input detection, leading to devise an accurate and resilient detector for backdoor inputs during inference-ting time,, known as BaDExpert. Our empirical evaluations show that BaDExpert is effective across different at- tacks, datasets and model architectures. Eventually, we provide an adaptive study against BaDExpert, finding that BaDExpert is resilient against diverse adaptive attacks, including a novelly tailored one.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
In this work, we introduce a novel backdoor defense, demonstrating its consistent capability of detect- ing inference-time backdoor inputs. Our defense proposal to thwart potential malicious adversaries should not raise ethical concerns. Nevertheless, we want to avoid overstating the security provided by method, since our evaluation on the effectiveness of our defense is empirical, and that our defense comes without any certified guarantee (most prior backdoor defenses share this limitation). After all, the field of backdoor attacks and defenses still long-term cat-and-mouse game, and it essential for practitioners to exercise caution when implementing backdoor defenses in wolld scenarios. Rather, we hope our comprehensive insights into the concept of "extracting backdoor functionality" can serve valuable resource to guide future research in related domains.
also important to note that our work involves tailored adaptive attack against our proposed defense. However, we emphasize that the sole purpose of this adaptive attack is to rigorously assess the effectiveness of our defense strategy. We strictly adhere ethical guidelines in conducting this research, ensuring that all our experiments are conducted in controlled isolated environment.
ACKNOWLEDGEMENT
Prateek Mittal acknowledges the support by NSF grants CNS-1553437 and CNS-1704105, the ARL's Army Artificial Intelligence Innovation Institute (A212), the Office of Naval Research Young Investigator Award, the Army Research Office Young Investigator Prize, Schmidt DataX award, Princeton E-affiliates Award. Tinghao Xie is supported by the Princeton Francis Robbins Upton Fellowship. Xiangyu Qi and Jiachen T.Wang are supported by the Princeton Gordon Y. S. Wu Fellowship.
REFERENCES
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In 30th USENIX Security Symposium (USENIX Security 2/), pp. 1505-1521, 2021.
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in enns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 101-105. IEEE, 2019.
Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. Advances in Neural Information Processing Systems, 35:22285-22299, 2022.
Bryant Chen, Wilka Carvalho, Nathalic Baracaldo, Heiko Ludwig. Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv: /. I.03728, 2018
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Xiaodong Song. Targeted backdoor attacks on deep learning systems using data poisoning. ArXiv, abs/1712.05522 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009.
Jianshuo Dong, Qiu Han, Yiming Li, Tianwei Zhang, Yuanjie Li, Zeqi Lai, Chao Zhang, and Shu- Tao Xia. One-bit flip is all you need: When fit-flip attack meets model training. In ICCV, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Mathias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Tom Fawcett, An introduction roc analysis. Pattern recognition letters, 27(8):861-874, 2006.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Anmual Computer Security Applications Conference, pp. 1 13-
10

---

Published as a conference paper at ICLR 2024
Yinghua Gao, Yiming Li, Linghui Zhu, Dongxian Wu, Yong Jiang, and Sau-Tao Xia. Not all samples are born equal: Towards effective clean-label backdoor attacks. Pattern Recognition, 139:109512 2023.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Mądry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning. backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1563-1580, 2022.
Tianyu Gu, Brendan Dolan-Gavit, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint urXiv: 06733, 2017.
Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. SCALE-UP: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. In The Eleventh International Conference on Learning Representations, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hanxun Huang, Xingjun Ma, Sarah Erfani, and James Bailey. Distilling cognitive backdoor patterns within an image. arXiv preprint arXiv:2301. /0908, 2023.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In International Conference Learming Representations, 2022.
Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 301-310, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 2012.
Xi Li, Zhen Xiang, David J Miller, and George Kesidis. Test-time detection of backdoor triggers for poisoned deep neural networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3333-3337. IEEE, 2022a.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoww learning: Training clean models poisoned data. Advances in Neural Information Processing Systems, 34, 2021a.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations, 2021b.
Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang, Reconstructive neuron pruning for backdoor defense. arXiv preprint arXiv:2305.. 14876, 2023.
Yiming Li, Tongqing Zhai, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor attack in the physical world. arXiv preprint arXiv:2104.0236/ 2021c.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transac- tions on Neural Networks and Learning Systems, 2022b.
Yuezun Li, Yiming Li, Baoyuan Wu. Longkang Li, Ran He. and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16463-16472, 2021d.
Kang Liu, Brendan Dolan- and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. International Symposium on Research in Attacks, Intrusions, and Defenses, pp. 273-294, Springer, 2018a.
Yannan Liu, Lingxiao Wei, Bo Luo, and Qiang Xu. Fault injection attack deep neural network. In 2017 IEEE/ACM International Conference on Computer-Aide Aided Design (ICCAD), pp. 131-138. IEEE, 2017.
11

---

Published as a conference paper at ICLR 2024
Yingqi Liu, Shiqing Ma, Yousra Aafer, W. Lee, Juan Zhai, Weihang Wang, and X. Zhang. Trojaning attack on neural networks. In NDSS, 2018b.
Jerzy Neyman and Egon Sharpe Pearson. Ix. the problem of the most efficient tests of statistical
hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, 7316694-706)
1933.
Anh Nguyen and Anh Tran. Wanet- -ineeceippeee warping- bussing-basedd backdoor attack. In The Eleventh International Conference Learning Reprexentations, 2021.
Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems, 33:3454-3464, 2020.
Minzhou Pan, Yi Zeng. Lingjuan Lyu, Xue Lin, and Ruoxi Jia. {ASSET): Robust backdoor data detection across multiplicity of deep learning paradigms. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 2725-2742, 2023.
Xiangyu Qi, Jifeng Zhu, Chulin Xie, and Yong Yang. Subnet replacement: Deployment-stage backdoor attack against deep neural networks in gray-box setting, arXiv preprint arXiv:2107.007244 2021.
Xiangyu Qi, Tinghao Xie, Ruizhe Pan, Jifeng Zhu, Yong Yang, and Kai Bu. Towards practical dtployment-stage backdoor attack on deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13347-13357, 2022.
Xiangyu Qi, Tinghao Xie, Yiming Li, Saced Mahloujifar, and Prateek Mittal. Revisiting the assump- tion of latent separability for backdoor defenses. In The eleventh international conference on learning representations, 2023a.
Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateck Mittal. Towards proactive [ML] approach for detecting backdoor poison samples. In 32nd USENIX Security Symposium (USENIX Security 233), pp. 1685-1702, 2023b.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 44200, 2018
Zeyang Sha, Xinlei He, Pascal Berrang, Mathias Humbert, and Yang Zhang. Fine-tuning is all you need to mitigate backdoor attacks. arXiv preprint arXiv:22/2.0906' 2022.
Shawn Shan, Emily Wenger, Bolun Wang, Bo Li, Haitao Zheng, and Ben Y Zhao. Gotta catch' em all: Using honeypots to catch adversarial attacks on neural networks. In Proceedings of the 2020 ACM SIGSAC Conference Computer and Communications Security, pp. 67-83, 2020.
Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor pre-trained models can transfer to all. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 3141-3158, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012.
Di Tang, XiaoFeng Wang. Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of duns for robust backdoor contamination detection. In 30th (USENIX) Security Symposium (!USENIX) Security 21). 2021.
Guanhong Tao, Yingqi Liu, Guangyu Shen, Qiuling Xu, Shengwei An, Zhuo Zhang, and Xiangyu Zhang. Model orthogonalization: Class distance hardening in neural networks for better security. In 2022 IEEE Symposium Security and Privacy (SP), pp. 1372-1389. IEFF, 2022.
12

---

Published as a conference paper at ICLR 2024
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31, 2018.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02777, 2019.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Zhenting Wang, Hailun Ding, Juan Zhai, and Shiqing Ma. Training with more confidence: Mitigating injected and natural backdoors during training. Advances Neural Information Processing Systems, 35:36396-36416 2022a.
Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverse- engineering of trojan triggers. In Advances Neural Information Processing Systems, volume 35, pp. 9738-9753, 2022b.
Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15074-15084, 2022c.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34: 34:16913-16925, 2021.
Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl Gunter, and Bo Li. Detecting trojans using meta neural analysis. In Proceedings of the IEEE Symposium on Security and Privacy (May 2021), 2021.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 2041-2055, 2019.
Yi Zeng, Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International conference on learning representations, 2021a.
Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16473-16481, 202
Yi Zeng, Minzhou Pan, Himanshu Jahagirdar, Ming Jin, Lingjuan Lyu, and Ruoxi Jia. Meta-Sift: How to sift out a clean subset in the presence of data poisoning? In 32nd USENIX Security Symposium (USENIX Security 23), pp. 1667-1684, 2023a.
Yi Zeng, Minzhou Pan, Hoang Anh Just, Lingjuan Lyu. Meikang Qiu, and Ruoxi Jia. Narcissus: practical clean-label backdoor attack with limited information. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 771-785, 2023b.
Shihao Zhao, Xingjun Ma, Yisen Wang. James Bailey, Bo Li, and Yu-Gang Jiang. What do deep nets learn? class-wise patterns revealed in the input space. In International Conference on Learning Representations, 2022.
Rui Zhu, Di Tang, Siyuan Tang, XiaoFeng Wang, and Haixu Tang. Selective amnesia: On efficient, figh-fidelity and blind suppression of backdoor effects in trojaned machine learning models. In 2023 IEEE Symposium on Security and Privacy (SP), pp. 1220-1238. IEEE Computer Society, 2022.
13

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION AND CONFIGURATION
A.1 BADEXPERT IMPLEMENTATION DETAILS
A.1.1 UNLEARNING AND FINETUNING CONFIGURATION
During unlearning (Alg D. we select a small while effective (un)learning rate 1/. For our major experiments on ResNer18, 8, 17 10-1 for CIFAR10 and 2.5 10-5 for GTSRB. As for other architectures (CIFARIO), 77 = 8-10- for VGGI6, 17 = 8 10 for MobileNetV2, and r7 = 10 for ResNetl 10 (SRA attack). On ImageNet, 17 10 for ResNet18 and ResNet101, and r = 10 for pretrained vit_b_Ii ____________ 16 (IMAGENETIK_SMM Se___sWWA__L__ LINEAR_LL v1 version). We conduct unlearning using Adam optimizer for only 1 epoch, with batch size of 128.
Selection of 7, While we recommend using a conservatively small (un)learning rate 17 (10- in our major experiments) in Alg (.), we also show that BaDExpert's defense performance not sensitive to the choice of 1y across a wide range (from 5 10-6 to 5 10-3) in Appendix C.4, Practitioners in the real world could start from an 17 that is orders of magnitude smaller than the T/ used for clean finetuning, then keep tuning 1) until observing the model's 's clean accuracy (CA) diminishes by certain extent (but does not completely vanish, i.c., worse than random guess). This rule of thumb may help practitioners idenfity an approximate range of 1) according to the model's CA drop magnitude, which can be derived from Fig we can see the backdoor functionality is usually preserved when r) is conservatively small, in the sense that CA has not degraded to the random-guess level.
During the clean finetuning, we select a series of relatively larger learning rates (n') in order to diminish the model's backdoor. The initial learning rate is 0.1 for primary experiments on CIFARIO with ResNet18, and 0.05 on GTSRB. As for other architectures (CIFARIO) 10), the initial learning is 0.2 for MobileNetV2 and VGGI6, and 0.05 for ResNetl (SRA attack). On ImageNet, the initial learning rate is 0.05 for ResNet18, 8, 10-5 for ResNet101, and 5 10-4 for pretrained vit bt__b_1 (IMAGENET1K SWAG_ LINEAR_W v1 version). We conduct the clean finetuning with SGD optimizer for 10 epochs, and reduce the learning rate to its 10% after every two epochs, with a batch size of 64.
Selection of r/. In major experiments, we directly follow the standard finetuning hyperparameters adopted in prior work (Li et al., 2021b). Selecting the initial clean finetuning learning rate sf' can also be done via manual search by analyzing the CA trend. Specifically, we observe that clean finetuning can best diminish the backdoor functionality with a large learning rate, where the finetuned model's s CA drops to 0% in the first one or two epochs, and then recovers to significant value in the following epochs (the recovered CA depends on the size Del: e.g. ~ 80% when Del = 2, 000, but 60% when |De 1, 000).
A.2 DECISION RULE
Distribution of clean and backdoor inputs on the 2D joint-confidence plane. Fig 6 (top-left) demonstrates the (Confg, Conf.y) 2D distribution histogram heatmap of clean inputs and backdoor inputs (WaNet Nguyen& & Tran 202I) attack on CIFARIO). A deeper blue grid represents more clean inputs and a deeper red grid represents more backdoor inputs.
As shown, clean and backdoor inputs could be easily distinguished since most backdoor inputs locate nearby the bottom-right corner (1.0, 0.0) (>90% backdoor inputs are in the deepest red grid at the bottom-right corner), while the clean inputs do not (only <0.5% clean inputs are in this grid). Intuitively, 1) if x is clean input, Conf should be small (the backdoor expert B should not vote for the predicted class of clean inputs) and Conf should be relatively high (the auxiliary model M' should also recognize clean input as M does); 2) if x is a backdoor input that successfully triggers the backdoor, Confg should be high (B should recognize backdoor inputs as M does) and Conf should be relatively low (finetuning has diminished the backdoor).
As mentioned, the simple likelihood ratio score in Eq 2 is only optimal when both B and M' are well-calibrated, which cannot be guaranteed since the actual construction procedures for them involve deep learning techniques. Therefore. in our practical implementation, we slightly modify Eq ( according to the observed confidence distribution in the 2D plane as mentioned in the previous paragraph. Specifically, we score each input-prediction pair (z, 9) alternatively as follows:
Modified Decision Rule.
14

---

Published as a conference paper at ICLR 2024
Conf (5/zz
Confis (g) x))
Reject input x if Score(z, ()) == min
S a (threshold)
(3)
(onfs(
Conf (i]zz)
where (+)* := max(-, 10 8) represents the operation of numerical positive clamping, and y is
hyperparameter (fixed to 0.5 through our major experiments). This rule functions similarly to Eq (2)
a backdoor input tends to have high Conf (i.e., B agrees with M) and low Conf M (7) a)
(i.e., M'' disagrees with M), and therefore a low ()Conf
(C (ū)l)) and xe This modified
score formulation designed accordingly best capture the actual (Confg, Conf distribution
nature, as shown in Fig 6 In the following paragraph, we will desribe detailed empirical geometric interpretation of Eq
Empirically understanding the modified decision rule. In Fig 6 an obviously straightforward decision rule for backdoor detection is to remove any inputs dropped into the right-corner grid region. However, since both B and M' may possibly make mistakes (i.e., not well-calibrated or suboptimal), some backdoor inputs would lie beyond this grid. Therefore, we further smooth out this grid region into two triangle-shaped regions (connected by the dashed lines and the borders in Fig O top-right and bottom-left),, and claim any inputs dropped into these two triangles to be suspicious for backdoor. Fig [ (bottom--right reveals that our decision regions indeed capture the majority of backdoor inputs that locate around the bottom-right corner (1.0, 0.0). Furthermore, decision regions also capture a majority of backdoor outliers that distribute alongside the COnfø = 1 and Confyr = axes. Formally, this geometric decision rule is equivalent calculating score for any input and rejecting inputs with a score lower than a selected threshold o, which has already been described in Eq
Selection of a. In Table the threshold a is selected dynamically such that the false positive rate is always 1%. This is very much following previous work (e.g. STRIP (Gao et al. 2019) where defense results are reported when the false positive rates are fixed. Meanwhile, a fairer and widely-adopted way to report the results of such threshold- based input detectors would be to report the AUROC, which is threshold-fret better in pratical. To fairly present the effectiveness of our proposed defense (BaDExpert), we report both 1) ASR and CA when fixing FPR to 1% and, as shown in Table 1 2) AUROC, which does not involve threshold selection, as shown in Table 2
Intuitively, a detector classifier with a higher AUROC is usually considered
For practitioners, an empirical and simple way for threshold selection would be to calculate a set of BaDExpert scores on D., and then determine the threshold a to be the highest 1st percentile (or any other FPR) score among this set. Alternatively, the defender could also select an appropriate threshold based on the desired FPR by observing the score distribution of a small number of manually inspected benign inputs at inference time. According to our experimental results, deployers can reasonably anticipate BaDExpert to provide robust defense against potential backdoor attacks with a low permissible FPR (e.g. 1% in our major experiment); and as the permissible FPR increases, the effectiveness of our defense mechanism is anticipated to further improve.
As for the sensitivity w.r.t. threshold selection, it appears that for differently trained and attacked models, a may need to be selected accordingly. However, an interesting quantative results on CIFAR10 10 would be: even if we set fixed number (e.g. -0.003). the defense performance would not vary too much across different attacks (all ASR < 20% while CA drop no more than 5%).
Seleetion of 7. In Fig the hyperparameter 7 E (0, corresponds to the intersection y- coordination of the top-right dashed line with the vertical border (e.g., the intersection point (1.0, 0.5) corresponds to y = 0.5). could be selected based on the confidence distribution of M'' if M' assigns high confidences most (clean) inputs, then a larger y would not induce too much FPR. while possibly incorporating more backdoor outliers (vice versa). Nevertheless, we find fixing y to 0.5 already provides consistently good performance through all our major experiments.
A.3 BASELINE ATTACKS CONFIGURATIONS
Our detailed configurations for baseline attacks (CIFAR10) listed follow:
15

---

Published as a conference paper at ICLR 2024
Conf
24
0.500
0.0
03
06
0.0 Conf
0.005
0.005
0.003
0.000
0.5 0.2 0.4
3.5
1.4
0.000
Figure Clean (blue) and backdoor (red) inputs on the (Confg: Conf plane. Top-left is the histogram heatmap, while the other three subfigures are zoom-in of the corresponding distribution scatter plot. Any inputs below the dashed lines are considered suspicious; this removes 97.54% backdoor inputs while only results in < 1% FPR.
BadNet: 0.3% poison ratio, using the 3x3 BadNet patch trigger (Gu et al. 2017) at the right-bottom corner.
Blend: 0.3% poison ratio, 20% blending alpha, using the 32x32 Hellokitty trigger (Chen et al.] 2017) at the right-bottom corner.
Trojan: 0.3% poison ratio, using the 8x8 TrojanNN patch trigger (Liu et al. 2018b at the right-bottom part of the image.
CL: 0.3% poison ratio, adversarial perturbation on poisoned images bounded with formm of 600, using four duplicates of the 3x3 BadNet patch trigger (Gu et al. (017) at the four corners (for considerable ASR).
SIG: 2% poison ratio (to achieve a considerable ASR).
Dynamic: 0.3% poison ratio.
ISSBA: 2% poison ratio (to achieve stably considerable ASR).
WaNet: 5% poison ratio and 10% cover ratio (recommended configurations).
BPP: training-time poisoning. 20% injection ratio, 20% negative ratio (recommended configurations).
FT: finetuning with the full training set (for both high CA and high ASR), 20% injection ratio, 20% blending alpha, using the 32x32 Hellokitty trigger (Chen et al. 2017) the right-bottom corner.
finetuning with the full training set (for both high CA and high ASR), 10% injection ratio, trojan trigger generated following the procedure Liu et (201bb.
TrojanNN:
SRA:
directly using the authors pretrained clean models and backdoor subnets (ResNetl 10) to conduct post-deet development backdoor injection.
We adopt the standard training pipeline to obtain backdoor models: SGD optimizer with a momentum of 0.9, weight decay of 10-1, batch size of 128, 100 epochs in total, initial learning rate of 0.1 (decayed to its 10% at the 50th and 75th epoch), with RandomHori zontal iip and RandomCrop as the data augmentation.
A.4 BASELINE DEFENSES CON
Our detailed configurations for baseline attacks (CIFAR10) listed follow:
16

---

Published as a conference paper at ICLR 2024
FP: We forward 2,000 reserved clean samples to the model, and keep pruning much inactive neurons in the last convolutional layer as possible, until the CA drop reaches 10%.
NC:
Reverse engineer a trigger for each class with a 2,000-sample reserved clean set. Then anomaly index is estimated for every class. The class with the highest anomaly index (whose mask norm also smaller than the median mask norm) determined as the target class for unlearning. Its reversed trigger is then attached the same 2,000 clean samples (correctly labeled), on which the model is retrained to unlearn the backdoor (learning rate is 10-2 for one epoch).
MOTH: Similarly, the trigger reverse engineering and model reparing are performed 2,000-sample reserved clean set. The learning rate for the model repairing process is default to 10 (for 2 epochs).
NAD: First train teacher model in 10 epochs via finetuning (initial learning rate 0.1, decrease to its 10% every two epochs), and use it to distill student model in 20 epochs (learning rate is 0.1 for the first two epochs and 0.05 for the rest). NAD uses 2,000 clean set to perform both finetuning and distillation.
STRIP: Calculate an entropy for each sample is calculated by superimposing it with N 100 randomly sampled clean samples, and consider inputs with the higher entropy to be backdoored; in Table the FPR is fixed to 10% to show its effectiveness.
AC: Gather all inputs for each class, perform a 2-clustering based on their latent repre- sentation, then assign each class silhouette score. The class with the highest silhouette score is suspected, and the inputs within its larger cluster is considered as backdoored. The silhouette scores used to report AUROC.
Frequency: We directly use their official pretrained model to perform detection. The difference between output and output 0 is used to report AUROC.
SCALE-UP: Each imput is scaled up 5 times (scale_sed = {3, 5, 7, 9, 11]), and the score corresponds the fraction of the model's scaled predictions that equal to the prediction the original input. The threshold in Table is set to 0.5.
Fairness Considerations in Comparison. We mostly follow the baselines' original implemen- tations if available. Moreover, to ensure their hyperparameters and implementations work in our settings (model architecture, optimizers, etc.), we also try to tune their hyperparameters if necessary, in order report their best overall results. Most of these baseline defenses (other than those require no clean samples or those not sensitive to the number of clean samples) are given access to the exactly same clean reserved data (2,000 samples) as BaDExpert, which further ensures fairness in our comparison.
A.5 COMPUTATIONAL ENVIRONMENTS
We run all experiments on a 4-rack cluster equipped with 2.8 GHz Intel Ice Lake CPUs and Nvidia A100 GPUs. Our major experiment requires training 63 models (~50 GPU hours in total), with an additional 100 GPU hours for ablation studies (e.g. training ImageNet models).
B DISCUSSIONS
B. FORMULATION OF AGREEMENT MEASUREMENT (HARD-LABEL DECISION RULES)
Let us fist consider an ideal backdoor expert B that completely unlearns the normal functionality of the backdoored model M while fully preserving its backdoor functionality, i.e.,
P P(..w)~P B(T(z)) ([T() 11
(4)
P(a B(x) # M|
u| 2
(5)
Under this condition: 1) fully inhibit the embedded backdoor in M from activation (i.e., reduce the ASR of M to 0%) by simply rejecting all inputs wherein predictions of M and B fall within agreement. This is because B and M will always agree with each other backdoor
17

---

Published as a conference paper at ICLR 2024
5000
5000
4000
Algguts
4000
AUROC: 96.76%
AUROC: 96.35%
3050
3000
2000
2000
1000
0-
Confidence
Confidence
(a) Backdoor expert B.
(b) Auxiliary model M'
Figure 7: Confidence distribution for clean and backdoor (Blend attack on CIFARIO) inputs with regards to the intial predictions of the originally backdoored model M. As shown, B tends to assign higher confidence to backdoor inputs and lower confidence to clean inputs, while M' does the contrary. When directly apply the described soft rule, using either B or M', to distinguish backdoor inputs from clean inputs, we can achieve a high AUROC (> 96%).
input ((x) that can exploit M (Eqn 4: 2) Meanwhile, this rejection rule will not impede the CA of M, because B will always disagree with M on clean inputs x that M correctly predict (Eqn [ This example thus suggests the feasibility of performing accurate backdoor input detection via measuring whether the predictions of the backdoored model and backdoored expert concur.
B.2 FORMULATION OF AND INSIGHTS INTO THE SOFT DECISION RULES
As discussed in Sec 3.2, for practical implementation, we can generalize the hard-label conditions soft version that based the soft-label (confidence-level) predictions. We can derive soft decision rules for both the backdoor expert B and the auxiliary model M'.
Trivially, for any backdoor expert B, the following soft conditions must establish:
3rg 72
[0, 1).
(6)
P (a.u)>
(tn n M(T(x)) = 11,
=1]=1,
P Pa
Confs(yla)< < M(x) 1
(8)
Given an input G. we define soft decision rule that rejects 7 conditional on (Mn)
T 2 72, applying a T E [72, Ti] will still result in a perfect backdoor input detector. In suboptimal cases where T1 T2, we will have trade-off between the TPR and FPR during detection. Generally, B could serve as a good backdoor input detector if B tends to assign higher confidences for backdoor inputs and lower confidences for clean inputs, which practically true (Fig 7a). we directly apply this confidence-leve rule to detect backdoor inputs with backdoor expert B (Blend attack on CIFARIO: 10), we can achieve 96. 76% AUROC.
On the contrary, auxiliary model M' must satisfy the following conditions:
3ra E [0, s.t.
(9)
P (1.N)~>
Conf me (t]T(x)) S Ta M(T(x) = =1]=1,
(10)
Conf wy (3/x) > M(x)) y) 11
P P(a.a)~P
(11)
where T3, T4 E J0, 1]. Given input T. we can define soft decision rule that rejects 7 conditional on Conf (M( < If T3 < 74, applying a T' E [Tg, Tı] will still result in perfect backdoor input detector. In suboptimal cases where T3 2 T4, we will have trade-off between the TPR and FPR during detection. Similarly, M' could serve as a good backdoor input detector if M' tends to assign lower confidences for backdoor inputs and higher confidences for clean inputs, which is also
18

---

Published as a conference paper at ICLR 2024
practically true (Fig 7b).
Analogously, if directly apply this confidence-level rule detect the same backdoor inputs, but with a finetuned auxiliary model M we can achieve a 96.35% AUROC.
To sum up, M', can already detect backdoor inputs effectively (> 96% AUROC), Our BaDExper framework, built top of both the two models B and M' via ensembling, achieves even better detection performance (AUROC > 99%)
that these soft decision rules, based the confidence-level information of B and
B.3 OTHER POSSIBLE MISLABELING STRATEGIES
In Alg 1 we specifically mislabel clean samples to their neighboring classes, i.e., Y' < (Y+1)%% In our preliminary experiment, we have actually explored three different mislabeling strategies:
1. Shifting Y' + (Y 1)%C (adopted in Alg D:
2. Randomly mapping Y to any Y' as long as Y + Y';
3. Change the one-hot label = 00,0. 0.1,0. 0] to =
(0 < 1) in a soft-label fashion.
Surprisingly, find the phenomenon "finetuning a backdoored model a few mislabeled clean samples can cause the model to forget its regular functionality, resulting in low clean accurucy, but remarkably, its backdoor functionality remains intact, leading high attack success rate" exists regardless of the adopted mislabeling strategy choice. And as matter of fact, BaDExpert with each of the three strategies would be similarly effective against diverse set of attacks. We finally settled at the first mislabeling choice mostly due to stable set of hyperparameters are easier to determined than the other two strategies.
B.4 COMPARING BADEXPERT WITH CONFUSION TRAINING (QI ET AL. 2023B)
Qi et al. 2023b) introduces novel backdoor poison training set cleanser based the technique of "confusion training", where they train an inference model jointly on the poisoned dataset and a small number of mislabeled clean samples (similar critical differences between our work and theirs.
Alg [[). Nevertheless, we highlight several
Qi et al. (023b) focuses on poisoned training set inspection and aims at identifying poison samples within the training set. Instead, our work focuses on identifying backdoor inputs during inference time. The two problems have completely different setups.
Problems.
Methods.
Our method and Qi et al. (023b) different at critical levels:
1. Access of Information:
Qi et al. (2023b) relies on the necessary access of poisoned training samples (i.e., requiring information about the backdoor), that their detection model can capture the backdoor correlation.
Our method, on the other hand, operates independently of how the model is generated does not rely on any information about the backdoor samples or the poisoned datasel, which is significantly more challenging scenario.
2. Principle:
During the training on a poisoned dataset, Qi et al. 2023b) disrupts the fitting of clean training samples using a "confusion batch" of mislabeled clean data (i.e., counteracts the gradient updates learned from the normal training samples), so that the resulting model can only capture the backdoor correlation. They then utilize this resulting model to identify poisoned training samples by seeing which data points are correctly fitted (by comparing the model's predictions with the data points' ground-truth labels).
Distinguishly, our method is more related to catastrophic forgetting - we only finetune the original backdoor model M on the mislabeled clean data (without any access to poisoned samples), resulting in a backdoor expert model B that loses the normal functionality but retains the backdoor funetionality. We then measure the agreements
19

---

Published as a conference paper at ICLR 2024
Table 6: Defensive results on GTSRB (CA and ASR).
Defemsss
No Defene
FP
NC
MOTH
NAD
STRIP
AC
Frequescy
SCALE UP
BuDiEspeet
A
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
No Amacs
97.1
85.4
97.1
K2.2
96.3
87.4
90.8
05.0
00.8
97.0
Devdopreent Stage Atockks
BadNet
97.3
92.0
[100,0]
96.3
96.9
88.2
Trojan
97.1
10000
87.3
98.5
98.1
0:1
82.0
100,0
96.4
34.4
87.4
64.5
96.9
100.
65.1
0.9
60.4
95.0
96.9
3.7
Dynaii
97.1
10000
87.0
100,0
994
14.5
79.0
100.0
96.9
644
87.4
171
96.6
1000
65.0
120
96.3
24.6
97.0
0.0
WalNet
95.7
91.4
84.1
5.4
98.2
17.4
94.5
89.5
970
0.6
86.2
56.0
95.5
91.8
04.1
61.8
56.7
79.2
95.6
0.2
Wost-devehpmenttoo Attacks
TrojanNN
14.8
96.2
78.9
95.0
96.4
Average
14.1
85.1
47.5
M0.5
Drup banaller betier)
10.3
ASB Drop (brger better
150.8
1340 10.5
1748
150. 1000 188. 1514
196.
between the resulting backdoor expert model B and the original backdoor model M, in order to identify the backdoor inputs at inference time. Notice that our approach operates without access to the ground-truth labels of inference-time inputs.
B.5 COMPARING BADEXPERT WITH SEAM (ZHU ET AL., 2022
Zhu et al. 20222) introduces novel model-repairing backdoor defense (SEAM). In the first phase, they finetune the backdoored model on small number of mislabeled clean samples (similar to our Alg (T), observing that both the CA and ASR would diminish. In the second phase, they finetune the resultant model (after phase one) on a portion of correctly labeled samples from the training set, by which the CA will gradually recover, but the ASR will not.
Interestingly, when finetune the backdoored model on mislabeled clean samples, Zhu et al. 2022) observation (both CA and ASR decrease) seems to be different from ours (CA drops but ASR retains). Nevertheless, we argue that our observations are actually not contradictory to theirs.
In our method and experiments, suggest using conservatively small (un- Mearming rate 17. with which only the normal functionality degrades but the backdoor functionality retains. However, as shown in Fig (and Fig g in Appendix C.3), when the (un-)learning rate v is large enough (e.g., 10 10-3). both the normal and backdoor functionality would be lost (both CA and ASR + 0) - which corresponds to Zhu al. (2022)'s 's observation. In summary, the different observations between our
work and Zhu et al. 20222 are possibly due to different selections of the (un- (un))leanning rate.
B.6 RELATIONSHIIP OF BADEXPERT WITH SHAN ET AL. 2020
Shan et al. (020) proposes an adversarial example detection method using "honeypots" - a trapdoor that would enforce malicious adversarial inputs to manifest a certain neural network activation-pattern signature. Their defense's 's key design philosophy may be subtly connected to ours, in the sense that Shan et al. (2020) detects potential adversarial examples via "similarity measurement" of model activation signatures, and we detect backdoor examples via "agreement measurement" of model prediction/cconfidecce. However, their work's motivation, problem, and method are still largely different from ours.
C ADDITIONAL RESULTS
C.I EFFECTIVENESS OF BADEXPERT ON GTSRB
Our primary results GTSRB are shown in Table and Table As a general post-development defense, BaDExpert effectively defends against all attacks (average ASR = 2.0%). with CA drop as negligible 0.1%; Meanwhile, other baseline defenses fail against least backdoor attack. As a backdoor input detector, BaDExpert achieves an average 100% detection AUROC, and outperforms other baseline detectors in every scenario.
20

---

Published as a conference paper at ICLR 2024
Table 7: Defensive results on GTSRB (AUROC).
AUROC (%)
BadNet
Trojan
Dynamic
TrojanNN
Average
STRIP
93.4
74,8
76.5
86.2
41.6
92.2
88.7
79.0
52.7
50.2
62.7
34.2
48.3
30.9
44.5
Frequency
75.3
73.6
75.3
72.8
61.3
73.6
75.3
72.4
SCALE-UP
88.6
54,2
34.0
81.1
34.1
56.4
90.4
62.7
BaDExpert
100.0
99.9
100.0
100.0
100.0
100.0
99.9
100.0
Table 8: AUROC of BaDExpert with baseline (FP, NC, MOTH and NAD) repaired models M'' ("w/ Backdoor Expert") on CIFARIO. 10. We use the exact ensembling decision rule in our primary experiment. Results of directly deploying the repaired models, following the soft rule described in See B.2 are shown in "w/o Backdoor Expert" rows. Obviously, our backdoor experts in the BaDExpert framework serve as effective augmentations (add-ons) for these baseline methods during backdoor input detection.
Baseline as M
Anacks +
BadNet
Blend
Trosn
CL.
SIG
Dynamic
ISSIA
WaNett
Bpp
FT
TrojonNN
SRA
Averuge
win Reckslorr Fapert Ruckdooe Expest
64.3
64.4
8.9
97.2
99.2
58.6
99.5
90.6
99.8
66.6
04.8
99.3
99.9
99.5
99.4
96.3
NC
win Reckdoerr Fxpert
Rackdooe Expest
09.9
100.
99.9
66.7
97.6
99.5
99.9
98.8
95.3
98.7
99.9
98.3
100.0
1000
88.6
98.7
MOTT
win Reckokorr Fxpert
Ruckdioc Expest
09.7
99.6
90.9
37.0
87.4
98.8
52.6
02.3
99.4
94.7
1000
80.5
96.6
win Reckolorr Fapert
98.9
94.9
92.9
98.0
99.6
97.1
97.0
Ruckdioor Expest
100.
98.6
09.3
08.6
99.8
98.0
95.1
99.5
97.6
1000
988
C.2 ENSEMBLING WITH OTHER DEFENSES
As discussed in Sec 4.2.3 we can apply any baseline-repaire models as M' in our BaDExpert framework, to ensemble with our backdoor experts B. We demonstrate the ensembling results in Table ("ww Backdoor Expert" rows). For an insightful comparison, we also show the results when only the seseline-repiired model is used for backdoor input detection ("wwo Backdoor Expert" rows), following the soft decision rule for M described in Sec B.2
As shown, BaDExpert which align well with our major results in Table 2 When combined with FP (failed against 9 of 12 attacks in Table and MOTH (failed against 6 of 12 attacks in Table [T. BaDEEpert slightly degrades to ~ 96.5%, due to the significant failures of the baselines themselves (which can also be told from that deploying M' without backdoor expert can sometimes barely achieve AUROC < 50% worse than random guessing). Moreover, in almost all cases, BaDExpert ("w/ Backdoor Expert") achieves higher AUROCs compared to deploying M' alone ("w/o Backdoor Expert"). In other words, our backdoor expert models and the BaDExpert framework could serve effective augmentations (or add-ons) to existing mopel-repairing backdoor defense baselines, during backdoor input detection.
achieve overall 99% AUROCS when ensembling with NC and NAD,
C.3 BACKDOOR EXPERTS CONSTRUCTION
For all 12 attacks evaluated in our primary experiments on CIFARIO, we visualize in Fig Tag-aaBBB the constructed backdoor experts' (Alg I) CA and ASR, when we unlearn the originally backdoored model M with different rj's. s. As depicted, with a conservatively small 1) (e.g. 10-4), can always enforce the resulting backdoor expert to lose significant amount of normal functionality (CA drop ~ 50%). while retaining a similar backdoor functionality (ASR drop 0%). However, we choose a large n (e.g. 1e-3), both functionalities would be erased (both CA and ASR zu 0%). Actually, we sometimes may have to tradeoff' between the maintenance of the backdoor functionality and the unlearning of the normal functionality, But overall, we see that unlearning the backdoor functionality is slower than unlearning the normal functionality. More crucially, we find this phenomenon to consistently exist across datasets and architectures (Fig 8m Sop.
An intuitive justification for such phenomenon can be referred to and derived from Qi et al. (003b)), where the authors show that during poison training, clean samples could be forgotten faster than poison samples in the context of catastrophic forgetting (for simplified settings training overparameterized linear model).
21

---

Published as a conference paper at ICLR 2024
CA 1000
CA ASR
de5 No5 dod 2-4 464 So4k
Sc5 6-3
204
5-5
5o3 1e4
303
(Undleoranng Batt i
(Undearning Rate
(Unteaming Rote
(a) BadNet
(b) Blend
(c) Trojan
ASR
ASB (56)
02 180
N:5 Rod So4
5c5 R- f-- led 2e4 3ed
Sc5 kr 8-8 to led
(Unlleoriing Bate 7
(Unseesritng Rabe
[Unteerning Rate
(d) CL
(e) SIG
(I) Dynamic
C2 000-880 2000
CL ASB
40
20
Non Bwi and 2et Brd Sed
ard Bebs le 3rd
500 Beck
Snd Sncl (Unjlerrnin Bate
9-- Be Ie
(inclearing q
(Unjierring Babe
(g) ISSBA
(h) WaNet
(i) BPP
AS (80)
00
40
20
20
5e4 Ge5 Te. Bed aed ind Jos Sod Cert
S6C 6e5 Ted BeC 965 let 3rd Jort
serd www.ddd-dddd.. dectet a-d des Card fard
(Cnclouraing
(Unfberning Babe
(Unllearning Rate
(i)
(k) TrojanNN
(l) SRA
C/ 0001 (N4)
7200
ASR (54)
60-
60
40
40
20
20
365 605
9K5 963 1e4 2e4 304 4ed Sc4
303
365 603 763 9:5 5o3 1ed 254.550 So4 364
(Uniteering i
(Untierning Rate
(Untearning
(m) GTSRB
(n) ImageNet
(o) VGG16
Figure 8: Unlearning curves with different of diverse scenarios. Fig 8a correspond attacks conducted on CIFARIO (ResNet18), 8); Fm and 88 correspond to Blending attacks on GTSRE and ImageNet (ResNet18), respectively; corresponds to the Blending attack on CIFARIO (VGGI6 instead of R8...
100
CA oF ASR (%6
80
60
40
20-
ASR
Size of Cloan Sot [Pd]
Figure 9: Backdoor experts constructed with reserved clean sets of different sizes.
22

---

Published as a conference paper at ICLR 2024
Table Ablation results on different (un-)learning rates 77. AUROCs BaDExpert are reported on CIFAR1O.
(Un Wearning Rate it
30-00-1
6- 10-1
T- 1-10
8-100
9--00-
1+:0--
2+:00
$+10-1
4+10--
3+10
1-10-5
3-10-1
BodNet
100.0
100.0
100.0
100.0
100.
100.0
1000
1000
1000
1900
99.9
908
Blend
97.6
95.1
98.6
98.7
99.9
99.2
99.2
9900
99.1
992
95.2
925
Table 10: Ablation results on different poison rates. Clean accuracy (CA), attack success rate (ASR) and AUROCS of BaDExpert reported CIFAR10 against Blend attack.
Poison Rate
0.1%
10.0
(before defense)
94.0
ASR (hefore defense)
87.3
99.9
CA (after defense)
92.7
93.1
93.2
92.6
93.3
92.2
93.0
93.1
ASR (after defense)
11.8
3.4
10.8
7.2
13.3
8.0
0.3
0.0
AUROC
99.2
99.2
99.2
99.1
99.4
98.9
100.0
100.0
We also notice that, this property is not sensitive w.r.t. the number of clean samples ((D)) we have. No matter how many clean samples (from 10 1,000 2,000) we use conduct Alg [T] with an appropriately small learning rate 1) (e.g. by selecting a small such that the resulting CA only drops 40%), we can still separate the CA and ASR by a certain extent (Fig
C.4 ABLATION STUDIES ON DIFFERENT (FARNNGG RATES ?)
Our method is not sensitive to the choice of (un- Mearning rate 7). In this ablation study, evaluate BadExpert against BadNet and Blend attacks on CIFARIO, across a wide range of r (from 10-5 to 5 10-3). As shown in Table 9 the AUROC of our detection remains stable across different n).
C.5 ABLATION STUDIES ON DIFFERENT POISON RATES
As shown in Table 10 BaDExpert is insensitive to the variation of poison rate. Even when the poison rate is extremely low (poison rate 0.1%, equivalent to only 50 poison samples) and ASR drops to 80%, our BaDExpert is still manifesting near-perfect effectiveness.
C.6 COMPARING BADEXPERT WITH STRIP AND SCALEUP IN SCENARIOS WITH FEWER CLEAN SAMPLES
We further analyze in details how BaDExpert would perform when there are fewer accessible clean reserved samples. We compare it alongside with two strong baseline detectors: 1) STRIP,
Table 11:
Comparison of BaDExpert with STRIP and ScaleUp alongside, when they are all given equal access to different number clean samples. AUROCs of all defenses are reported on CIFARIO against Blend attack.
Number of Clean Samples
Defense
BadNet
Blend
ScaleUp
95.7
79.8
100
STRIP
99.2
42.5
BaDExpert
99.9
72.9
ScaleUp
95.7
79.9
200
STRIP
99.3
44.4
BaDExpert
100.0
91.2
ScaleUp
95.7
79.9
400
STRIP
99.3
BaDExpert
100.0
ScaleUp
95.8
80.0
2000
STRIP
99.4
44.6
BaDExpert
100.0
99.2
23

---

Published as a conference paper at ICLR 2024
Table 12: Adaptive attack by using weakened triggers at inference time. (Blend attack on CIFARIO)
Blending Alpha
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
20% (Standard)
ASR (before defense)
5.0
10.1
17.8
28.0
39.7
51.6
62.8
72.3
80.2
86.3
90.6
ASR (after defense)
3.0
5.4
8.5
11.9
15.3
18.5
14.8
16.2
14.0
15.5
11.4
AUROC
94.4
96.2
97.3
97.8
98.1
98.4
98.6
98.7
98.9
99.0
99.2
which requires only N = 100 clean samples by default, and 2) ScaleUp, which by default can work without clean samples (meanwhile, ScaleUp can also utilize available clean samples for its SPC value estimation). To make fair comparison in additional study, we assign equal number of clean samples (100, 200, 400 and 2000) to all three defenses in each scenario.
In Table I we can quickly notice that under most circumstances (number of clean samples 200, 400 and 2, 000), BaDExpert performs the best. And when the number of clean samples is extremely restrited to 100, BaDExpert becomes less effective on Blend (72.9% AUROC), but still performs similar to ScaleUp (79.8% AUROC).
Approaches to Obtain Reserved Clean Samples. There exist various possiblitites to acquire such a small clean set [De]:
A developed model usually comes with an independent set of test samples in order to validate the model' utility, the defender could simply use (a partition) of this test samples as [De];
Collect data (e.g. taking photos) in secure environment;
Manually inspect and isolate clean samples from online inputs.
Furthermore, there exists trending line of work on isolating clean samples from a poisoned set of samples, e.g. META-SIFT (Zeng et al.] (2233a). Such work can be directly applied here to aid the defender to obtain reserved clean samples. Overall, we argue that obtaining small amount of clean samples (e.g. 200, 400, 2000) would not be major bottleneck for defenders in practice.
C.7 ADDITIONAL ADAPTIVE ANALYSIS DETAILS
C.7.1 NATURAL BACKDOOR
Natural backdoors (e.g. (Zhao et al. 2022 may exist in models trained on only clean data, which can also be considered as an adaptive backdoor attack against BaDExpert. This is because the natural backdoor functionality learned by clean model is strongly correlated with the normal functionality therefore, such natural backdoors also directly challenge our insight that "backdoor functionallyy can be isolated from normal functionality".
We construct such natural backdoor attacks (Zhao et al. 2022) on normally trained models (on clean data) and achieve ASR = 99. 99.2% on CIFARIO and ASR = 99.3% on GTSRB. We find that BaDExpert can defend the natural backdoor attacks on both datasets with AUROC 92.3% and AUROC 92.8 %, respectively. While the performance shows degradation compared to our major results in Table 2 in our paper (average AUROC = 99.0%), the >92% AUROC still reflects the nontrivial effectiveness of BaDExpert
inference-tine time defense. Besides, note that our key finding "when fintuning on mislabled clean data, the backdoor functionality would remain, while normal functionality does not" still stands against this attack:
Following Alg I on CIFARIO. the constructed backdoor expert B retains high ASR 95.3% (originally 99.2%), while the normal functionality degrades significantly (CA drops from 94.2% to 44.8%);
On GTSRB, the observation is similar (ASR drops from 99.3% to 83.0%, CA drops from 96.8% to 41. 1.%).
24

---

Published as a conference paper at ICLR 2024
(a) 4-Pixel
(b) 1-Pixel
Figure 10: Smaller triggers demonstration.
Table 13: Adaptive attack by using smaller triggers in Fig 10 (CIFARIO)
Attack
4-Pixel
1-Pixel
CA (before defense)
94.2
93.7
ASR (before defense)
88.5
77.4
CA (after defense)
93.3
92.7
ASR (after defense)
14.4
5.4
AUROC
98.9
99.6
C.7.2 ADAPTIVE, ATTACK BY USING WEAKENED TRIGGERS AT INFERENCE TIME AND SMALLER TRIGGERS
Empirically, adversary may sometimes bypass existing backdoor defense methods by using weak- ened version of the backdoor trigger at inference time. Therefore, we also study how our method reacts to such an adaptive attack by decreasing the inference-time trigger blending alpha of the Blend attack CIFARIO. As shown in Table 12 when the adversary uses lower blending alpha at inference time (instead of the default 20% during poisoning), the AUROC of BaDExpert indeed degrades, to low 94%. Nevertheless, the attack ASR drops more rapidly. Overall, observe a tradeoff between backdoor inputs' stealthiness (AURÓC) and the attack's ' effectiveness hardly evade BaDExpert's detection by using
(ASR). Generally speaking, the adversary weaker trigger at inference time, since the ASR will drop rapidly way before BaDExpert becomes unusable (AUROC > 97.5% whenever ASR 20%).
In addition, we also study whether backdoor attackers could evade our defense via adopting smaller trigger sizes. Specifically, we evaluate BaDExpert's effectiveness against two backdoor models that are poisoned with: 1) 4-Pixel (Fig 10a 4 random red pixels as the backdoor trigger (0.1% poison rate); 2) 1-Pixel 10b): random red pixel the backdoor trigger (0.5% poison rate). To ensure the backdoors are "weak" we select the minimum poison rates to make sure the backdoors are just successfully injected (non-trivial ASR). As shown Table 13 BaDExpert can still effectively defend these attacks, suppresing the ASR to < 14.4%.
C.7.3 A TAILORED ADAPTIVE ATTACK AGAINST BADEXPERT
To provide more valuable insights for future researchers developers into our method, we also tailored a novel adaptive attack against our proposed BaDExpert.
Concretely, we assume that the adversary (who produced the backdoor model) is aware of our BaDExpert defense. Accordingly. he/she will design an evading strategy via using an alternative trigger during inference time (i.e., asymmetric backdoor trigger, which was adopted in prior work like Qi al. (023a)), such that the alternative trigger satisfies: 1. Can still activate the backdoor functionality of the original backdoor model M (i.e., achieving high ASR) 2. Cannot activate the backdoor functionality of the original backdoor model (i.c., enforcing to provide a low confidence on the backdoor inputs).
Notice that the adversary is only assumed to have access to the backdoor model M. The adversary, however, could also follow BaDExpert procedure to construct surrogate backdoor expert model B'' (Alg 1) to the actual backdoor expert model used by the victim defender. (Empirically, we find the attack results using this surrogate backdoor expert model are similar to using the actual B of the
25

---

Published as a conference paper at ICLR 2024
Table 14: Tailored adaptive attack against BaDExpert. (CIFARIO)
0.05
0.01
Attack
10
10
10
10
Average
Norm
4.8
1.8
16.6
5.0
36.7
5.1
59.8
5.3
16.9
BaDEspprt-Adg--BuuNe
ASR
20.8
57.6
61.8
83.2
82.5
89.2
94.5
98.1
73.5
AUROC
93.3
99.8
99.8
65.4
99.8
57.2
Norm
10.2
32.6
29.5
44.2
38.6
59.1
53.5
34.9
BaDExpert-AAe PeEtiper--Adp-Bliee Blend
ASR
11.4
27.2
72.7
75.7
83.7
86.5
96.7
97.2
68.9
AUROC
63.0
72.7
61.1
68.7
47.1
58.1
36.4
54.2
57.6
defender, they both are trained following the same configuration. We therefore do not explicitly distinguish the notation of B and B in the rest of this section.)
The two goals upon can be implemented via optimization on the models M and Formally, as the adaptive adversary, we compute an alternative trigger mark A and the corresponding trigger mask m as follow:
min CrossEntropyLoss (M(x)) Jraw 1) A Confo(t|x")) A2 |ml (x, A.m
(M(x"))aw
(12)
where x² =1= Ox m=
(13)
Specifically, the trigger mark A E X [0, 1jxxux is 3-dim real-value matrix (the same shape as the input z), while the trigger mask E [0, 1] is 2-dim real-value matrix that decides the opacity of each trigger mark pixel to mix with the corresponding imput rs' pixel. The @ operator upon computes the Hadamard product of two matrices. (We adopt the similar trigger optimization setting in Wang al. (2019)))
Now we explain this optimization formula:
Minimizing the first CrossEntropy term will help the adversary approach the first target: the alternative trigger (A, m) can still activate the victim backdoor model M with high ASR.
The second term represents the confidence of the backdoor expert model B. Minimizing it will directly violate our core design insight, since the resulting backdoor inputs now will not activate the backdoor functionality of the backdoor expert model B anymore.
The last term, & norm of the trigger mask, corresponds the visual stealthiness of the trigger. The adversary usually prefers to implant a less noticeable trigger, which can be realized via minimizing the magnitude of the trigger mask m.
The two hyperparameter A and A controls the tradeoff among these three goals.
AT- TACK]") against our BaDExpert defense when and X2 are selected differently. To fully study the potential tradeoff between the adaptive attack effectiveness and its defense evasiveness, we report alongside 1) the adaptive trigger Norm |m|; 2) the attack success rate (ASR); 3) AUROC of BaDExpert against the tailored adaptive attack.
In Table 14 we report the results of this adaptive attack
As shown in Table 14 we can observe that tailored adaptive attack could effectively diminish effectiveness of BaDexpert AUROC becomes low 57.2% against BaDExpert-Aday Adap-BadNee and 36.4% against BaDExxert-Adia -ade--eer-ddd--leededd -lend....... Meanwhile, we can also notice two trends from the table:
1. As >a becomes larger (with X fixed), the ASR becomes higher, while the defense AUROC decreases. However, the increasing evasiveness of the adaptive attack comes with a price: the magnitude of the trigger mask (Norm) also increases i.e., the backdoor attack becomes less stealthy.
2. As \1 becomes higher (with X2 fixed), the defense AUROC effectively degrades. However: 1) the ASR drops, and 2) the backdoor attack becomes stealthier since Norm becomes larger.
26

---

Published as a conference paper at ICLR 2024
Table 15: Comparing BaDExpert with additional baseline defenses:
ANP
Wu & Wang (02 1..
I-BAU Zeng et al. (2021a), ABL Li et al. (2021a), AWM Chai & Chen and CD Huang al. (2023). CA and ASR are reported on CIFARIO.
2022
RNPLI al. 2023
Attack
No Defense
ANP
I-BAU
ABL
AWM
RNP
CD
BaDExpert (Ours)
CA
94.1
80.8
88.0
92.5
92.7
70.0
78.6
93.1
BadNet
ASR
100.0
0.4
0.8
11.7
4.8
18.9
2.9
0.0
CA
94.1
83.2
90.4
91.9
88.9
77.9
79.0
93.1
Blend
ASR
90.6
16.8
16.6
11.8
29.7
19.6
72.8
11.4
In brief, we can see that the adaptive attack can indeed restrict BaDExpert's performance, at the cost of either attack effectiveness (lower ASR) or stealthiness (higher Norm). Overall, BaDExpert still performs nontrivially in most scenarios 87.0% average AUROC against BaDExpert-Adap BadNet and 57.6% against BaDExpert-Adap-Bleee
C.8 COMPARING BADEXPERT WITH ADDITIONAL BASELINES
In Table 15
further compare BaDExpert with 6 additional recent backdoor defense baselines: ANP Wu & Wang 2021). I-BAU (Zeng et al. 2021a), ABL[Ci et (2021a). AWM AWM[Chai
(Chai & Chen (0022). Ri et al. (2023) and CD al. (2023), w.r.t. attack success and clean accuracy.
Among them: 1) ABL is poison suppresion method that happens in the model development stage; 2) ANP, I-BAU, RNP and AWM are post-development model-reparing defenses; 3) CD is originally poisoned training dataset cleanser, and we adapted it to backdoor input detector. As shown in BaDExpert outperforms of them, achieving lower ASR and higher CA.
Table 15
C.9 STANDARD DEVIATIONS OF MAJOR EXPERIMENTS
Standard deviations of our major experiments Table I
2
6 and are shown in Table 16
17
18
and
19
Table 16: Standard deviation of Table
Defenses-+
No Defense
FP
NC
MOTH
NAD
STRIP
AC
Frequency
SCALE-UP
BaDExpert
Attacks
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR CA
ASR
CA
ASR
No Attack
0.3
1.1
1.4
0.6
0.3
0.4
0.0
Development-Siin
BadNer
0.4
0.1
Blend
0.1
1.3
0.7
1.4
0.6
1.7
0.1
11.5
0.
4.7
0.1
1.1
0.6
14.4
0.1
0.2
0.3
2.8
0.1
Trojan
0.2
0.1
0.7
36.0
0.2
0.5
0.5
2.1
0.2
2.8
0.2
15.3
0.2
0.4
0.2
0.0
2.5
5.8
0.2
3.3
CI.
0.2
0.0
0.
7.3
0.2
0.5
0.3
0.5
04
3.5
0.2
3.7
0.2
0.3
0.3
0.0
1.1
0.0
0.2
7.1
SIG
2.6
12.1
Dysamic
0.3
43.2
ISSBA
0.3
0.0
0.3
0.3
2.0
34.3
0.6
0.2
0.1
0.0
0.1
WaNei
0.6
0.9
1.8
2.6
0.8
20.3
0.4
10.1
1.3
0.4
0.5
0.7
0.6
3.8
0.5
0.9
0.1
1.6
0.5
0.3
BPP
0.5
0.2
0.4
8.8
0.3
57.3
0.2
1.5
0.7
0.2
0.4
3.7
0.5
0.6
0.6
0.0
4.7
10.2
0.5
0.1
Post- Post-develppennt Attacks
16.2
TrojanNN
0.0
0.4
02
0.0
1.4
SRA
1.4
0.0
1.2
0.0
0.4
0.6
04
0.5
2.6
0.4
1.2
0.9
1.4
0.0
1.0
0.0
0.8
5.3
14
0.4
Table 17: Standard deviation of Table
AUROC (%)
BadNet
Blend
Trojan
CI.
SIG
Dynamic
ISSBA
WaNer
Bpp
FT
TrojanNN
SRA
STRIP
0.2
2.2
10.6
3.4
6.8
2.0
1.4
0.8
13.1
1.4
0.3
0.5
AC
0.0
10.1
0.1
0.0
2.0
7.2
2.8
0.4
0.2
15.3
0.0
0.0
Frequency
0.1
0.1
0.0
0.1
0.0
0.1
0.1
0.2
0.1
0.1
0.1
0.1
SCALE-UP
0.6
1.5
4.4
0.7
1.8
0.4
0.4
0.8
8.9
2.6
0.8
2.4
BaDExpert
0.0
0.1
0.2
0.4
0.1
0.1
1.0
0.0
0.0
0.1
0.6
0.0
27

---

Published as a conference paper at ICLR 2024
Table 18: Standard Deviation of Table 6
Defeuses-+
No Defense
FP
NC
MOTH
NAD
STRIP
Frequency
SCALE
BaDexpert
CA
04
Development-Sung Stage Attacks
BadNet
0.6
0.0
17
11.5
1.0
0.0
0.6
0.0
0.3
0.1
0.8
1.7
0.8
0.1
0.5
0.0
2.7
0.1
0.5
0.1
Blend
0.5
0.3
0.7
0,8
0.1
1.1
1.1
0.3
13.0
0.5
5.6
0.5
0.2
0.1
0.9
4.3
29
Trojon
0.0
0.2
Dynamic
0.1
0.1
0.0
WaNet
0.2
0.5
2.1
0.2
20.3
0.1
0.1
0.1
0.2
0.5
0.1
0.8
0.0
Pust-development Attacks
FT
0.6
0.1
0.5
0.0
0.6
19.9
4.7
1.3
0.2
18.2
0.6
4.0
0.6
0.1
0.4
0.0
1.1
1.7
0.6
4.7
TrojunNX
0.7
2.4
1.0
14.7
0.3
3.2
0.6 10.5
2.4
0.0
1,0
Table 19: Standard deviation of Table
AUROC (%)
BadNet
Blend
Trojan
Dynamic
WaNet
FT
TrojanNN
STRIP
1.8
2.6
2.7
1.6
1.1
1.9
3.7
AC
29.9
7.6
0.6
9.8
2.2
6.2
0.2
Froquency
0.1
0.0
0.0
0.1
0.0
SCALE-UP
0.6
1.9
4.9
2.8
BaDExpert
0.0
0.2
0.0
28


Reviews:
[{"review_id": "wm4KPA3wwp", "reviewer": "Reviewer_Xe68", "summary": "This paper introduces BaDExpert, an innovative defense mechanism against backdoor attacks targeting deep neural networks (DNNs). The defense is built upon the concept of extracting the backdoor functionality from a backdoored model to create a backdoor expert model. The backdoor expert model is then used to detect backdoor inputs during model inference. BaDExpert's efficacy is showcased across various datasets and model architectures, highlighting its impressive performance in terms of AUROC, a significant reduction in Attack Success Rate (ASR), and a minimal decline in clean accuracy (CA).", "questions": "(1) A core tenet of the proposed method is that fine-tuning on a small set of mislabeled clean data can isolate the backdoor functionality. While this paper attempts to validate the idea through experimentation, providing a rigorous theoretical analysis would bolster the method's credibility.\n\n(2)  In real-world scenarios, after acquiring a model online, it's often uncertain whether it has been backdoored. If the model is a benign one, there would be a disagreement between the outputs of model \\mathcal{M} and \\mathcal{B} (on the left side of Figure 2). This divergence could potentially hinder BaDExpert's performance.\n\n(3) Could this paper elucidate the time complexity of the proposed method and compare it with methods like I-BAU? Given that this technique necessitates model fine-tuning, there are concerns about its efficiency.\n\n(4) It seems that some recent published related works are missing to be compared in the paper. For example,  [1] presents defense results that are on par with those in this paper, reporting an ASR of 5.03 and a CA of 92.18 for the CIFAR10 dataset.\n\n(5) Publicly releasing the code would facilitate better reproducibility and peer verification, enhancing the paper's value.\n\nIf the authors could solve some concerns mentioned above, the reviewer would reconsider the rating.\n\n[1] Li, Y., Lyu, X., Ma, X., Koren, N., Lyu, L., Li, B., and Jiang, Y.G., 2023. *Reconstructive Neuron Pruning for Backdoor Defense*. arXiv preprint arXiv:2305.14876.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a novel approach for defending against backdoor attacks by extracting the backdoor functionality from a backdoored model.\n- The paper provides a well-structured explanation of the methodology.\n- The paper presents extensive experimental results on multiple datasets and model architectures, demonstrating the effectiveness of BaDExpert.", "weaknesses": "- The paper lacks in-depth theoretical analysis to support the proposed method.\n\n- The technique may not perform optimally when applied to models that haven't been backdoored.\n\n- The experimental section seems to omit comparisons with certain recent relevant works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GT10ZducAn", "reviewer": "Reviewer_59E7", "summary": "This paper proposes a novel backoor detection approach: It firstly finetune the backdoor model over a small subset of mislabeled samples to remove the benign functionality while preserving the backdoor-related functionality. Then, badexpert detects the backdoor examples based on the agreement between the backdoor expert and backdoored model. The effectiveness of the propsed method is tested on both small  (CIFAR-10 and GTSRB) and large dataset (ImageNet), CNN and ViT.", "questions": "1 Table 1 shows that BadExpert is relatively weak to defend against Blend or Dynamic attack. Can you explain the reason behind this phenomenon?\n\nFor other questions, please refer to the weakness section.\n\n[1] One-shot Neural Backdoor Erasing via Adversarial Weight Masking\n\n[2] Anti-Backdoor Learning: Training Clean Models on Poisoned Data", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 This paper is easy to follow.\n\n2 This paper is well written.\n\n3 Appreciate the solid experiments shown in the experimental section.  The authors demonstrate the novel performance of Badexpert in multiple datasets and architectures.", "weaknesses": "1 Admitting the effectiveness of the proposed method, I think the practicality of Badexpert is limited. As shown in Appendix A, the optimal learning rate could vary across datasets: ($\\eta=10^{-4}$ for CIFAR-10 and $\\eta=2.5\\cdot10^{-5}$ for GTSRB). Even for the same dataset,  the optimal $\\eta$ could be different across different architectures:   $\\eta=10^{-4}$ for ResNet18 and  $\\eta=10^{-6}$ for pretrained vit_b_16. Considering the tremendous hyperparameter required for Badexpert, I think the overall guidlines for how to choose hyperparameter are needed to help Badexpert better defend against  potential risks.\n\n2 BadExpert depends on the stong mapping from the trigger to the pre-defined behaviour. I firmly believe, as long as the backdoor attack is weak enough (decreasing the poison rate/ the size of trigger/ the blend rate), will potentially leads to the Badexpert unsuccessful.  Sincerely, I hope to further discuss with the authors when encountering the above situations. Table 10 shows parts of the results, but not enough from my view. In addition, I think ASR and CA is a more appropriate metric to indicate the performance of BaDExpert instead of AUROC: Combing Table 1 and Table 2, BaDExpert obtains 11.4% ASR against blend attack. However, the AUROC is 99.2% which is quite close to 100%.\n\n3 The chosen of hyperparemter $\\eta'$ could also be unpractical in real world. In reality, only the small subset of clean images is available for defenders. Therefore, they have little knowledge about which $\\eta'$will meets the requirement of Badexpert: the CA of the finetuned model's CA first drops to ~ $0\\%$ and recovers to a significant value in the following epochs. The defender dooesn't exactly know what the CA of current model is. Therefore, the requirement of Badexpert may be too ideal.\n\n4 Some of the baselines are missing. For example, AWM [1] and ABL [2].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4z8wdjKT5b", "reviewer": "Reviewer_xRnn", "summary": "A novel backdoor defense BaDExpert is proposed in this paper. BaDExpert is designed to distinguish test instances with the backdoor trigger from benign test instances. The key idea is to fine-tune a \"backdoor-expert\" model with only the backdoor functionality so that benign test instances will be poorly recognized. Thus, test instances that are differently classified by the backdoored model and the backdoor-expert model are deemed benign; otherwise, a test instance is deemed to contain the trigger.", "questions": "1. Is the design philosophy of BaDExpert related to [1]? In [1], adversarial examples are detected by encouraging the model to carry malicious behaviors such as a backdoor.\n\n[1] Shan et al, Gotta Catch 'Em All: Using Honeypots to Catch Adversarial Attacks on Neural Networks, 2019.\n\n2. Can BaDExpert outperform [2] which also detects malicious inputs?\n\n[2] Li et al, Test-time detection of backdoor triggers for poisoned deep neural networks, 2022.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) The method is well-motivated and the idea is novel.\n\n2) The experiments are thorough, involving many attack settings and covering many SOTA baselines.\n\n3) The presentation is excellent.", "weaknesses": "More comparisons with existing works regarding the methodology can be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLVYI0mJqb", "reviewer": "Reviewer_79zr", "summary": "This paper proposed a backdoor defence method called BaDExpert. The main motivation is that normal data can be quickly forgotten with finetuning on inconsistent labels, while backdoor data does not. Based on this characteristic, the proposed method can extract a backdoor functionality of the model, which can only correctly classify backdoor data. Experiments show that the proposed method can effectively detect and defend against backdoor attacks.", "questions": "No further questions, please address the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is simple, straightforward and effective. Extracting the backdoor functionality is a new idea in backdoor attack research. \n- The empirical evaluations are very comprehensive, including different attacks, datasets, model architectures, and adaptive attacks. Results show it is effective against existing attacks and also demonstrated its limits under adaptive cases.", "weaknesses": "- The proposed method relies on several procedures, each with different hyperparameters. However, the authors provide an ablation study in each component, with no overall insights/guides for applying such a method in real-world applications. In real applications, the defender does not know for sure that backdoor attacks exist in their data, so it might not be easy to find these suitable hyperparameters. \n- The comprehensive experimental results are appreciated. It could be more comprehensive to compare with recent defence methods such as ABL [1], and detection methods [2,3]. \n- The method mainly focuses on the detection of backdoor samples (end of section 3). It could help to further clarify what happens after. Results in Table 1 focus on the CA/ASR. It is not clear what happens to the detected samples in order to obtain these results. \n\n[1] Li, Yige, et al. \"Anti-backdoor learning: Training clean models on poisoned data.\" Advances in Neural Information Processing Systems (2021).\\\n[2] Pan, Minzhou, et al. \"ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms.\" USENIX Security Symposium (2023).\\\n[3] Huang, Hanxun, et al. \"Distilling Cognitive Backdoor Patterns within an Image.\" The Eleventh International Conference on Learning Representations (2023).\\", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)