BAN: Detecting Backdoors Activated by Adversarial Neuron Noise
Authors: Xiaoyun Xu
Zhuoran Liu
Stefanos Koffas
Shujian Yu
Stjepan Picek
Venue: NeurIPS 2024 poster (2024)
Keywords: backdoor detection
backdoor trigger inversion
AI Security
Primary area: safety_in_machine_learning

Abstract: Backdoor attacks on deep learning represent a recent threat that has gained significant attention in the research community. 
Backdoor defenses are mainly based on backdoor inversion, which has been shown to be generic, model-agnostic, and applicable to practical threat scenarios. State-of-the-art backdoor inversion recovers a mask in the feature space to locate prominent backdoor features, where benign and backdoor features can be disentangled.  However, it suffers from high computational overhead, and we also find that it overly relies on prominent backdoor features that are highly distinguishable from benign features. To tackle these shortcomings, this paper improves backdoor feature inversion for backdoor detection by incorporating extra neuron activation information. In particular, we adversarially increase the loss of backdoored models with respect to weights to activate the backdoor effect, based on which we can easily differentiate backdoored and clean models. Experimental results demonstrate our defense, BAN, is 1.37$\times$ (on CIFAR-10) and 5.11$\times$ (on ImageNet200) more efficient with an average 9.99\% higher detect success rate than the state-of-the-art defense BTI DBF. Our code and trained models are publicly available at https://github.com/xiaoyunxxy/ban.

BAN: Detecting Backdoors Activated by Adversarial Neuron Noise
Xiaoyun Xu
Zhuoran Liu" Radboud University wwlluu@cs.cr ru.nl
Radboud University xiaoyun-. xu@ru.m nl
Stefanos Koffas Delfi University of Technology s koffas@tudel.t.nll
Shujian Yu Vrije Universiteit Amsterdam s yu38vu.
Stjepan Picek Radboud University st epan. weeannpicceuu...m ml
Abstract
Backdoor attacks on deep learning represent a recent threat that has gained signifi- cant attention in the research community. Backdoor defenses mainly based backdoor inversion, which has been shown to be generic, model-agnostic, and ap- plicable to practical threat scenarios. State-of-the-an backdoor inversion recovers a mask in the feature space to locate prominent backdoor features, where benign and backdoor features can be disentangled. However, it suffers from high computational overhead, and we also find that it overly relies on prominent backdoor features that are highly distinguishable from benign features. To tackle these shortcomings, this paper improves backdoor feature inversion for backdoor detection by incorporating extra neuron activation information. In particular, we adversarially increase the loss of backdoored models with respect to weights to activate the backdoor effect, based on which we can easily differentiate backdoored and clean models. Experimental results demonstrate our defense, BAN, is 1.37 x (on CIFAR- 10) and 5. x (on ImageNet200) more efficient with an average 9,99% higher detect success rate than the ofte--of-the-art defense BTIIDBF, Our code and trained models are publicly available https: //gi thub. com/xtt////oooo xiaoyunxxybaan.
Introduction
Deep neural networks (DNNss are known be vulnerable backdoor attacks, setting where the attacker trains a malicious DNN to perform well on normal inputs but behave inconsistently when receiving malicious inputs that stamped with triggers [9]. The malicious model is obtained by training with poisoned data [9, 6], by tampering with the training process [22, 2]. or by directly altering the model's weights [14]. Backdoors are proposed for various domains, from computer vision [9, 22, 6, 26, 40, 25] to graph data [42] and neuromorphic data [1]. Still, backdoors in computer vision received most of the attention of the research community, which also means there is significant variety of triggers. For instance, trigger can be pixel patch [9], an existing image [22], dynamic perturbation [25], image warping [26]. Various backdoor attacks target different stages of the machine learning model pipeline, inducing several threat scenarios in which defenses are developed by exploiting different knowledge.
Trigger inversion is principled method that makes minimal assumptions about the backdoor attacks in the threat model [38, 39, 44], and it has clear advantages training-time [18] run-time defenses [24, 19, 11]. Inpul space trigger inversion is first introduced by Neural Cleanse (NC) [36].
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

where potential triggers for all target classes are reversed by minimizing the model loss of clean inputs. Median absolute deviations of reversed triggers from all classes are then calculated detect triggers. More recently, input space trigger inversion methods have been shown to be ineffective against feature space backdoor attacks [26, 38]. To address this problem, FeatureRE. [38] proposes a detection method using feature space triggers. The authors observe that features of backdoored and benign inputs are separable in the feature space by hyperplanes. Unicorn [39] formally defines and analyzes different spaces for trigger inversion problems. An invertible input space transformation function is proposed to map the input space to others, such as feature space, and to reconstruct the backdoor triggers. These trigger inversion methods can be considered as an optimization problem for the targeted class. They optimize the input images to mislead the model under various constraints and recover strong triggers in different spaces. The state-of- offte-of-the-arre the-art trigger inversion method, BTI- DBF [44], increases the inversion efficiency by relaxing the dependency of trigger inversion on the target labels. BTI-DBF trains trigger generator by minimizing the differences between benign samples and their generated poisoned version in decoupled benign features while maximizing the differences in remaining backdoor features. Feature space trigger inversion defenses are generic and effective against most backdoor attacks. However, we show that BTI-DBF may fail against BadNets, as its backdoor is not prominent in the feature space (see Section 3.4). In addition, existing works still suffer from huge computational overhead and overly rely on prominent backdooor features.
To resolve this shortcoming, we propose backdoor defense called detecting Backdoors activated by Adversarial neuron Noise (BAN). Our defense is inspired by the finding that backdoored models are more sensitive adversarial noise than benign models [43, 10], and neuron noise can be adversarially manipulated to indicate backdoor activations [41]. Specifically, BAN generates adversarial neuron noise, where the weights of the victim model are adversarially perturbed maximize the classification loss on a clean data set. Simultaneously, trigger inversion is conducted in the victim model's 's feature space to calculate the mask for benign and backdoor features. Clean inputs with masked feature maps are then fed to the adversarially perturbed model, based on the outputs of which backdoored models can be differentiated. Figure presents t-SNE visualization of the feature space for backdoored and clean models when they are perturbed with different levels of adversarial neuron noise. It can be observed that the backdoored model misclassifies parts of the data from all classes as the target class when the adversarial neuron noise increases, while the clean model has fewer misclassifications under the same level of noise. By leveraging the induced difference, BAN can successfully detect and mitigate backdoor attacks in both input and feature space.
We make the following contributions:
We find generalization shortcoming in current trigger inversion-based detection methods (FeatureRE [38]. Unicorn [39], BTI-DBF [44]). In particular, feature space detections overly rely on highly distinguishable prominent backdoor features. We provide an in-depth analysis of trigger inversion- that are exploited by ttate--f-thhe-rt defenses to distinguish feature space backdoors may not be suitable for the identification of input space backdoors.
backdoor defenses, showing that prominent backdoor features
We propose detecting Backdoors activated by Adversarial neuron Noise (BAN) to mitigate this generalization shortcoming by introducing neuron noise into feature space trigger inversion. BAN includes an adversarial learning process to incorporate neuron activation information into the inversion-based backdoor detection. Experimental results demonstrate that BAN is 1.37 x (on CIFAR- 10) and 5.11x (on ImageNer200) more efficient with a 9.99% higher detect success rate than the state-of-the-art defense BTI-DBF [44].
We also exploit the neuron noise to further design a simple yet effective defense for removing the backdoor, such that we build a workable framework.
2 Backdoors
Attacks. Backdoor attacks 19, 22, 2, 26, 25, 35, 8, 20, 3] refer to injecting a secret functionality into the victim model that is activated through malicious inputs that contain the trigger. To this end, substantial research has been proposed by poisoning small percentage of training data using small and static triggers [9, 6, 22). Early attacks generate backdoors in input space, where BadNets [9] is the first backdoor attack in DNNs. Blend [6] proposed three injection strategies to blend translucent images into inputs of DNNs as triggers. The authors controlled the transparency of the trigger to allow the trade-off between strength of attack and invisibility. Although these attacks work well,
2

---

0.0
c= 0.1
c 02
=
Backdoor:
Beniinn:
Neuron noise incressing
Torgeted Jabe.
Figure 1: The feature plots of backdoor and benign models with neuron noise using R8 on CIFAR-10. The darker blue represents the target label. As noise increases, the backdoor model identifies more inputs from each class as the target label. The clean model has fewer errors, and there no significant increase in the number of misclassifications to the target class.
their triggers are still perceptible to humans and can be easily detected by backdoor defenses, such as Activation Clustering (AC) [4] and NC [36].
Dynamic and imperceptible triggers [26, 25, 40, 8), including feature space backdoor triggers [20, 25], are explored to bypass both human observers and input space defenses. IAD [25] designs input- specific triggers. To evaluate the uniqueness of dynamic triggers, the authors designed a cross-trigger test to determine whether the trigger of one input is reusable others. WaNet [26] proposes warping- based triggers, which are unnoticeable and smooth the input space. Bpp [40] exploits vulnerabilities in the human visual system and, by using image quantization and dithering, introduces invisible triggers that are stealthier than previous attacks.
Adaptive backdoor attacks [31, 28, 24] are built to systematically evaluate defenses, where attacks discourage the indistinguishability of latent representations of poisoned and benign inputs in the feature space. Adap-Blen [28] divides the trigger image into 16 pieces and randomly applies only 50% of the trigger during data poisoning. They use the full trigger image at inference time to mislead the poisoned model. SSDT [24] considers the combination of source spercffc and dynamic- -rigger backdoors. Only the inputs from victim classes (source) with the dynamic triggers are classified to the target labels, which encourages the generalization of more diverse trigger patterns from different inputs. In this paper, we evaluate BAN against various types of attacks, including input space, feature space, and adaptive attacks, to provide systematic evaluation resembling practical threats.
Defenses. Trigger inversion- based backdoor defense considered one of the most practical and general defenses against backdoors [38, 39, 44]. The recovered trigger is used to determine whether the model is backdoored. For example, NC [36] reverses input space triggers to detect backdoors by selecting significantly smaller triggers in size. Other methods, such as ABS [21] and FeatureRE [38], usually determine whether there is a backdoor based on the attack success rate of the trigger. More specifically, given DNN model and small set of clean data D. (xn, Vn JN NC recovers the potential trigger by solving the following objective:
min C((((1 - m) @ x + m 2 t), + A/mm. m.t
where (x, y) De and m is the trigger mask, is the trigger pattern, and yt is the target label. The mask m determines whether the pattern will replace the pixel. C is the cross-entropy loss function. Most prior works [12, 30, 21, 38. 39] follow this design conduct trigger inversion for all possible target labels. For example, Tabor [12] adds more constraints to the NC optimization problem according to the overly large (large size triggers but no overlaps with the true trigger) and overlaying (overlap the true trigger but with redundant noise) problem of NC. Moving from input space to feature, FeatureRE [38] utilizes feature space constraint according to an observation that neuron activation values representing the backdoor behavior are orthogonal to others. Unicorn [39] proposes a transformation function that projects from the input space to other spaces, such as feature space [26] and numerical space [40]. Then, the authors conduct trigger inversion in different spaces.
3

---

Unlike previous NC-style methods, recent works [44, 37] explore different optimization objectives to avoid the time-consuming optimization for all possible target classes or to avoid the fixed mask- pattern design as advanced attacks utilize more complex and dynamic triggers. BTI-DBF [44] takes advantage of the prior knowledge that benign and backdoored features are decoupled the feature space. It distinguishes them by the optimization objective, where benign features contribute to the correct predictions and backdoored features lead to wrong predictions. Based on the decoupled features, BTI-DBF trains trigger generator by minimizing the difference between benign samples and their generated version according to the benign features and maximizing the difference according to the backdoored features. Feature space backdoor defenses are developed based the fact that backdoor features are highly distinguishable from benign features. However, this finding is not consistently valid for input space attacks where the feature difference is small (See Sections 3.4).
3
BAN Method
3.1
The Pipeline of Training Backdoor Models
For brevity, consider an L-layer fully connected network f (similar principles apply to convolutional networks) that has 1 = I + 12 + ... + 4r. neurons. Xn E RRd and yn E {0, 1 are the image and its label in dı and d, dimensional spaces, respectively. The attacker creates a poisoned dataset Dp by poisoning generators Gx and Gy for subset of the whole training dataset, i.e., D.UDD Do De is the original clean data. Db is the poisoned backdoor data, D, = {xx', y')
Dj = DD Cx(x), Gy (v), (x, y) E D DeJ. In all-to-one attacks, (y) V/, V is the attacker- specified target class. In our experiments, we consider the dirty-label attack. In all-to-all attacks, ususally Gy (y + 1) [44, 26, 40], which is also what we chose in our experiments. In the training stage, the backdoor is injected into the model by training with Dp,, ie., minimizing the training loss D, to find the optimal weights and bias (w", b,):
min LD, (w, b) w.b
(,(x:,,b)).r
(x.y)ED,
where {(: :) is the cross-entropy In the inference stage, the backdoored model predicts an unseen input & as its true label ÿ but predicts (x as (ū) ( w)b) Gy)
3.2 Threat Model
Attacker's goal. We consider the attacker to be the pre-trained model's provider. The attacker aims to inject stealthy backdoors into the pre-trained models. The pre-trained models perform well on clean inputs but predict the attacker-chosen target label when receiving backdoor inputs.
Attacker knowledge. The attacker has white-box access to the model, including training data, architectures, hyperparameters, and model weights.
Defender's goal and knowledge. The main goal is to detect whether a given model is backdoored and then remove the potential backdoor according to the detection results. Following [38, 19,,399444 44), we assume the defender has white-box access to the model and holds few local clean samples. However, the defender does not have access to the training data and has no knowledge of the backdoor trigger.
3.3 Detection with Neuron Noise
Based on previous findings that adversarial noise can activate backdoors [43, 41, 10], we design two-step method for backdoor detection. First, we search for noise on neurons that can activate the potential backdoor. Then, we decouple the benign and backdoored features using learnable mask of the latent feature (the output before the final classification layer).
Neuron Noise. In backdoor models, there are two types of neurons: benign and backdoor [21]. The backdoor neurons build shortcut from G (x) to Gy (3)). Neuron noise generated noises added on neurons to maximize model loss on clean samples in an adversarial manner [23]. The noise on benign neurons evenly misleads prediction to all classes, while the noise on backdoor neurons tends to mislead prediction to the target label due to the backdoor shorteut, as shown in Figure 1. Therefore, backdoor models with noise behave differently from benign models with noise, as there no backdoor neurons in the benign models.

---

Given the ju neuron connecting the in layer and (i - 1)m layer, we denote its weight and bias with Wij and byy, respectively. Neuron noise can be added the neuron by multiplying its weight and bias with small number: (1 diz) Waj and (1 Ezy)ay- Then, the output of the neuron is:
hij ((1 + dig) w. b + (1+ E1y)b1g).
(3)
where h(s-1) is the output of the previous layer and (.)) is the nonlinear function (activation). The noise on all neurons are represented by
ót2.1. and f...t.- The ā and are optimized via maximization to increase the cross-entropy loss on the clean data:
Ōi,,, ,
[EI.1,... 1,, ÊI.L,."
max CD. ((1 0) w, E) b),
ó,E
(4)
S B(wee) 16 R c)
where S' is the ball function of the radius in the dimensional space. and share the ball function as the maximum noise size. The maximization in Eq. (4) can be solved by PGD [23] algorithm with random start, as PGD can better explore the entire searching space mitigate local minima [23].
Feature Decoupling with Mask. The neuron noise activates the backdoor (see Figure 1) and misleads the predictions to
Accuracy w/ and w/o feature mask
0.8
the target label. However, the performance has high variance
w/o mask
0.7
w/ mask
when searching for noise multiple times, which conjecture is caused by random initialization. Therefore, inspired by [44], further introduce feature decoupling process enhance the effect of noise on backdoored features but maintain a de- creased effect on benign features. Specifically, the network fi o ... o fI.-1 and ft. where fr-1
0.6
is decomposed into extracts the latent features from x. while fr. the classification
0.1-
layer. Then, we use a mask m on top of the latent feature g(x) to decouple benign and backdoored features. The optimization objective can be written as:
0.0
Benign
IAD
Blend WaNet
Bpp BadNets
Figure 2: Model's clean accuracy
min [( ((f u) ((L_-CC((((())( +)+ |m) m
with (red dots) and without (blue dots) the mask defined in Eq. (6). Only the backdoored models are af-
(5)
where @ is the element-wise product. This optimization divides
the latent features into two parts through the mask while main-
fected by the noise.
taining a relatively small size of m. Note that the regularizer for m in Eq. (5) is necessary. Otherwise, m will become a dense matrix full ones to focus on positive part of Eq. (5) because maintaining only the positive part (without penalty of |ml) already satisfies the optimization objective. Finally, we apply the negative mask (1 - m) on top of latent feature of f to enhance the backdoor effect. The final output is:
fi (9(x: (1 (1+))) Ow,, w. (1+ + 0 (1-
(6)
In Figure 2, the blue dots show the accuracy after adding noise to the model. The red dots show the accuracy (with noise) while applying the feature mask to the model using Eq. (6). Our feature masks do not affect the performance of benign models. However, we see that the performance is significantly decreased for backdoored models. This decrease caused by the model only using backdoor features (through the negative mask), which means the backdoor is activated more frequently. Finally, suspect model determined backdoored if the prediction using Eq. (6) has a high attack success rate. For all-to-one attacks, the misclassification will be concentrated into one label, which is the target label. For all-to-all attacks, we can do the same as for all-to-one attacks but evaluate the prediction for each class independently.
3.4 Improving BTI-DBF
This section shows that the most recent work, BTI-DBF [44], may fail to capture the backdoor features by its decoupling method in Table 2. We show how to patch BTI-DBF using simple solution. The BTI-DBF the original version, and BTI-DBF= our improved version. The main pipeline of BTI-DBF consists of two steps: (1) decoupling benign and backdoor features and (2) trigger inversion
5

---

by minimizing the distance between benign features and maximizing the distance between backdoor features. We found that the defense's 's first step may introduce enrors in the decoupled features. Similar to our Eq. (5), the decoupling of BTI-DBF can be written as:
min (Jr. o (x(x) @ m),y) - C(Jz. o (g(x) o (1 (1- m)),
However, this equation has no constraints on the feature mask m. Overall, the optimization objective to decrease the loss. Obviously, BTI-DBF's decoupling encourages the norm of the mask to increase so that the loss will focus on the positive part and ignore the negative part because the negative part goes against the overall objective. Finally, the mask will be dense matrix that is full of ones. The fL (x) @ (1 m)) is ignored due to multiplying by zero. We propose a simple solution to fix the problem by adding a regularizer of the size of the mask to the loss, i.e., we use our Eq. (5) as the first step of BTI-DBF*. According to Table 2, BTI-DBF* successfully overcomes BTI-DBF's shortcomings.
3.5 Backdoor Defense
After determining whether a suspect model is backdoored, we can fine-tume the backdoored model to remove the backdoor. However, standard fine-tuning using clean data does not effectively remove the backdoor because it does not activate it. Therefore, we propose using optimized neuron noise to fine-tune the model. In the optimization of the neuron noise, the objective is to increase the loss of [xf(x): : y). We consider both the benign and backdoor neurons to contribute to the increase in loss when optimizing the noise. Indeed, on benign neurons, the neuron noise misleads f to any result other than the true label while the noise misleads f to the target label Gy (v) on backdoor neurons. decrease the loss between noise output and the true label. The loss for our noise fine-tuning can be written as:
Therefore, straightforward method
min L(f w. b)b v) + x2 (1 + O (1+ + b). w.b
(8)
4 Experimental Results
Datasets and Architectures. The datasets for our experiments include CIFAR-10 [16), GTSRB [33], Tiny-lmageNer [17], and subset of ImageNet- K [7]. The ImageNet subset contains 200 classes, which is referred to as ImageNet200. BAN is evaluated using three architectures: ResNet18 [13]. VGG16 [32], and DenseNet121 [15]. Please refer to Appendix B.1 for more details.
Attack Baselines. Our experiments conducted using seven attacks: BadNets [9], Blend [6]. WaNet [26), IAD [25], BppAttack [40], Adap-Blend [28], and SSDT [24], which are commonly used in other works [44, 38, 39, 46, 29]. The BadNets [9] and Blend [6] are designed for input space. WaNet [26], IAD [25], and Bpp [40] are designed for feature space. Adap-Blend [28] and SSDT [24] (for adaptive evaluation) are state-of-the-art attacks that have been recently introduced to bypass backdoor defenses. The main idea of Adap-Blend [28) and SSDT [24] is to obscure the latent separation in features of benign and backdoor samples. More details be found in Appendix B.2.
Loss with positive negative mask
Defense Baselines. BAN is compared to five representa- tive methods: Neural Cleanse (NC) [36]. Tabor [12), Fea- tureRE [38], Unicomn [39], and BTI-DBF [44]. NC and Tabor are designed for input space attacks, while the other three are designed for feature space and dealing with the latest advanced attacks. BAN uses only 1% and 5% of training data for de- tection and defense, respectively. Note the data used for defense is not used for training models, i.e., the defender has no knowledge of the model to be defended. More details can be found in Appendices B.3 and B.4. We use bold font to denote the best results.
mask loss
2
benign
backdoored
BadNets
WaNet
IAD
Bpp
Figure 3: BadNets features are weaker when using the mask to dis- entangle the benign and backdoor features. Defenses that are biased towards large differences may not work in cases like BadNets.
4.1 The Performance of Backdoor Detection
Main Results. In Table 1, BAN shows better results on CIFAR- 10 than all basclines, especially on advanced attacks. Results on

---

Table 1: The detection results under different model architectures CIFAR- 10. The "Bd." refers to the number of models the defense identifies as backdoored. The "Ace." refers to detection success accuracy. The best results are marked in bold. BTI-DBF refers to improved version (details in Section 3.4).
NC
Tabor
FeatureRE
Unicorn
BTI STI-DDF*
Ours
Model
Attack
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
No Attack
0
100%
0
100 %
2
90%
6
70%
0
100%
0
100
BadNets
20
100
%
20
100 %
14
70%
18
90%
18
90%
20
100 %
Blend
20
100%
20
100 %
20
100%
19
95%
20
100%
18
90%
ResNet18
WaNet
11
55%
8
40%
15
75%
20
100%
90%
100%
IAD
0
0%
0
0%
15
75%
11
55%
20
100%
20
100 %
Bpp
0% 0
5%
12
60%
17
85 %
20
100% %
20
100%
No Attack
0
100%
0
100 %
3
85%
6
70%
6
70%
100%
BadNets
18
90%
16
80%
13
65%
16
80%
18
90%
19
95%
Blend
19
95%
19
95%
16
80%
18
90%
16
80%
85%
VGGI6
WaNet
50%
9
45%
12
60%
18
90%
16
80%
100%
IAD
0
0%
0
0%
8
40%
17
85%
20
100%
20
100%
Bpp
9
45%
10
50%
5
2.5%
15
75%
14
70%
18
90 %
No Attack
0
100%
0
100 %
5
75%
8
60%
3
85%
100% %
BadNets
18
90%
20
100 %
19
95%
15
75%
17
85%
20
100 %
Blend
20
100%
20
100 %
12
60%
18
90%
19
95%
20
100%
DenseNet121
WaNet
13
65%
10
50%
20
100%
17
85%
70%
95%
IAD
0
0%
0
0%
14
70%
16
80%
14
70%
19
95%
Bpp
0
0%
0
0%
16
80%
8
40%
16
80%
20
100%
Average
60.56%
59.17%
72.5%
78.61%
86.39%
97.22% %
other datasets are presented in Appendix C. We note that the advanced detection methods (FeatureRE. Unicorn, and BTI-DBF) perform worse than NC on simple attacks (BadNets and Blend). We hypothesize this is because the backdoor features generated by BadNets are less obvious on feature channels than advanced attacks the feature space, such as WaNet and IAD. Figure 3 shows that BadNets features are weaker than features from advanced attacks using the feature mask in Eq. (5). Specifically, we optimize the feature mask to disentangle the benign and backdoor features for four attacks. Then, we compute two cross-entropy loss values using the positive mask (m) and the negative mask (1 m) for benign and backdoor features, respectively. The average loss values and standard deviation for four models for each attack are plotted in Figure 3. The negative loss value of BadNets much smaller than others, which means BadNets features are weaker than others with regard to misleading the model to the backdoor target. Recent defenses usually add more regularizers to their losses and optimization objectives to counteract powerful backdoor attacks. These regularizers encourage the capturing of strong features but omit weak ones. Thus, recent advanced detections can perform worse on BadNets than NC.
Time consumption. BAN is efficient and scalable as we do not iterate over all target classes. Figure 4 demonstrates that BAN uses substantially less time than all baselines. BAN also more scalable for larger architectures or datasets. BIT-DBF (76.90s) is 1.37 slower than BAN (55.95s) on CIFAR-10 with ResNet1 8, and BIT-DBF (5.792.51s) without pre-training is 5.1 5.11x slower than ours 132.855) on ImageNet200 with ResNet18. FeatureRE (297.74g 7.7)) is 5.32 slower than ours on CIFAR- 10 with ResNet18, and it (6,053.62s) is 45.14x slower on CIFAR-10 with DenseNet121 than ours 10s4.10s).
4.2 The Performance of Backdoor Defense
A complete backdoor defense framework should include both detection and defense. The goal of defense to decrease the attack success rate (ASR) of backdoor triggers. Detection before the defense is also necessary because the defense usually decreases the performance on benign inputs [47, 44, 38, 46, 48]. In Section 3.5, we propose a simple and effective fine-tuning method using the noise that activates the backdoor. Table 3 compares BAN with three baselines: plain fine- tuning, FeatureRE [38], and BTI-DBF(U) [44], Plain fine-tuning refers to training the backdoor model using the same hyperparameters BAN but without the noise loss in Eq. (8). The FeatureRE [38]
7

---

(w/ pro)
Tabor
4-
0-
200
400
800
5000 10000 15000
CIFAR-10
Imag-ImageNet
ImageNet200
Figure 4: Time consumption of detection baselines on ResNet18 8 (in seconds) for all three datasets.
BAN uses significantly less time than the baselines.
Table 2: The detection results of BTI-DBF and BTI-DBI* using ResNet18 and CIFAR-10.
BTI-DBF
BTI-DBF
Ours
Attack
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
No Attack
0
100%
0
100%
0
100
BadNets
5
25%
18
90%
20
100%
Blend
0
0%
20
100%
18
90%
WaNet
35%
90%
20
100%
IAD
19
95%
20
100%
20
100% %
Bpp
20
100%
20
100 %
20
100 %
and BUI-DBF(U) [44] refer to the defense methods from the respective paper. We use default hyperparameters for FeatureRE [38] and BTI-DBF(U) [44]. For plain fine-tuning and BAN, we use small learning rate (0.005) to avoid jumping out of the current optimal parameters the well-trained model. Then, we use X (0.5) for Eq. (8) for the trade-off between robustness against backdoor and clean accuracy. We fine-tune for short schedule of 25 epochs, as the model is well-trained. Table in Appendix B.4 shows defense performance with standard deviation on different hyperparameters, which supports our choice. Tables 3 and demonstrate that our fine-tuning method effectively removes the backdoor while preserving high accuracy in benign inputs. We provide comparison between our find-tuning with ANP [41] in Table 12, Appendix C.4, as ANP uses the neuron noise for pruning backdoor neurons.
4.3 Defense against All-To-All Attacks
Previous works [38, 39] usually only consider all-to-one attacks, which limits the application in practical situations. this section, we evaluate our fine-tuning method under three all-to-all attacks: WaNet-All [26], IAD-AlI [25], and Bpp-All [40]. FeatureRE is designed for all-to-one attacks, so we use target label 0 here for FeatureRE. FeatureRE is included to show that the all-to-one defense does not work in the all-to-all setting. BAN is capable of handling all-to-all attacks because directly explore the neurons themselves instead of optimizing for the potential targeted label. Table demonstrates the effectiveness of our method.
Table 3: Defense against 5 attacks using ResNet18. BA refers benign accuracy on clean data.
No defense
Fine-tuning
FeatureRE
BTI-DBF(U)
Ours
Attack
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BadNets
93.37
99.41
92.93
87.81
93.15
99.79
91.26
13.12
92.06
Blend
94.60
100.00
93.07
99.99
93.20
39.28
91.86
100.00
92.72
4.10
WaNet
93.57
99.37
93.05
1.10
93.67
0.03
90.30
4.89
92.05
0.91
IAD
93.11
97.88
94.11
0.46
92.73
0.0
89.54
1.59
92.78
1.48
Bpp
94.29
99.93
93.85
4.46
94.21
98.13
90.61
2.73
92.54
2.58
Average
93.80
99.32
93.40
38.76
93.39
47.45
90.71
24.47
92.43
2.21

---

Table 4: Defense of backdoor attacks Tiny-Im ImygeNet and ImageNet200 using ResNet18.
No defense
Fine-tume tuning
BTI- BTI-DBF(U)
Ours
Dataset
Attack
BA
ASR
BA
ASR
BA
ASR
BA
ASR
WaNet
58.32
99.85
51.53
1.3
39.49
0.96
50.69
0.86
Tiny-ImageNe
IAD
58.54
99.32
51.86
1.72
38.79
0.60
50.04
0.76
Bpp
60.63
99.89
57.72
0.15
46.84
0.40
57.66
0.10
WaNet
77.01
99.74
66.71
0.78
63.47
1.0
69.95
0.58
ImageNet200
IAD
76.72
99.75
69.91
0,42
64.33
1.24
72.18
1.30
Bpp
78.56
99.88
70.89
0.82
67.02
3.10
72.59
2.68
Table 5: Defense against all-to-all backdoor attacks. BA refers to benign accuracy clean data.
No defense
Fine-tuning
FeatureRE
BTI-DBF(U)
Ours
Attack
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
WaNet-All
93.60
91.86
92.65
18.03
93.33
91.96
91.30
1.72
92.29
1.11
IAD-All
92.96
90.62
93.19
3.72
93.06
91.20
91.36
3.72
92.31
1.13
App-All
94.45
84.68
93.90
1.58
94.32
83.87
90.16
2.05
93.23
1.38
Average
93.67
89.05
93.25
7.78
93.57
89.01
90.94
2.49
92.61
1.21
4.4 Evaluation under Adaptive Attack
Attackers may design specific attack for defense when they know its details [34]. In this section, evaluate BAN against two attacks that attempt to bypass the difference between backdoor and benign features: Adap-Blend [28] and SSDT attack [24]. Both Adap-Blend [28] and SSDT attack [24] try to obscure the difference between benign and backdoor latent features. Adap-Blen [28] achieves it by randomly applying 50% of the trigger poison the training data, while SSDT attack [24] utilizes source-specific and dynamic-triggers to reduce the impact of triggers on samples from non-victim classes. The source- speccffc attack refers to backdoor triggers that mislead the model to the target class only when applied to victim class samples. Table 6 demonstrates our approach is resistant to these two attacks, while other methods fail. The reason that the backdoor features of SSDT are close to benign features in the feature space. It is difficult for other methods to distinguish between backdoor and benign features created by SSDT. Our detection method directly analyzes the model itself using neuron noise, which captures the difference between backdoor and benign models concerning parameters. See Appendix C.3 for the mitigation results of our method against these adaptive attacks.
Table 6: The detection results under adaptive attacks using CIFAR-10 and ResNet18.
NC
Tabor
FeatureRE
Unicom
BTI-DBF
Ours
Attack
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
Adap-Patch
18
90%
15
75%
85%
100% %
100%
100%
SSDT
0
0%
0
0%
0
0%
0%
20
100%
20
100%
4.5 Analysis on Prominent Features
We provide additional analysis of the phenomenon that backdoor features are more prominent for advanced attacks (WaNet, IAD, and Bpp) than weaker attacks (BadNets, Blend). Table demonstrates that previous decoupling methods cannot easily pick up backdoor features from weak attacks, such as BadNets. particular, when detecting without the L1 regularizer (i.e., w/o norm), the negative feature loss of BadNets is high with a very large L₁ mask norm, while the Bpp has an even higher negative loss with much smaller mask norm. The high negative loss of BadNets is actually from the sparse feature mask rather than backdoor features, i.e., there are too many zeros in (1 m). This indicates that BadNet backdoor features are less prominent than Bpp features, making it more challenging decouple BadNets features.

---

Table 7: Loss values when feeding the benign or backdoor features into the final classification layer: The mask optimized using Eq. (5) (w/ norm) or Eq. (7) (w/o norm). The shape the feature mask is 512 x 4 x 4, so the maximal L1 norm is 8192.
Attack
BA
ASR
L mask norm
Positive feature loss
Negative feature loss
BadNets (w/ norm)
93.47
99.71
2258,90
0.21
0.26
BadNets (wo norm)
93.47
99.71
8054.45
0.14
2.17
Blend (w/ norm)
94.60
100.00
2084.62
0.15
0.20
Blend (w/ norm)
94.60
100.00
8117.90
0.04
2.22
WaNet (w/ norm)
93.88
99.63
7400.97
0.06
2.34
WaNet (w/o norm)
93.88
99.63
7702.56
0.05
2.39
IAD (w/ norm)
93.82
99.64
7898.91
0.03
2.25
IAD (w/o norm)
93.82
99.64
7895.16
0.03
2.25
Bpp (w/ norm)
94.56
99.97
7147.68
0.09
2.80
Bpp (w/o norm)
94.56
99.97
7260.31
0.09
2.78
Conclusions and Future Work
This paper proposes effective yet efficient backdoor defense, BAN, that utilizes the adversarial neuron noise and the mask the feature space. BAN is motivated by the observation that traditional defenses outperformed the latest feature space defenses input space backdoor attacks. To this end, we provide an in-depth analysis showing that feature space defenses are overly dependent on prominent backdoor features. Experimental studies demonstrate BAN's effectiveness and efficiency against various types of backdoor attacks. We also show BAN's resistance to potential adaptive attacks. Future studies could explore more practical detection method without assuming access to local benign samples and better strategies for decoupling features because fixed mask in the feature always aligned with the benign and backdoor features.
Acknowledgments and Disclosure of Funding
This work was partly funded by the Dutch Research Council (NWO) through the PROACT project 125..1215.18.0144) and the CiCS project of the research programme Gravitation (024.006.037).
References
[1] Gorka Abad, Oguzhan Ersoy, Stjepan Picek, and Aitor. Urbieta. Sneaky spikes: Uncovering stealthy backdoor attacks in spiking neural networks with neuromorphic data. In NDSS, 2024.
[2] Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In USENIX Security, 2021.
[3] M. Barni, K. Kallas, and B. Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In ICIP, 2019.
[4] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. SafeAl Workshop @ AAAI, 2018. 2018.
[5] Erh AtrhCChng Chen, Pin-Yu Chen, I (Chung, Che-Rung Lee, et al. Data-driven lipschitz continuity: cost-effective approach to improve adversarial robustness. arXiv preprint arXiv:2406. /9622, 2024.
[6] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv, 2017.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
10

---

[8] Khoa Doan, Yingjie Lao. Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. ICCV, 2021.
[9] Tianyu Gu, Kang Liu, Brendan Dolan Gavit,, and Siddharth Garg. Badnets: Evaluating backdooring attacks deep neural networks. IEEE Access, 7, 2019.
[10] Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-box backdoor detection using adversarial extreme value analysis. In ICLR, 2022.
[11] Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. SCALE-UP: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency, In ICLR, 2023.
[12] Wenbo Guo, Lun Wang. Yan Xu, Xinyu Xing, Min Du, and Dawn Song. Towards inspecting and eliminating trojan backdoors in deep neural networks. In ICDM, 2020.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[14] Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. Handerafted backdoors in deep neural networks. NeurlPS, 2022.
[15] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[16] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[17] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
[18] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurlPS, 2021.
[19] Yinshan Li, Hua Ma, Zhi Zhang, Yansong Gao, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Yifeng Zheng. Said F. Al-Sarawi, and Derek Abbott. Ntd: Non-transferability enabled deep learning backdoor detection. TIFS, 19:104-119, 2024.
[20] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, 2021.
[21] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In CCS, 2019.
[22] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS, 2018.
[23] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[24] Xiaoxing Mo, Yechao Zhang, Leo Yu Zhang, Wei Luo, Nan Sun, Shengshan Hu, Shang Gao, and Yang Xiang. Robust backdoor detection for deep learning via topological evolution dynamics. In SP, 2024.
[25] Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurlPS, 2020.
[26] Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible surping-based backdoor attack. In ICLR, 2021.
[27] Ren Pang, Zheng Zhang, Xiangshan Gao, Zhaohan Xi, Shouling Ji, Peng Cheng, and Ting Wang. Trojanzoo: Towards unified, holistic, and practical evaluation of neural backdoors. In Euro S&P, 2022.
(28) Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In ICLR, 2023.

---

[29] Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards proactive ML approach for detecting backdoor poison samples. In USENIX Security, 2023.
[30] Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimiza- tion. In ICML, 2021.
[31] Reza Shokri et al. Bypassing backdoor detection algorithms in deep learning, In Euro S&P, 2020.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[33] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Newral Networks, pages 323-332, 2012.
[34] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In NeurlPS, 2020.
[35] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv: 091..0227.1112212 2019.
[36] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In SP, 2019.
[37] Hang Wang. Zhen Xiang, David J Miller, and George Kesidis. Mm-bd: Post-training detection of backdoor attacks with arbitrary backdoor pattern types using maximum margin statistic. In SP, 2024.
[38] Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverse- engineering of trojan triggers. In NeurIPS, 2022.
[39] Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. UNICORN: A unified backdoor trigger inversion framework. In ICLR, 2023.
[40] Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In CVPR, 2022.
[41] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurlPS, 2021.
[42] Jing Xu, Stefanos Koffas, Oguzhan Ersoy, and Stjepan Picek. Watermarking graph neural networks based on backdoor attacks. In Euro S&P, 2023.
[43] Xiaoyun Xu, Oguzhan Ersoy, Behrad Tajalli, and Stjepan Picek. Universal soldier: Using universal adversarial perturbations for detecting backdoor attacks. In DSN-W, 2024.
[44] Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren. Towards reliable and efficient backdoor trigger inversion via decoupling benign features. In ICLR, 2024.
[45] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In ECCV, 2022.
[46] Hong Zhu, Yue Zhao, Shengzhi Zhang, and Kai Chen. Neuralsanitizer: Detecting backdoors in neural networks. TIFS, 19:4970 4985, 2024.
[47] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-awar minimization. In ICCV, 2023.
[48] Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: lightweight and effecttve backdoor defense via purifying poisoned features. In NewrIPS, 2023.
12

---

[49] Rui Zhu, Di Tang, Siyuan Tang, Zihao Wang, Guanhong Tao, Shiqing Ma, XiaoFeng Wang, and Haixu Tang. Gradient shaping: Enhancing backdoor attack against reverse engineering, In NDSS, 2024.
13

---

A Discussion
Limitations. Similar to existing works [36, 21, 12, 38, 39, 44], BAN assumes local small and benign dataset. This scenario is common, considering that benign samples are available online, and the model provider may provide some samples verify the model's performance. addition, our fine-tuning with neuron noise may slightly decrease the performance in benign inputs, similar to other defense methods [38, 44, 47, 41, 18, 36, 48]. However, BAN provides better performance and requires less time consumption.
Broader Impacts. This paper proposes complete defense that includes detecting and removing DNN backdoors. We believe our approach has positive impact on the security of DNNs. A possible negative impact is overconfidence guarantee always to remove the backdoor.
robustness against backdoor attacks, as there no theoretical
Additional Details for Experimental Settings B
B.1 Datasets
CIFAR-10. The CIFAR-10 [16] contains 50,000 training images and 10,000 testing images with the size of x 32 x 32 in 10 classes
GTSRB. The GTSRB [33] contains 39,209 training images and 12,630 testing images in 43 classes. In our experiments, the images are resized to 32 x 32.
Tiny-Image. Tiny-Imann ImageNet [17] contains 100,000 training images and 10,000 testing images with the size 3 x 64 64 in 200 classes.
ImageNet. ImageNet [7] contains over 1.2 million high-resolution images in 1,000 classes. Our experiments subset of the full ImageNet dataset, i.e., 200 randomly selected classes. Each class has 1300 training images and 50 testing images. The ImageNet images are scaled to 3 x 224 x 224.
B.2 Backdoor Models
BAN and defense baselines are evaluated using well-known attacks: BadNets [9], Blend [6], WaNet [26]. IAD [25], BppAttack [40], Adap-Blend (28), and SSDT [24].
For all backdoored models, we use SGD with momentum of 0.9, weight decay of 5 x 10-1, and learning rate 0.01. We train for 200 epochs, and the learning rate is divided by 10 at the 100-th and 150-th epochs. Same to [41], we use 90% of training data for training backdoor models and 10% for the validation set.
BadNets. We use a 3 x pattern to build the backdoor trigger. The poisoning rate is 5%.
Blend. We the random Gaussian noise and blend ratio 0.2 for backdoor training. The poisoning
rate is 5%.
-landBlend. We use the "hellokitty33.ppp" and blend ratio of 0.2 to build triggers. The poisoning rate is 5%.
SSDT. SSDT is a Source-Specific attack, which only misleads the victim classes to the targeted class. We class as the victim and class the target class.
For other attacks and hyperparameters not mentioned, we use the default settings from the papers or their official open-source implementations.
B.3 Defense Baselines
BAN is compared with five representative methods, including Neural Cleanse (NC) [36], Tabor [12], FeatureRE [38], Unicom [39], and BTI-DBF [44]. In this section, we describe the hyperparameters of these defenses.
NC and Tabor. We use the implementation and default hyperparameters from TrojanZoo [27] for NC and Tabor. 1 % of the training set used to conduct 100 epochs the trigger inversion. The learning rate is 0.01.
14

---

Adverarrial neuron noise
Clean input +
D ASR
Masked feature
Figure 5: Illustrative diagram of BAN
FeatureRE. We first tried the default hyperparameters from the FeatureRE paper, but they could not work even for BadNets. We assume this is because the constraints, including feature mask size and similarity between original images and trigger images, are limited to very small size. Therefore, we relaxed the constraints on masks and norms to find stronger triggers, including loss std bound=1.2, 2, p loss bound=0.2, loss std bound=1.22 mask size=0.06, and learning rate=0.001. We use 1% of the training set and run FeatureRE for 400 epochs for each class.
powerful in our case, and the recovered triggers can mislead any model to the target label when applied to inputs. Every model, including benign models, thought of as backdoored. Therefore, we slightly increase thread values of the constraints on Unicorn optimization to find proper triggers, including loss std bound=0.5, SSIM loss bound=0.1, and mask size=0.01. We use 1% of the training set and run Unicorn 40 epochs for each class.
Unicorn. The default hyperparameters for Unicorn
BTI-DBE We follow the default settings in the BTI-DBF [44] paper. Note that BTI-DBF uses 5% of training samples for defenses.
B.4 BAN Settings
Detection. We epsilon (c) to limit the noise added neurons. Otherwise, the model weights destroyed by the noise. The values are 0.3, 0.3, 0.2, and 0.1 for CIFAR-10, GTSRB, Tiny-ImageNet, and ImageNet200, respectively, e values are decided according to the size of the images. We use smaller for larger datasets. We use the 30-step PGD algorithm to solve the optimization in Eq. (4) to find the noise, i.e., and E. We use SGD and the learning rate of c/30 for the PGD optimization. Then, we use Adam and the learning rate of 0.01 to search for 20 epochs for the feature mask using Eq. (5). The A for Eq. (5) equals 0.75. The two optimizations above use 1% of the training set. Note that this 1% of data is not used to train the backdoor models. The elements in the mask are clamped into continuous values between and 1. Figure 5, we present an illustrative diagram of BAN.
Defense. Our defense uses Eq. (8) to remove the backdoor. Due to limited access to benign samples and time consumption, we only use 5% of training data and 25 epochs for our fine-tuning. The trade-off hyperparameter (A2) is 0.5. Then, we use the most commonly used hyperparameters for the optimizer. We use SGD with momentum=0.9, weight decay=5e-4, and learning rate=0.005 as the optimizer. The plain fine-tuning uses the same hyperparameters as BAN.
Table 8 shows the ablation results fine-tuning epochs, A2 and learning rate. Considering both performances on benign accuracy and removing the backdoor, our hyperparameters (LR-0.005, >==0.5 and epoch=25) show the best results.
Hardware, All experiments are run on a single machine with 4 RTX A6000 (48GB) and RTX A5000 (24GB) GPUs, CUDA 12.0.
15

---

Table 8: Ablation of hyperparameters for our defense. Each entry is the average of runs.
Ours
Attack
L.R
Epoch
A2
BA
ASR
0.005
0.2
93.26 + 0.14
2.53
0.93
0.005
0.5
92.64 + 0.19
2.28 = 0.72
0.005
25
0.8
92.19 0.13
2.89 0.94
0.005
25
1.0
92.04 0.29
2.74 0.49
0.005
10
0.5
92.97 0.29
2.62 = 0.44
Bpp
0.005
50
0.5
92.57
2.80 = 0.73
0.005
0.5
92.11
2.34 = 0.17
0,001
25
0.5
0
2.89 1.01
0,010
0.5
91.99 0.22
2.52 0.64
0.020
25
0.5
90.05 = 0.37
1.91 = 0.55
Additional Experimental Results
C.1 The Performance under Different Datasets
Table 9 shows the detection results using two larger datasets, Tiny-ImageNet and Imagenet200 (please refer Appendix C.2 for results GTSRB). The ImageNet200 is subset of ImageNet- 1K. We train 10 models for each case, 60 models in total. BAN still performs well under the two larger datasets. In addition, BAN is also scalable in terms of time consumption. To conduct backdoor detection on ResNet18 models trained using CIFAR- 10, Tiny- ImageNe,,,,,, and ImageNet200, BAN takes around 55, 129, and 1,120 seconds, respectively. We not apply other baselines (except for BTI-DBF) to larger datasets due to high computational requirements. For example, FeatureRE costs more than hours for detection in one class of ImageNet200 (default settings), meaning FeatureRE needs more than 400 hours for full detection on ImageNet200.
Table 9: The detection results using ResNetl8 under different datasets.
BTI-DBF
BTI-DBF
Ours
Dataset
Attack
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
No Attack
0
100%
100%
100%
Imageet
WaNet
4
40%
6
60%
10
100%
IAD
9
90%
8
80%
10
100%
No Attack
3
70%
0
100%
0
100%
ImageNet200
WaNet
60%
90%
10
100%
IAD
100%
10
100%
100%
C.2 Dection GTSRB
Table 10 shows the detection success rate for BAN on GTSRB. Notice that NC and BTI-DBF* perform worse against advanced attacks, while BAN precisely detects backdoor models.
C.3 Defense of Adaptive Attacks
Table 11 shows the fine-tuning results using our defense method against the two adap-blend and SSDT. The results show that fine-tuning with neuron noise can remove the backdoor with slight decrease clean accuracy.
C.4 Comparsion with ANP
ANP [41] uses a similar neuron noise for pruning backdoor neurons. Their optimization objective includes neuron noise and neuron mask. The neuron noise is optimized to fool the network, while
16

---

Table 10:
The detection results on GTSRB with ResNet18.
Table 11: Fine-tuning for adaptive attacks.
NC
BTI-DEF*
Ours
Attack
No defense
Ours
Bd.
Acc.
Bd.
Acc.
Bd.
Acc.
Attack
BA
ASR
BA
ASR
BadINets
20
100%
20
100%
20
100%
WaNet
40%
100%
Adap-Patch
94.20
99.75
90.45
10.24
IAD
0
0%
20
100%
20
100%
SSDT
93.86
90.30
93.29
0.90
Bpp
13
65%
14
70%
20
100%
the mask controls a trade-off for clean cross-entropy loss. compare our fine-tuning with ANP in Table 12. The hyperparameters of ANP are the default from the original paper.
Table 12: Comparison with ANP [41] on CIFAR-10 using ResNet18.
BadNets
Blend
WaNet
IAD
Bpp
Defense
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
No defense
93.37
99.41
94.60
93.57
99.37
92.83
97.10
94.29
99.93
ANP
93.16
2.13
94.17
97.24
93.06
13.84
92.50
0.44
93.80
4.77
Ours
92.26
2.04
92.61
1.04
92.18
0.87
92.36
1.47
92.82
2.16
C.5
Different Poisoning Rates
BAN is effective against BadNets with different poisoning rates, as shown in Table 13. We use BadNets because it is relatively weak a low poisoning rate (i.e., hard be mitigated), while more advanced attacks may still be strong at low poisoning rate.
Table 13: The performance of BAN fine-tuning under different poisoning rates of BadNets using CIFAR-10.
Poisoning Rate
BA
pos. feature loss neg. feature loss
Mitigated BA
Mitigated ASR
0.01
93.48
98.69
0.38
0.17
92.07
2.73
0.05
93.37
99.41
0.37
0.35
92.06
1.97
0.10
90.98
100.00
0.35
2,06
90.29
2.17
0.15
90.32
100.00
0.39
2.23
90.16
1.71
0.20
89.34
100.00
0.44
2.43
90.39
1.01
0.25
88.09
100.00
0.56
2.81
89.55
1.54
0.30
86.09
100.00
0.62
3.13
88,83
1.08
0.40
82.39
100.00
0.67
3.51
88.75
1.67
0.50
77.83
99.97
0.84
4.27
86.87
3.56
C.6 Against Backdoors on the MLP
We evaluate the defense performance simple model architecture. In particular, 4-layer MLP is trained with benign samples and with BadNets. Table 14 demonstrates that BAN is effective on MLP, where the "num. to target" refers the number of samples (in 5000 validation samples) that classified as the backdoor target after our detection. We also find that the positive feature loss (i.e., benign feature loss) is very close the negative loss (i.e., potential backdoor feature loss), which indicates that the backdoor features are more challenging to decouple from benign ones.
C.7 Discussion about Relationship between the Neuron Noise and Lipschitz Continuity
small trigger that changes output benign input into malicious target label can be related to the high Lipschitz constant [45] and neural network with high robustness tends to have lower local Lipschitz constant. Moreover, a larger local Lipschitz constant implies steeper output around trigger-inserted points, leading to smaller trigger effective radius making trigger inversion less
17

---

Table 14: The performance of BAN on the 4-layer MLP.
MLP
BA
ASR
Pos. feature loss
Neg. feature loss
Num. to target Mitigated BA Mtigated ASR
Benign
53.24
1.95
2.29
BadNets 47.04
100.00
2.03
2.34
3607
45.13
7.11
effective [49]. Thus, the concepts of adversarial activation noise and Lipschitz continuity related, and the local Lipschitz constant can serve as an upper bound for the trigger's effective influence.
In addition, introducing theoretical tools like the Lipschitz constant for backdoor defense may also be very tricky in practice because it needs approximation for implementation. For example, [45] evaluates the channel-wise Lipschitz constant by its upper bound but does not thoroughly discuss the relationship between the channel-wise Lipschitz constant and the network-wisc Lipschitz constant, where the theorem of Lipschitz continuity really relies Recent work also mentions that empirically estimating the Lipschitz constant is hard from observed data, which usually leads to overestimation [5]. Methods relying Lipschitz continuity may not require heavy computational load and are also related to our approach. Our method emphasizes more on fine-tuning with the guidance of neuron noise, rather than tuning the trained model.
C.8 Choosing the A1
As discussed in the method section, the mask norm (I, norm) with lambda in Eq. (5) is ensure that the optimization objective is decoupling between benign and backdoor features. Without the constraint of the mask norm in Eq. (5), the optimization objective will be simply increasing the mask norm unless there are extremely strong backdoor features. The λ value selection is implemented by checking the value of the mask norm. Table 15 provides an example λ selection on CIFAR- 10, where the maximal mask norm is 8192. It can be observed that the mask is almost full of ones when A1 smaller than 0.7, and the negative feature loss (backdoor feature) is high, based on which can pick a value of Å greater than 0.7. Note that the selection is unaware of potential backdoors.
Table 15: Feature loss values with different À
Mask norm
Pos. feature loss
Neg. feature loss
0.0
3188.62
0.27
2.30
8188.75
0.28
2.30
0.2
8184.30
0.27
2.30
0.3
8175.50 50
0.29
2,30
0.4
8152.40
0.25
2.30
0.5
8131.26
0.27
2.29
8055.07
0.21
2.27
0.7
7898.25
0.26
2.24
0.8
596.85
0.99
0.23
0.9
22.08
2.33
0.28
C.9 Swin Transformer
Table 16 provides experimental results of 12-layer Swin transformer on CIFAR-10. For BadNets and Blend, we train the backdoored network using Adam the optimizer. For WaNet, IAD, and Bpp. we use the default setting. All backdoored models can detected by BAN.
C.10 Clean Label Attack
We provide a clean-label backdoor experiment following [35], where we take default hyperparameters. Table 17 demonstrates the detection performance of 20 models, where we randomly select one model from the 20 for the mitigation experiment. Experimental results demonstrate that BAN is also effective against the clean-label attack.
18

---

Table 16: Results on Swin Transformer.
No defense
FT
BTI-DBB(()
BAN Fine-tune
Attack
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BadNets
86.7
97.43
82.23
44.07
85.21
99.99
83.92
2.91
Blend
85.46
100.00
80.13
100.00
83.9
100.00
79.18
26.20
WaNet
78.25
31.03
75.63
2.3
73.83
2.61
75.28
3.1
IAD
76.35
88.71
74.58
54.63
74.29
61.16
76.26
9.8
Bpp
79.19
63.74
75.35
12.99
74.66
11.04
74.25
6.91
Table 17: Clean label attack on CIFAR-10.
Attack
BA
ASR
Detection Success Accuracy
Mitigated BA
Mitigated ASR
LC [35]
92.72
100.00
90.00
89.27
6.16
Table 18: Results different target classes CIFAR-10.
No defense
BAN Fine-tune
Target
BA
ASR
BA
ASR
93.41
100.00
92.57
1.56
93.51
100.00
92.53
0.84
93.54
99.99
91.49
1.08
93.59
99.99
92.14
1.93
4
93.84
99.98
92.13
1.49
93.52
100.00
91.84
3.29
93,46
100.00
92.48
0.93
93.56
100.00
92.36
0.73
93.57
99.89
92.31
0.58
93.36
100.00
91.65
2.40
C.11 Semantically Similar Classes
We provide additional analysis on image form semantically similar classes. From ImageNet200. we select class n02096294 (Australian terrier) as the target class, since there are 19 kinds of terriers in our ImageNer200 datasets, such as n02094433 (Yorkshire terrier) and n02095889 (Sealyham terrier). We train 10 backdoor networks using BadNets with different target classes. Table 18 demonstrates that BAN effective against the backdoor regardless of semantically similar classes.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In section 1, introduce two drawbacks of existing methods and list our contribution in folds.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix A, we describe the limitations of our method and discuss the broader impacts.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
We focus on empirical results so we do not provide theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Appendix B, we provide all details about experiments. The link to anonymous repository (code and trained models) is also provided in the abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibilityy may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
21

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The link to our anonymous repository (code and trained models) is provided in the abstract. We also introduce how to run our code to the experimental results in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / cii.ps.cc/publii ceroolicgguidee gtt////idee/eeeeeeg CCodeSubmeaaacc ssi asionnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix B, we describe all details about the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The error bars are provided in Figure 2 and 3, and Table 8. Each entry in the figures or tables above averages runs with standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the time-con consucoosumption in Figure 4 and B.4, which demonstrates that our method is much more efficient than other methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics //tturi///eerpps...t. wwtppbliccccccccccc/..ccc. h diness??
Answer: [Yes]
Justification: We follow the "NeurIPS Code of Ethics".. We do not include human participants in experiments. We use open-source source data and code, and there are no privacy or copyright issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
In Appendix A, we discuss the potential positive and negative impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not overlap with pretrained language models, image generators, etc.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all relevant literature and assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code and trained models are provided. We also describe how prepare the environment and run our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not include crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not include crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "T9pYrclVNi", "reviewer": "Reviewer_yW41", "summary": "The authors propose a novel technique for detecting backdoor attacks on neural networks by incorporating extra neuron activation information to reduce the overhead from prior backdoor feature inversion methods. The experimental results show a higher detection rate on the tested datasets when compared to the prior work.", "questions": "1. What is the impact of $\\lambda_2$ on the fine-tuning loss found in Equation 8?", "limitations": "Yes, in appendix A.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to follow. The authors address a significant topic with the widespread adoption of neural networks for a wide variety of tasks. The experimental results affirm the design choices made by the authors in Section 3. Overall, it is a well written paper that addresses a significant area.", "weaknesses": "1. A figure of the proposed method outlined in Sections 3.1 and 3.3 could be a helpful tool to visualize the proposed method.\n2. The in figure text in Figures 2 and 3 are too small and hard to read.\n3. Some tables use % and others don't when report accuracy, e.g. Table 2 and Table 3 with no % symbol for the BA columns.\n4. I think a table or figure further emphasizing the proposed changes would greatly improve the strength of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work. We address your comments as follows.\n\n**Q1.** A figure of the proposed method outlined in Sections 3.1 and 3.3 could be a helpful tool to visualize the proposed method.\n\n**A1:** We will add a figure that illustrates the design of our method. \n\n**Q2.** The in figure text in Figures 2 and 3 are too small and hard to read. Some tables use \\% and others don't when report accuracy, e.g. Table 2 and Table 3 with no \\% symbol for the BA columns.\n\n**A2:** We will increase the font size and revise the table coherence. \n\n\n**Q3.** I think a table or figure further emphasizing the proposed changes would greatly improve the strength of this paper.\n\n**A3:** We will revise the details of different methods and show the differences between our method and baselines.\n\n**Q4.** What is the impact of $\\lambda_2$ on the fine-tuning loss found in Equation 8?\n\n**A4:** The hyperparameter $\\lambda_2$ controls the trade-off between benign accuracy (BA) and attack success rate (ASR).\nIncreasing the $\\lambda_2$ value will decrease BA but provide better defense performance.\nTable 7 shows the results with different $\\lambda_2$ values, ranging from 0.2 to 1, where $\\lambda_2=0.5$ provides the best defense performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zCPUNxNKrF", "reviewer": "Reviewer_hJLG", "summary": "This paper focuses on the backdoor defense task. The proposed detection method includes neuron noise, which leverages the differing robustness between regular and backdoored neurons, and a feature decoupling process using a mask. Additionally, a backdoor defense method is proposed, which achieves improved efficiency and overall performance.", "questions": "1, I might have a misunderstanding, but my key question is that the paper emphasizes that previous methods overly rely on prominent backdoor features. However, the Feature Decoupling with Mask at the neuron level seems to rely even more on these prominent backdoor features. Some previous works, like fine pruning [1], show that neurons might not necessarily be decoupled. \n\n2, Additionally, in Section 4.4, the paper claims that \"the reason is that the backdoor features of SSDT are close to benign features in the feature space. It is difficult for other methods to distinguish between backdoor and benign features created by SSDT.\" However, the Feature Decoupling with Mask approach may not be effective in this context.\n\n3, There is a minor issue that the experimental improvement does not seem significant, especially in Table 3 and Table 4.\n\n4, Is the detection performance related to the pattern of the trigger or the training strength of the backdoor?\n\n5, I'm curious about when a model architecture becomes less redundant for a given dataset. For instance, when we train DenseNet121 on CIFAR-10, the initial features are likely to be sparse. On the other hand, training an MLP model on CIFAR-10 makes feature decoupling more challenging. I wonder if using smaller models makes feature decoupling less effective.\n\n6 I also believe that selecting the lambda value in Equation (5) is quite challenging and tricky. This is because it is essential to maintain performance and ensure it doesn't degrade significantly after applying the mask (1-m).\n\nI would like to change my score if the questions are well answered.\n\n[1] Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks", "limitations": "The limitations have been discussed.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The exhibited efficiency is impressive, and the motivation behind neuron noise is intriguing. The dimensions of conducted experiments are relatively thorough.", "weaknesses": "The neuron noise approach appears reasonable and interesting, but the feature decoupling with a mask is a little questionable.\n\nAdditionally, the study does not include enough baselines, such as fine pruning, which can obscure the distinction between backdoored and regular neurons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We address your comments as follows.\n\n**Q1.**  I might have a misunderstanding, but ... be decoupled.\n\n**A1:**  Our defense aims to activate the backdoor by neuron noise such that the backdoor networks behave differently from benign networks. The feature mask we use in Eq. (5) is improved by the penalty of mask norm, and it does depend on the prominent backdoor. However, we apply the feature mask to activated backdoors (networks with neuron noise), which makes the decoupling easier. The existence of backdoored neurons is validated in [A,B,C,D]. As there are both backdoor and benign neurons in backdoored networks and only benign neurons in benign networks, it is evident that they will be different when applying noise to those neurons. Note that using neuron noise is insufficient for backdoor detection, as shown in Figure 2.\n\n**Q2.**  Additionally, in Section 4.4, the paper claims that ...  in this context\n\n**A2:** The feature decoupling methods, including other methods such as BTI-DBF and featureRE, may not always be effective.\nHowever, different from other methods, we apply the feature mask to a network with neuron noise. The noise activates the potential backdoor, which makes our decoupling easier.\n\n**Q3.** There is a minor issue that ... Table 3 and Table 4\n\n**A3:** We agree with the reviewer that performance improvement is sometimes not substantial. Our aim is to mitigate the generalization shortcomings of previous approaches. In our experiments, BAN is the only effective approach for all types of attacks.\n\n**Q4:** Is the detection performance ... backdoor?\n\n**A4:**  BAN is designed to be unaware of the trigger pattern, and we included backdoor attacks that are based on different patterns in experiments, including BadNets, Blend, IAD, WaNet, BppAttack, Adap-blend, and SSDT.\n\nRegarding the backdoor training strength, we provide experimental results with different poisoning rates of BadNets in the following table since the poisoning rate is closely related to backdoor strength. We use BadNets because it is relatively weak at a low poisoning rate, while more advanced attacks may still be strong at a low poisoning rate. All models in the table are successfully detected and mitigated. Our new results indicate that a stronger backdoor is easier to detect.\n\n\n|PR| BA | ASR | pos. feature loss | neg. feature loss|Mtigation(BA)|Mtigation(ASR)|\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| 0.01 | 93.48 | 98.69 | 0.38 | 0.17 | 92.07 | 2.73 |\n| 0.05 | 93.37 | 99.41 | 0.37 | 0.35 | 92.06 | 1.97 |\n| 0.10 | 90.98 | 100 | 0.35 | 2.06 | 90.29 | 2.17 |\n| 0.15 | 90.32 | 100 | 0.39 | 2.23 | 90.16 | 1.71 |\n| 0.20 | 89.34 | 100 | 0.44 | 2.43 | 90.39 | 1.01 |\n| 0.25 | 88.09 | 100 | 0.56 | 2.81 | 89.55 | 1.54 |\n| 0.30 | 86.09 | 100 | 0.62 | 3.13 | 88.83 | 1.08 |\n| 0.40 | 82.39 | 100 | 0.67 | 3.51 | 88.75 | 1.67 |\n| 0.50 | 77.83 | 99.97 | 0.84 | 4.27 | 86.87 | 3.56 |\n\n**Q5:** I'm curious about ... makes feature decoupling less effective\n\n**A5:**  To validate this hypothesis, we trained a 4-layer MLP with the BadNets attack and another benign MLP for the CIFAR-10 dataset. In the table, the ``num. to target'' refers to the number of samples (in 5000 validation samples) that are classified as the backdoor target after our detection. After BAN detection, we find BadNets MLP classifies 3607 samples as the target, while for benign MLP, it is 419.  It means that BAN detects the backdoor.\n\nWe also find that the positive feature loss (benign) is very close to the negative loss (potential backdoo). It indicates that the backdoor features are more challenging to decouple from benign ones, as the reviewer hypothesized. Note that the performance drop of MLP might also make features challenging to recognize.\n\n|MLP| BA | ASR | pos. feature loss | neg. feature loss| num. to target|Mtigation(BA)|Mtigation(ASR)|\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n|Benign| 53.24 | - | 1.95 | 2.29 | 419 | -| - |\n|BadNets| 47.04 | 100 |2.03 | 2.34 | 3607| 45.13 | 7.11|\n\n**Q6.** I also believe... applying the mask (1-m)\n\n**A6:** In practice, selecting the lambda value is easy. As discussed in the method section, the motivation for using the mask norm ($L_1$ norm) with lambda in Eq. (5) is to ensure the optimization objective is decoupling between benign and backdoor features.\nWithout the constraint of the mask norm in Eq. (5), the optimization objective will be simply increasing the mask norm unless there are extremely strong backdoor features. Therefore, we choose a lambda value for Eq. (5) such that the mask norm does not change significantly while optimizing. \n\nThis selection is easy because we only need to check the value of the mask norm. For example, the following table shows the $L_1$ mask norm and positive and negative feature losses. The feature size is $512\\times 4\\times 4$, which means the maximum of the mask norm is 8192 (every value in the mask is one). It is clear that the mask is almost full of ones when $\\lambda_1$ is smaller than 0.7, and the negative feature loss (backdoor feature) is ignored. Therefore, in this case, we need the $\\lambda_1$ values to be greater than 0.7. The selection of the lambda values is unaware of potential backdoors.\n\n| $\\lambda_1$ | mask norm | Pos. feature loss | Neg. feature loss |\n| ---- | ---- |  ---- | ---- |\n| 0.0 | 8188.62 | 0.268| 2.30 |\n| 0.1 | 8188.75 | 0.28 | 2.30 |\n| 0.2 | 8184.30 | 0.27 | 2.30 |\n| 0.3 | 8175.50 | 0.29 | 2.30 |\n| 0.4 | 8152.40 | 0.25 | 2.30 |\n| 0.5 | 8131.26 | 0.27 | 2.29 |\n| 0.6 | 8055.07 | 0.21 | 2.27 |\n| 0.7 | 7898.25 | 0.26 | 2.24 |\n| 0.8 | 596.85 | 0.99 | 0.23 |\n| 0.9 | 22.08 | 2.33 | 0.28 |\n\n\n\n\n[A] ABS: Scanning neural networks for back-doors by artificial brain stimulation, CCS 2019\n\n[B] Backdoor Scanning for Deep Neural Networks through K-Arm Optimization, ICML 2021\n\n[C] Reconstructive neuron pruning for backdoor defense, ICML 2023\n\n[D] Pre-activation distributions expose backdoor neurons, NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6mVFTCLV0a", "reviewer": "Reviewer_U3ov", "summary": "This paper addresses the problem of efficient backdoor defense using the backdoor inversion approach. The authors leverage the past work “TOWARDS RELIABLE AND EFFICIENT BACKDOOR TRIGGER INVERSION VIA DECOUPLING BENIGN FEATURES” (BTI-DBF) which recovers a mask in the feature space to locate prominent backdoor features and decouples benign and backdoor features in the mask. The authors make contributions by improving the BTI-DBF method and by incorporating extra neuron activation information into their method called “Backdoors activated by Adversarial neuron Noise” or BAN.", "questions": "The authors should answer the questions posed as weaknesses of the paper.", "limitations": "The limitations of the work lie in the accuracy and computational complexity of BAN. The authors describe the limitations in Appendix A and focus the limitations on the availability of clean (benign) samples and a slight decrease in the accuracy for benign inputs after fine-tuning.\n\nThe authors state, “We also exploit the neuron noise to further design a simple yet effective defense for removing the backdoor, such that we build a complete defense framework.” The statement “we build a complete defense framework” is an overstatement in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper lie in \n\n(1)\tstudying the trigger inversion-based detection methods\n\n(2)\toptimizing the performance of the BTI-DBF method by adding a regularizer to the loss function\n\n(3)\tincorporating adversarial activation noise into the BAN method", "weaknesses": "The weaknesses of the paper lie in \n\n(1)\tmissing explanations of the performance. For example, why is featureRE in Table 3 performing better than BAN? What would be the impact of a lambda value other than 0.5? Why is the Blend attack always better detected by BTI-DBF* than by BAN? \n\n(2)\tmissing relationship between the adversarial activation noise approach and input/feature perturbation approach?  How is the adversarial activation noise approach different from assuming Lipschitz continuity on the estimated input-output function?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work. We address your comments as\nfollows.\n\n**Q1.** missing explanations of the performance. For example, why is featureRE in Table 3 performing better than BAN? What would be the impact of a lambda value other than 0.5? Why is the Blend attack always better detected by BTI-DBF* than by BAN?\n\n**A1:** We agree that featureRE provides slightly better benign accuracy but fails to mitigate the backdoor ASR for BadNets, Blend, and Bpp, while our method is consistently effective.\nRegarding the difference between featureRE and BAN, we observed that commonly (e.g., BTI-DBF or BAN), including featureRE, there is a trade-off between benign accuracy and backdoor deletion, where the backdoor deletion task harms benign accuracy. \n\nAbout the $\\lambda$ values:\nWe provided experiments with different values, as shown in Table 7 in the appendix (we tested 0.2, 0.5, 0.8, and 1). \nWe saw that as the lambda value increased, the BA dropped. \n\nFor the Blend attack, as shown in Table 3, the Blend backdoor is only better detected by BTI-DBF* on ResNet18, but our method performs better on VGG16 and DenseNet121.\n\n\n**Q2.** missing relationship between the adversarial activation noise approach and input/feature perturbation approach? How is the adversarial activation noise approach different from assuming Lipschitz continuity on the estimated input-output function?\n\n**A2:** Thank you for this intriguing question. From the literature, it is known that for a backdoor attack, if we have a small trigger that changes the output of a benign input into a malicious target label, this behavior can be related to the high Lipschitz constant [A] and a neural network with high robustness then tends to have a lower local Lipschitz constant.\nMoreover, a larger local Lipschitz constant implies steeper output around trigger-inserted points, leading to a smaller trigger effective radius and making trigger inversion less effective [B].\nThus, the concepts of adversarial activation noise and Lipschitz continuity are related, and the local Lipschitz constant can serve as an upper bound for the trigger’s effective influence. We will include a discussion about this in the revised version of the document.\n\nIn addition, introducing theoretical tools like the Lipschitz constant for backdoor defense may also be very tricky in practice because it needs approximation for implementation. For example,  [A] evaluates the channel-wise Lipschitz constant by its upper bound but does not thoroughly discuss the relationship between the channel-wise Lipschitz constant and the network-wise Lipschitz constant, where the theorem of Lipschitz continuity really relies on. In another recent paper [C], it is also mentioned that empirically estimating the Lipschitz constant is hard from observed data, which usually leads to overestimation. We admit that methods relying on Lipschitz continuity may not require heavy computational load and are also related to our approach. However, our method emphasizes more on fine-tuning with the guidance of neuron noise, rather than tuning the trained model.\n\n\n\n[A] Zheng, R., Tang, R., Li, J., Liu, L. Data-Free Backdoor Removal Based on Channel Lipschitzness. ECCV 2022.\n\n[B] Rui Zhu, Di Tang, Siyuan Tang, Zihao Wang, Guanhong Tao, Shiqing Ma, Xiaofeng Wang, Haixu Tang. Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering. NDSS 2024.\n\n[C] Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness\n\n**Q3.** The authors state, “We also exploit the neuron noise to further design a simple yet effective defense for removing the backdoor, such that we build a complete defense framework.” The statement “we build a complete defense framework” is an overstatement in the paper.\n\n**A3:** We agree with the reviewer that this claim may not be accurate in some scenarios, and we rephrase the sentence as follows to avoid confusion: \"We also leverage the neuron noise to design a simple yet effective defense that removes the backdoor.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M3tcAGvD9G", "reviewer": "Reviewer_Eja2", "summary": "This paper provides an in-depth analysis of the SOTA trigger inversion-based backdoor defenses and finds that they suffer from high computational overhead and rely on prominent backdoor features. The authors tackle the challenges based on the previous findings on adversarial noise incorporating activation information and propose to improve the backdoor feature inversion for backdoor detection. The experiments show its efficiency and effectiveness compared to the SOTA methods.", "questions": "None.", "limitations": "1. The reliance on prominent backdoor features is expected to be illustrated in detail, and the reason why the proposed method is able to overcome it with only a mask regularizer is unclear.  Some case visualizations are expected.\n2.  The conclusion from Equation 7 that ignoring the second term is expected to be illustrated with evidence such as the separate loss values on it.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is intuitively reasonable and empirically effective. Both backdoor detection and defense are carefully considered.\n2. The paper writing is clear and well-structured.\n3. The experiments are comprehensive.", "weaknesses": "1. The motivations of high computational overhead and reliance on prominent backdoor features are not well-illustrated in the method section. Only the limitations of BTI-DBF are discussed. \n2. The novelty is limited. The finding on neuron noise is previously proposed, and an additional mask regularizer based on BTI-DBF is a commonly used technique, which was used in NC as well. The idea of backdoor defense is similar to ANP, with the change in learning from the noise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We address your comments as follows.\n\n**Q1.** The motivations of high computational overhead and reliance on prominent backdoor features are not well-illustrated in the method section. Only the limitations of BTI-DBF are discussed.\n\n**A1:** High computational overhead:\nMost existing methods conduct trigger inversion for every class to detect potential backdoors (e.g., NC, FeatureRE, or Unicorn), so detection algorithms need to be executed multiple times, each using a different class as the optimization objective.\nOur detection does not rely on class-wise optimization and only executes once to detect the potential backdoor, which significantly reduces the time consumption.\nSimilarly, BTI-DBF also only executes once for detection, but BTI-DBF needs to pre-train a U-Net generator to generate trigger images, which is computationally heavier.\n\nProminent backdoor features:\nThe reliance on prominent backdoor features is illustrated in Figure 3. \nWe show that benign and backdoor features are very close for BadNets but prominent for advanced attacks, such as WaNet, IAD, and BPP. Please also refer to the analysis under Q3. \n \n\n**Q2.** The novelty is limited. The finding on neuron noise is previously proposed, and an additional mask regularizer based on BTI-DBF is a commonly used technique, which was used in NC as well. The idea of backdoor defense is similar to ANP, with the change in learning from the noise.\n\n**A2:** Our main finding is that prominent backdoor features may not be suitable for identifying input space backdoors. \nBased on the findings, we show the generalization shortcomings of previous approaches, and we fix current approaches by introducing the regularizer with the further help of neuron noise.\nWe would argue that both our findings and defenses are novel because we recognized and then tried to solve common and previously unknown shortcomings from previous research.\nTechnically, we introduced a novel feature space regularizer (Eq. (5)) combined with adversarial neuron noise, which strategically increases the feature difference and, more importantly, helps the defense generalization.\n\n\n**Q3.** The reliance on prominent backdoor features is expected to be illustrated in detail, and the reason why the proposed method is able to overcome it with only a mask regularizer is unclear. Some case visualizations are expected.\n\n**A3:** We will revise and improve the contents in Section 3.3 and Section 4.1, together with Figure 1, to better illustrate our findings on prominent backdoor features. \nRegarding BAN, the mask regularizer's utility is limited, and it is only effective when combined with the adversarial neuron noise.\n\nWe also provide new experimental analysis in the table below, showing that previous decoupling methods cannot easily pick up backdoor features of weak attacks, such as BadNets, as their backdoor features are not prominent. \nFor example, when detecting without the $L_1$ regularizer (i.e., w/o norm), the negative feature loss of BadNets is high with a very large $L_1$ mask norm, while the Bpp has an even higher negative loss with a much smaller mask norm.\nNote that for BadNets, the negative loss is high with a high mask norm where almost all features are included. \nIt indicates that BadNet backdoor features are less prominent than Bpp features, making it more challenging to decouple BadNets features. We will provide a discussion in the revised draft. \n\n(The table is also presented in the global rebuttal.)\nA note on the table: The shape of the feature mask is $512\\times4\\times4$, which means the maximum of the mask $L_1$ norm is 8192.\n\n|Attack| BA | ASR | $L_1$ mask norm | pos. feature loss | neg. feature loss|\n| ---- | ---- | ---- | ---- | ---- | ---- |\n|BadNets (w/ norm)| 93.47 | 99.71 | 2258.90 | 0.21 | 0.26 |\n|BadNets (w/o norm)| 93.47 | 99.71 | 8054.45 | 0.14 | 2.17 |\n|Blend (w/ norm) | 94.60 | 100 | 2084.62 | 0.15 | 0.20 | \n|Blend (w/o norm) | 94.60 | 100 | 8117.90 | 0.04 | 2.22 |\n|WaNet (w/ norm)| 93.88 | 99.63 | 7400.97 | 0.06 | 2.34 |\n|WaNet (w/o norm)| 93.88 | 99.63 | 7702.56 | 0.05 | 2.39 |\n|IAD (w/ norm)| 93.82 | 99.64 | 7898.91 | 0.03 | 2.25 |\n|IAD (w/o norm)| 93.82 | 99.64 | 7895.16 | 0.03 | 2.25 | \n|Bpp (w/ norm)| 94.56 | 99.97 | 7147.68 | 0.09 | 2.80 |\n|Bpp (w/o norm)| 94.56 | 99.97 | 7260.31 | 0.09 | 2.78 |\n\n\n**Q4.** The conclusion from Equation 7 that ignoring the second term is expected to be illustrated with evidence such as the separate loss values on it.\n\n**A4:** The table above shows the $L_1$ mask norm and separated feature losses.\nWhen optimizing without the penalty of mask norm, for relatively weaker attacks (BadNet and Blend), the mask is very dense because the $L_1$ mask norm values are close to the upper bound (8192).\nThe negative feature loss values are also large for BadNets and Blend, but this is because the negative feature mask, i.e., (1-mask), is close to zero.\nAlmost no feature is used to get the negative feature loss, i.e., (1-mask)$\\times$feature in Eq. (5) leads to the high loss value.\nIn other words, the high negative feature loss (without the mask penalty) is not because of backdoor features on BadNets and Blend.\n\nWith the penalty of mask norm, the optimization of the feature mask focuses more on finding backdoor features rather than only increasing the proportion of benign features.\nIn the table, when optimizing with the penalty of mask norm, BadNets and Blend provide small negative feature loss values compared to advanced attacks (WaNet, IAD, and Bpp), even with a larger (1-mask).\nA small mask norm means the mask provides a larger proportion on the negative mask, i.e., (1-mask).\nThat is to say, there are fewer backdoor features, or the backdoor features are less prominent.\nIn conclusion, the feature decoupling using the mask is not effective when the backdoor features are not prominent.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pPFv6gDAbP", "reviewer": "Reviewer_cerE", "summary": "The paper proposes a novel detection and defense method for backdoored models. The new method is motivated by the finding that existing state-of-the-art trigger-inversion methods, like BTI-DBF, rely on strong backdoor features, which might not always be present, such as for  BadNet-type triggers. As a solution, the paper proposes BAN (Backdoors activated by Adversarial neuron Noise), a novel detection method based on the findings that models with backdoors are more sensitive to adversarial noise than benign ones, allowing the identification of neurons responsible for the backdoor function. Practically, BAN computes neuron-specific noise inspired by adversarial attacks like PGD to maximize the model's classification loss on clean and unseen data. The method tries to decouple benign and backdoor features by optimizing a sparse neuron mask on the loss behavior of the model. Intuitively, a backdoored model tends to predict the target class label for clean inputs and the adversarial perturbed neurons, whereas a benign model shows fewer misclassifications under comparable neuron noise. Evaluated against common backdoor attacks and defense methods, BAN demonstrates improvements to existing approaches and robustness against adaptive attacks.", "questions": "- L303: \"FeatureRE is designed for all-to-one attacks, so we use target label 0 here for FeatureRE.\" -> Does it make sense to include an approach designed for all-to-one attacks in an all-to-all setting? The comparison here with the other approaches seems misleading.\n- Table 4: What are the BA (benign accuracy) and ASR (attack success rate) of models trained only on clean data without any poisoned data samples? It would be interesting to compare the defended backdoored models to a clean model. Since ImageNet (and its subsets) are complex datasets to learn, there will also be misclassifications that might, by chance, lead to the prediction of target class labels (even without any backdoor integration). Given ASR of the defended models, it would be interesting to see a baseline ASR (achieved only by random misclassifications) to assess if the remaining ASR after applying the defense methods is due to remaining backdoor behavior or simple random model behavior.\n- The proposed method is based on the assumption that under neuron noise, a backdoored model tends to predict the target class label. In contrast, the predicted class labels for the benign model are rather equally distributed. This makes sense for datasets with clearly distinguishable classes, e.g., CIFAR10. Let us assume that there exist two classes in the dataset that are semantically very similar, e.g., two visually similar (but still different) dog breeds among other clearly separable classes (bug, plane, etc.). Let us further assume that the backdoor target is to change the label from the one dog class to the other one. If we now apply the neuron noise approach to a benign and a backdoored model, can we still assume that only the backdoored model will tend to predict the other dog class (the target class) under neuron noise? Or will the benign model under neuron noise behave similarly, since the other dog class is probably close in the feature space, and adding noise to the neuron activations could lead to predictions for the second dog class (which happens to be also the backdoor target)? Phrased differently, can we assume the proposed BAN method also works reliably on datasets with semantically similar classes, if the backdoor targets label changes from one class to the other?", "limitations": "Limitations are sufficiently discussed in Appx. A. Since the paper proposes a novel defense method for backdoor attacks, no negative societal impact is expected.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The proposed defense method adds an interesting aspect to existing backdoor defense methods by improving the feature decoupling between benign and backdoor features. Whereas the decoupling optimization is conceptionally simple (which is not bad at all), it demonstrates strong results on the evaluation benchmarks and beats existing detection and defense methods markedly.\n- The paper is well-written and easy to follow. Most parts of the proposed method is sufficiently placed in existing research, and the open research problems and the proposed solution are clearly presented and described. I enjoyed reading the paper.\n- The evaluation is comprehensive and compares the proposed method against numerous attacks and defenses. It also includes the all-to-all setting, which is often ignored in literature. Evaluations of adaptive attacks underline the effectiveness and robustness of the approach.\n- Empirical solutions and theoretical considerations sufficiently support all claims in the paper.", "weaknesses": "- The evaluation focuses on common (comparably shallow) CNN architectures. However, given that ViT also plays an increasing role in image classification, showing the effectiveness of the approach on ViT architectures in contrast to traditional CNNs would further support the efficiency claims of the method. \n- The evaluation further focuses on dirty label attacks, i.e., attacks that change the label of a training example. However, the method's efficiency on clean-label attacks is not demonstrated. Including 1-2 clean label attacks in the evaluation would further strengthen the results.\n- Some contributions of the paper should be stated more clearly and set apart from existing approaches in literature. For example, the feature decoupling process described in 3.3 is quite similar to the method by Xu et al. [1], and it is unclear to me (by only reading this paper) what exactly distinguishes the proposed method from the one in the existing literature. \n\nSmall remarks:\n- Some captions are missing details. For example, the caption of Table 5 does not clearly state the investigated setting. Which dataset and model architecture are used here? \n- The font in the figures could be increased. Some legends and texts are hard to read without zooming in.\n- There is a typo in line 142: \"experiemnts\" -> experiments.\n- Table 7 seems like a sensitivity analysis (measuring the sensitivity of the method to the hyperparameter selection) instead of an ablation study (impact of deactivating certain parts of the method).\n\n[1] Xu et al. \"Towards Reliable and Efficient Backdoor Trigger Inversion via Decoupling Benign Features\". ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. We address your comments as follows.\n\n\n**Q1.** The evaluation only focuses on CNN architectures.\n\n**A1:** We provide ViT results on a 12-layer Swin transformer in the following table.\nFor BadNets and Blend, we train the backdoored network using Adam as the optimizer.\nFor WaNet, IAD, and Bpp, we use the default setting.\nAll models in the following table are successfully detected as backdoored models.\nIn addition, we notice that the attack performance of WaNet, IAD, and Bpp is not as good as BadNets and Blend.\nWe conjecture this is because of training with SGD.\n\n\n| Attack | No def. (BA) | No def. (ASR) |FT (BA) | FT (ASR)| BTI-DBF (BA) | BTI-DBF (ASR) | Ours (BA) | Ours (ASR) |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| BadNets | 86.7 | 97.43 | 82.23 | 44.07 | 85.21 | 99.99 | 83.92 | 2.91 |\n| Blend | 85.46 | 100 | 80.13 | 100 | 83.9 | 100 | 79.18 | 26.20 |\n| WaNet | 78.25 | 31.03 | 75.63 | 2.3 | 73.83 | 2.61 | 75.28 | 3.1 |\n| IAD | 76.35 | 88.71 | 74.58 | 54.63 | 74.29 | 61.16 | 76.26 | 9.8 |\n| Bpp | 79.19 | 63.74 | 75.35 | 12.99 | 74.66 | 11.04 | 74.25 | 6.91 |\n\n**Q2.** The evaluation further focuses on dirty label attacks\n\n**A2:** Our submitted draft did not include clean-label attacks because it has a stricter threat model, which leads to weaker attacks.\nHere, we provide clean-label backdoor experiments to address the reviewer's concern using a well-known clean-label attack [A].\nWe take default hyperparameters from the original paper.\nThe following table demonstrates the detection and mitigation performance on 20 networks.\nWe use one of the 20 networks for the mitigation experiment.\nExperimental results demonstrate that BAN is also effective against the clean-label attack.\n\n\n| Attack | BA | ASR | Detection Success Accuracy| Mitigation (BA) | Mitigation (ASR) |\n| ---- | ---- | ---- | ---- | ---- | ---- |\n|Clean Label[A] | 92.72 | 100 | 90\\% | 89.27 | 6.16 |\n\n\n\n**Q3.** Does it make sense to include an approach (FeatureRE) designed for all-to-one attacks in an all-to-all setting?\n\n**A3:** The results of featureRE in Table 5 are to show that methods based on all-to-one settings may not work for all-to-all.\nTherefore, we believe it is not misleading since the featureRE relies on the assumption that backdoor attacks are always all-to-one. We will clarify this in the revised version.\n\n\n**Q4.** Some contributions of the paper should be stated more clearly and set apart from existing approaches in literature. \n\n**A4:** BTI-DBF [1] assumes the benign and backdoor features can be decoupled by a mask for benign features and (1-mask) for backdoor features.\nThen, they conduct trigger inversion based on the feature mask.\nHowever, we show in the following table that their mask will be a dense matrix full of ones.\nThe feature's shape is $512\\times4\\times4$ (ResNet18), meaning the maximum mask norm is 8192.\nThe reason is that the benign and backdoor features are not prominent to each other for weaker attacks, such as BadNets and Blend.\nIn the BadNets (w/o norm), the negative feature loss is relatively larger because the (1-mask) is almost zero, and the loss is calculated by the feature $\\times$ (1-mask).\nTo solve this problem, we apply a $L_1$ maks norm to the feature mask decoupling.\nIn addition, the decoupling performance can be further improved by adversarial neuron noise. \nWe use the neuron noise to activate the backdoor such that the backdoor features are more prominent to the feature mask.\n\n\n|Attack| BA | ASR | L_1 mask norm | pos. feature loss | neg. feature loss|\n| ---- | ---- | ---- | ---- | ---- | ---- |\n|BadNets (w/ norm)| 93.47 | 99.71 | 2258.90 | 0.21 | 0.26 |\n|BadNets (w/o norm)| 93.47 | 99.71 | 8054.45 | 0.14 | 2.17 |\n|Blend (w/ norm) | 94.60 | 100 | 2084.62 | 0.15 | 0.20 | \n|Blend (w/o norm) | 94.60 | 100 | 8117.90 | 0.04 | 2.22 |\n\n\n**Q5.** Table 4: What are the BA (benign accuracy) and ASR (attack success rate) of models trained only on clean data without any poisoned data samples? It would be interesting to compare the defended backdoored models to a clean model. \n\n**A5:** We provide ASR on a benign model using CIFAR-10 and ResNet18 in the following table.\nThe ASR of other attacks is omitted because attacks such as IAD train a generator for the backdoor trigger, which does not apply to a benign model.\nIn the table, it is clear that ASR for the benign model is very low since the benign accuracy is high.\n\n|trigger type | BA | ASR | \n| ---- | ---- |  ---- |\n| BadNets | 94.77 | 0.56 |\n| Blend | 94.77 | 0.02 |\n\n\n**Q6.** can we assume the proposed BAN method also works reliably on datasets with semantically similar classes?\n\n**A6:** The classes with semantic similarities are actually widespread.\nThe phenomenon makes classification results of these similar classes closely related. For example, Table 5 of IB-RAR [B] shows cats and dogs (CIFAR-10 classes) tend to be classified as each other under adversarial attacks.\nFor ImageNet200, we use class n02096294 (Australian terrier) as the target class. \nThere are 19 kinds of terriers in our ImageNet200 datasets,  such as n02094433 (Yorkshire terrier) and n02095889 (Sealyham terrier).\n\nTo further support this observation, we train 10 backdoor networks using BadNets with different target classes.\nIn the following table, BAN is contentiously effective against the backdoor regardless of semantically similar classes. \n\n| target | BA | ASR | mitigation (BA)| mitigation (ASR) |\n| ---- | ---- |  ---- | ---- |  ---- |\n| 0 | 93.41 | 100 | 92.57 | 1.56 |\n| 1 | 93.51 | 100 | 92.53 | 0.84 |\n| 2 | 93.54 | 99.99 | 91.49 | 1.08 |\n| 3 | 93.59 | 99.99 | 92.14 | 1.93 |\n| 4 | 93.84 | 99.98 | 92.13 | 1.49 |\n| 5 | 93.52 | 100 | 91.84 | 3.29 |\n| 6 | 93.46 | 100 | 92.48 | 0.93 |\n| 7 | 93.56 | 100 | 92.36 | 0.73 |\n| 8 | 93.57 | 99.89 | 92.31 | 0.58 |\n| 9 | 93.36 | 100 | 91.65 | 2.40 |\n\n[A] Label-consistent backdoor attacks. \n\n[B] IB-RAR: Information Bottleneck as Regularizer for Adversarial Robustness", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and ACs,
 
We thank you for evaluating and providing thorough feedback on our work. We are glad to see that most reviewers rated the paper positively, agreeing on the topic relevance and results provided in our work, such as "The authors address a significant topic", "The experimental results affirm the design choices made by the authors" (reviewer yW41), "The exhibited efficiency is impressive, and the motivation behind neuron noise is intriguing" (reviewer hJLG), "The proposed method is intuitively reasonable and empirically effective" (reviewer Eja2), and "Empirical solutions and theoretical considerations sufficiently support all claims in the paper" (reviewer cerE).
 
We made our best efforts to address the remaining concerns, and have written individual responses.
We are confident that the reviewer feedback and the incorporated additional experiments and discussions have even further strengthened our paper.

We include new experiments to illustrate the phenomenon that prominent backdoor features exist for advanced attacks (WaNet, IAD, and BPP) but may not be for weaker attacks (BadNets, Blend).
Experimental results demonstrate that previous decoupling methods cannot easily pick up backdoor features from weak attacks, such as BadNets, as their backdoor features may not be prominent.
For example, when detecting without the $L_1$ regularizer (i.e., w/o norm), the negative feature loss of BadNets is high with a very large $L_1$ mask norm, while the Bpp has an even higher negative loss with a much smaller mask norm.
The high negative loss of BadNets is actually from the sparse feature mask rather than backdoor features, i.e., there are too many zeros in (1-mask).
It indicates that BadNet backdoor features are less prominent than Bpp features, making it more challenging to decouple BadNets features. We will add a discussion in the revised draft. 

A note on the table: The shape of the feature mask is $512\times4\times4$, which means the maximum of the mask $L_1$ norm is 8192.

|Attack| BA | ASR | $L_1$ mask norm | pos. feature loss | neg. feature loss|
| ---- | ---- | ---- | ---- | ---- | ---- |
|BadNets (w/ norm)| 93.47 | 99.71 | 2258.90 | 0.21 | 0.26 |
|BadNets (w/o norm)| 93.47 | 99.71 | 8054.45 | 0.14 | 2.17 |
|Blend (w/ norm) | 94.60 | 100 | 2084.62 | 0.15 | 0.20 | 
|Blend (w/o norm) | 94.60 | 100 | 8117.90 | 0.04 | 2.22 |
|WaNet (w/ norm)| 93.88 | 99.63 | 7400.97 | 0.06 | 2.34 |
|WaNet (w/o norm)| 93.88 | 99.63 | 7702.56 | 0.05 | 2.39 |
|IAD (w/ norm)| 93.82 | 99.64 | 7898.91 | 0.03 | 2.25 |
|IAD (w/o norm)| 93.82 | 99.64 | 7895.16 | 0.03 | 2.25 | 
|Bpp (w/ norm)| 94.56 | 99.97 | 7147.68 | 0.09 | 2.80 |
|Bpp (w/o norm)| 94.56 | 99.97 | 7260.31 | 0.09 | 2.78 |


Except for providing an additional prominent feature analysis, we also discuss and analyze different poisoning strengths, different architectures (i.e., less redundant MLP and popular Swin ViT), details on hyperparameter selections, and clean-label backdoors. 
New experimental analyses and discussions can be found in the responses to each reviewer. 

We are looking forward to hearing from you, and we remain at your disposal should you have any comments/suggestions.

Best regards,

Authors of BAN

Meta-review: This work proposes a defense against backdoor attacks that improve trigger-detection-based defenses. The opinions about this paper are all positive (three accept, one weak accept, and one borderline accept). The reviewers had various concerns with this paper. However, the majority of them were solved by the rebuttal. The only relevant weakness that persisted after the rebuttal is that the novelty of this paper is incremental as it is a combination and improvement of previously proposed techniques. Nevertheless, the reviewer agrees that its effectiveness and efficiency surpass the competitors' approaches and can be accepted. The authors are recommended to add the clarifications and extra experiments made in the rebuttal to the paper.
Decision: Accept (poster)