Title: BAN: Detecting Backdoors Activated by Adversarial Neuron Noise
Venue: NeurIPS 2024
Accepted: yes
Keywords: backdoor detection

backdoor trigger inversion

AI Security
Abstract: Backdoor attacks on deep learning represent a recent threat that has gained significant attention in the research community. 
Backdoor defenses are mainly based on backdoor inversion, which has been shown to be generic, model-agnostic, and applicable to practical threat scenarios. State-of-the-art backdoor inversion recovers a mask in the feature space to locate prominent backdoor features, where benign and backdoor features can be disentangled.  However, it suffers from high computational overhead, and we also find that it overly relies on prominent backdoor features that are highly distinguishable from benign features. To tackle these shortcomings, this paper improves backdoor feature inversion for backdoor detection by incorporating extra neuron activation information. In particular, we adversarially increase the loss of backdoored models with respect to weights to activate the backdoor effect, based on which we can easily differentiate backdoored and clean models. Experimental results demonstrate our defense, BAN, is 1.37$\times$ (on CIFAR-10) and 5.11$\times$ (on ImageNet200) more efficient with an average 9.99\% higher detect success rate than the state-of-the-art defense BTI DBF. Our code and trained models are publicly available at https://github.com/xiaoyunxxy/ban.

BAN: Detecting Backdoors Activated by Adversarial Neuron Noise

Abstract

Backdoor attacks on deep learning represent a recent threat that has gained significant attention in the research community. Backdoor defenses are mainly based on backdoor inversion, which has been shown to be generic, model-agnostic, and applicable to practical threat scenarios. State-of-the-art backdoor inversion recovers a mask in the feature space to locate prominent backdoor features, where benign and backdoor features can be disentangled. However, it suffers from high computational overhead, and we also find that it overly relies on prominent backdoor features that are highly distinguishable from benign features. To tackle these shortcomings, this paper improves backdoor feature inversion for backdoor detection by incorporating extra neuron activation information. In particular, we adversarially increase the loss of backdoored models with respect to weights to activate the backdoor effect, based on which we can easily differentiate backdoored and clean models. Experimental results demonstrate our defense, BAN, is 1.37 × (on CIFAR-10) and 5.11 × (on ImageNet200) more efficient with an average 9.99% higher detect success rate than the state-of-the-art defense BTI-DBF. Our code and trained models are publicly available at https://github.com/xiaoyunxxy/ban .

1 Introduction

Deep neural networks (DNNs) are known to be vulnerable to backdoor attacks, a setting where the attacker trains a malicious DNN to perform well on normal inputs but behave inconsistently when receiving malicious inputs that are stamped with triggers [9]. The malicious model is obtained by training with poisoned data [9, 6], by tampering with the training process [22, 2], or by directly altering the model's weights [14]. Backdoors are proposed for various domains, from computer vision [9, 22, 6, 26, 40, 25] to graph data [42] and neuromorphic data [1]. Still, backdoors in computer vision received most of the attention of the research community, which also means there is a significant variety of triggers. For instance, a trigger can be a pixel patch [9], an existing image [22], dynamic perturbation [25], or image warping [26]. Various backdoor attacks target different stages of the machine learning model pipeline, inducing several threat scenarios in which defenses are developed by exploiting different knowledge.

Trigger inversion is a principled method that makes minimal assumptions about the backdoor attacks in the threat model [38, 39, 44], and it has clear advantages to training-time [18] or run-time defenses [24, 19, 11]. Input space trigger inversion is first introduced by Neural Cleanse (NC) [36],

where potential triggers for all target classes are reversed by minimizing the model loss of clean inputs. Median absolute deviations of reversed triggers from all classes are then calculated to detect triggers. More recently, input space trigger inversion methods have been shown to be ineffective against feature space backdoor attacks [26, 38]. To address this problem, FeatureRE [38] proposes a detection method using feature space triggers. The authors observe that features of backdoored and benign inputs are separable in the feature space by hyperplanes. Unicorn [39] formally defines and analyzes different spaces for trigger inversion problems. An invertible input space transformation function is proposed to map the input space to others, such as feature space, and to reconstruct the backdoor triggers. These trigger inversion methods can be considered as an optimization problem for the targeted class. They optimize the input images to mislead the model under various constraints and recover strong triggers in different spaces. The state-of-the-art trigger inversion method, BTIDBF [44], increases the inversion efficiency by relaxing the dependency of trigger inversion on the target labels. BTI-DBF trains a trigger generator by minimizing the differences between benign samples and their generated poisoned version in decoupled benign features while maximizing the differences in remaining backdoor features. Feature space trigger inversion defenses are generic and effective against most backdoor attacks. However, we show that BTI-DBF may fail against BadNets, as its backdoor is not prominent in the feature space (see Section 3.4). In addition, existing works still suffer from huge computational overhead and overly rely on prominent backdoor features .

To resolve this shortcoming, we propose a backdoor defense called detecting B ackdoors activated by A dversarial neuron N oise (BAN). Our defense is inspired by the finding that backdoored models are more sensitive to adversarial noise than benign models [43, 10], and neuron noise can be adversarially manipulated to indicate backdoor activations [41]. Specifically, BAN generates adversarial neuron noise, where the weights of the victim model are adversarially perturbed to maximize the classification loss on a clean data set. Simultaneously, trigger inversion is conducted in the victim model's feature space to calculate the mask for benign and backdoor features. Clean inputs with masked feature maps are then fed to the adversarially perturbed model, based on the outputs of which backdoored models can be differentiated. Figure 1 presents a t-SNE visualization of the feature space for backdoored and clean models when they are perturbed with different levels of adversarial neuron noise. It can be observed that the backdoored model misclassifies parts of the data from all classes as the target class when the adversarial neuron noise increases, while the clean model has fewer misclassifications under the same level of noise. By leveraging the induced difference, BAN can successfully detect and mitigate backdoor attacks in both input and feature space.

We make the following contributions:

We find a generalization shortcoming in current trigger inversion-based detection methods (FeatureRE [38], Unicorn [39], BTI-DBF [44]). In particular, feature space detections overly rely on highly distinguishable prominent backdoor features. We provide an in-depth analysis of trigger inversion-based backdoor defenses, showing that prominent backdoor features that are exploited by state-of-the-art defenses to distinguish feature space backdoors may not be suitable for the identification of input space backdoors.

We propose detecting Backdoors activated by Adversarial neuron Noise (BAN) to mitigate this generalization shortcoming by introducing neuron noise into feature space trigger inversion. BAN includes an adversarial learning process to incorporate neuron activation information into the inversion-based backdoor detection. Experimental results demonstrate that BAN is 1.37 × (on CIFAR-10) and 5.11 × (on ImageNet200) more efficient with a 9.99% higher detect success rate than the state-of-the-art defense BTI-DBF [44].

We also exploit the neuron noise to further design a simple yet effective defense for removing the backdoor, such that we build a workable framework.

2 Backdoors

Attacks. Backdoor attacks [9, 22, 2, 26, 25, 40, 35, 8, 20, 3] refer to injecting a secret functionality into the victim model that is activated through malicious inputs that contain the trigger. To this end, substantial research has been proposed by poisoning a small percentage of training data using small and static triggers [9, 6, 22]. Early attacks generate backdoors in input space, where BadNets [9] is the first backdoor attack in DNNs. Blend [6] proposed three injection strategies to blend translucent images into inputs of DNNs as triggers. The authors controlled the transparency of the trigger to allow the trade-off between strength of attack and invisibility. Although these attacks work well,

Backdoor:

Benign:

Neuron noise increasing

Targeted label

𝜖 = 0.0

𝜖 = 0.2

𝜖 = 0.1

𝜖 = 0.3

their triggers are still perceptible to humans and can be easily detected by backdoor defenses, such as Activation Clustering (AC) [4] and NC [36].

Dynamic and imperceptible triggers [26, 25, 40, 8], including feature space backdoor triggers [20, 25], are explored to bypass both human observers and input space defenses. IAD [25] designs inputspecific triggers. To evaluate the uniqueness of dynamic triggers, the authors designed a cross-trigger test to determine whether the trigger of one input is reusable to others. WaNet [26] proposes warpingbased triggers, which are unnoticeable and smooth in the input space. Bpp [40] exploits vulnerabilities in the human visual system and, by using image quantization and dithering, introduces invisible triggers that are stealthier than previous attacks.

Adaptive backdoor attacks [31, 28, 24] are built to systematically evaluate defenses, where attacks discourage the indistinguishability of latent representations of poisoned and benign inputs in the feature space. Adap-Blend [28] divides the trigger image into 16 pieces and randomly applies only 50% of the trigger during data poisoning. They use the full trigger image at inference time to mislead the poisoned model. SSDT [24] considers the combination of source-specific and dynamic-trigger backdoors. Only the inputs from victim classes (source) with the dynamic triggers are classified to the target labels, which encourages the generalization of more diverse trigger patterns from different inputs. In this paper, we evaluate BAN against various types of attacks, including input space, feature space, and adaptive attacks, to provide a systematic evaluation resembling practical threats.

Defenses. Trigger inversion-based backdoor defense is considered one of the most practical and general defenses against backdoors [38, 39, 44]. The recovered trigger is used to determine whether the model is backdoored. For example, NC [36] reverses input space triggers to detect backdoors by selecting significantly smaller triggers in size. Other methods, such as ABS [21] and FeatureRE [38], usually determine whether there is a backdoor based on the attack success rate of the trigger. More specifically, given a DNN model f and a small set of clean data D c = { x n , y n } N n =1 , NC recovers the potential trigger by solving the following objective:

where ( x , y ) ∈ D c and m is the trigger mask, t is the trigger pattern, and y t is the target label. The mask m determines whether the pattern will replace the pixel. L is the cross-entropy loss function. Most prior works [12, 30, 21, 38, 39] follow this design to conduct trigger inversion for all possible target labels. For example, Tabor [12] adds more constraints to the NC optimization problem according to the overly large (large size triggers but no overlaps with the true trigger) and overlaying (overlap the true trigger but with redundant noise) problem of NC. Moving from input space to feature, FeatureRE [38] utilizes feature space constraint according to an observation that neuron activation values representing the backdoor behavior are orthogonal to others. Unicorn [39] proposes a transformation function that projects from the input space to other spaces, such as feature space [26] and numerical space [40]. Then, the authors conduct trigger inversion in different spaces.

Unlike previous NC-style methods, recent works [44, 37] explore different optimization objectives to avoid the time-consuming optimization for all possible target classes or to avoid the fixed maskpattern design as advanced attacks utilize more complex and dynamic triggers. BTI-DBF [44] takes advantage of the prior knowledge that benign and backdoored features are decoupled in the feature space. It distinguishes them by the optimization objective, where benign features contribute to the correct predictions and backdoored features lead to wrong predictions. Based on the decoupled features, BTI-DBF trains a trigger generator by minimizing the difference between benign samples and their generated version according to the benign features and maximizing the difference according to the backdoored features. Feature space backdoor defenses are developed based on the fact that backdoor features are highly distinguishable from benign features. However, this finding is not consistently valid for input space attacks where the feature difference is small (See Sections 3.4).

3 BAN Method

3.1 The Pipeline of Training Backdoor Models

For brevity, consider an L -layer fully connected network f (similar principles apply to convolutional networks) that has l = l 1 + l 2 + · · · + l L neurons. x n ∈ R d x and y n ∈ { 0 , 1 } d y are the n th image and its label in d x and d y dimensional spaces, respectively. The attacker creates a poisoned dataset D p by poisoning generators G X and G Y for a subset of the whole training dataset, i.e., D p = D c ∪D b . D c is the original clean data. D b is the poisoned backdoor data, D b = { ( x ′ , y ′ ) | x ′ = G X ( x ) , y ′ = G Y ( y ) , ( x , y ) ∈ D - D c } . In all-to-one attacks, G Y ( y ) = y t , y t is the attackerspecified target class. In our experiments, we consider the dirty-label attack. In all-to-all attacks, ususally G Y ( y ) = ( y + 1) [44, 26, 40], which is also what we chose in our experiments. In the training stage, the backdoor is injected into the model by training with D p , i.e., minimizing the training loss on D p to find the optimal weights and bias ( w ∗ , b ∗ ) :

where ℓ ( · , · ) is the cross-entropy. In the inference stage, the backdoored model predicts an unseen input ˆ x as its true label ˆ y but predicts G X (ˆ x ) as G Y (ˆ y ) : f ( G X (ˆ x ); w , b ) = G Y (ˆ y ) .

3.2 Threat Model

Attacker's goal. We consider the attacker to be the pre-trained model's provider. The attacker aims to inject stealthy backdoors into the pre-trained models. The pre-trained models perform well on clean inputs but predict the attacker-chosen target label when receiving backdoor inputs.

Attacker knowledge. The attacker has white-box access to the model, including training data, architectures, hyperparameters, and model weights.

Defender's goal and knowledge. The main goal is to detect whether a given model is backdoored and then remove the potential backdoor according to the detection results. Following [38, 39, 44], we assume the defender has white-box access to the model and holds a few local clean samples. However, the defender does not have access to the training data and has no knowledge of the backdoor trigger.

3.3 Detection with Neuron Noise

Based on previous findings that adversarial noise can activate backdoors [43, 41, 10], we design a two-step method for backdoor detection. First, we search for noise on neurons that can activate the potential backdoor. Then, we decouple the benign and backdoored features using a learnable mask of the latent feature (the output before the final classification layer).

Neuron Noise. In backdoor models, there are two types of neurons: benign and backdoor [21]. The backdoor neurons build a shortcut from G X ( x ) to G Y ( y ) . Neuron noise is generated noises added on neurons to maximize model loss on clean samples in an adversarial manner [23]. The noise on benign neurons evenly misleads prediction to all classes, while the noise on backdoor neurons tends to mislead prediction to the target label due to the backdoor shortcut, as shown in Figure 1. Therefore, backdoor models with noise behave differently from benign models with noise, as there are no backdoor neurons in the benign models.

Given the j th neuron connecting the i th layer and ( i -1) th layer, we denote its weight and bias with w ij and b ij , respectively. Neuron noise can be added to the neuron by multiplying its weight and bias with a small number: (1 + δ ij ) w ij and (1 + ξ ij ) b ij . Then, the output of the neuron is:

where h ( i -1) is the output of the previous layer and σ ( · ) is the nonlinear function (activation). The noise on all neurons are represented by δ = [ δ 1 , 1 , · · · , δ l 1 , 1 , · · · , δ 1 ,L , · · · , δ l L ,L ] and ξ = [ ξ 1 , 1 , · · · , ξ l 1 , 1 , · · · , ξ 1 ,L , · · · , ξ l L ,L ] . The δ and ξ are optimized via a maximization to increase the cross-entropy loss on the clean data:

where S is the ball function of the radius ϵ in the l dimensional space. δ and ξ share the ball function S as the maximum noise size. The maximization in Eq. (4) can be solved by PGD [23] algorithm with a random start, as PGD can better explore the entire searching space to mitigate local minima [23].

Feature Decoupling with Mask. The neuron noise activates the backdoor (see Figure 1) and misleads the predictions to the target label. However, the performance has high variance when searching for noise multiple times, which we conjecture is caused by random initialization. Therefore, inspired by [44], we further introduce a feature decoupling process to enhance the effect of noise on backdoored features but maintain a decreased effect on benign features. Specifically, the network f is decomposed into g = f 1 ◦ · · · ◦ f L -1 and f L , where f L -1 extracts the latent features from x , while f L is the classification layer. Then, we use a mask m on top of the latent feature g ( x ) to decouple benign and backdoored features. The optimization objective can be written as:

where ⊙ is the element-wise product. This optimization divides the latent features into two parts through the mask while maintaining a relatively small size of m . Note that the regularizer

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

w/o mask

w/ mask

Benign

IAD

Blend WaNet

for m in Eq. (5) is necessary. Otherwise, m will become a dense matrix full of ones to focus on the positive part of Eq. (5) because maintaining only the positive part (without penalty of | m | ) already satisfies the optimization objective. Finally, we apply the negative mask (1 -m ) on top of latent feature of f to enhance the backdoor effect. The final output is:

In Figure 2, the blue dots show the accuracy after adding noise to the model. The red dots show the accuracy (with noise) while applying the feature mask to the model using Eq. (6). Our feature masks do not affect the performance of benign models. However, we see that the performance is significantly decreased for backdoored models. This decrease is caused by the model only using backdoor features (through the negative mask), which means the backdoor is activated more frequently. Finally, a suspect model is determined backdoored if the prediction using Eq. (6) has a high attack success rate. For all-to-one attacks, the misclassification will be concentrated into one label, which is the target label. For all-to-all attacks, we can do the same as for all-to-one attacks but evaluate the prediction for each class independently.

3.4 Improving BTI-DBF

This section shows that the most recent work, BTI-DBF [44], may fail to capture the backdoor features by its decoupling method in Table 2. We show how to patch BTI-DBF using a simple solution. The BTI-DBF is the original version, and BTI-DBF* is our improved version. The main pipeline of BTI-DBF consists of two steps: (1) decoupling benign and backdoor features and (2) trigger inversion

by minimizing the distance between benign features and maximizing the distance between backdoor features. We found that the defense's first step may introduce errors in the decoupled features. Similar to our Eq. (5), the decoupling of BTI-DBF can be written as:

However, this equation has no constraints on the feature mask m . Overall, the optimization objective is to decrease the loss. Obviously, BTI-DBF's decoupling encourages the norm of the mask to increase so that the loss will focus on the positive part and ignore the negative part because the negative part goes against the overall objective. Finally, the mask will be a dense matrix that is full of ones. The f L ◦ ( g ( x ) ⊙ (1 -m )) is ignored due to multiplying by zero. We propose a simple solution to fix the problem by adding a regularizer of the size of the mask to the loss, i.e., we use our Eq. (5) as the first step of BTI-DBF*. According to Table 2, BTI-DBF* successfully overcomes BTI-DBF's shortcomings.

3.5 Backdoor Defense

After determining whether a suspect model is backdoored, we can fine-tune the backdoored model to remove the backdoor. However, standard fine-tuning using clean data does not effectively remove the backdoor because it does not activate it. Therefore, we propose using optimized neuron noise to fine-tune the model. In the optimization of the neuron noise, the objective is to increase the loss of L ( f ( x ) , y ) . We consider both the benign and backdoor neurons to contribute to the increase in loss when optimizing the noise. Indeed, on benign neurons, the neuron noise misleads f to any result other than the true label y , while the noise misleads f to the target label G Y ( y ) on backdoor neurons. Therefore, a straightforward method is to decrease the loss between noise output and the true label. The loss for our noise fine-tuning can be written as:

4 Experimental Results

Datasets and Architectures. The datasets for our experiments include CIFAR-10 [16], GTSRB [33], Tiny-ImageNet [17], and a subset of ImageNet-1K [7]. The ImageNet subset contains 200 classes, which is referred to as ImageNet200. BAN is evaluated using three architectures: ResNet18 [13], VGG16 [32], and DenseNet121 [15]. Please refer to Appendix B.1 for more details.

Attack Baselines. Our experiments are conducted using seven attacks: BadNets [9], Blend [6], WaNet [26], IAD [25], BppAttack [40], Adap-Blend [28], and SSDT [24], which are commonly used in other works [44, 38, 39, 46, 29]. The BadNets [9] and Blend [6] are designed for input space. WaNet [26], IAD [25], and BppAttack [40] are designed for feature space. Adap-Blend [28] and SSDT [24] (for adaptive evaluation) are state-of-the-art attacks that have been recently introduced to bypass backdoor defenses. The main idea of Adap-Blend [28] and SSDT [24] is to obscure the latent separation in features of benign and backdoor samples. More details can be found in Appendix B.2.

Defense Baselines. BAN is compared to five representative methods: Neural Cleanse (NC) [36], Tabor [12], FeatureRE [38], Unicorn [39], and BTI-DBF [44]. NC and Tabor are designed for input space attacks, while the other three are designed for feature space and dealing with the latest advanced attacks. BAN uses only 1% and 5% of training data for detection and defense, respectively. Note the data used for our defense is not used for training models, i.e., the defender has no knowledge of the model to be defended. More details can be found in Appendices B.3 and B.4. We use bold font to denote the best results.

4.1 The Performance of Backdoor Detection

Main Results. In Table 1, BAN shows better results on CIFAR10 than all baselines, especially on advanced attacks. Results on

BadNets

WaNet

IAD

0

1

2

3

mask loss

Loss with positive or negative mask

benign

backdoored

other datasets are presented in Appendix C. We note that the advanced detection methods (FeatureRE, Unicorn, and BTI-DBF) perform worse than NC on simple attacks (BadNets and Blend). We hypothesize this is because the backdoor features generated by BadNets are less obvious on feature channels than advanced attacks in the feature space, such as WaNet and IAD. Figure 3 shows that BadNets features are weaker than features from advanced attacks using the feature mask in Eq. (5). Specifically, we optimize the feature mask to disentangle the benign and backdoor features for four attacks. Then, we compute two cross-entropy loss values using the positive mask ( m ) and the negative mask ( 1 -m ) for benign and backdoor features, respectively. The average loss values and standard deviation for four models for each attack are plotted in Figure 3. The negative loss value of BadNets is much smaller than others, which means BadNets features are weaker than others with regard to misleading the model to the backdoor target. Recent defenses usually add more regularizers to their losses and optimization objectives to counteract powerful backdoor attacks. These regularizers encourage the capturing of strong features but omit weak ones. Thus, recent advanced detections can perform worse on BadNets than NC.

Time consumption. BAN is efficient and scalable as we do not iterate over all target classes. Figure 4 demonstrates that BAN uses substantially less time than all baselines. BAN is also more scalable for larger architectures or datasets. BIT-DBF (76.90s) is 1.37 × slower than BAN (55.95s) on CIFAR-10 with ResNet18, and BIT-DBF (5,792.51s) without pre-training is 5.11 × slower than ours (1,132.85s) on ImageNet200 with ResNet18. FeatureRE (297.74s) is 5.32 × slower than ours on CIFAR-10 with ResNet18, and it (6,053.62s) is 45.14 × slower on CIFAR-10 with DenseNet121 than ours (134.10s).

4.2 The Performance of Backdoor Defense

A complete backdoor defense framework should include both detection and defense. The goal of defense is to decrease the attack success rate (ASR) of backdoor triggers. Detection before the defense is also necessary because the defense usually decreases the performance on benign inputs [47, 44, 38, 46, 48]. In Section 3.5, we propose a simple and effective fine-tuning method using the noise that activates the backdoor. Table 3 compares BAN with three baselines: plain finetuning, FeatureRE [38], and BTI-DBF(U) [44]. Plain fine-tuning refers to training the backdoor model using the same hyperparameters as BAN but without the noise loss in Eq. (8). The FeatureRE [38]

CIFAR-10

Tiny-ImageNet

ImageNet200

Ours

BTI-DBF

BTI-DBF (w/ pre)

NC

FeatureRE

Tabor

Unicorn

and BTI-DBF(U) [44] refer to the defense methods from the respective paper. We use default hyperparameters for FeatureRE [38] and BTI-DBF(U) [44]. For plain fine-tuning and BAN, we use a small learning rate (0.005) to avoid jumping out of the current optimal parameters of the well-trained model. Then, we use λ 2 (0.5) for Eq. (8) for the trade-off between robustness against backdoor and clean accuracy. We fine-tune for a short schedule of 25 epochs, as the model is well-trained. Table 8 in Appendix B.4 shows defense performance with standard deviation on different hyperparameters, which supports our choice. Tables 3 and 4 demonstrate that our fine-tuning method effectively removes the backdoor while preserving high accuracy in benign inputs. We provide a comparison between our find-tuning with ANP [41] in Table 12, Appendix C.4, as ANP uses the neuron noise for pruning backdoor neurons.

4.3 Defense against All-To-All Attacks

Previous works [38, 39] usually only consider all-to-one attacks, which limits the application in practical situations. In this section, we evaluate our fine-tuning method under three all-to-all attacks: WaNet-All [26], IAD-All [25], and Bpp-All [40]. FeatureRE is designed for all-to-one attacks, so we use target label 0 here for FeatureRE. FeatureRE is included to show that the all-to-one defense does not work in the all-to-all setting. BAN is capable of handling all-to-all attacks because we directly explore the neurons themselves instead of optimizing for the potential targeted label. Table 5 demonstrates the effectiveness of our method.

4.4 Evaluation under Adaptive Attack

Attackers may design a specific attack for defense when they know its details [34]. In this section, we evaluate BAN against two attacks that attempt to bypass the difference between backdoor and benign features: Adap-Blend [28] and SSDT attack [24]. Both Adap-Blend [28] and SSDT attack [24] try to obscure the difference between benign and backdoor latent features. Adap-Blend [28] achieves it by randomly applying 50% of the trigger to poison the training data, while SSDT attack [24] utilizes source-specific and dynamic-triggers to reduce the impact of triggers on samples from non-victim classes. The source-specific attack refers to backdoor triggers that mislead the model to the target class only when applied to victim class samples. Table 6 demonstrates our approach is resistant to these two attacks, while other methods fail. The reason is that the backdoor features of SSDT are close to benign features in the feature space. It is difficult for other methods to distinguish between backdoor and benign features created by SSDT. Our detection method directly analyzes the model itself using neuron noise, which captures the difference between backdoor and benign models concerning parameters. See Appendix C.3 for the mitigation results of our method against these adaptive attacks.

4.5 Analysis on Prominent Features

We provide additional analysis of the phenomenon that backdoor features are more prominent for advanced attacks (WaNet, IAD, and Bpp) than weaker attacks (BadNets, Blend). Table 7 demonstrates that previous decoupling methods cannot easily pick up backdoor features from weak attacks, such as BadNets. In particular, when detecting without the L 1 regularizer (i.e., w/o norm), the negative feature loss of BadNets is high with a very large L 1 mask norm, while the Bpp has an even higher negative loss with a much smaller mask norm. The high negative loss of BadNets is actually from the sparse feature mask rather than backdoor features, i.e., there are too many zeros in (1 m ). This indicates that BadNet backdoor features are less prominent than Bpp features, making it more challenging to decouple BadNets features.

5 Conclusions and Future Work

This paper proposes an effective yet efficient backdoor defense, BAN, that utilizes the adversarial neuron noise and the mask in the feature space. BAN is motivated by the observation that traditional defenses outperformed the latest feature space defenses on input space backdoor attacks. To this end, we provide an in-depth analysis showing that feature space defenses are overly dependent on prominent backdoor features. Experimental studies demonstrate BAN's effectiveness and efficiency against various types of backdoor attacks. We also show BAN's resistance to potential adaptive attacks. Future studies could explore a more practical detection method without assuming access to local benign samples and better strategies for decoupling features because a fixed mask in the feature is not always aligned with the benign and backdoor features.

Acknowledgments and Disclosure of Funding

This work was partly funded by the Dutch Research Council (NWO) through the PROACT project (NWA.1215.18.014) and the CiCS project of the research programme Gravitation (024.006.037).

References

Gorka Abad, Oguzhan Ersoy, Stjepan Picek, and Aitor. Urbieta. Sneaky spikes: Uncovering stealthy backdoor attacks in spiking neural networks with neuromorphic data. In NDSS , 2024.

Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In USENIX Security , 2021.

M. Barni, K. Kallas, and B. Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In ICIP , 2019.

Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. SafeAI Workshop @ AAAI , 2018.

Erh-Chung Chen, Pin-Yu Chen, I Chung, Che-Rung Lee, et al. Data-driven lipschitz continuity: A cost-effective approach to improve adversarial robustness. arXiv preprint arXiv:2406.19622 , 2024.

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv , 2017.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In ICCV , 2021.

Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access , 7, 2019.

Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-box backdoor detection using adversarial extreme value analysis. In ICLR , 2022.

Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. SCALE-UP: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. In ICLR , 2023.

Wenbo Guo, Lun Wang, Yan Xu, Xinyu Xing, Min Du, and Dawn Song. Towards inspecting and eliminating trojan backdoors in deep neural networks. In ICDM , 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. Handcrafted backdoors in deep neural networks. NeurIPS , 2022.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR , 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 2015.

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurIPS , 2021.

Yinshan Li, Hua Ma, Zhi Zhang, Yansong Gao, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Yifeng Zheng, Said F. Al-Sarawi, and Derek Abbott. Ntd: Non-transferability enabled deep learning backdoor detection. TIFS , 19:104-119, 2024.

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV , 2021.

Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In CCS , 2019.

Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS , 2018.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

Xiaoxing Mo, Yechao Zhang, Leo Yu Zhang, Wei Luo, Nan Sun, Shengshan Hu, Shang Gao, and Yang Xiang. Robust backdoor detection for deep learning via topological evolution dynamics. In SP , 2024.

Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurIPS , 2020.

Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In ICLR , 2021.

Ren Pang, Zheng Zhang, Xiangshan Gao, Zhaohan Xi, Shouling Ji, Peng Cheng, and Ting Wang. Trojanzoo: Towards unified, holistic, and practical evaluation of neural backdoors. In Euro S&P , 2022.

Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In ICLR , 2023.

Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards a proactive ML approach for detecting backdoor poison samples. In USENIX Security , 2023.

Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. In ICML , 2021.

Reza Shokri et al. Bypassing backdoor detection algorithms in deep learning. In Euro S&P , 2020.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks , pages 323-332, 2012.

Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In NeurIPS , 2020.

Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771 , 2019.

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In SP , 2019.

Hang Wang, Zhen Xiang, David J Miller, and George Kesidis. Mm-bd: Post-training detection of backdoor attacks with arbitrary backdoor pattern types using a maximum margin statistic. In SP , 2024.

Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverseengineering of trojan triggers. In NeurIPS , 2022.

Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. UNICORN: A unified backdoor trigger inversion framework. In ICLR , 2023.

Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In CVPR , 2022.

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurIPS , 2021.

Jing Xu, Stefanos Koffas, O˘ guzhan Ersoy, and Stjepan Picek. Watermarking graph neural networks based on backdoor attacks. In Euro S&P , 2023.

Xiaoyun Xu, Oguzhan Ersoy, Behrad Tajalli, and Stjepan Picek. Universal soldier: Using universal adversarial perturbations for detecting backdoor attacks. In DSN-W , 2024.

Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren. Towards reliable and efficient backdoor trigger inversion via decoupling benign features. In ICLR , 2024.

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In ECCV , 2022.

Hong Zhu, Yue Zhao, Shengzhi Zhang, and Kai Chen. Neuralsanitizer: Detecting backdoors in neural networks. TIFS , 19:4970-4985, 2024.

Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In ICCV , 2023.

Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. In NeurIPS , 2023.

Rui Zhu, Di Tang, Siyuan Tang, Zihao Wang, Guanhong Tao, Shiqing Ma, XiaoFeng Wang, and Haixu Tang. Gradient shaping: Enhancing backdoor attack against reverse engineering. In NDSS , 2024.

A Discussion

Limitations. Similar to existing works [36, 21, 12, 38, 39, 44], BAN assumes a local small and benign dataset. This scenario is common, considering that benign samples are available online, and the model provider may provide some samples to verify the model's performance. In addition, our fine-tuning with neuron noise may slightly decrease the performance in benign inputs, similar to other defense methods [38, 44, 47, 41, 18, 36, 48]. However, BAN provides better performance and requires less time consumption.

Broader Impacts. This paper proposes a complete defense that includes detecting and removing DNN backdoors. We believe our approach has a positive impact on the security of DNNs. A possible negative impact is overconfidence in robustness against backdoor attacks, as there is no theoretical guarantee always to remove the backdoor.

B Additional Details for Experimental Settings

B.1 Datasets

CIFAR-10. The CIFAR-10 [16] contains 50,000 training images and 10,000 testing images with the size of 3 × 32 × 32 in 10 classes.

GTSRB. The GTSRB [33] contains 39,209 training images and 12,630 testing images in 43 classes. In our experiments, the images are resized to 3 × 32 × 32 .

Tiny-ImageNet. Tiny-ImageNet [17] contains 100,000 training images and 10,000 testing images with the size 3 × 64 × 64 in 200 classes.

ImageNet. ImageNet [7] contains over 1.2 million high-resolution images in 1,000 classes. Our experiments use a subset of the full ImageNet dataset, i.e., 200 randomly selected classes. Each class has 1300 training images and 50 testing images. The ImageNet images are scaled to 3 × 224 × 224 .

B.2 Backdoor Models

BAN and defense baselines are evaluated using seven well-known attacks: BadNets [9], Blend [6], WaNet [26], IAD [25], BppAttack [40], Adap-Blend [28], and SSDT [24].

For all backdoored models, we use SGD with a momentum of 0.9, weight decay of 5 × 10 -4 , and a learning rate of 0.01. We train for 200 epochs, and the learning rate is divided by 10 at the 100-th and 150-th epochs. Same to [41], we use 90% of training data for training backdoor models and 10% for the validation set.

BadNets. We use a 3 × 3 pattern to build the backdoor trigger. The poisoning rate is 5%.

Blend. We use the random Gaussian noise and blend ratio 0.2 for backdoor training. The poisoning rate is 5%.

Adap-Blend. We use the 'hellokitty_32.png' and blend ratio of 0.2 to build triggers. The poisoning rate is 5%.

SSDT. SSDT is a Source-Specific attack, which only misleads the victim classes to the targeted class. We use class 1 as the victim and class 0 as the target class.

For other attacks and hyperparameters not mentioned, we use the default settings from the papers or their official open-source implementations.

B.3 Defense Baselines

BAN is compared with five representative methods, including Neural Cleanse (NC) [36], Tabor [12], FeatureRE [38], Unicorn [39], and BTI-DBF [44]. In this section, we describe the hyperparameters of these defenses.

NC and Tabor. We use the implementation and default hyperparameters from TrojanZoo [27] for NC and Tabor. 1% of the training set is used to conduct 100 epochs of the trigger inversion. The learning rate is 0.01.

Masked feature

ASR

Clean input

: Adversarial neuron noise

FeatureRE. We first tried the default hyperparameters from the FeatureRE paper, but they could not work even for BadNets. We assume this is because the constraints, including feature mask size and similarity between original images and trigger images, are limited to a very small size. Therefore, we relaxed the constraints on masks and norms to find stronger triggers, including loss std bound=1.2, p loss bound=0.2, loss std bound=1.2, mask size=0.06, and learning rate=0.001. We use 1% of the training set and run FeatureRE for 400 epochs for each class.

Unicorn. The default hyperparameters for Unicorn are too powerful in our case, and the recovered triggers can mislead any model to the target label when applied to inputs. Every model, including benign models, is thought of as backdoored. Therefore, we slightly increase thread values of the constraints on Unicorn optimization to find proper triggers, including loss std bound=0.5, SSIM loss bound=0.1, and mask size=0.01. We use 1% of the training set and run Unicorn 40 epochs for each class.

BTI-DBF. We follow the default settings in the BTI-DBF [44] paper. Note that BTI-DBF uses 5% of training samples for defenses.

B.4 BAN Settings

Detection. We use epsilon ( ϵ ) to limit the noise added to neurons. Otherwise, the model weights are destroyed by the noise. The ϵ values are 0.3, 0.3, 0.2, and 0.1 for CIFAR-10, GTSRB, Tiny-ImageNet, and ImageNet200, respectively. ϵ values are decided according to the size of the images. We use smaller ϵ for larger datasets. We use the 30-step PGD algorithm to solve the optimization in Eq. (4) to find the noise, i.e., δ and ξ . We use SGD and the learning rate of ϵ/ 30 for the PGD optimization. Then, we use Adam and the learning rate of 0.01 to search for 20 epochs for the feature mask using Eq. (5). The λ 1 for Eq. (5) equals 0.75. The two optimizations above use 1% of the training set. Note that this 1% of data is not used to train the backdoor models. The elements in the mask are clamped into continuous values between 0 and 1. In Figure 5, we present an illustrative diagram of BAN.

Defense. Our defense uses Eq. (8) to remove the backdoor. Due to limited access to benign samples and time consumption, we only use 5% of training data and 25 epochs for our fine-tuning. The trade-off hyperparameter ( λ 2 ) is 0.5. Then, we use the most commonly used hyperparameters for the optimizer. We use SGD with momentum=0.9, weight decay=5e-4, and learning rate=0.005 as the optimizer. The plain fine-tuning uses the same hyperparameters as BAN.

Table 8 shows the ablation results on fine-tuning epochs, λ 2 , and learning rate. Considering both performances on benign accuracy and removing the backdoor, our hyperparameters (LR=0.005, λ 2 =0.5 and epoch=25) show the best results.

Hardware. All experiments are run on a single machine with 4 RTX A6000 (48GB) and 4 RTX A5000 (24GB) GPUs, CUDA 12.0.

C Additional Experimental Results

C.1 The Performance under Different Datasets

Table 9 shows the detection results using two larger datasets, Tiny-ImageNet and Imagenet200 (please refer to Appendix C.2 for results on GTSRB). The ImageNet200 is a subset of ImageNet-1K. We train 10 models for each case, 60 models in total. BAN still performs well under the two larger datasets. In addition, BAN is also scalable in terms of time consumption. To conduct backdoor detection on ResNet18 models trained using CIFAR-10, Tiny-ImageNet, and ImageNet200, BAN takes around 55, 129, and 1,120 seconds, respectively. We do not apply other baselines (except for BTI-DBF) to larger datasets due to high computational requirements. For example, FeatureRE costs more than 2 hours for detection in one class of ImageNet200 (default settings), meaning FeatureRE needs more than 400 hours for a full detection on ImageNet200.

C.2 Dection on GTSRB

Table 10 shows the detection success rate for BAN on GTSRB. Notice that NC and BTI-DBF* perform worse against advanced attacks, while BAN precisely detects all backdoor models.

C.3 Defense of Adaptive Attacks

Table 11 shows the fine-tuning results using our defense method against the two adap-blend and SSDT. The results show that fine-tuning with neuron noise can remove the backdoor with a slight decrease in clean accuracy.

C.4 Comparsion with ANP

ANP [41] uses a similar neuron noise for pruning backdoor neurons. Their optimization objective includes neuron noise and neuron mask. The neuron noise is optimized to fool the network, while

the mask controls a trade-off for clean cross-entropy loss. We compare our fine-tuning with ANP in Table 12. The hyperparameters of ANP are the default from the original paper.

C.5 Different Poisoning Rates

BAN is effective against BadNets with different poisoning rates, as shown in Table 13. We use BadNets because it is relatively weak at a low poisoning rate (i.e., hard to be mitigated), while more advanced attacks may still be strong at a low poisoning rate.

C.6 Against Backdoors on the MLP

We evaluate the defense performance on simple model architecture. In particular, a 4-layer MLP is trained with benign samples and with BadNets. Table 14 demonstrates that BAN is effective on MLP, where the 'num. to target' refers to the number of samples (in 5000 validation samples) that are classified as the backdoor target after our detection. We also find that the positive feature loss (i.e., benign feature loss) is very close to the negative loss (i.e., potential backdoor feature loss), which indicates that the backdoor features are more challenging to decouple from benign ones.

C.7 Discussion about Relationship between the Neuron Noise and Lipschitz Continuity

A small trigger that changes the output of a benign input into a malicious target label can be related to the high Lipschitz constant [45] and a neural network with high robustness tends to have a lower local Lipschitz constant. Moreover, a larger local Lipschitz constant implies steeper output around trigger-inserted points, leading to a smaller trigger effective radius making trigger inversion less

effective [49]. Thus, the concepts of adversarial activation noise and Lipschitz continuity are related, and the local Lipschitz constant can serve as an upper bound for the trigger's effective influence.

In addition, introducing theoretical tools like the Lipschitz constant for a backdoor defense may also be very tricky in practice because it needs approximation for implementation. For example, [45] evaluates the channel-wise Lipschitz constant by its upper bound but does not thoroughly discuss the relationship between the channel-wise Lipschitz constant and the network-wise Lipschitz constant, where the theorem of Lipschitz continuity really relies on. Recent work also mentions that empirically estimating the Lipschitz constant is hard from observed data, which usually leads to overestimation [5]. Methods relying on Lipschitz continuity may not require heavy computational load and are also related to our approach. Our method emphasizes more on fine-tuning with the guidance of neuron noise, rather than tuning the trained model.

C.8 Choosing the λ 1

As discussed in the method section, the mask norm ( L 1 norm) with lambda in Eq. (5) is to ensure that the optimization objective is decoupling between benign and backdoor features. Without the constraint of the mask norm in Eq. (5), the optimization objective will be simply increasing the mask norm unless there are extremely strong backdoor features. The λ 1 value selection is implemented by checking the value of the mask norm. Table 15 provides an example of λ 1 selection on CIFAR-10, where the maximal mask norm is 8192. It can be observed that the mask is almost full of ones when λ 1 is smaller than 0.7, and the negative feature loss (backdoor feature) is high, based on which we can pick a value of λ 1 greater than 0.7. Note that the selection is unaware of potential backdoors.

C.9 Swin Transformer

Table 16 provides experimental results of a 12-layer Swin transformer on CIFAR-10. For BadNets and Blend, we train the backdoored network using Adam as the optimizer. For WaNet, IAD, and Bpp, we use the default setting. All backdoored models can be detected by BAN.

C.10 Clean Label Attack

We provide a clean-label backdoor experiment following [35], where we take default hyperparameters. Table 17 demonstrates the detection performance of 20 models, where we randomly select one model from the 20 for the mitigation experiment. Experimental results demonstrate that BAN is also effective against the clean-label attack.

C.11 Semantically Similar Classes

We provide an additional analysis on image form semantically similar classes. From ImageNet200, we select class n02096294 (Australian terrier) as the target class, since there are 19 kinds of terriers in our ImageNet200 datasets, such as n02094433 (Yorkshire terrier) and n02095889 (Sealyham terrier). We train 10 backdoor networks using BadNets with different target classes. Table 18 demonstrates that BAN is effective against the backdoor regardless of semantically similar classes.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In section 1, we introduce two drawbacks of existing methods and list our contribution in 3 folds.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Appendix A, we describe the limitations of our method and discuss the broader impacts.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We focus on empirical results so we do not provide theoretical analysis.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Appendix B, we provide all details about our experiments. The link to our anonymous repository (code and trained models) is also provided in the abstract.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The link to our anonymous repository (code and trained models) is provided in the abstract. We also introduce how to run our code to the experimental results in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Appendix B, we describe all details about the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The error bars are provided in Figure 2 and 3, and Table 8. Each entry in the figures or tables above averages 5 runs with a standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe the time-consumption in Figure 4 and B.4, which demonstrates that our method is much more efficient than other methods.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the 'NeurIPS Code of Ethics'. We do not include human participants in experiments. We use open-source data and code, and there are no privacy or copyright issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In Appendix A, we discuss the potential positive and negative impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work does not overlap with pretrained language models, image generators, etc.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all relevant literature and assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code and trained models are provided. We also describe how to prepare the environment and run our code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not include crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not include crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel detection and defense method for backdoored models. The new method is motivated by the finding that existing state-of-the-art trigger-inversion methods, like BTI-DBF, rely on strong backdoor features, which might not always be present, such as for  BadNet-type triggers. As a solution, the paper proposes BAN (Backdoors activated by Adversarial neuron Noise), a novel detection method based on the findings that models with backdoors are more sensitive to adversarial noise than benign ones, allowing the identification of neurons responsible for the backdoor function. Practically, BAN computes neuron-specific noise inspired by adversarial attacks like PGD to maximize the model's classification loss on clean and unseen data. The method tries to decouple benign and backdoor features by optimizing a sparse neuron mask on the loss behavior of the model. Intuitively, a backdoored model tends to predict the target class label for clean inputs and the adversarial perturbed neurons, whereas a benign model shows fewer misclassifications under comparable neuron noise. Evaluated against common backdoor attacks and defense methods, BAN demonstrates improvements to existing approaches and robustness against adaptive attacks.\n\nSTRENGTHS:\n- The proposed defense method adds an interesting aspect to existing backdoor defense methods by improving the feature decoupling between benign and backdoor features. Whereas the decoupling optimization is conceptionally simple (which is not bad at all), it demonstrates strong results on the evaluation benchmarks and beats existing detection and defense methods markedly.\n- The paper is well-written and easy to follow. Most parts of the proposed method is sufficiently placed in existing research, and the open research problems and the proposed solution are clearly presented and described. I enjoyed reading the paper.\n- The evaluation is comprehensive and compares the proposed method against numerous attacks and defenses. It also includes the all-to-all setting, which is often ignored in literature. Evaluations of adaptive attacks underline the effectiveness and robustness of the approach.\n- Empirical solutions and theoretical considerations sufficiently support all claims in the paper.\n\nWEAKNESSES:\n- The evaluation focuses on common (comparably shallow) CNN architectures. However, given that ViT also plays an increasing role in image classification, showing the effectiveness of the approach on ViT architectures in contrast to traditional CNNs would further support the efficiency claims of the method. \n- The evaluation further focuses on dirty label attacks, i.e., attacks that change the label of a training example. However, the method's efficiency on clean-label attacks is not demonstrated. Including 1-2 clean label attacks in the evaluation would further strengthen the results.\n- Some contributions of the paper should be stated more clearly and set apart from existing approaches in literature. For example, the feature decoupling process described in 3.3 is quite similar to the method by Xu et al. [1], and it is unclear to me (by only reading this paper) what exactly distinguishes the proposed method from the one in the existing literature. \n\nSmall remarks:\n- Some captions are missing details. For example, the caption of Table 5 does not clearly state the investigated setting. Which dataset and model architecture are used here? \n- The font in the figures could be increased. Some legends and texts are hard to read without zooming in.\n- There is a typo in line 142: \"experiemnts\" -> experiments.\n- Table 7 seems like a sensitivity analysis (measuring the sensitivity of the method to the hyperparameter selection) instead of an ablation study (impact of deactivating certain parts of the method).\n\n[1] Xu et al. \"Towards Reliable and Efficient Backdoor Trigger Inversion via Decoupling Benign Features\". ICLR 2024\n\nQUESTIONS:\n- L303: \"FeatureRE is designed for all-to-one attacks, so we use target label 0 here for FeatureRE.\" -> Does it make sense to include an approach designed for all-to-one attacks in an all-to-all setting? The comparison here with the other approaches seems misleading.\n- Table 4: What are the BA (benign accuracy) and ASR (attack success rate) of models trained only on clean data without any poisoned data samples? It would be interesting to compare the defended backdoored models to a clean model. Since ImageNet (and its subsets) are complex datasets to learn, there will also be misclassifications that might, by chance, lead to the prediction of target class labels (even without any backdoor integration). Given ASR of the defended models, it would be interesting to see a baseline ASR (achieved only by random misclassifications) to assess if the remaining ASR after applying the defense methods is due to remaining backdoor behavior or simple random model behavior.\n- The proposed method is based on the assumption that under neuron noise, a backdoored model tends to predict the target class label. In contrast, the predicted class labels for the benign model are rather equally distributed. This makes sense for datasets with clearly distinguishable classes, e.g., CIFAR10. Let us assume that there exist two classes in the dataset that are semantically very similar, e.g., two visually similar (but still different) dog breeds among other clearly separable classes (bug, plane, etc.). Let us further assume that the backdoor target is to change the label from the one dog class to the other one. If we now apply the neuron noise approach to a benign and a backdoored model, can we still assume that only the backdoored model will tend to predict the other dog class (the target class) under neuron noise? Or will the benign model under neuron noise behave similarly, since the other dog class is probably close in the feature space, and adding noise to the neuron activations could lead to predictions for the second dog class (which happens to be also the backdoor target)? Phrased differently, can we assume the proposed BAN method also works reliably on datasets with semantically similar classes, if the backdoor targets label changes from one class to the other?\n\nLIMITATIONS:\nLimitations are sufficiently discussed in Appx. A. Since the paper proposes a novel defense method for backdoor attacks, no negative societal impact is expected.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

comments: SUMMARY:
This paper provides an in-depth analysis of the SOTA trigger inversion-based backdoor defenses and finds that they suffer from high computational overhead and rely on prominent backdoor features. The authors tackle the challenges based on the previous findings on adversarial noise incorporating activation information and propose to improve the backdoor feature inversion for backdoor detection. The experiments show its efficiency and effectiveness compared to the SOTA methods.

STRENGTHS:
1. The proposed method is intuitively reasonable and empirically effective. Both backdoor detection and defense are carefully considered.
2. The paper writing is clear and well-structured.
3. The experiments are comprehensive.

WEAKNESSES:
1. The motivations of high computational overhead and reliance on prominent backdoor features are not well-illustrated in the method section. Only the limitations of BTI-DBF are discussed. 
2. The novelty is limited. The finding on neuron noise is previously proposed, and an additional mask regularizer based on BTI-DBF is a commonly used technique, which was used in NC as well. The idea of backdoor defense is similar to ANP, with the change in learning from the noise.

QUESTIONS:
None.

LIMITATIONS:
1. The reliance on prominent backdoor features is expected to be illustrated in detail, and the reason why the proposed method is able to overcome it with only a mask regularizer is unclear.  Some case visualizations are expected.
2.  The conclusion from Equation 7 that ignoring the second term is expected to be illustrated with evidence such as the separate loss values on it.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of efficient backdoor defense using the backdoor inversion approach. The authors leverage the past work “TOWARDS RELIABLE AND EFFICIENT BACKDOOR TRIGGER INVERSION VIA DECOUPLING BENIGN FEATURES” (BTI-DBF) which recovers a mask in the feature space to locate prominent backdoor features and decouples benign and backdoor features in the mask. The authors make contributions by improving the BTI-DBF method and by incorporating extra neuron activation information into their method called “Backdoors activated by Adversarial neuron Noise” or BAN.\n\nSTRENGTHS:\nThe strengths of the paper lie in \n\n(1)\tstudying the trigger inversion-based detection methods\n\n(2)\toptimizing the performance of the BTI-DBF method by adding a regularizer to the loss function\n\n(3)\tincorporating adversarial activation noise into the BAN method\n\nWEAKNESSES:\nThe weaknesses of the paper lie in \n\n(1)\tmissing explanations of the performance. For example, why is featureRE in Table 3 performing better than BAN? What would be the impact of a lambda value other than 0.5? Why is the Blend attack always better detected by BTI-DBF* than by BAN? \n\n(2)\tmissing relationship between the adversarial activation noise approach and input/feature perturbation approach?  How is the adversarial activation noise approach different from assuming Lipschitz continuity on the estimated input-output function?\n\nQUESTIONS:\nThe authors should answer the questions posed as weaknesses of the paper.\n\nLIMITATIONS:\nThe limitations of the work lie in the accuracy and computational complexity of BAN. The authors describe the limitations in Appendix A and focus the limitations on the availability of clean (benign) samples and a slight decrease in the accuracy for benign inputs after fine-tuning.\n\nThe authors state, “We also exploit the neuron noise to further design a simple yet effective defense for removing the backdoor, such that we build a complete defense framework.” The statement “we build a complete defense framework” is an overstatement in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the backdoor defense task. The proposed detection method includes neuron noise, which leverages the differing robustness between regular and backdoored neurons, and a feature decoupling process using a mask. Additionally, a backdoor defense method is proposed, which achieves improved efficiency and overall performance.\n\nSTRENGTHS:\nThe exhibited efficiency is impressive, and the motivation behind neuron noise is intriguing. The dimensions of conducted experiments are relatively thorough.\n\nWEAKNESSES:\nThe neuron noise approach appears reasonable and interesting, but the feature decoupling with a mask is a little questionable.\n\nAdditionally, the study does not include enough baselines, such as fine pruning, which can obscure the distinction between backdoored and regular neurons.\n\nQUESTIONS:\n1, I might have a misunderstanding, but my key question is that the paper emphasizes that previous methods overly rely on prominent backdoor features. However, the Feature Decoupling with Mask at the neuron level seems to rely even more on these prominent backdoor features. Some previous works, like fine pruning [1], show that neurons might not necessarily be decoupled. \n\n2, Additionally, in Section 4.4, the paper claims that \"the reason is that the backdoor features of SSDT are close to benign features in the feature space. It is difficult for other methods to distinguish between backdoor and benign features created by SSDT.\" However, the Feature Decoupling with Mask approach may not be effective in this context.\n\n3, There is a minor issue that the experimental improvement does not seem significant, especially in Table 3 and Table 4.\n\n4, Is the detection performance related to the pattern of the trigger or the training strength of the backdoor?\n\n5, I'm curious about when a model architecture becomes less redundant for a given dataset. For instance, when we train DenseNet121 on CIFAR-10, the initial features are likely to be sparse. On the other hand, training an MLP model on CIFAR-10 makes feature decoupling more challenging. I wonder if using smaller models makes feature decoupling less effective.\n\n6 I also believe that selecting the lambda value in Equation (5) is quite challenging and tricky. This is because it is essential to maintain performance and ensure it doesn't degrade significantly after applying the mask (1-m).\n\nI would like to change my score if the questions are well answered.\n\n[1] Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks\n\nLIMITATIONS:\nThe limitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The authors propose a novel technique for detecting backdoor attacks on neural networks by incorporating extra neuron activation information to reduce the overhead from prior backdoor feature inversion methods. The experimental results show a higher detection rate on the tested datasets when compared to the prior work.

STRENGTHS:
The paper is well written and easy to follow. The authors address a significant topic with the widespread adoption of neural networks for a wide variety of tasks. The experimental results affirm the design choices made by the authors in Section 3. Overall, it is a well written paper that addresses a significant area.

WEAKNESSES:
1. A figure of the proposed method outlined in Sections 3.1 and 3.3 could be a helpful tool to visualize the proposed method.
2. The in figure text in Figures 2 and 3 are too small and hard to read.
3. Some tables use % and others don't when report accuracy, e.g. Table 2 and Table 3 with no % symbol for the BA columns.
4. I think a table or figure further emphasizing the proposed changes would greatly improve the strength of this paper.

QUESTIONS:
1. What is the impact of $\lambda_2$ on the fine-tuning loss found in Equation 8?

LIMITATIONS:
Yes, in appendix A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12